The CIHost Saga Continues

kiltboy writes "CIhosting had a major failure effectively eliminating 48,000 e-commerce sites. They claim it was a DNS failure but customers are complaining of old data being restored and some pages just being gone. MSNBC has picked up the story here along with some human interest stories. " I've talked personally with several people who've been dealing with this, and as people know, we've had hosting issues before. It's one of the most frustrating aspects of working on the Internet, but can anything be done about it? What do you think?

Comments

An interesting problem

[jd]

It doesn't cost -much- to run a decent web-hosting service, if you cut the -right- corners. It costs a lot, in the long-term, if you cut the wrong ones.

A web-hosting service can cut corners on software, by using Open Source products such as Apache (optionally with IBM's GUI, SGI's patches, and/or one of the acceperators such as Squid), OpenSSL, Perl, [PHP | Zope], Minivend, [Sendmail | Postfix | Qmail | Cyrus IMAP], Cyrus LDAP, [SSH | OpenSSH], [Linux | FreeBSD | OpenBSD | NetBSD], Heartbeat (for High Availability), etc.

They can also cut corners -to some degree- on hardware. eg: You don't -need- to buy hardware RAID solutions, as you can do that in software. You don't -need- to buy watchdog cards, as you can do that in software, too. If you use ReiserFS, you can use the increase in performance to go for disks that aren't necessarily as fast.

How often you make backups depends on the volatility of your data, NOT on how much you want to spend. If your data can be expected to change daily, then backup daily. If it's likely to be stable for a few weeks, backup weekly. If it's continuously on the move, then backup hourly.

How to make backups - buy tapes. They're cheap, they're reliable, and they store a lot. Tape drives are also a lot cheaper than R/W CD-ROM drives. If you've got the cash, get twice the number of tape drives you normally would, then Cron the jobs to run in the background. Stripe your data across half your tape drives. Use the remaining drives for the next backup cycle. That way, you give yourself more time to swap in fresh tapes, and if you forget before the next backup, you're OK.

How long to keep tape backups: Forever, if you can afford it. As long as your budget can possibly allow, otherwise. It's vitally important to be able to backtrack as far as possible.

All in all, there's never any excuse for mishandling data, on account of expense. You CAN make things as cheap as you like, WITHOUT compromising the integrity of the system or your ability to recover from a catastrophic failure.

As for DNS', routers, etc: ALWAYS have TWO of everything. That doesn't mean you have to splash out on a vast number of machines. You can always use your fileserver as your secondary DNS, and a software router can always sit on a web server (though that's not really good practice, for security reasons). And ALWAYS have High Availability wherever applicable. DNS doesn't really need this, as most OS' can search multiple DNS servers.

Again, there's no excuse for not having backup systems. Once you've got the basic machines, you can have them multitask as much as you like, so they can always act as backups for something else.

It's negligence that leads to disasters like this. And I include the times that my own failure to backup has led to significant loss of data. It's a lesson I learned well. If others haven't, well, don't hire them to host your web services until they have.

I think the worst example of negligence like this that I've personally seen was at NASA Langley. The admins backed up -officially- daily. In practice, it was whenever they felt like it. One visitor to the center picked up a hard disk (in use at the time), shook it, and asked what it was. The disk, needless to say, crashed. It turned out that there was a vast amount of critical research data on it and the admins hadn't backed it up in 3 months.