Slashdot Mirror
The CIHost Saga Continues
kiltboy writes "CIhosting had a major failure effectively eliminating 48,000 e-commerce sites. They claim it was a DNS failure but customers are complaining of old data being restored and some pages just being gone. MSNBC has picked up the story here along with some human interest stories. " I've talked personally with several people who've been dealing with this, and as people know, we've had hosting issues before. It's one of the most frustrating aspects of working on the Internet, but can anything be done about it? What do you think?
It's about money. If you want something done right, or at least done so that you do not complain, then you'd better be prepared to do it yourself. Otherwise, you get what you pay for.
Web hosting is big business, and to make the price competitive, corners get cut. The problem is when failures occur, you see the underbelly of your cost savings. It costs money to hire good staff. It costs money to make (frequent) backups. It costs money to provide redundant equipment.
Now, an out of the box solution may run great for a while, giving all involved a sense of security. A smoothly running computer needs little more than a baby-sitter in terms of administration and tech support. It's when all hell breaks loose that you find out where your money has been going. A trained staff costs more, but will get you back up and will keep you there. The cheap and untrained baby-sitter will, at best, be on-hold with someone elses tech support.
The mettle of your staff and contracted hosting company is tested and proven during a crisis. How they handle that crisis is what you pay for. Their response to this matter tells much of their commitment to their customers.
-- What you do today will cost you a day of your life.
A web-hosting service can cut corners on software, by using Open Source products such as Apache (optionally with IBM's GUI, SGI's patches, and/or one of the acceperators such as Squid), OpenSSL, Perl, [PHP | Zope], Minivend, [Sendmail | Postfix | Qmail | Cyrus IMAP], Cyrus LDAP, [SSH | OpenSSH], [Linux | FreeBSD | OpenBSD | NetBSD], Heartbeat (for High Availability), etc.
They can also cut corners -to some degree- on hardware. eg: You don't -need- to buy hardware RAID solutions, as you can do that in software. You don't -need- to buy watchdog cards, as you can do that in software, too. If you use ReiserFS, you can use the increase in performance to go for disks that aren't necessarily as fast.
How often you make backups depends on the volatility of your data, NOT on how much you want to spend. If your data can be expected to change daily, then backup daily. If it's likely to be stable for a few weeks, backup weekly. If it's continuously on the move, then backup hourly.
How to make backups - buy tapes. They're cheap, they're reliable, and they store a lot. Tape drives are also a lot cheaper than R/W CD-ROM drives. If you've got the cash, get twice the number of tape drives you normally would, then Cron the jobs to run in the background. Stripe your data across half your tape drives. Use the remaining drives for the next backup cycle. That way, you give yourself more time to swap in fresh tapes, and if you forget before the next backup, you're OK.
How long to keep tape backups: Forever, if you can afford it. As long as your budget can possibly allow, otherwise. It's vitally important to be able to backtrack as far as possible.
All in all, there's never any excuse for mishandling data, on account of expense. You CAN make things as cheap as you like, WITHOUT compromising the integrity of the system or your ability to recover from a catastrophic failure.
As for DNS', routers, etc: ALWAYS have TWO of everything. That doesn't mean you have to splash out on a vast number of machines. You can always use your fileserver as your secondary DNS, and a software router can always sit on a web server (though that's not really good practice, for security reasons). And ALWAYS have High Availability wherever applicable. DNS doesn't really need this, as most OS' can search multiple DNS servers.
Again, there's no excuse for not having backup systems. Once you've got the basic machines, you can have them multitask as much as you like, so they can always act as backups for something else.
It's negligence that leads to disasters like this. And I include the times that my own failure to backup has led to significant loss of data. It's a lesson I learned well. If others haven't, well, don't hire them to host your web services until they have.
I think the worst example of negligence like this that I've personally seen was at NASA Langley. The admins backed up -officially- daily. In practice, it was whenever they felt like it. One visitor to the center picked up a hard disk (in use at the time), shook it, and asked what it was. The disk, needless to say, crashed. It turned out that there was a vast amount of critical research data on it and the admins hadn't backed it up in 3 months.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
I think everyone is missing two important points: first is that CIHost, despite being down for several days, isn't guilty for being down so long, per se. It's dumb, but not un-survivable; alternative service could be found, and if they were a great service, they'd have little to worry about. But their attitude is not an apologetic and a forthright one, but one of arrogance and obfuscation - arrogance, because they see the problem as being a minor glitch, and obfuscation, because they can't seem to get even simple information, like the status of their site and my site, up. I have not recieved one e-mail, one call, one notice, or even a web page informing me of their status. I note, though, that their own CIHost main page was up before mine. When I look at that, when I can't reach their customer service site, when I can't get my own mail because of this - I'm mad. Down, I can understand. Down & no info why or when it'll be fixed, I don't.
The second point, one that I think everyone is saying but skirting around, is that there's no good, reliable information on web hosts. Lists make money off of a) advertising from the very people they're rating, or, sometimes b) money to rate other people higher. No one has an objective list which rates the customer service, the time up and down, the overall service, that I know of - and it's almost impossible to make one, because the good ones can so rapidly become bad ones. Information about these things is so subjective as well - several people have already complained of having "bad service" while not detailing what happened.
The first problem is CIHost's fault. They can (but most likely won't) change. The second problem, maybe we can work on - compiling a list of decent web hosts, and keeping track of problems and sucesses. Any thoughts?
Whatever you do... don't read this.
Check out Kimmel and Silverman, the Computer Lemon Law Attorneys.. They advertise: Simply call Kimmel & Silverman, The Lemon Law Attorneys at 1-800-LEMON-LAW (800-536-6652). You can also fill out our form and submit by e-mail. We'll do the rest, quickly and efficiently to get you a NEW COMPUTER or FULL REFUND at absolutely no charge to you. That's right, Kimmel & Silverman's service is FREE, win or lose!!
I have no connection with them, but I called them, and they say they handle this sort of thing.
I've been co-locating my web server at Global Online Electronic Services. My server is more of an educational toy for my friends and I than anything. So we suffer no great loss if something goes wrong. However, if I ever need to host anything serious, I'll probably still stick with GOES.
:) I became aquainted with the owner, Norm, at a fair at the local college. The cool thing about it is that Norm is an ISP because that's what he loves to do. He loves having his own T3, servers, and racks of modems.
:)
GOES is a local ISP in Hackettstown, NJ (where M&M's come from
Norm's a BSD guy, but he's got at least one member of his tech support staff that's a Linux nut. They're very knowledgable, know me by name, and eager to help. In fact, they seem genuinely interested in what we're doing.
Point: I understand that a small ISP may not suit everyone, and not all local ISP's are alike. However, it's certainly worth considering because there are benefits to dealing with someone who will actually know who you are once you start with them. And, if they're local, you can always harass them in person if something goes wrong
Anyway, I moved ~1000 miles south as the car [sic] flies since the time I started doing business there. I'm using Linux so it's not as if I ever need to visit the box anyway.
Oh, another bit...I did have a problem with the server not rebooting once. It refused to detect the SCSI adapter all of a sudden and I figured the adapter was dead. Norm called the owner of the computer store located in the same building. He came up and fixed it--then wouldn't even let me pay him for his time because all he had to do was re-seat the SCSI card. You don't get that kind of support when you're dealing with a big companies...
numb