Slashdot Mirror
The CIHost Saga Continues
kiltboy writes "CIhosting had a major failure effectively eliminating 48,000 e-commerce sites. They claim it was a DNS failure but customers are complaining of old data being restored and some pages just being gone. MSNBC has picked up the story here along with some human interest stories. " I've talked personally with several people who've been dealing with this, and as people know, we've had hosting issues before. It's one of the most frustrating aspects of working on the Internet, but can anything be done about it? What do you think?
I feel that a company is supposed to provide a service that is stable, secure, and efficient. Any time those goals are violated in any way the person has a legitimate legal complaint. Would you be pleased or very impressed at all if for example you had your automobile serviced and then as you were driving out of the service center the engine just dropped out of the vehicle? Hardly.
Slashdot social engineering at it's finest
According to the story, the recorded message on C|Host's customer service line said something about entering domain names and IP addresses into the DNS server by hand.
This actually sounds like a Microsoft solution when something goes wrong. Since usually most items are dependant on preconfigured settings and gui administration most of the really interesting automation usually cannot happen natively. Batch files after all can only do so much and WSH is only so effective at some of the more independent abilities of shell and perl.
I'm guessing that these are customer IP addresses, but even that smacks of technical incompetence. Pretty scary. Hint: if you're not going to make recoverable backups, at least spend an hour or two learning rudimentary shell scripting, if not Perl.
I agree. Most of the settings for various system files that I have seen have been in hiden and system directories that are not usually thought as part of the normal file system (Netware) and such. When people do backups its not totally intuitive for the average person to get something of that nature done easily or properly without the necessary training. An MCSE will not usually cut it because little is done to teach out of contingency management.
Slashdot social engineering at it's finest
It's about money. If you want something done right, or at least done so that you do not complain, then you'd better be prepared to do it yourself. Otherwise, you get what you pay for.
Web hosting is big business, and to make the price competitive, corners get cut. The problem is when failures occur, you see the underbelly of your cost savings. It costs money to hire good staff. It costs money to make (frequent) backups. It costs money to provide redundant equipment.
Now, an out of the box solution may run great for a while, giving all involved a sense of security. A smoothly running computer needs little more than a baby-sitter in terms of administration and tech support. It's when all hell breaks loose that you find out where your money has been going. A trained staff costs more, but will get you back up and will keep you there. The cheap and untrained baby-sitter will, at best, be on-hold with someone elses tech support.
The mettle of your staff and contracted hosting company is tested and proven during a crisis. How they handle that crisis is what you pay for. Their response to this matter tells much of their commitment to their customers.
-- What you do today will cost you a day of your life.
This may be off-topic, but....
Which virtual hosting services have people had success with? Which ones should be avoided like the plague? I'm sure there's enough experience here to answer this question.
Personally, I just signed up with a new, inexpensive service, and have gotten the impression that they are completely incompetant, and I may be looking to switch. Any advice?
As far as the people who, at least the article seems to indicate, lost their entire site and fear that there is no backup, I have a piece of advice:
"Never trust a backup that isn't in your hands."
What, do these people just schlop their pages on the server and not keep a frequently-updated copy of everything on their own media? That's just stupid, plain and simple.
Ok, wonder why sites like UserFriendly and such make fun of idio... er customers calling in? Read the above! Your cow-orker calls tech support to yell at them. In long entire chain of command how much power do you think the lowly techie he is yelling at has? Tech support gets dumped on because they are a convient target. If you want something changed go higher up the food chain and give the techies a break.
I would ask all of these companies, (all the people who signed contracts with the web hosting services) "Have you ever used a computer?" Any professional in the IT industry should know that computers are not 100% reliable, and anyone who has ever used a PC will have seen a few GPFs, or BSODs. When you contract with someone to provide computer services, you don't ask them "Do your systems crash?" You ask them "What do you do when the system crashes?"
Unfortunately, most people want to be lied to. Despite the fact that it's an OBVIOUS lie, that anyone with an ounce of common sense would disbeliev, they want to hear "Our systems never crash." If they don't hear that, they keep on looking until they find someone who will promise that. Of course, there's never anything like that actually in the contract...
However they aren't fundamental in running a business.
;)
Businesses will always try to cut corners, unfortunately backups and high availability, which should be the cornerstone of this kind of "operation" are often overlooked.
Just today in fact I came into work and had a disk crashed on a Jamaica JBOD attached to a Hewlett-Packard K370. True, since it's not an array it took a while to recover everything, but being as it was a development database, I managed to restore it back to the state it was at 2am last night.
Without those backups I'm be typing my resume now instead of posting to Slashdot
But there's no excuse for a company like this to backup their customers' data, the technology is out there--StorageTek, ATL, they all have scalable solutions to back up terrabytes worth of data, but that costs money and apparently they didn't think providing a decent service to their paying customers was worth it.
If I had an account there, I'd most likely find another provider that knew about system backups and high availability, it is *your* business.
A web-hosting service can cut corners on software, by using Open Source products such as Apache (optionally with IBM's GUI, SGI's patches, and/or one of the acceperators such as Squid), OpenSSL, Perl, [PHP | Zope], Minivend, [Sendmail | Postfix | Qmail | Cyrus IMAP], Cyrus LDAP, [SSH | OpenSSH], [Linux | FreeBSD | OpenBSD | NetBSD], Heartbeat (for High Availability), etc.
They can also cut corners -to some degree- on hardware. eg: You don't -need- to buy hardware RAID solutions, as you can do that in software. You don't -need- to buy watchdog cards, as you can do that in software, too. If you use ReiserFS, you can use the increase in performance to go for disks that aren't necessarily as fast.
How often you make backups depends on the volatility of your data, NOT on how much you want to spend. If your data can be expected to change daily, then backup daily. If it's likely to be stable for a few weeks, backup weekly. If it's continuously on the move, then backup hourly.
How to make backups - buy tapes. They're cheap, they're reliable, and they store a lot. Tape drives are also a lot cheaper than R/W CD-ROM drives. If you've got the cash, get twice the number of tape drives you normally would, then Cron the jobs to run in the background. Stripe your data across half your tape drives. Use the remaining drives for the next backup cycle. That way, you give yourself more time to swap in fresh tapes, and if you forget before the next backup, you're OK.
How long to keep tape backups: Forever, if you can afford it. As long as your budget can possibly allow, otherwise. It's vitally important to be able to backtrack as far as possible.
All in all, there's never any excuse for mishandling data, on account of expense. You CAN make things as cheap as you like, WITHOUT compromising the integrity of the system or your ability to recover from a catastrophic failure.
As for DNS', routers, etc: ALWAYS have TWO of everything. That doesn't mean you have to splash out on a vast number of machines. You can always use your fileserver as your secondary DNS, and a software router can always sit on a web server (though that's not really good practice, for security reasons). And ALWAYS have High Availability wherever applicable. DNS doesn't really need this, as most OS' can search multiple DNS servers.
Again, there's no excuse for not having backup systems. Once you've got the basic machines, you can have them multitask as much as you like, so they can always act as backups for something else.
It's negligence that leads to disasters like this. And I include the times that my own failure to backup has led to significant loss of data. It's a lesson I learned well. If others haven't, well, don't hire them to host your web services until they have.
I think the worst example of negligence like this that I've personally seen was at NASA Langley. The admins backed up -officially- daily. In practice, it was whenever they felt like it. One visitor to the center picked up a hard disk (in use at the time), shook it, and asked what it was. The disk, needless to say, crashed. It turned out that there was a vast amount of critical research data on it and the admins hadn't backed it up in 3 months.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
as a very unsatisfied, and even baffled, customer i have to say that these guys are right on the money!!! and msnbc picking up on the story too. haha.. i hope these guys realize what theyve gotten themselves into. jay
I think everyone is missing two important points: first is that CIHost, despite being down for several days, isn't guilty for being down so long, per se. It's dumb, but not un-survivable; alternative service could be found, and if they were a great service, they'd have little to worry about. But their attitude is not an apologetic and a forthright one, but one of arrogance and obfuscation - arrogance, because they see the problem as being a minor glitch, and obfuscation, because they can't seem to get even simple information, like the status of their site and my site, up. I have not recieved one e-mail, one call, one notice, or even a web page informing me of their status. I note, though, that their own CIHost main page was up before mine. When I look at that, when I can't reach their customer service site, when I can't get my own mail because of this - I'm mad. Down, I can understand. Down & no info why or when it'll be fixed, I don't.
The second point, one that I think everyone is saying but skirting around, is that there's no good, reliable information on web hosts. Lists make money off of a) advertising from the very people they're rating, or, sometimes b) money to rate other people higher. No one has an objective list which rates the customer service, the time up and down, the overall service, that I know of - and it's almost impossible to make one, because the good ones can so rapidly become bad ones. Information about these things is so subjective as well - several people have already complained of having "bad service" while not detailing what happened.
The first problem is CIHost's fault. They can (but most likely won't) change. The second problem, maybe we can work on - compiling a list of decent web hosts, and keeping track of problems and sucesses. Any thoughts?
Whatever you do... don't read this.
A lot of companies would be better off using Small ISP's. Smaller ISP's are almost always willing to give you more bang for your buck. They don't charge enourmous amounts of money and you have a better chance of getting a human on the other end rather then an automated attendent. When you call, ask them how long they have been around... the average lifespan of bankrupt/closed down companies in this world is about 3 years. If they live past then, they are pretty well as solid as the next guy.
If you are a small company chances are your webpage is not going to garner gobs of bandwidth at one time... even the little ISP with a 128K ISDN connection can easily serve up your pages to people in a reasonable amount of time. Ie. At home I have a 56K modem... at work I have a T1. Slashdot loads at the same speed for both. I get 6k per second on my home machine... and I get 80-100 on my work machine at peak so its not my server because they are both using the same pipe.
People need to get out of this frame of mind that bigger is better. It isn't... with bigger you have red tape. You have cutbacks where they have 3 guys tech supporting 10,000 users. Sure with the smaller ISP you may not get 24/7 tech service.. or you may even pay a little bit more for one specific thing but in a lot of cases cheaper isn't better. If your business hosting needs are dependent on 24/7 support then you need a Network Service Provider. If your business was so dependent on uptime that it would kill your profit margin by being down for 8 hours because your ISP staff is asleep then you need the bigger guy... even then you can find medium sized companies that have higher levels of tech support options available. I would be willing to bet that some of the best webhosting companies out there are small timers who will stay small because they won't sacrifice quantity for quality.
Just my 2 cents... keep them in mind when you need a business (or even personal) internet solution.
- Xabbu
- Jimbob
Check out Kimmel and Silverman, the Computer Lemon Law Attorneys.. They advertise: Simply call Kimmel & Silverman, The Lemon Law Attorneys at 1-800-LEMON-LAW (800-536-6652). You can also fill out our form and submit by e-mail. We'll do the rest, quickly and efficiently to get you a NEW COMPUTER or FULL REFUND at absolutely no charge to you. That's right, Kimmel & Silverman's service is FREE, win or lose!!
I have no connection with them, but I called them, and they say they handle this sort of thing.
I've been co-locating my web server at Global Online Electronic Services. My server is more of an educational toy for my friends and I than anything. So we suffer no great loss if something goes wrong. However, if I ever need to host anything serious, I'll probably still stick with GOES.
:) I became aquainted with the owner, Norm, at a fair at the local college. The cool thing about it is that Norm is an ISP because that's what he loves to do. He loves having his own T3, servers, and racks of modems.
:)
GOES is a local ISP in Hackettstown, NJ (where M&M's come from
Norm's a BSD guy, but he's got at least one member of his tech support staff that's a Linux nut. They're very knowledgable, know me by name, and eager to help. In fact, they seem genuinely interested in what we're doing.
Point: I understand that a small ISP may not suit everyone, and not all local ISP's are alike. However, it's certainly worth considering because there are benefits to dealing with someone who will actually know who you are once you start with them. And, if they're local, you can always harass them in person if something goes wrong
Anyway, I moved ~1000 miles south as the car [sic] flies since the time I started doing business there. I'm using Linux so it's not as if I ever need to visit the box anyway.
Oh, another bit...I did have a problem with the server not rebooting once. It refused to detect the SCSI adapter all of a sudden and I figured the adapter was dead. Norm called the owner of the computer store located in the same building. He came up and fixed it--then wouldn't even let me pay him for his time because all he had to do was re-seat the SCSI card. You don't get that kind of support when you're dealing with a big companies...
numb