The CIHost Saga Continues
kiltboy writes "CIhosting had a major failure effectively eliminating 48,000 e-commerce sites. They claim it was a DNS failure but customers are complaining of old data being restored and some pages just being gone. MSNBC has picked up the story here along with some human interest stories. " I've talked personally with several people who've been dealing with this, and as people know, we've had hosting issues before. It's one of the most frustrating aspects of working on the Internet, but can anything be done about it? What do you think?
CIhost has an office just down the street from my house, and when I applied there, i noticed a lot of lax behavior. from everyone in the building. Their equipment was not kept clean or orderly, and according to one tech I spoke with, they only run backups every week or so, sometimes as long as 3 weeks! This is NOT flamebait or BS, I got this info from people in the company.
=======
There was never a genius without a tincture of madness.
Not to mention that they tend to tell you they'll call you right back, and then don't; they'll tell you something has been fixed, but they haven't tested it (and of course it's not fixed); and so forth. I would avoid Digex at all costs if I were you.
--- Dirtside | "Spirituality" is the irrational belief in the supernatural
"Destroy science and religion. Science would re-emerge exactly the same; but not religion." - Penn Jillette, paraphrased
According to the story, the recorded message on C|Host's customer service line said something about entering domain names and IP addresses into the DNS server by hand.
I'm guessing that these are customer IP addresses, but even that smacks of technical incompetence. Pretty scary. Hint: if you're not going to make recoverable backups, at least spend an hour or two learning rudimentary shell scripting, if not Perl.
--
how to invest, a novice's guide
I feel that a company is supposed to provide a service that is stable, secure, and efficient. Any time those goals are violated in any way the person has a legitimate legal complaint. Would you be pleased or very impressed at all if for example you had your automobile serviced and then as you were driving out of the service center the engine just dropped out of the vehicle? Hardly.
Slashdot social engineering at it's finest
It's about money. If you want something done right, or at least done so that you do not complain, then you'd better be prepared to do it yourself. Otherwise, you get what you pay for.
Web hosting is big business, and to make the price competitive, corners get cut. The problem is when failures occur, you see the underbelly of your cost savings. It costs money to hire good staff. It costs money to make (frequent) backups. It costs money to provide redundant equipment.
Now, an out of the box solution may run great for a while, giving all involved a sense of security. A smoothly running computer needs little more than a baby-sitter in terms of administration and tech support. It's when all hell breaks loose that you find out where your money has been going. A trained staff costs more, but will get you back up and will keep you there. The cheap and untrained baby-sitter will, at best, be on-hold with someone elses tech support.
The mettle of your staff and contracted hosting company is tested and proven during a crisis. How they handle that crisis is what you pay for. Their response to this matter tells much of their commitment to their customers.
-- What you do today will cost you a day of your life.
I have been useing AIT for almost 2 years now without a single problem. There connections are wicked fast and dirt cheap. They offer virtual both NT and unix virtual servers and colocation of anything you want.
http://www.aitcom.net
This may be off-topic, but....
Which virtual hosting services have people had success with? Which ones should be avoided like the plague? I'm sure there's enough experience here to answer this question.
Personally, I just signed up with a new, inexpensive service, and have gotten the impression that they are completely incompetant, and I may be looking to switch. Any advice?
to prarphrase an old lawyer joke... What do you have when you've got 18,000 e-commerce sites wiped with no backup? An idea whose time has come.
As far as the people who, at least the article seems to indicate, lost their entire site and fear that there is no backup, I have a piece of advice:
"Never trust a backup that isn't in your hands."
What, do these people just schlop their pages on the server and not keep a frequently-updated copy of everything on their own media? That's just stupid, plain and simple.
Ok, wonder why sites like UserFriendly and such make fun of idio... er customers calling in? Read the above! Your cow-orker calls tech support to yell at them. In long entire chain of command how much power do you think the lowly techie he is yelling at has? Tech support gets dumped on because they are a convient target. If you want something changed go higher up the food chain and give the techies a break.
So from reading the article it seems that we've lost a site that sells tennis balls that go on your car antenna, and a meeting place for celtic musicians.
I don't know about you, but that sounds like acceptable losses to me. You know the old saying, if you're gonna make an omlette, you've gotta break a couple tennis balls.
Hotnutz.com
I would ask all of these companies, (all the people who signed contracts with the web hosting services) "Have you ever used a computer?" Any professional in the IT industry should know that computers are not 100% reliable, and anyone who has ever used a PC will have seen a few GPFs, or BSODs. When you contract with someone to provide computer services, you don't ask them "Do your systems crash?" You ask them "What do you do when the system crashes?"
Unfortunately, most people want to be lied to. Despite the fact that it's an OBVIOUS lie, that anyone with an ounce of common sense would disbeliev, they want to hear "Our systems never crash." If they don't hear that, they keep on looking until they find someone who will promise that. Of course, there's never anything like that actually in the contract...
However they aren't fundamental in running a business.
;)
Businesses will always try to cut corners, unfortunately backups and high availability, which should be the cornerstone of this kind of "operation" are often overlooked.
Just today in fact I came into work and had a disk crashed on a Jamaica JBOD attached to a Hewlett-Packard K370. True, since it's not an array it took a while to recover everything, but being as it was a development database, I managed to restore it back to the state it was at 2am last night.
Without those backups I'm be typing my resume now instead of posting to Slashdot
But there's no excuse for a company like this to backup their customers' data, the technology is out there--StorageTek, ATL, they all have scalable solutions to back up terrabytes worth of data, but that costs money and apparently they didn't think providing a decent service to their paying customers was worth it.
If I had an account there, I'd most likely find another provider that knew about system backups and high availability, it is *your* business.
I know that keeping web servers up isn't the easiest thing in the world, from my past experience. It's not rocket science, but sometimes hardware failures happen. Sometimes you lost power, and not everbody can afford a backup generator to run their web server on.
However, I've never dealt with a hosting company with as many problems as CIHost.In the past two months or so, our web site has been offline at least 8 times for no apparent reason. For about two weeks, their bandwidth was almost completely saturated. And now, we've got this issue.
When we first heard about CIHost, we checked into them, and they were rated highly on at least two sites we checked. With this kind of service they sure won't keep a rating like that for long.
A web-hosting service can cut corners on software, by using Open Source products such as Apache (optionally with IBM's GUI, SGI's patches, and/or one of the acceperators such as Squid), OpenSSL, Perl, [PHP | Zope], Minivend, [Sendmail | Postfix | Qmail | Cyrus IMAP], Cyrus LDAP, [SSH | OpenSSH], [Linux | FreeBSD | OpenBSD | NetBSD], Heartbeat (for High Availability), etc.
They can also cut corners -to some degree- on hardware. eg: You don't -need- to buy hardware RAID solutions, as you can do that in software. You don't -need- to buy watchdog cards, as you can do that in software, too. If you use ReiserFS, you can use the increase in performance to go for disks that aren't necessarily as fast.
How often you make backups depends on the volatility of your data, NOT on how much you want to spend. If your data can be expected to change daily, then backup daily. If it's likely to be stable for a few weeks, backup weekly. If it's continuously on the move, then backup hourly.
How to make backups - buy tapes. They're cheap, they're reliable, and they store a lot. Tape drives are also a lot cheaper than R/W CD-ROM drives. If you've got the cash, get twice the number of tape drives you normally would, then Cron the jobs to run in the background. Stripe your data across half your tape drives. Use the remaining drives for the next backup cycle. That way, you give yourself more time to swap in fresh tapes, and if you forget before the next backup, you're OK.
How long to keep tape backups: Forever, if you can afford it. As long as your budget can possibly allow, otherwise. It's vitally important to be able to backtrack as far as possible.
All in all, there's never any excuse for mishandling data, on account of expense. You CAN make things as cheap as you like, WITHOUT compromising the integrity of the system or your ability to recover from a catastrophic failure.
As for DNS', routers, etc: ALWAYS have TWO of everything. That doesn't mean you have to splash out on a vast number of machines. You can always use your fileserver as your secondary DNS, and a software router can always sit on a web server (though that's not really good practice, for security reasons). And ALWAYS have High Availability wherever applicable. DNS doesn't really need this, as most OS' can search multiple DNS servers.
Again, there's no excuse for not having backup systems. Once you've got the basic machines, you can have them multitask as much as you like, so they can always act as backups for something else.
It's negligence that leads to disasters like this. And I include the times that my own failure to backup has led to significant loss of data. It's a lesson I learned well. If others haven't, well, don't hire them to host your web services until they have.
I think the worst example of negligence like this that I've personally seen was at NASA Langley. The admins backed up -officially- daily. In practice, it was whenever they felt like it. One visitor to the center picked up a hard disk (in use at the time), shook it, and asked what it was. The disk, needless to say, crashed. It turned out that there was a vast amount of critical research data on it and the admins hadn't backed it up in 3 months.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
Well, rule #1 should be to make your own backups and don't trust anybody.
Ideally, you should do your development on your own machine and only upload 'production' code to the machine that hosts your domain and has all the bandwidth and then run a cron job each day or so to dump your database, tar and gzip it all, and ftp it someplace else.
Also, a cron job that runs a little script to inform you of the state of things and emails it to you every few hours is a good idea if your site is not a full time gig.
Of course, since you're paying somebody to do all this for you you really shouldn't have to worry about it but that's life.
Mind you, I only use server space there -- I haven't tried to colocate whole machines -- but they recently moved their entire NOC and they did a pretty damn fine job of it.
As a result, I trust them even more.
I have no
Data is fundamental to running a business. If a business loses data, it loses money. Since it is a given that systems will fail, backups are a requirement for businesses. Any business that cuts costs here will eventually learn why it's bad. If they're lucky, it will be a painful lesson; often the lesson is a fatal one.
The reason why backups are fundamental to sys administration is because the sysadmin is responsible for the data - like you said, lose the data, lose your job.
as a very unsatisfied, and even baffled, customer i have to say that these guys are right on the money!!! and msnbc picking up on the story too. haha.. i hope these guys realize what theyve gotten themselves into. jay
I think one of the largest culprits in bad events like this is lack of documentation/planning.
It's not that the hosting company is negligent, but that the users assume certain things about the hosting company without getting it in writing.
ie: Did the hosting company specify that they had daily backups? Weekly? Did they state that they had a disaster recovery plan? Did you ask? Did you state *your* needs as the customer, or did you just assume, as many people do, 'They are a web hosting company, so they'll have daily backups, a super-fast disaster recovery plan, fault tolerant systems, and enough staff/resources to deal with problems efficiently.).
This is not the case in a great many companies.
As for backups, you should have your own off-site backups, and if it's e-commerce related, you should damn well have your OWN disaster recovery plan. Your own backups. Your own copy of the records, either that, or contractual obligations that your provider will supply those services.
Even before they crashed I was getting pissed at CiHost. PHP / MySQL web pages that should have worked fine, and did work fine at home, simply hung on their machines. They talked about all these wonderful things they could do (e-commerce etc) but never gave the details. Now this.
So, where do I go? Any recommendations for a *good* provider with decent backups, uptime, big pipes, available tech support, reasonable prices and (most importantly) MySQL and PHP support?
I think everyone is missing two important points: first is that CIHost, despite being down for several days, isn't guilty for being down so long, per se. It's dumb, but not un-survivable; alternative service could be found, and if they were a great service, they'd have little to worry about. But their attitude is not an apologetic and a forthright one, but one of arrogance and obfuscation - arrogance, because they see the problem as being a minor glitch, and obfuscation, because they can't seem to get even simple information, like the status of their site and my site, up. I have not recieved one e-mail, one call, one notice, or even a web page informing me of their status. I note, though, that their own CIHost main page was up before mine. When I look at that, when I can't reach their customer service site, when I can't get my own mail because of this - I'm mad. Down, I can understand. Down & no info why or when it'll be fixed, I don't.
The second point, one that I think everyone is saying but skirting around, is that there's no good, reliable information on web hosts. Lists make money off of a) advertising from the very people they're rating, or, sometimes b) money to rate other people higher. No one has an objective list which rates the customer service, the time up and down, the overall service, that I know of - and it's almost impossible to make one, because the good ones can so rapidly become bad ones. Information about these things is so subjective as well - several people have already complained of having "bad service" while not detailing what happened.
The first problem is CIHost's fault. They can (but most likely won't) change. The second problem, maybe we can work on - compiling a list of decent web hosts, and keeping track of problems and sucesses. Any thoughts?
Whatever you do... don't read this.
If it seems too good to be true, generaly it's not true.
I wouldn't trust a web provider who had that kind of promotion of being very reliable or competent. Maybe I'm just cynical like that. Of course, my current provider is cut-rate, but doesn't offer huge promos either. They have basic service for basic prices and I've been happy thus far.
In Soviet Russia...michael would be rotting in Siberia!
A lot of companies would be better off using Small ISP's. Smaller ISP's are almost always willing to give you more bang for your buck. They don't charge enourmous amounts of money and you have a better chance of getting a human on the other end rather then an automated attendent. When you call, ask them how long they have been around... the average lifespan of bankrupt/closed down companies in this world is about 3 years. If they live past then, they are pretty well as solid as the next guy.
If you are a small company chances are your webpage is not going to garner gobs of bandwidth at one time... even the little ISP with a 128K ISDN connection can easily serve up your pages to people in a reasonable amount of time. Ie. At home I have a 56K modem... at work I have a T1. Slashdot loads at the same speed for both. I get 6k per second on my home machine... and I get 80-100 on my work machine at peak so its not my server because they are both using the same pipe.
People need to get out of this frame of mind that bigger is better. It isn't... with bigger you have red tape. You have cutbacks where they have 3 guys tech supporting 10,000 users. Sure with the smaller ISP you may not get 24/7 tech service.. or you may even pay a little bit more for one specific thing but in a lot of cases cheaper isn't better. If your business hosting needs are dependent on 24/7 support then you need a Network Service Provider. If your business was so dependent on uptime that it would kill your profit margin by being down for 8 hours because your ISP staff is asleep then you need the bigger guy... even then you can find medium sized companies that have higher levels of tech support options available. I would be willing to bet that some of the best webhosting companies out there are small timers who will stay small because they won't sacrifice quantity for quality.
Just my 2 cents... keep them in mind when you need a business (or even personal) internet solution.
- Xabbu
- Jimbob
i just got myself a T1, sure its expensive but the only downtime is self-incurred.
Don't count on that! You'll need a backup from another provider at least. The big frame relay meltdown last year nailed a lot of people who believed the same thing you do.
A week of downtime is what is unacceptable.
Is that specified in your contract? If it's not in your contract, you essentially told them that it is acceptable. If you don't require them to do it, why should they spend extra money to do it?. If it is in your contract, then they took your money and lied about what you'd get - time to sue sue sue.
Check out Kimmel and Silverman, the Computer Lemon Law Attorneys.. They advertise: Simply call Kimmel & Silverman, The Lemon Law Attorneys at 1-800-LEMON-LAW (800-536-6652). You can also fill out our form and submit by e-mail. We'll do the rest, quickly and efficiently to get you a NEW COMPUTER or FULL REFUND at absolutely no charge to you. That's right, Kimmel & Silverman's service is FREE, win or lose!!
I have no connection with them, but I called them, and they say they handle this sort of thing.
I've been co-locating my web server at Global Online Electronic Services. My server is more of an educational toy for my friends and I than anything. So we suffer no great loss if something goes wrong. However, if I ever need to host anything serious, I'll probably still stick with GOES.
:) I became aquainted with the owner, Norm, at a fair at the local college. The cool thing about it is that Norm is an ISP because that's what he loves to do. He loves having his own T3, servers, and racks of modems.
:)
GOES is a local ISP in Hackettstown, NJ (where M&M's come from
Norm's a BSD guy, but he's got at least one member of his tech support staff that's a Linux nut. They're very knowledgable, know me by name, and eager to help. In fact, they seem genuinely interested in what we're doing.
Point: I understand that a small ISP may not suit everyone, and not all local ISP's are alike. However, it's certainly worth considering because there are benefits to dealing with someone who will actually know who you are once you start with them. And, if they're local, you can always harass them in person if something goes wrong
Anyway, I moved ~1000 miles south as the car [sic] flies since the time I started doing business there. I'm using Linux so it's not as if I ever need to visit the box anyway.
Oh, another bit...I did have a problem with the server not rebooting once. It refused to detect the SCSI adapter all of a sudden and I figured the adapter was dead. Norm called the owner of the computer store located in the same building. He came up and fixed it--then wouldn't even let me pay him for his time because all he had to do was re-seat the SCSI card. You don't get that kind of support when you're dealing with a big companies...
numb