ISP Recovers in 72 Hours After Leveling by Tornado

aldheorte writes "Amazing story of how an ISP in Jackson, TN, whose main facility was completely leveled by a tornado, recovered in 72 hours. The story is a great recounting of how they executed their disaster recovery plan, what they found they had left out of that plan, data recovery from destroyed hard drives, and perhaps the best argument ever for offsite backups. (Not affiliated with the ISP in question)"

Comments

Nice work!

[Tebriel]

This is what happens when people make intelligent plans and the modify them as they see other plans work or fail. I'm glad to see that this was a work in progress rather than some arcane plan in a binder somewhere that no one ever looked at.

Re:Nice work!

[blackp]

One of the problems with a plan in a binder somewhere, is that the tornado would have probably taken out the binder as well.

Fire...

[Shut the fuck up!]

...is a good enough argument for off site backups. If you don't have them, your backup plan is not enough.

Re:Fire...

[Zathrus]

Everyone should have off-site backups. It's not very expensive (>100 dollars for tapes)

Er, for how much data? For your personal computer, maybe (but the tape drive will cost you considerably more than that $100), but I don't think you're going to back up a few hundred gigs of business data on ~$100 of tapes. And I suspect you meant 100... although if the latter then you're almost certainly correct!

It's not very hard (drive tapes to site). It's not difficult to get the backups if you need them (drive to site with tapes)

If your offsite backup is within convienent driving distance then odds are it's not far enough offsite. A flood, tornado, hurricane, earthquake, or other large scale natural disaster could conceivably destroy both your onsite and offsite backups if they're within a few miles. The flipside is that the further the distance the more the inconvienence on an ongoing basis and the more likely you are to stop doing backups.

There's far more to be considered here, but I'm not the DR expert (my wife is... seriously). It does make sense to have offsite backups, but you have to have some sense about those too.

Cool, but could be better

[MicroBerto]

While that's awesome, I still think that small businesses and big ones should both have offsite tape backups. Even if this means the owner brings back and forth a case of tapes to his home once a week or so. That alone would have saved much of this trouble.

Then I've seen the other end of the spectrum - a 6 Billion dollar corporation's world HQ IT center... wow. They have disaster recovery sessions and planning like I never would have imagined. Very cool facility, but it has to be like that. Some day if they get burned, it's all over.

Truly stunning

[dbarclay10]

What amazes me isn't that these people were able to restore service to their customers in 72 hours. They used standard systems administration techniques. BGP was specifically mentioned.

No, what amazes me is that this is news. The IT industry is so full of idiots and morons and MCSEs that taking basic precautions earns you a six-figure salary and news coverage. These folks didn't even have off-site backups, it was luck that they were able to resume business operations (ie: billing) so soon.

Moral of the story? When automobile manufacturers start getting press coverage for doing a great job because unlike their competition, they install brakes in their vehicles, you know that the top-tier IT managers and executives have switched industries.

make sure off-site is far enough away

[DiveX]

Many companies in the World Trade Center thought that off-site backup meant the other building.

Re:Poor tech support

[koa]

Actually.. I ran a technical support department for a small ISP for a couple years.

It amazing how accurate you are in reguards to customer viewpoint on downtime.

After having done it myself, I actually have MUCH more respect for technicul support engineers/supervisors becuase within reason most "downtime" is fixed even before the customer knows about it (i.e. small blips in service).

And the majority of people who purchase an ISP's services have absolutely no idea what it takes to respond to an outtage.

Not good enough

[vasqzr]

When you go to a DRP seminar, they make the claim that the majority of business that are knocked out for longer than 48 hours go out of business within 1 year.