Slashdot Mirror

← Back to Stories (view on slashdot.org)

ISP Recovers in 72 Hours After Leveling by Tornado

Posted by CmdrTaco on from the now-that-is-what-i-call-disaster-recovery dept.

aldheorte writes "Amazing story of how an ISP in Jackson, TN, whose main facility was completely leveled by a tornado, recovered in 72 hours. The story is a great recounting of how they executed their disaster recovery plan, what they found they had left out of that plan, data recovery from destroyed hard drives, and perhaps the best argument ever for offsite backups. (Not affiliated with the ISP in question)"

15 of 258 comments (clear)

  1. That's fricking awesome by Anonymous Coward · · Score: 5, Funny

    "So, ah, your ISP here.. what's your uptime for the last year?"

    "99.18% for our service, and 96.2% for our building."

  2. Poor tech support by dswensen · · Score: 5, Funny

    And I'm sure every minute of those 72 hours was characterized by irate phone calls to tech support.

    "Are you guys down again? You're down more than you're up! I'm going to find another service... etc..."

    "Ma'am our facilities have been entirely leveled by a tornado, we'll be back up in 72 hours."

    "72 HOURS?! I have photos of my grandchildren I have to mail! Worst ISP ever! Let me speak to your supervisor!"

    "Ma'am our supervisor was also leveled by the tornado."

    *click*

    Not that I work tech support for an ISP and am bitter...

  3. Elephant Insurance by Bob+Vila's+Hammer · · Score: 5, Funny

    When your business gets pelted with the equivalent force of 100,000 elephants, you better have a friggin contingency plan.

    --


    --"The perfect example of the man of action is the suicide." - William Carlos Williams
  4. Fire... by Shut+the+fuck+up! · · Score: 5, Insightful

    ...is a good enough argument for off site backups. If you don't have them, your backup plan is not enough.

    1. Re:Fire... by Zathrus · · Score: 5, Insightful

      Everyone should have off-site backups. It's not very expensive (>100 dollars for tapes)

      Er, for how much data? For your personal computer, maybe (but the tape drive will cost you considerably more than that $100), but I don't think you're going to back up a few hundred gigs of business data on ~$100 of tapes. And I suspect you meant 100... although if the latter then you're almost certainly correct!

      It's not very hard (drive tapes to site). It's not difficult to get the backups if you need them (drive to site with tapes)

      If your offsite backup is within convienent driving distance then odds are it's not far enough offsite. A flood, tornado, hurricane, earthquake, or other large scale natural disaster could conceivably destroy both your onsite and offsite backups if they're within a few miles. The flipside is that the further the distance the more the inconvienence on an ongoing basis and the more likely you are to stop doing backups.

      There's far more to be considered here, but I'm not the DR expert (my wife is... seriously). It does make sense to have offsite backups, but you have to have some sense about those too.

  5. Before someone else says it... by wo1verin3 · · Score: 5, Funny

    No, in Russia Tornado does not own you. Neither does ISP. It is not, step 1) tornado step 2) ??? step 3) ISP recovers. There is not a beowulf cluster of these, and the tornado doesn't run Linux.

    1. Re:Before someone else says it... by Trigun · · Score: 5, Funny

      the tornado doesn't run Linux.

      No, it runs .NET. There's a lot of huffing and puffing, nobody knows too much about it, and in the end your business is in shambles and half your IT staff is no longer.

      -3 Stupid.

  6. Re:Amazing is an innapropriate adjective by HardCase · · Score: 5, Funny
    I realize that slashdot is mostly populated by high-school educated "IT people", who give a shit about logs and backups and think plugging a PC and monitor into a powerbar is "computer science". To these people, the prospect of plugging in a bunch of computers and restoring backup tapes is exhillirating and exciting. The highlight of their lives.

    But, as a programmer, I just dont care.



    When I was a sophomore, working on my electrical engineering degree, I worked for a small, network-centric company that employed what seemed to be an abnormal number of snooty programmers and technical writers. Maybe it wasn't so abnormal.



    Me: "Hi, IT support."
    Stratjakt: "Hey, I know you're just a high-school educated 'IT person', but you need to get one of your cable monkeys up here and find out why I can't see the network!"
    Me:: "OK, but let's check a couple of things quickly before I dispatch a technician. It may save some time."
    Stratjakt: "Hey, I'm a programmer! I just don't care!"
    Me: "I understand...I realize that my mundane existance doesn't have the exhilaration and exitedness of the thrilling, edge-of-your-seat world of a computer programmer, but there are just a few simple things that we could do to resolve this problem that will be faster than you waiting for a technician."
    Stratjakt: "I just don't care."
    Me: "No problem, I'll dispatch a technican."


    An hour later...


    Technician: "Stratjakt is all fixed up. I plugged his network cable back into the jack."

  7. Truly stunning by dbarclay10 · · Score: 5, Insightful

    What amazes me isn't that these people were able to restore service to their customers in 72 hours. They used standard systems administration techniques. BGP was specifically mentioned.

    No, what amazes me is that this is news. The IT industry is so full of idiots and morons and MCSEs that taking basic precautions earns you a six-figure salary and news coverage. These folks didn't even have off-site backups, it was luck that they were able to resume business operations (ie: billing) so soon.

    Moral of the story? When automobile manufacturers start getting press coverage for doing a great job because unlike their competition, they install brakes in their vehicles, you know that the top-tier IT managers and executives have switched industries.

    --

    Barclay family motto:
    Aut agere aut mori.
    (Either action or death.)
    1. Re:Truly stunning by HardCase · · Score: 5, Interesting
      No, what amazes me is that this is news. The IT industry is so full of idiots and morons and MCSEs that taking basic precautions earns you a six-figure salary and news coverage. These folks didn't even have off-site backups, it was luck that they were able to resume business operations (ie: billing) so soon.


      I agree, although maybe not so vehemently. For the IT managers who need a clue, the article is evidence that a sound disaster recovery plan works. Obviously, in the case of the ISP, the plan wasn't completely sound, but the other, possibly more important, point of the article is that the ISP's management recognized that their recovery plan was incomplete. Based on the lessons they learned, they made changes.


      I work for a large (~20,000 employees) company, with about 10,000 employees at one site. The IT department (actually the entire company as well) has a disaster recovery plan in place. But beyond having a plan, we also have drills. As an example, we are in the flight path of the local airport (possibly not the best place in the world for a manufacturing site). What happens if a plane crashes smack in the middle of the plant? Hopefully we'll never know for sure, but the drills that we've run showed strong and weak points of the disaster plan. The strong points were emphasized, the weak points were revised and the disaster plan continues as a work in progress.


      Specifics aside, and maybe this is just stating the obvious, but considering a disaster recovery plan to be a continuously evolving procedure could be one of its strongest points.


      -h-

  8. Re:However... by MachineShedFred · · Score: 5, Funny

    I, for one, welcome our new Tornado-beating ISP overlords.

    --
    Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
  9. But... by macshune · · Score: 5, Funny

    Can they recover from the slashdot effect???

    The slashdot effect differs from a tornado in a few subtle ways:

    1) You can't see it coming (unless you pay money to be a subscriber)

    2) It doesn't hurt anything, except for webservers, the occasional OC line lit up like New Year's Eve, spammers, and the odd *IAA executive.

    3) A tornado doesn't typically smell like armpits, cheetos, empty 64oz soda cups, burning plastic, your parent's basement and/or too much cologne for that first date.

    4) It travels at the speed of light, a lot quicker than a tornado.

    5) Does not require specific atmospheric conditions to be present...just a link on the front page.

    Anything else?

    1. Re:But... by BMonger · · Score: 5, Interesting

      Hmmm... what if a website admin did become a subscriber. Could they theoretically take the RSS feed to know when a new post was made, pull the article text, scan it for their domain and if their domain was linked to just have a script auto-block referers from slashdot for like 24 hours or so? Somebody less lazy than me might look into that. Then you could sell it for like $100! It'd be like paying the mob not to beat you up! But only if somebody affiliated with slashdot wrote it I guess.

  10. I live in Jackson.... by Daniel+Wood · · Score: 5, Interesting

    I am also a former Aeneas customer.
    Unless Aeneas has made some major changes they are quite certainly the worst ISP I have ever worked with. Aeneas has contracts with the Jackson-Madison County School System to provide internet service district wide. The quality of such service is, bar none, the worst I have experienced.
    I did some volunteer work at a local Elementary school helping teachers work out any lingering computing problems they had(Virii, printer drivers, misconfigured ip settings, file transfer to a new computer, etc). The internet service I experienced while I was there lead me to believe I was on a 128k ISDN line. Not until I went to the server room did I realize that I was, infact, on a T1. Now this is during the middle of summer, mabye four other persons were in the building, three of which were in the same room as myself. The service was also intermittent, having several dead periods while I was working. Needless to say, I remained unimpressed by said experience.

    When I was an Aeneas dialup customer, in 1998, the service provided by Aeneas was also subpar. The dialup speeds were averaging 21.6kbps, where as when I switched to U.S. Internet(now owned by Earthlink) my dialup speeds were always above 26.4kbps(Except on Mother's Day). There were frequent disconnections, and they had a limit of 150hrs/month.

    I'm not supprised how easy it is to restore subpar service. All they had to do was tie together the strings that are their backbone.

  11. Been there, done that, Northridge Quake by Tsu+Dho+Nimh · · Score: 5, Interesting
    I was playing minute-person at a "disaster recovery" meeting (the first one) where high-level suits were figuring out what to do in case of a disaster at their multi-state bank. Their core assumptions were initially as follows:
    • They would all survive whatever it was. (I was looking out the window, and seeing jetliners coming in for a landing ... a few feet too low and the meeting would have been over).
    • All critical equipment would survive in repairable condition.
    • Public services would not be affected over a wide area or for a long time.
    • Critical personnel would be available as needed, as would the transportation to get them there.
    • The disaster plan only needed to be distributed to managers, who would instruct people what to do to recover.

    That was on a Monday. The next Monday was the Northridge quake.

    • One critical person woke up with his armoir on top of him, and a 40-foot chasm between him and the freeway.
    • One of their buildings was so badly damaged that they were banned from entering ... and there was mission-critical info on those desktop PCs. Had it not been a holiday, the casualty toll would have been horrendous.
    • The building with their backups was on the same power grid as the one with no power and the generators could only power the computers, not the AC they also needed.
    • None of the buildings had food or water for the staff who had to sleep over, nor did they have working toilets or even cots to nap on.
    • One of the local competitors was back in business Tuesday morning, because their disaster plan worked. They rolled up the trailers, swapped some cables and were going again.

    They came into the next meeting a couple of weeks after the quake with a whole new perspective on disaster planning and training:

    • Anyone who survives knows what the disaster plan is and copies of it are all over the place.
    • Critical equipment is redundant and "offsite" backups are out of the quake zone.
    • They have generators and fuel enough to last a couple of weeks for the critical equipment and it's support, survival supplies for the critical staff. This is rotated regularly to keep it form going stale.
    • They cross-trained like mad.
    • They started testing the plan regularly.