Slashdot Mirror

← Back to Stories (view on slashdot.org)

ISP Recovers in 72 Hours After Leveling by Tornado

Posted by CmdrTaco on from the now-that-is-what-i-call-disaster-recovery dept.

aldheorte writes "Amazing story of how an ISP in Jackson, TN, whose main facility was completely leveled by a tornado, recovered in 72 hours. The story is a great recounting of how they executed their disaster recovery plan, what they found they had left out of that plan, data recovery from destroyed hard drives, and perhaps the best argument ever for offsite backups. (Not affiliated with the ISP in question)"

24 of 258 comments (clear)

  1. Heh by B3ryllium · · Score: 4, Funny

    Hopefully no one was hurt when the trailer park got levelled.

  2. That's fricking awesome by Anonymous Coward · · Score: 5, Funny

    "So, ah, your ISP here.. what's your uptime for the last year?"

    "99.18% for our service, and 96.2% for our building."

  3. Poor tech support by dswensen · · Score: 5, Funny

    And I'm sure every minute of those 72 hours was characterized by irate phone calls to tech support.

    "Are you guys down again? You're down more than you're up! I'm going to find another service... etc..."

    "Ma'am our facilities have been entirely leveled by a tornado, we'll be back up in 72 hours."

    "72 HOURS?! I have photos of my grandchildren I have to mail! Worst ISP ever! Let me speak to your supervisor!"

    "Ma'am our supervisor was also leveled by the tornado."

    *click*

    Not that I work tech support for an ISP and am bitter...

    1. Re:Poor tech support by koa · · Score: 4, Insightful

      Actually.. I ran a technical support department for a small ISP for a couple years.

      It amazing how accurate you are in reguards to customer viewpoint on downtime.

      After having done it myself, I actually have MUCH more respect for technicul support engineers/supervisors becuase within reason most "downtime" is fixed even before the customer knows about it (i.e. small blips in service).

      And the majority of people who purchase an ISP's services have absolutely no idea what it takes to respond to an outtage.

      --
      ....move along....nothing to see here....
  4. Nice work! by Tebriel · · Score: 4, Insightful

    This is what happens when people make intelligent plans and the modify them as they see other plans work or fail. I'm glad to see that this was a work in progress rather than some arcane plan in a binder somewhere that no one ever looked at.

    --
    The Blaster Master Fighting for Truth, Justice, and Evil Pie since 1979
  5. Elephant Insurance by Bob+Vila's+Hammer · · Score: 5, Funny

    When your business gets pelted with the equivalent force of 100,000 elephants, you better have a friggin contingency plan.

    --


    --"The perfect example of the man of action is the suicide." - William Carlos Williams
  6. Fire... by Shut+the+fuck+up! · · Score: 5, Insightful

    ...is a good enough argument for off site backups. If you don't have them, your backup plan is not enough.

    1. Re:Fire... by Zathrus · · Score: 5, Insightful

      Everyone should have off-site backups. It's not very expensive (>100 dollars for tapes)

      Er, for how much data? For your personal computer, maybe (but the tape drive will cost you considerably more than that $100), but I don't think you're going to back up a few hundred gigs of business data on ~$100 of tapes. And I suspect you meant 100... although if the latter then you're almost certainly correct!

      It's not very hard (drive tapes to site). It's not difficult to get the backups if you need them (drive to site with tapes)

      If your offsite backup is within convienent driving distance then odds are it's not far enough offsite. A flood, tornado, hurricane, earthquake, or other large scale natural disaster could conceivably destroy both your onsite and offsite backups if they're within a few miles. The flipside is that the further the distance the more the inconvienence on an ongoing basis and the more likely you are to stop doing backups.

      There's far more to be considered here, but I'm not the DR expert (my wife is... seriously). It does make sense to have offsite backups, but you have to have some sense about those too.

  7. Tornad'oh! by AtariAmarok · · Score: 4, Funny

    Let the OZ jokes flow:

    "Bring me the router of the wicked switch of the Qwest!"

    Although, I am starting to wonder. Has anyone checked to see if this ISP has a record of resisting RIAA subpeonas? Perhaps the RIAA levelled it after acquiring cloudbuster equipment.

    --
    Don't blame Durga. I voted for Centauri.
  8. Compare and contrast... by ptomblin · · Score: 4, Interesting

    A couple of friends of mine were badly burned because the web hosting company they were using lost all their data (customer and their own) in one humungous crash, and didn't have any backups. They didn't even have a spare copy of their customer database, so they couldn't even contact their customers to tell them what was going on. Nor could they tell what customers they had and how much service they'd paid for, etc.

    --
    The next Cmdr Taco duplicate will be ready soon, but subscribers can beat the rush and see it early!
  9. Before someone else says it... by wo1verin3 · · Score: 5, Funny

    No, in Russia Tornado does not own you. Neither does ISP. It is not, step 1) tornado step 2) ??? step 3) ISP recovers. There is not a beowulf cluster of these, and the tornado doesn't run Linux.

    1. Re:Before someone else says it... by Trigun · · Score: 5, Funny

      the tornado doesn't run Linux.

      No, it runs .NET. There's a lot of huffing and puffing, nobody knows too much about it, and in the end your business is in shambles and half your IT staff is no longer.

      -3 Stupid.

  10. Cool, but could be better by MicroBerto · · Score: 4, Insightful
    While that's awesome, I still think that small businesses and big ones should both have offsite tape backups. Even if this means the owner brings back and forth a case of tapes to his home once a week or so. That alone would have saved much of this trouble.

    Then I've seen the other end of the spectrum - a 6 Billion dollar corporation's world HQ IT center... wow. They have disaster recovery sessions and planning like I never would have imagined. Very cool facility, but it has to be like that. Some day if they get burned, it's all over.

    --
    Berto
  11. Re:And then gets slashdotted by cindik · · Score: 4, Interesting

    That's actually interesting - how many sites have contingency plans for the /. effect? How many businesses? It's not just /., but just about any media can refer people to a real business site. For small companies, this could bring them down for some time. Imagine the "Bruce Almighty" effect, only with some business with a small-to-medium capacity connection, bombarded just because someone used http://www.slashdotme.com/ or spam@.me.into.oblivion.org in their movie. The fact that so many sites are taken down by the /. effect causes me to believe that few sites and those who run them are truly prepared.

    --
    Terrycloth Lobster
  12. Re:Amazing is an innapropriate adjective by HardCase · · Score: 5, Funny
    I realize that slashdot is mostly populated by high-school educated "IT people", who give a shit about logs and backups and think plugging a PC and monitor into a powerbar is "computer science". To these people, the prospect of plugging in a bunch of computers and restoring backup tapes is exhillirating and exciting. The highlight of their lives.

    But, as a programmer, I just dont care.



    When I was a sophomore, working on my electrical engineering degree, I worked for a small, network-centric company that employed what seemed to be an abnormal number of snooty programmers and technical writers. Maybe it wasn't so abnormal.



    Me: "Hi, IT support."
    Stratjakt: "Hey, I know you're just a high-school educated 'IT person', but you need to get one of your cable monkeys up here and find out why I can't see the network!"
    Me:: "OK, but let's check a couple of things quickly before I dispatch a technician. It may save some time."
    Stratjakt: "Hey, I'm a programmer! I just don't care!"
    Me: "I understand...I realize that my mundane existance doesn't have the exhilaration and exitedness of the thrilling, edge-of-your-seat world of a computer programmer, but there are just a few simple things that we could do to resolve this problem that will be faster than you waiting for a technician."
    Stratjakt: "I just don't care."
    Me: "No problem, I'll dispatch a technican."


    An hour later...


    Technician: "Stratjakt is all fixed up. I plugged his network cable back into the jack."

  13. Truly stunning by dbarclay10 · · Score: 5, Insightful

    What amazes me isn't that these people were able to restore service to their customers in 72 hours. They used standard systems administration techniques. BGP was specifically mentioned.

    No, what amazes me is that this is news. The IT industry is so full of idiots and morons and MCSEs that taking basic precautions earns you a six-figure salary and news coverage. These folks didn't even have off-site backups, it was luck that they were able to resume business operations (ie: billing) so soon.

    Moral of the story? When automobile manufacturers start getting press coverage for doing a great job because unlike their competition, they install brakes in their vehicles, you know that the top-tier IT managers and executives have switched industries.

    --

    Barclay family motto:
    Aut agere aut mori.
    (Either action or death.)
    1. Re:Truly stunning by HardCase · · Score: 5, Interesting
      No, what amazes me is that this is news. The IT industry is so full of idiots and morons and MCSEs that taking basic precautions earns you a six-figure salary and news coverage. These folks didn't even have off-site backups, it was luck that they were able to resume business operations (ie: billing) so soon.


      I agree, although maybe not so vehemently. For the IT managers who need a clue, the article is evidence that a sound disaster recovery plan works. Obviously, in the case of the ISP, the plan wasn't completely sound, but the other, possibly more important, point of the article is that the ISP's management recognized that their recovery plan was incomplete. Based on the lessons they learned, they made changes.


      I work for a large (~20,000 employees) company, with about 10,000 employees at one site. The IT department (actually the entire company as well) has a disaster recovery plan in place. But beyond having a plan, we also have drills. As an example, we are in the flight path of the local airport (possibly not the best place in the world for a manufacturing site). What happens if a plane crashes smack in the middle of the plant? Hopefully we'll never know for sure, but the drills that we've run showed strong and weak points of the disaster plan. The strong points were emphasized, the weak points were revised and the disaster plan continues as a work in progress.


      Specifics aside, and maybe this is just stating the obvious, but considering a disaster recovery plan to be a continuously evolving procedure could be one of its strongest points.


      -h-

  14. Re:However... by MachineShedFred · · Score: 5, Funny

    I, for one, welcome our new Tornado-beating ISP overlords.

    --
    Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
  15. But... by macshune · · Score: 5, Funny

    Can they recover from the slashdot effect???

    The slashdot effect differs from a tornado in a few subtle ways:

    1) You can't see it coming (unless you pay money to be a subscriber)

    2) It doesn't hurt anything, except for webservers, the occasional OC line lit up like New Year's Eve, spammers, and the odd *IAA executive.

    3) A tornado doesn't typically smell like armpits, cheetos, empty 64oz soda cups, burning plastic, your parent's basement and/or too much cologne for that first date.

    4) It travels at the speed of light, a lot quicker than a tornado.

    5) Does not require specific atmospheric conditions to be present...just a link on the front page.

    Anything else?

    1. Re:But... by BMonger · · Score: 5, Interesting

      Hmmm... what if a website admin did become a subscriber. Could they theoretically take the RSS feed to know when a new post was made, pull the article text, scan it for their domain and if their domain was linked to just have a script auto-block referers from slashdot for like 24 hours or so? Somebody less lazy than me might look into that. Then you could sell it for like $100! It'd be like paying the mob not to beat you up! But only if somebody affiliated with slashdot wrote it I guess.

    2. Re:But... by BMonger · · Score: 4, Funny

      Ohhh! Or even better yet! Have your site auto-post the articles text in a slashdot comment plus block the slashdot referer header for 24 hours! (patent pending)

  16. Re:And then gets slashdotted by Fishstick · · Score: 4, Informative

    that's computerworld receiving the /.ing

    the isp is here

    picture of the aftermath here

    --

    There is much cruelty in the universe, John.
    Yeah, we seem to have the tour map.

  17. I live in Jackson.... by Daniel+Wood · · Score: 5, Interesting

    I am also a former Aeneas customer.
    Unless Aeneas has made some major changes they are quite certainly the worst ISP I have ever worked with. Aeneas has contracts with the Jackson-Madison County School System to provide internet service district wide. The quality of such service is, bar none, the worst I have experienced.
    I did some volunteer work at a local Elementary school helping teachers work out any lingering computing problems they had(Virii, printer drivers, misconfigured ip settings, file transfer to a new computer, etc). The internet service I experienced while I was there lead me to believe I was on a 128k ISDN line. Not until I went to the server room did I realize that I was, infact, on a T1. Now this is during the middle of summer, mabye four other persons were in the building, three of which were in the same room as myself. The service was also intermittent, having several dead periods while I was working. Needless to say, I remained unimpressed by said experience.

    When I was an Aeneas dialup customer, in 1998, the service provided by Aeneas was also subpar. The dialup speeds were averaging 21.6kbps, where as when I switched to U.S. Internet(now owned by Earthlink) my dialup speeds were always above 26.4kbps(Except on Mother's Day). There were frequent disconnections, and they had a limit of 150hrs/month.

    I'm not supprised how easy it is to restore subpar service. All they had to do was tie together the strings that are their backbone.

  18. Been there, done that, Northridge Quake by Tsu+Dho+Nimh · · Score: 5, Interesting
    I was playing minute-person at a "disaster recovery" meeting (the first one) where high-level suits were figuring out what to do in case of a disaster at their multi-state bank. Their core assumptions were initially as follows:
    • They would all survive whatever it was. (I was looking out the window, and seeing jetliners coming in for a landing ... a few feet too low and the meeting would have been over).
    • All critical equipment would survive in repairable condition.
    • Public services would not be affected over a wide area or for a long time.
    • Critical personnel would be available as needed, as would the transportation to get them there.
    • The disaster plan only needed to be distributed to managers, who would instruct people what to do to recover.

    That was on a Monday. The next Monday was the Northridge quake.

    • One critical person woke up with his armoir on top of him, and a 40-foot chasm between him and the freeway.
    • One of their buildings was so badly damaged that they were banned from entering ... and there was mission-critical info on those desktop PCs. Had it not been a holiday, the casualty toll would have been horrendous.
    • The building with their backups was on the same power grid as the one with no power and the generators could only power the computers, not the AC they also needed.
    • None of the buildings had food or water for the staff who had to sleep over, nor did they have working toilets or even cots to nap on.
    • One of the local competitors was back in business Tuesday morning, because their disaster plan worked. They rolled up the trailers, swapped some cables and were going again.

    They came into the next meeting a couple of weeks after the quake with a whole new perspective on disaster planning and training:

    • Anyone who survives knows what the disaster plan is and copies of it are all over the place.
    • Critical equipment is redundant and "offsite" backups are out of the quake zone.
    • They have generators and fuel enough to last a couple of weeks for the critical equipment and it's support, survival supplies for the critical staff. This is rotated regularly to keep it form going stale.
    • They cross-trained like mad.
    • They started testing the plan regularly.