What Hurricane Sandy Taught IT About Disaster Preparedness

StewBeans writes: The National Oceanic and Atmospheric Administration Climate Prediction Center is calling for calmer than normal storm activity this hurricane season, which runs through Nov. 30. But it's likely that data centers and IT companies in NYC are still taking disaster preparedness seriously. Three years ago, Hurricane Sandy devastated homes, businesses, transportation, and communication in New York, and taught many companies (the hard way) how to keep the lights on when the lights were literally off for weeks on end. Alphonzo Albright, former CIO of the Office of Information Technology in New York City, gives a behind-the-scenes account of what life and business were like in the dark, cold days following Hurricane Sandy in NYC. He also shares tips for other tech leaders to create their own Business Continuity Plan in case this year's storms take a turn for the worse.

Comments

Re:Geographic diversity

[Lumpy]

A week? most data disaster you are down for at least 30 days. Hell you cant get an order for servers in from DELL even on rush faster than 2 weeks.

If your company can survive zero revenue and 100% loss for 30 days, you either are sitting on a mountain of money, or your business is more of a hobby than anything else.

Oh and if you lose your accounting data due to lack of a bomb proof plan, expect fines in the high 6 figure range.

Re:Geographic diversity

[nine-times]

Do companies actually bulk-order from Dell any more? This is actually the most I've heard about Dell for months.

I know Google manufactures their own computers, for the most part.

So you think just because Google builds their own servers, it must be that everyone else does the same? There are a few companies out there that aren't Google, and yes, many of them still buy from Dell or HP.

Re:Geographic diversity

[dgatwood]

A week? most data disaster you are down for at least 30 days. Hell you cant get an order for servers in from DELL even on rush faster than 2 weeks.

Maybe true, but you can get a cloud server deployed in a matter of minutes, and you can use that as a temporary (expensive) alternative to servers under your complete control.

If your company can survive zero revenue and 100% loss for 30 days, you either are sitting on a mountain of money, or your business is more of a hobby than anything else.

You're making a lot of assumptions that aren't necessarily valid. The amount of downtime and the impact depends heavily on the nature of the company, and in particular, whether sales/income depends on maintaining continuous operation of the business. Take, for example, a company that makes software:

• On the development side, even if a company's entire repository went away tomorrow, and even if half the development team died, a typical software company could still get back all but the last few days' work (and perhaps a few old branches) by configuring a github instance on Amazon's Elastic Cloud and having the remaining developers push all of the branches from their local checkouts. Downtime would be minimal.
• On the distribution site, most software companies would be completely unaffected, because distribution is usually handled by a large third-party merchant (Apple, Google, etc.).

So unless a software company requires critical server infrastructure beyond what they get for free via iCloud, etc., it probably needs very little in the way of disaster preparedness, because the very nature of the work and the tools involved lends itself to being prepared for a disaster automatically.

On the opposite end of the spectrum, cloud service providers and Internet service companies must have disaster preparedness plans in place, or else everybody who uses their services is screwed. And if they're down for even a couple of days, they're probably going out of business. If Facebook went down for a week, Google+ would become the #1 social network.