What Hurricane Sandy Taught IT About Disaster Preparedness

StewBeans writes: The National Oceanic and Atmospheric Administration Climate Prediction Center is calling for calmer than normal storm activity this hurricane season, which runs through Nov. 30. But it's likely that data centers and IT companies in NYC are still taking disaster preparedness seriously. Three years ago, Hurricane Sandy devastated homes, businesses, transportation, and communication in New York, and taught many companies (the hard way) how to keep the lights on when the lights were literally off for weeks on end. Alphonzo Albright, former CIO of the Office of Information Technology in New York City, gives a behind-the-scenes account of what life and business were like in the dark, cold days following Hurricane Sandy in NYC. He also shares tips for other tech leaders to create their own Business Continuity Plan in case this year's storms take a turn for the worse.

Comments

Geographic diversity

[Todd Knarr]

First rule: have facilities capable of running your business in more than one location. Everywhere is susceptible to disaster of one sort or another, but if you pick areas far apart that aren't geographically similar they probably won't both suffer disasters at the same time.

Second rule: the probability of disaster taking out your main facilities is 100%. It will happen. The only question is exactly when it'll happen, and the only constant in the answer is that it won't be at a good time. If anyone in your organization doesn't like this, remind them that reality doesn't really care what they like.

Re:Geographic diversity

[turbidostato]

I should add a rule zero then: Take your time to properly understand your costs and revenues so you can make a sensible investment. Maybe it ends up being cheaper just to close door for a week every 30 years than your A-Bomb-proof continuity plan.

And then a zero-plus: Make sure you get business-aligment in written. Maybe the board member that agreed to your investment-sensible less-than-A-Bomb-proof continuity plan wants you as scapegoat once the shit hits the fan.

Re:Geographic diversity

[Lumpy]

A week? most data disaster you are down for at least 30 days. Hell you cant get an order for servers in from DELL even on rush faster than 2 weeks.

If your company can survive zero revenue and 100% loss for 30 days, you either are sitting on a mountain of money, or your business is more of a hobby than anything else.

Oh and if you lose your accounting data due to lack of a bomb proof plan, expect fines in the high 6 figure range.

Re:Geographic diversity

[nine-times]

Take your time to properly understand your costs and revenues so you can make a sensible investment. Maybe it ends up being cheaper just to close door for a week every 30 years than your A-Bomb-proof continuity plan.

This is an amazingly difficult concept to get people to understand. I've had way too many conversations with people who are sure they need an instantaneous failure-proof disaster recovery plan. They believe their servers should be constantly in sync with multiple copies in various places, such that in the even of a short internet outage, their servers will fail over to an outside copy, and then fail back when the outage ends, automatically and without skipping a beat. Unfortunately, they're willing to spend approximately $0 to achieve this, but that should be fine, because "the cloud" is pretty much free, right?

It's a similar problem with security. Everyone wants all of their data to be completely secure without any possibility of being compromised under any circumstances, but they also want it to be as convenient as if the data is unsecured, and they don't expect to pay extra for any of it.

I always try to explain that it's about trade-offs. I can make your data much more secure than it is now, but it'll cost you money, and you'll have to jump through extra hoops to get access to your own data. I can replicate what you need to a remote server, yes, but then you have to pay for the remote server. Depending on exactly what we're talking about, it might not be a real-time sync, or it might not result in anything like an automatic failover. Those things might require special software or services or licenses. Pay enough, and yes, I can probably get you a real-time sync with automatic failover and fail-back, but even then, you could still have an outage. The system that keeps everything in sync and triggers the failover could be the component that fails. Or if there's a total blackout on the east cost, it might not matter that there's a complete replica automatically started on the west coast, if all your employees are on the east coast and without power.

It's trade-offs. Spend enough money and put up with enough limitations, and you'll get something that does what you want, although imperfectly. Most of the time, for most businesses, it doesn't make sense. "Good enough" is good enough. But people don't like to be told, "A pretty secure network with a pretty good disaster recovery plan is appropriate for you." It makes them feel unimportant, which most executives and business owners can't live with. They want to know that they should have the best thing possible.

Re:Geographic diversity

[fustakrakich]

Show me Google's manufacturing plants, please.

Aren't they on the North Pole? I hope they can float...

Re:Geographic diversity

[tlambert]

"I know Google manufactures their own computers, for the most part."

As a former Google employee, I must say you are full of shit.

Show me Google's manufacturing plants, please.

As a former Google employee myself, I'm bound by my NDA from naming the East Asia contractors who build the actual equipment. Google generally only provides the reference implementation.

Do you think Dell builds their own boards? They don't. The majority of their server class motherboards are manufactured by ASUS, based on Intel reference designs (Intel also no longer manufactures desktop motherboards, as of Haswell -- yields were too low).

If you are curious about who made your motherboard, and run Windows, use the following command:
wmic baseboard det product,Manufacturer,version,serialnumber

(If you want a GUI version, download "Speccy", run it, and either look for the "Motherboard" section in the "Summary" view, or click on the "Motherboard" list item to get only that information by itself).

Other OS's have their own commands, as an exercise for the student.

P.S.: If the information has been obfuscated, you can usually back-track by looking at the BIOS vendor and version information, and then using searches for updated/same versions of the BIOS based on that, to see which platforms the BIOS vendor says it's for. You are welcome.