Slashdot Mirror
NYC Data Center Needs Focus On Fuel
Nerval's Lobster writes "Who knew that the most critical element of operating a data center in New York City was ensuring a steady supply of diesel fuel? In the wake of Hurricane Sandy, the challenges facing data center operators in the affected zones include pumping water from basements, waiting for utility power to be restored, and managing fuel-truck deliveries. And it's become increasingly clear which companies had the resources and foresight to plan for a disaster like Sandy, and which are simply reacting. Here's the latest on providers around the New York area." And remember, having fuel for machines sometimes only means it's time to start the manual labor.
Cell sites I worked in Africa run a pair of Cummins generators as their main power. In the unlikely event that both of these fail at the same time, there's a chance that the main grid might be working well enough to take over. But the fuel is the biggest priority on these sites.
At the moment, we have tankers on site at all of our NY DCs. 2 of them are on generator only, so we're topping up the tanks every 4 hours. The generators will need full overhauls when we're back on the grid properly, but for now we're keeping our clients serviced which is what matters.
If this was their first time, or it had been a few years, I would expect there to be mass hysteria and a general failure of DR plans. Mostly because DR plans are theoretical and costly to test and not tested very often. However I think Irene gleaned much useful information for those developing these DR plans which led to NY data centers being better prepared. I'd love to read the DR plans in 6 months and compare the new changes from lessons learned during and after this storm.
One of my buds is IT director for a company that resupplies generators. The logisitics for it are crazy as you route trucks on available streets, deal with priority of the customers (hospitals front of the line) and optimize resupply into mostly empty tanks before they actually empty, etc... And you have >10 days of this 24/7 after a storm.
After Hurricane Allison here, some companies in this sector went out of business before the power came back on. We were better prepared for Ike, but I think that's because facilities are more willing to sign contracts with the pricey but reputable businesses.
This is the most expensive way to get fuel: you need a massive amount for a short time, and you need it consistently during that time.
Which makes me question the wisdom of urban datacenters in the first place.
It's nothing particularly special to the datacenter being in a city. Get a big enough disaster and you will be knocked off the net, wherever you are and whatever steps you take to prevent it. The best mitigation strategy is to have geographically distributed datacenters that can run replicated services that are structured so that losing one site entirely is not crippling (though it might hurt a lot). This isn't a simple thing to do — for most types of service, you have to design the overall service from the ground up to work that way — but it's not too bad. (And at least with the Cloud you don't have to build your own collection of geographically-separated datacenters or pound your head against the problems you usually get with hosting providers when you're desperate for service in a hurry.)
Mind you, rents in NYC are so high that I'd guess that nobody's going there for the cheapness of the service. Instead, it's probably for the proximity to the Wall St. exchanges. The fun thing with that? Any datacenter which could satisfy the latency requirements for people in that business would have been deeply hit by this particular disaster, and all the places that could easily offer continuous service this time would never be able to normally compete for this particular type of business. To double the fun, the exchanges were closed for two days anyway. That's modern capitalism at work...
"Little does he know, but there is no 'I' in 'Idiot'!"
30 years ago, I retired from the Navy, along with a friend who was a Machinist Mate on Navy submarines. I didn't see him for a couple years. Turns out that he had spent 10 of those years servicing batteries on diesel submarines, and ended up servicing large batteries for the phone company -- about the same size batteries that had been on the boats. He switched to servicing the diesels (because of his prior experience on submarines with their diesel generators), and one of the first to deploy natural gas fired diesels for sustained power, first for the phone company, and then for a local utility.
There's the solution. Don't pipe flammable liquid to the top floor, pipe the natural gas. Contrary to popular belief, it is a safer fuel, and requires less maintenance. Automatic shutoff valves work better, less explosive volume released on a tubing breakage, and it doesn't rot the pipes like liquid diesel fuel, disperses to atmosphere in the case of a leak, and doesn't make everything around it flammable, when it does leak.
Engines run at least 4 times as long between between overhaul cycles, and it doesn't dilute the lube oil.
Natural gas is going to be a HUGE change for this countries infrastructure, both in common usage, as well as emergency failover services.
This guy has made tons of money, by the way. Has a condo and car in San Diego, LA, Seattle, New York, and Atlanta -- cheaper than hotels and taxis, and doesn't know what else to do with his money.