Slashdot Mirror
NYC Data Center Needs Focus On Fuel
Nerval's Lobster writes "Who knew that the most critical element of operating a data center in New York City was ensuring a steady supply of diesel fuel? In the wake of Hurricane Sandy, the challenges facing data center operators in the affected zones include pumping water from basements, waiting for utility power to be restored, and managing fuel-truck deliveries. And it's become increasingly clear which companies had the resources and foresight to plan for a disaster like Sandy, and which are simply reacting. Here's the latest on providers around the New York area." And remember, having fuel for machines sometimes only means it's time to start the manual labor.
Which makes me question the wisdom of urban datacenters in the first place.
Once refined, it degrades. Oxidation, bacteria. Algae. Amazing things grow in diesel. You can add preservatives, but these only go so far. Keep it too long, and it's unusable. If you don't use it, you have to dispose of it. There's a large disincentive to keep diesel around.
Please do not read this sig. Thank you.
Everyone already knew that the on-site fuel supply is the limiting factor of power availability in a disaster. Even fuel delivery contracts mean nothing in a disaster or wide-spread outages - hospitals, EMS and other government services will trump the fuel delivery contract, if a hospital needs fuel, they are going to get the fuel that's been "guaranteed" for your datacenter.
There's no reason to spend big $$ creating a flood proof, earthquake proof, tornado proof, airplane crash proof datacenter in the middle of a city when you can have a disaster recovery site 1000 miles away that's not subject to the same type of disaster. (except maybe an asteroid strike, but there are few datacenters on the moon). No matter how disaster-proof you make your datacenter, mother nature (or man) will always find a way to create a disaster you didn't plan for -- even if that "disaster" is a typo in a router configuration file that takes down the network, or a contractor accidentally shorting out the emergency power cutoff switch wiring when bolting a rack to the wall.
Some of these servers were the "cloud".
The truth is that all men having power ought to be mistrusted. James Madison
Think. It is a fire hazard. As in "it starts leaking because of event X happening". And where does stuff go when it leaks? Yes, it goes down.
That is why fuel is stored DOWN in the basement. *Water*, on the other hand, you can store on the roof.
If you don't want to worry about your tank getting flooded, then make it water tight with air inlet well up high out of the water. There are safe and reliable ways of doing this without storing your fuel on the roof!!
You also need to make sure to bolt the damn tank down. Fuel and air are more buoyant than water or mud. More than a few flooded gas stations have suffered damage only because the tanked popped out of the ground.
One of my buds is IT director for a company that resupplies generators. The logisitics for it are crazy as you route trucks on available streets, deal with priority of the customers (hospitals front of the line) and optimize resupply into mostly empty tanks before they actually empty, etc... And you have >10 days of this 24/7 after a storm.
After Hurricane Allison here, some companies in this sector went out of business before the power came back on. We were better prepared for Ike, but I think that's because facilities are more willing to sign contracts with the pricey but reputable businesses.
This is the most expensive way to get fuel: you need a massive amount for a short time, and you need it consistently during that time.
Which makes me question the wisdom of urban datacenters in the first place.
It's nothing particularly special to the datacenter being in a city. Get a big enough disaster and you will be knocked off the net, wherever you are and whatever steps you take to prevent it. The best mitigation strategy is to have geographically distributed datacenters that can run replicated services that are structured so that losing one site entirely is not crippling (though it might hurt a lot). This isn't a simple thing to do — for most types of service, you have to design the overall service from the ground up to work that way — but it's not too bad. (And at least with the Cloud you don't have to build your own collection of geographically-separated datacenters or pound your head against the problems you usually get with hosting providers when you're desperate for service in a hurry.)
Mind you, rents in NYC are so high that I'd guess that nobody's going there for the cheapness of the service. Instead, it's probably for the proximity to the Wall St. exchanges. The fun thing with that? Any datacenter which could satisfy the latency requirements for people in that business would have been deeply hit by this particular disaster, and all the places that could easily offer continuous service this time would never be able to normally compete for this particular type of business. To double the fun, the exchanges were closed for two days anyway. That's modern capitalism at work...
"Little does he know, but there is no 'I' in 'Idiot'!"
30 years ago, I retired from the Navy, along with a friend who was a Machinist Mate on Navy submarines. I didn't see him for a couple years. Turns out that he had spent 10 of those years servicing batteries on diesel submarines, and ended up servicing large batteries for the phone company -- about the same size batteries that had been on the boats. He switched to servicing the diesels (because of his prior experience on submarines with their diesel generators), and one of the first to deploy natural gas fired diesels for sustained power, first for the phone company, and then for a local utility.
There's the solution. Don't pipe flammable liquid to the top floor, pipe the natural gas. Contrary to popular belief, it is a safer fuel, and requires less maintenance. Automatic shutoff valves work better, less explosive volume released on a tubing breakage, and it doesn't rot the pipes like liquid diesel fuel, disperses to atmosphere in the case of a leak, and doesn't make everything around it flammable, when it does leak.
Engines run at least 4 times as long between between overhaul cycles, and it doesn't dilute the lube oil.
Natural gas is going to be a HUGE change for this countries infrastructure, both in common usage, as well as emergency failover services.
This guy has made tons of money, by the way. Has a condo and car in San Diego, LA, Seattle, New York, and Atlanta -- cheaper than hotels and taxis, and doesn't know what else to do with his money.