Slashdot Mirror
NYC Data Center Needs Focus On Fuel
Nerval's Lobster writes "Who knew that the most critical element of operating a data center in New York City was ensuring a steady supply of diesel fuel? In the wake of Hurricane Sandy, the challenges facing data center operators in the affected zones include pumping water from basements, waiting for utility power to be restored, and managing fuel-truck deliveries. And it's become increasingly clear which companies had the resources and foresight to plan for a disaster like Sandy, and which are simply reacting. Here's the latest on providers around the New York area." And remember, having fuel for machines sometimes only means it's time to start the manual labor.
Data center on the coast - and they're surprised by what happened?
Good God! I saw this coming for years - I guess I AM GOD!
Listen to me my sheep, if you're on the coast, you will be flooded and wiped out by hurricanes!
And more, my sheep, you will be knocked out by tornadoes in the Mid-West!
And yet, my sheep - oh, fuck it! - my morons - you will be taken out by Earthquakes on the West Coast!
And for you in the Middle East... You'll be taken out by terrorists!
And in a few years by the Omicron Persei 7 peoples - not '*8" because Lurrrr hasn't taken them over yet - poor, poor, bastard.
*Read it with the Professor Farnsworth voice from "Furturama"
Good luck turning a profit.
you can't store 3 days of fuel at high floors that can be a big fire hazard.
How can you not have a multi-day supply of diesel on hand?
Some places did, but in the city, you can't store it above ground - which means that that week's supply of diesel is now mixed in with the ocean.
I was under the impression that a fuel supply was a standard part of the contingency plan for any data center.
I recently visited a new data center opening near me. The operator had contracts with several fuel suppliers that in the event of a power outage, the first one to get a full tank truck through their front gate got paid, and would keep getting paid for each additional truck that was needed. Any latecomers would be turned away, effectively making it an exclusive contract upon arrival.
You do not have a moral or legal right to do absolutely anything you want.
Once refined, it degrades. Oxidation, bacteria. Algae. Amazing things grow in diesel. You can add preservatives, but these only go so far. Keep it too long, and it's unusable. If you don't use it, you have to dispose of it. There's a large disincentive to keep diesel around.
Please do not read this sig. Thank you.
Everyone already knew that the on-site fuel supply is the limiting factor of power availability in a disaster. Even fuel delivery contracts mean nothing in a disaster or wide-spread outages - hospitals, EMS and other government services will trump the fuel delivery contract, if a hospital needs fuel, they are going to get the fuel that's been "guaranteed" for your datacenter.
There's no reason to spend big $$ creating a flood proof, earthquake proof, tornado proof, airplane crash proof datacenter in the middle of a city when you can have a disaster recovery site 1000 miles away that's not subject to the same type of disaster. (except maybe an asteroid strike, but there are few datacenters on the moon). No matter how disaster-proof you make your datacenter, mother nature (or man) will always find a way to create a disaster you didn't plan for -- even if that "disaster" is a typo in a router configuration file that takes down the network, or a contractor accidentally shorting out the emergency power cutoff switch wiring when bolting a rack to the wall.
Some of these servers were the "cloud".
The truth is that all men having power ought to be mistrusted. James Madison
Cell sites I worked in Africa run a pair of Cummins generators as their main power. In the unlikely event that both of these fail at the same time, there's a chance that the main grid might be working well enough to take over. But the fuel is the biggest priority on these sites.
At the moment, we have tankers on site at all of our NY DCs. 2 of them are on generator only, so we're topping up the tanks every 4 hours. The generators will need full overhauls when we're back on the grid properly, but for now we're keeping our clients serviced which is what matters.
If this was their first time, or it had been a few years, I would expect there to be mass hysteria and a general failure of DR plans. Mostly because DR plans are theoretical and costly to test and not tested very often. However I think Irene gleaned much useful information for those developing these DR plans which led to NY data centers being better prepared. I'd love to read the DR plans in 6 months and compare the new changes from lessons learned during and after this storm.
So you move that Cloud to another Cloud.
-- By all means let's be open-minded, but not so open-minded that our brains drop out.
I was visiting 1 summer st once, and I asked the guy who set the place up "why don't you use natural gas? You can just get it from your local utilities, or have it delivered by truck." and he looked at me with the "you poor idiot" look people give me some times. Then he said "Nobody uses natural gas because you can't tell if the supply upstream has been disrupted."
I didn't feel like explaining "no, dude, you pump it into a pressurized tank and monitor the flow, *and* you have a diesel generator as a backup / alternate." Do these places typically run with just one generator?
So I ask again -- Why not use natural gas? The pipes are underground and typically pretty safe in a storm.
Lastly -- why the heck did these guys put the pumps *outside* the tank? Why not put them in the tank?
If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York. Now, of course, you generally want a backup site that is more than 100 miles away, in the event of actual disasters, but you definitely want your primary facilities as close to your users as you can get.
Yea, That is nice. Maybe later.
I am busy making sure that these servers that transmit data to hundreds of millions of people keep going no matter what kind of pounding it takes or how drenched in fluid it becomes, just like the chicks in the videos that the servers are transmitting.
One of my buds is IT director for a company that resupplies generators. The logisitics for it are crazy as you route trucks on available streets, deal with priority of the customers (hospitals front of the line) and optimize resupply into mostly empty tanks before they actually empty, etc... And you have >10 days of this 24/7 after a storm.
After Hurricane Allison here, some companies in this sector went out of business before the power came back on. We were better prepared for Ike, but I think that's because facilities are more willing to sign contracts with the pricey but reputable businesses.
This is the most expensive way to get fuel: you need a massive amount for a short time, and you need it consistently during that time.
...there are few datacenters on the moon...
The latency is a real bitch.
-
If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York. Now, of course, you generally want a backup site that is more than 100 miles away, in the event of actual disasters, but you definitely want your primary facilities as close to your users as you can get.
Sure, there will always be datacenters in NYC, but that doesn't change the fact that instead of putting all your money into trying to build a datacenter impervious to all hazards, you're better off having a second site far from your primary site. Real-world constraints mean that you can't build that perfectly impervious datacenter when you're subject to real estate prices, building codes, fire codes, and Murphy's Law -- there will always be a disaster that the facility can't handle.
I would have thought it was obvious, but the middle of the city is where the telecommunications infrastructure is. It doesn't exist in a barn a hundred miles from any major city. And fuel has a shelf life. Diesel will slowly oxidize over time, and so the time you can keep it in the tank is about 12 months. You'll burn about 72 gallons an hour per megawatt (as a rough average). So a 2 megawatt data center will need about 3,500 gallons of diesel per day. A gallon takes up 231 cubic inches of space, so a single day's worth of fuel would need a tank with a capacity of 67,375 cubic feet. The average height of a floor in a skyscraper is 12.5 feet. In New York city, the average city block is 264 feet. That means that even if you filled an entire floor of a skyscraper with nothing but diesel fuel, you'd still get less than a week's worth of fuel guys. At a rate of perhaps $100 per month per square foot... you're talking about $83 million a year for a floor of an outlying area just to store that diesel fuel. Mind you, near Wall St., that price is probably going to be double or triple. The price of fuel is peanuts compared to this; 7 days of fuel for a 2MW plant would cost you only $94,000, plus transport costs.
So as you can see, this isn't a question of them not stocking enough fuel -- the cost of storing that fuel is prohibitively expensive.
And that, people, is why they didn't load up on fuel ahead of the storm. You can't simply pay $83 million a year for your data center to protect against a threat that might only materialize once a decade, and be severe enough to deny fuel deliveries or power restoration for that long of a time frame. Generator backup is a short term solution. There is no long term solution for disaster recovery, at least not one that's cost effective. Not in an urban setting.
The only time you can justify spending that kind of cash is if you're supporting critical infrastructure like phones, hospitals, and emergency services. Everyone else plans for a couple day supply and leaves it at that.
#fuckbeta #iamslashdot #dicemustdie
I agree. I recently has to do a disaster recovery plan for a law firm I administrate. They wanted to be able to weather any event without disruption. I took disruptions that I knew of within the last ten year, and expanding to situations that has happened in other parts of the town. Flooding wasn't really an issue because they are on a hill that would require most of the state to be under water before they were. The biggest concerns was power and internet outages, something happening to the building making it unsafe or uninhabitable, and something happening outside the building requiring a sustained evacuation.
Long story short, they ignored it, purchased a portable gas powered generator and extension cords long enough to make it into the server room. They wanted backups stored offsite but wouldn't provide any way of reading them. Off site ended up being one of the partner's home too. The Generator, I think was one that one of the partners purchased during the last storm that we lost power on and he wanted to unload it because it hadn't been needed in the last 3 years. And the one time we needed to use it, I spent about 2 hours draining the fuel and cleaning the carburetor because someone wanted it fueled up and ready to go at a moment's notice while in storage.
But they did have a point, why spend more money to stay open for 3 days then you wold lose by shutting down or being partially shut down those 3 days.
30 years ago, I retired from the Navy, along with a friend who was a Machinist Mate on Navy submarines. I didn't see him for a couple years. Turns out that he had spent 10 of those years servicing batteries on diesel submarines, and ended up servicing large batteries for the phone company -- about the same size batteries that had been on the boats. He switched to servicing the diesels (because of his prior experience on submarines with their diesel generators), and one of the first to deploy natural gas fired diesels for sustained power, first for the phone company, and then for a local utility.
There's the solution. Don't pipe flammable liquid to the top floor, pipe the natural gas. Contrary to popular belief, it is a safer fuel, and requires less maintenance. Automatic shutoff valves work better, less explosive volume released on a tubing breakage, and it doesn't rot the pipes like liquid diesel fuel, disperses to atmosphere in the case of a leak, and doesn't make everything around it flammable, when it does leak.
Engines run at least 4 times as long between between overhaul cycles, and it doesn't dilute the lube oil.
Natural gas is going to be a HUGE change for this countries infrastructure, both in common usage, as well as emergency failover services.
This guy has made tons of money, by the way. Has a condo and car in San Diego, LA, Seattle, New York, and Atlanta -- cheaper than hotels and taxis, and doesn't know what else to do with his money.
I would have thought it was obvious, but the middle of the city is where the telecommunications infrastructure is. It doesn't exist in a barn a hundred miles from any major city. And fuel has a shelf life. Diesel will slowly oxidize over time, and so the time you can keep it in the tank is about 12 months. You'll burn about 72 gallons an hour per megawatt (as a rough average). So a 2 megawatt data center will need about 3,500 gallons of diesel per day. A gallon takes up 231 cubic inches of space, so a single day's worth of fuel would need a tank with a capacity of 67,375 cubic feet. The average height of a floor in a skyscraper is 12.5 feet. In New York city, the average city block is 264 feet. That means that even if you filled an entire floor of a skyscraper with nothing but diesel fuel, you'd still get less than a week's worth of fuel guys. At a rate of perhaps $100 per month per square foot... you're talking about $83 million a year for a floor of an outlying area just to store that diesel fuel. Mind you, near Wall St., that price is probably going to be double or triple. The price of fuel is peanuts compared to this; 7 days of fuel for a 2MW plant would cost you only $94,000, plus transport costs.
While I agree with your point in general, your math is off. I can see our own 1MW generator and 3-day fuel tank from my office, which is no where near the size of tank you calculated -- it should consume around 1/4 of a city block using your figures.
A week of fuel (24,000 gallons) is 3200 cubic feet, with a 10 foot high tank, that's only 320 square feet, at $100/month that's still a pricey $384,000/year. But since that 2MW of power is enough to power 1000 servers (assuming 1000 watts/server, half the power goes to cooling), that's only $384 per server per year or $32/month per server.
Each server rack consumes around 6 square feet of space (allowing room in front and behind the rack), so that 6 square feet would cost $7200/year, or $171/server/year if you put 42 servers in a rack. ($14/server/month). So spending $32/month to provide a week of backup power isn't that out of line if you really want a server close to your office.
But $100 sounds pretty high for unfinished office space in Manhattan even in a class-A building. If you put your datacenter into a cheaper class-B or class-C building - rent would be closer to $40/sq ft.
This is one of those situations where the invisible hand of the free market fails miserably:
Company A: plans properly for contingencies, but has to charge $10 more per customer.
Company B: thinks they have planned for contingencies as well as A (but accredit the cheaper pricetag to their managerial prowess), makes the same claims about uptime as company A, and undercuts company A.
Company A goes out of business as B steals all of A's customers.
Disaster hits, but due to company A having gone out of business 2 years ago, B can legitimately say "no one could have planned for this!" and likely gets away with it...
Except that company As are still in business - given a bunch of places did just fine. So the invisible hand seems to have done just fine.
Note these are NYC data centers if you were buying the cheaper option you weren't in one of them anyway.
Uh... Gulf States? Experience?
First, building an electricity-hungry data center in NYC, with its notorious electricity problems and in the path of coastal storms... well... let's just say it's not the brightest star in the cosmos of ideas.
In fact, it kind of reminds me of... what was it? Oh, yeah. A major Southern coastal city that was built... mostly below sea level! Yeah! That's the ticket!
I don't understand why they're using deisel rather than natural gas. With natural gas they could have the generators on the roof. Illinois Secretary of State mainframes have two backup generators, both natural gas. I've lost electricity a whole lot of times, but I've never had a natural gas disruption in my six decades. Not even in 2006 when we were hit with two tornados and electricity was out for a week; I heated my apartment with the oven (the gas furnace needs electricity for blowers and theromostats).
Free Martian Whores!