Slashdot Mirror
NYC Data Center Needs Focus On Fuel
Nerval's Lobster writes "Who knew that the most critical element of operating a data center in New York City was ensuring a steady supply of diesel fuel? In the wake of Hurricane Sandy, the challenges facing data center operators in the affected zones include pumping water from basements, waiting for utility power to be restored, and managing fuel-truck deliveries. And it's become increasingly clear which companies had the resources and foresight to plan for a disaster like Sandy, and which are simply reacting. Here's the latest on providers around the New York area." And remember, having fuel for machines sometimes only means it's time to start the manual labor.
Data center on the coast - and they're surprised by what happened?
Good God! I saw this coming for years - I guess I AM GOD!
Listen to me my sheep, if you're on the coast, you will be flooded and wiped out by hurricanes!
And more, my sheep, you will be knocked out by tornadoes in the Mid-West!
And yet, my sheep - oh, fuck it! - my morons - you will be taken out by Earthquakes on the West Coast!
And for you in the Middle East... You'll be taken out by terrorists!
And in a few years by the Omicron Persei 7 peoples - not '*8" because Lurrrr hasn't taken them over yet - poor, poor, bastard.
*Read it with the Professor Farnsworth voice from "Furturama"
... you would know. The blogs about keeping data centers running in New Orleans during Katrina were incredible.
I know if my business needed to be up and online 24-7 to keep the money rolling in. I'd have at LEAST 3 layers of backup in place for every single situation i could plan for.
Fire, flood, power outage, earthquake, tidalwave, and more...
Because mother nature does not give a fuck about your business. If you don't do these things nobody else will.
How can you not have a multi-day supply of diesel on hand?
I want to delete my account but Slashdot doesn't allow it.
you can't store 3 days of fuel at high floors that can be a big fire hazard.
I was under the impression that a fuel supply was a standard part of the contingency plan for any data center.
I recently visited a new data center opening near me. The operator had contracts with several fuel suppliers that in the event of a power outage, the first one to get a full tank truck through their front gate got paid, and would keep getting paid for each additional truck that was needed. Any latecomers would be turned away, effectively making it an exclusive contract upon arrival.
You do not have a moral or legal right to do absolutely anything you want.
Once refined, it degrades. Oxidation, bacteria. Algae. Amazing things grow in diesel. You can add preservatives, but these only go so far. Keep it too long, and it's unusable. If you don't use it, you have to dispose of it. There's a large disincentive to keep diesel around.
Please do not read this sig. Thank you.
hahahahahaha, thanks i needed a good laugh.
What's the point in trying to keep these servers operating, and risking another fire hazard. Not to mention all the pollution and contamination. I'd be renting a bunch of cloud servers on Amazon, or move what you can to another location.
-- By all means let's be open-minded, but not so open-minded that our brains drop out.
Thanks, I've learned a lot about various digital tools!
Everyone already knew that the on-site fuel supply is the limiting factor of power availability in a disaster. Even fuel delivery contracts mean nothing in a disaster or wide-spread outages - hospitals, EMS and other government services will trump the fuel delivery contract, if a hospital needs fuel, they are going to get the fuel that's been "guaranteed" for your datacenter.
There's no reason to spend big $$ creating a flood proof, earthquake proof, tornado proof, airplane crash proof datacenter in the middle of a city when you can have a disaster recovery site 1000 miles away that's not subject to the same type of disaster. (except maybe an asteroid strike, but there are few datacenters on the moon). No matter how disaster-proof you make your datacenter, mother nature (or man) will always find a way to create a disaster you didn't plan for -- even if that "disaster" is a typo in a router configuration file that takes down the network, or a contractor accidentally shorting out the emergency power cutoff switch wiring when bolting a rack to the wall.
Cell sites I worked in Africa run a pair of Cummins generators as their main power. In the unlikely event that both of these fail at the same time, there's a chance that the main grid might be working well enough to take over. But the fuel is the biggest priority on these sites.
At the moment, we have tankers on site at all of our NY DCs. 2 of them are on generator only, so we're topping up the tanks every 4 hours. The generators will need full overhauls when we're back on the grid properly, but for now we're keeping our clients serviced which is what matters.
If this was their first time, or it had been a few years, I would expect there to be mass hysteria and a general failure of DR plans. Mostly because DR plans are theoretical and costly to test and not tested very often. However I think Irene gleaned much useful information for those developing these DR plans which led to NY data centers being better prepared. I'd love to read the DR plans in 6 months and compare the new changes from lessons learned during and after this storm.
Ok, so this is a geeknet story submitted by a geeknet employee.
Why make it look like a user submitted the article?
Not that it actually matters.
Don't know something? Look it up. Still don't know? Then ask.
(Please keep in mind that I write this with a somewhat cruddy memory as to historical events.) I'm on the fence with how to feel on this one. Being SO dependent on computers and data centers is a relatively recent phenomenon. Not saying they haven't been important before, but it's really blossomed over the last 10 to 15 years. Prior to Katrina in the gulf region, I can't recall a storm of this magnitude in the last decade. (Although the same northeast region got hit with "Hazel" back in 1954) And this one hit in a spot that's so packed with data centers and other "critical" devices. Katrina should have been a great lesson in disaster recovery. But I'd imagine the ol' "that'll never happen here, only other places" mentality kicked in. There's no reason they shouldn't have a week of fuel on hand. Now, having said that, you can have a very good disaster plan. But you can't plan for everything and it's very hard to plan for something of this magnitude. OK, yes, you can plan for it, but you have to balance your plan with your budget. For major hardware and large company needs, I'd go at least a good week. Smaller vendors? I'd say a good 3-4 days. Plan for which is your most critical equipment and *must* have power to maintain minimal company functionality.
I was visiting 1 summer st once, and I asked the guy who set the place up "why don't you use natural gas? You can just get it from your local utilities, or have it delivered by truck." and he looked at me with the "you poor idiot" look people give me some times. Then he said "Nobody uses natural gas because you can't tell if the supply upstream has been disrupted."
I didn't feel like explaining "no, dude, you pump it into a pressurized tank and monitor the flow, *and* you have a diesel generator as a backup / alternate." Do these places typically run with just one generator?
So I ask again -- Why not use natural gas? The pipes are underground and typically pretty safe in a storm.
Lastly -- why the heck did these guys put the pumps *outside* the tank? Why not put them in the tank?
Actually, the irony to me in this situation is that various sites in Manhattan have been trying to store additional diesel for years. However, the city would deny them because they feared thousands of gallons of diesel would be a tempting terrorist target..
If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York. Now, of course, you generally want a backup site that is more than 100 miles away, in the event of actual disasters, but you definitely want your primary facilities as close to your users as you can get.
I was once told that a place could generate power locally for less than it cost for them to get it from their utility.
If that's the case, why not just have urban data centers use local generation to dump *into* the grid when they feel like running through their local supply of fuel or testing their generators under load or whatever?
That's a much better case than dealing with 3 year old fuel oil or discovering that your generators don't work when you need them.
Don't fight for your country, if your country does not fight for you.
Yea, That is nice. Maybe later.
I am busy making sure that these servers that transmit data to hundreds of millions of people keep going no matter what kind of pounding it takes or how drenched in fluid it becomes, just like the chicks in the videos that the servers are transmitting.
So, this is a new Slashdot story that just links to 2 older Slashdot stories?
Didn't I just read the same thing here yesterday or the day before? An intentional dupe? Are the editors all on vacation this week?
One of my buds is IT director for a company that resupplies generators. The logisitics for it are crazy as you route trucks on available streets, deal with priority of the customers (hospitals front of the line) and optimize resupply into mostly empty tanks before they actually empty, etc... And you have >10 days of this 24/7 after a storm.
After Hurricane Allison here, some companies in this sector went out of business before the power came back on. We were better prepared for Ike, but I think that's because facilities are more willing to sign contracts with the pricey but reputable businesses.
This is the most expensive way to get fuel: you need a massive amount for a short time, and you need it consistently during that time.
...there are few datacenters on the moon...
The latency is a real bitch.
-
If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York. Now, of course, you generally want a backup site that is more than 100 miles away, in the event of actual disasters, but you definitely want your primary facilities as close to your users as you can get.
Sure, there will always be datacenters in NYC, but that doesn't change the fact that instead of putting all your money into trying to build a datacenter impervious to all hazards, you're better off having a second site far from your primary site. Real-world constraints mean that you can't build that perfectly impervious datacenter when you're subject to real estate prices, building codes, fire codes, and Murphy's Law -- there will always be a disaster that the facility can't handle.
Um, some would say the Lord's wrath for the sins of Americans has big ears and can shoot the jumper.
Me? the Lord needs do nothing special to make us miserable. We do that well all by ourselves. Special Wrath not required.
deleting the extra space after periods so i can stay relevant, yeah.
Exactly. This is why your monday-quarterback "have a disaster recovery site 1000 miles away" is just as totally frigging useless.
You've left straw all over the place. Sweep up and sit down.
If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York.
I don't think OP was saying that you shouldn't have datacenters in NYC, persay, but rather that you'd be better off spending most of your "disaster-proofing" budget on off-site disaster recovery equipment/services.
Don't put all your eggs in one basket, and such.
An enigma, wrapped in a riddle, shrouded in bacon and cheese
Anyone with 1/2 a brain would know.
Now, it may be hard to mitigate the problem for the long haul, but not knowing? No excuse.
---- Booth was a patriot ----
Exactly. This is why your monday-quarterback "have a disaster recovery site 1000 miles away" is just as totally frigging useless.
You've left straw all over the place. Sweep up and sit down.
Care to elaborate? How is geographically diversity not significantly better than no geographical diversity?
I would have thought it was obvious, but the middle of the city is where the telecommunications infrastructure is. It doesn't exist in a barn a hundred miles from any major city. And fuel has a shelf life. Diesel will slowly oxidize over time, and so the time you can keep it in the tank is about 12 months. You'll burn about 72 gallons an hour per megawatt (as a rough average). So a 2 megawatt data center will need about 3,500 gallons of diesel per day. A gallon takes up 231 cubic inches of space, so a single day's worth of fuel would need a tank with a capacity of 67,375 cubic feet. The average height of a floor in a skyscraper is 12.5 feet. In New York city, the average city block is 264 feet. That means that even if you filled an entire floor of a skyscraper with nothing but diesel fuel, you'd still get less than a week's worth of fuel guys. At a rate of perhaps $100 per month per square foot... you're talking about $83 million a year for a floor of an outlying area just to store that diesel fuel. Mind you, near Wall St., that price is probably going to be double or triple. The price of fuel is peanuts compared to this; 7 days of fuel for a 2MW plant would cost you only $94,000, plus transport costs.
So as you can see, this isn't a question of them not stocking enough fuel -- the cost of storing that fuel is prohibitively expensive.
And that, people, is why they didn't load up on fuel ahead of the storm. You can't simply pay $83 million a year for your data center to protect against a threat that might only materialize once a decade, and be severe enough to deny fuel deliveries or power restoration for that long of a time frame. Generator backup is a short term solution. There is no long term solution for disaster recovery, at least not one that's cost effective. Not in an urban setting.
The only time you can justify spending that kind of cash is if you're supporting critical infrastructure like phones, hospitals, and emergency services. Everyone else plans for a couple day supply and leaves it at that.
#fuckbeta #iamslashdot #dicemustdie
30 years ago, I retired from the Navy, along with a friend who was a Machinist Mate on Navy submarines. I didn't see him for a couple years. Turns out that he had spent 10 of those years servicing batteries on diesel submarines, and ended up servicing large batteries for the phone company -- about the same size batteries that had been on the boats. He switched to servicing the diesels (because of his prior experience on submarines with their diesel generators), and one of the first to deploy natural gas fired diesels for sustained power, first for the phone company, and then for a local utility.
There's the solution. Don't pipe flammable liquid to the top floor, pipe the natural gas. Contrary to popular belief, it is a safer fuel, and requires less maintenance. Automatic shutoff valves work better, less explosive volume released on a tubing breakage, and it doesn't rot the pipes like liquid diesel fuel, disperses to atmosphere in the case of a leak, and doesn't make everything around it flammable, when it does leak.
Engines run at least 4 times as long between between overhaul cycles, and it doesn't dilute the lube oil.
Natural gas is going to be a HUGE change for this countries infrastructure, both in common usage, as well as emergency failover services.
This guy has made tons of money, by the way. Has a condo and car in San Diego, LA, Seattle, New York, and Atlanta -- cheaper than hotels and taxis, and doesn't know what else to do with his money.
At my work we had our generators on the roof above 6th floor. It had a small tank that can run for a couple of hours. In the basement, a larger tank that had fuel for 2 days.
Sadly when the power went away, so did the power to the pump that were supposed to lift it to the root. So the our electrician all of the suddon got busy.
At another workplace we had two huge generators with flywheels for starting them, plenty of fuel and even our own redundant line to the powerplant some miles away. Sadly when the installation got upgraded, someone rewired the part that monitored the power coming in so it monitored itself. So after it had fired up and run for some time and synchronized the phases to itself. It shut down and we lost power again. That went on for the most of the day and almost drained our UPS batteries before they figured out what was wrong.
ah the memories.
I was working at a federal government department that was planning for a prolonged power outage and they were happy to have signed a contract with a diesel supplier should such a problem happen. This was happening shortly after the blackout in 2003. They were quiet proud of themselves for finding this solution to keep their "mission critical" web application up and running. This only lasted a couple of days until the city emergency planning committee came along and informed them that in such emergency the department would not be getting diesel supplies until emergency locations such as hospitals have been served.
As others have said, the coast is probably a bad place for a data center. Somewhere geologically stable, with reliable power and a good backplane connection would be best.
Basic lessons learned from Katrina and pretty much every other hurricane over the last two decades... The East coast of North America is not immune to the effects of hurricanes, all the way up to the coast of Newfoundland and Labrador. If, as a server admin, you don't understand this basic fact you need to spend a great deal of time over the next few months pulling your head out of your collective asses.
and then the lag will suck big time
fine lets put the data center on mars can you take 40min ping times?
Having an office in NYC is great... Having a datacenter is not. If you must have a datacenter please for the love of god leave it to the professionals. Purchase colocation space in either 111 8th ave or 60Hudson. Add to that redundant circuits from the office to the datacenter with plenty of bandwidth. Just don't forget to roll in Virtual Desktops for your people so they can work from anywhere that has internet. And for real long term recovery duplicate your datacenter somewhere other than a small island barely above sea level that is not only a target for terrorism but mother nature ain't to pleased with it either.
"The only time you can justify spending that kind of cash is if you're supporting critical infrastructure like phones, hospitals, and emergency services."
Which is why I can't understand why some hospitals in New York City had to evacuate due to power loss. I heard that one hospital evacuated the goddamn NICU.
I work for a major hospital system in a hurricane-prone area. We have serious contingency plans. Multiple redundant on-site generators capable of powering the whole system. 45 days of on-site diesel storage that is rotated through our fleet of delivery trucks so the fuel in storage is never more than 6 months old. We test the generators every month. We do a full fail-over test 3 times per year. Once at the beginning of hurricane season, once in the middle, then a third time at some other random time of year.
The generators are raised. If those generators flood, then we have much larger concerns than trying to supply power to the hospital - you would have to smash third-floor windows and use boats to get people in and out of the buildings. The generators are housed and shielded so their air intake can handle torrential rainfall.
Whoever was responsible for disaster planning for these hospitals ought to be facing criminal charges.
I would have thought it was obvious, but the middle of the city is where the telecommunications infrastructure is. It doesn't exist in a barn a hundred miles from any major city. And fuel has a shelf life. Diesel will slowly oxidize over time, and so the time you can keep it in the tank is about 12 months. You'll burn about 72 gallons an hour per megawatt (as a rough average). So a 2 megawatt data center will need about 3,500 gallons of diesel per day. A gallon takes up 231 cubic inches of space, so a single day's worth of fuel would need a tank with a capacity of 67,375 cubic feet. The average height of a floor in a skyscraper is 12.5 feet. In New York city, the average city block is 264 feet. That means that even if you filled an entire floor of a skyscraper with nothing but diesel fuel, you'd still get less than a week's worth of fuel guys. At a rate of perhaps $100 per month per square foot... you're talking about $83 million a year for a floor of an outlying area just to store that diesel fuel. Mind you, near Wall St., that price is probably going to be double or triple. The price of fuel is peanuts compared to this; 7 days of fuel for a 2MW plant would cost you only $94,000, plus transport costs.
While I agree with your point in general, your math is off. I can see our own 1MW generator and 3-day fuel tank from my office, which is no where near the size of tank you calculated -- it should consume around 1/4 of a city block using your figures.
A week of fuel (24,000 gallons) is 3200 cubic feet, with a 10 foot high tank, that's only 320 square feet, at $100/month that's still a pricey $384,000/year. But since that 2MW of power is enough to power 1000 servers (assuming 1000 watts/server, half the power goes to cooling), that's only $384 per server per year or $32/month per server.
Each server rack consumes around 6 square feet of space (allowing room in front and behind the rack), so that 6 square feet would cost $7200/year, or $171/server/year if you put 42 servers in a rack. ($14/server/month). So spending $32/month to provide a week of backup power isn't that out of line if you really want a server close to your office.
But $100 sounds pretty high for unfinished office space in Manhattan even in a class-A building. If you put your datacenter into a cheaper class-B or class-C building - rent would be closer to $40/sq ft.
"The only time you can justify spending that kind of cash is if you're supporting critical infrastructure like phones, hospitals, and emergency services."
Which is why I can't understand why some hospitals in New York City had to evacuate due to power loss. I heard that one hospital evacuated the goddamn NICU.
...
Whoever was responsible for disaster planning for these hospitals ought to be facing criminal charges.
Hospitals, even publicly funded ones, don't have unlimited funds to implement disaster recovery plans. If it comes down to reducing services during the 99.9% of the time when there is no disaster versus spending money to ride out the 0.1% of the time when there's a major disaster, spending the money to help more people now isn't necessarily worse than planning to evacuate during an uncommon disaster.
I'd question the need for a hospital to have 45 days of fuel on-site... keeping the lights on does little good when you run out of drugs, antiseptics, fresh water, bandages, oxygen, and all of the other supplies you need to keep the facility running.
Do you keep a 45 day supply of all of the rest of your consumables on-site? My only experience is with a small rural/regional hospital and they had 3 days of fuel on-site, and 4 -7 days of most consumables.
If you only need 1MW of power, that means you have 80,000 gallons of fuel in storage - at 7mpg that means your delivery trucks are traveling over 500,000 miles every 6 months. What kind of deliveries are you making that puts a million miles on your fleet each year?
Short memories...big in the tech news during Katrina was the story of Intercosmos keeping their operations going in New Orleans. The emergency fuel came but the truck could not fit into the parking structure where the generator was located. They hand trucked 55 gallon drums to keep it going.
"We also have to figure out a way to move 21 large barrels of diesel up nine stories to the generator cage in the parking garage without a truck. Im hoping that we will be able to find someone with a truck to help us later in the week. Otherwise we will need to wheel them up on a handtruck individually, a Herculean conquest no matter which way you look at it!" http://interdictor.livejournal.com/2005/09/17/
In the 90's when datacenter aggregation in Manhattan became a design point _the_ holy grail for disaster was off-site redundancy.
So...all this whining and crying is bullshit to my ears
A major disaster is the time when you really need your hospital to function properly. So spending some money on disaster preparedness is prudent.
Yes, we keep large amounts of emergency supplies of consumables on site : medical gases, medications, food, linens, antiseptics, needles, lines, etc.
This hospital system has multiple facilities spread across a large metro area. We have centralized laundry service and to some extent centralized food service. The fleet of trucks transports laundry and food between those service facilities and the hospitals. I forget how many trucks there are. It's some large number. Also some support services are somewhat centralized like engineering and parts of biomedical. Yes, we have backup stores at each facility for linens, and local food stores and food-prep capability as well. If the roads are down, then the food may not be quite as varied as usual, but we will be able to feed our patients.
Evacuating critical patients is dangerous - there is a chance that the process will kill them. Trying to avoid killing people is a good thing and is something that we spend lots of money on.
Telling injured hurricane victims that show up in the E.D. "Sorry, dude, but this storm really screwed us up, man. We can't treat you." isn't acceptable.
I just googled to source the numbers to get a ballpark figure. So yeah, undoubtedly they're off -- people spend months preparing reports for their data center on fuel consumption, storage costs, location, etc. I spent 20 minutes. But putting the numbers together shows that even if you lowball all the numbers by 50%, storage cost is still massively eclipsing fuel cost.
People on slashdot here were being 'armchair CTOs' and saying how if it was their data center, they'd have bought all that extra fuel. "Well it was obvious Sandy was going to hit! Duh. Buy fuel." And of course when I see commentary like that, I feel a strong compulsion to point out that there were hundreds of professionals involved in the design, build, and maintenance of each of these data centers. They didn't just make an idiot mistake. And running the numbers (above) shows why they made the decisions they did. They may not be exact, but they're ballpark -- close enough to prove that the armchair CTOs were wrong, and the pros were right... and why they were right.
#fuckbeta #iamslashdot #dicemustdie
I just googled to source the numbers to get a ballpark figure. So yeah, undoubtedly they're off -- people spend months preparing reports for their data center on fuel consumption, storage costs, location, etc. I spent 20 minutes. But putting the numbers together shows that even if you lowball all the numbers by 50%, storage cost is still massively eclipsing fuel cost.
Sure, I understand the fuel usage numbers were an estimate, but your calculations weren't off by 50% - you were claiming that it would take an entire 264x264 foot office floor (70,000 square feet) to store a week's worth of fuel. I was pointing out that it only takes about 300 square feet - you were off by a factor of 233, not by 50%. Instead of an unaffordable $83M in rent and an entire office floor to store a week's worth of fuel, it only takes an 18 foot by 18 foot room and $400K/year. That's a huge difference.
Your key mistakes were in dividing cubic inches by 12 instead of by 12*12*12 (or 1728) to convert from in^3 to ft^3, and again where you assumed that 67,000 ft^3 of fuel would need 69,000 square feet of office space to store, ignoring the 12 foot ceilings that you noted earlier.
If you did devote an entire 70,000 square foot 10 foot high office floor to fuel storage, you'd have 700,000 ft^3 of fuel storage, or 5,000,000 gallons, enough to power the 2MW generator for 1400 days. Of course, it would weigh 35M lbs, so the building would need to be purpose built (or retrofitted) to support the weight.
A major disaster is the time when you really need your hospital to function properly. So spending some money on disaster preparedness is prudent.
Yes, we keep large amounts of emergency supplies of consumables on site : medical gases, medications, food, linens, antiseptics, needles, lines, etc.
Telling injured hurricane victims that show up in the E.D. "Sorry, dude, but this storm really screwed us up, man. We can't treat you." isn't acceptable.
Sure, I understand that a hospital needs to function for some time during/after a disaster, but 45 days of supplies seems like expensive overkill - do you have millions of gallons of water on site and your own treatment plant on-site since it's not likely that your municipal plant will withstand a 30+ day power outage.
What hospital is this?
Even just combined heat & power systems running on small natural gas turbines. Our hospital uses one for most of the electricity, all of our heat and hot water, and a decent amount of our cooling. The electric grid is largely a backup for our natural gas supply...
Seem to recall stories on /. about companies contracting for delivery of fuel oil, if necessary by helicopter, to city data centers leading up to Y2K. Then there was a blog allegedly by the data center guy in post Katrina Nawlins. What was that saying "The hazards of computing are only limited by the imagination."
Who knew that the most critical element of operating a data center in New York City was ensuring a steady supply of diesel fuel?
Uhm, anyone that was conscious for Katrina...
We found these heaps of backup tapes down in the basement. We just shoveled them into the boilers and kept the generators running.
Have gnu, will travel.
The Lance Armstrong solution:
http://www.youtube.com/watch?v=ZmzeKNiMkAs
Of course he would have to be moved up out of the basement.
Off-topic, still a Lance fan, he never failed a drug test during the event so what is the problem. This is as stupid as disqualifying the Olympic badminton players for losing on purpose, within the rules, to improve their chances of medaling. Cycling and long-distance running are brain sports. You calculate when to draft and when to sprint. You work within the rules. You don't do maximum exertion all the time. Ditto any sport where there are multiple heats before the finals, disqualify anyone who paces themselves?
>Diesel will slowly oxidize over time, and so the time you can keep it in the tank is about 12 months.
you're way off. I've stored and used 10 year old diesel fuel without any problems. You're thinking of gasoline which has about a 12-24 month shelf life.
Some of these centre designers need to work in bad places before getting their credentials. Common sense does not reign supreme. It is not just diesel supply that is the problem; generators, pumps and electrical gear in basements is a disaster. The Japanese nuclear disaster would probably have been averted if the generators were on the roof instead of in the basement in a Tsunami prone area - loose power, loose pumps and you loose a nuclear reactor.
Nos Morituri te salutamus
Uh... Gulf States? Experience?
First, building an electricity-hungry data center in NYC, with its notorious electricity problems and in the path of coastal storms... well... let's just say it's not the brightest star in the cosmos of ideas.
In fact, it kind of reminds me of... what was it? Oh, yeah. A major Southern coastal city that was built... mostly below sea level! Yeah! That's the ticket!
It's peanut butter jelly sour grapes time! There you go! There you go!
"Who knew that the most critical element of operating a data center in New York City was ensuring a steady supply of diesel fuel?"
I did. Or would have, would have been a major item on the short list if they had hired me. But as a seasoned sysadmin wandering jobless in two major cities for months with the only 'tech' job available being telephone support in a boiler room for minimum wage plusabit I said "F*CK EYE-TEE and moved to a small town.
Where I now clean sewers for a living. And I apply the same dedication to the task as I did to maintaining linux systems, developing disaster plans. The sewers are very CLEAN here and when there is a stoppage our response time is AWESOME.
<blink>down the rabbit hole</blink>
I dont pretend to be the end all of computer and IT and I never will. But here is a truthful statement as seen from the bean counters.
Nothing will change.
Thats right, nothing. Yes a few companies (maybe even big ones) might do something different in their DR plans. They will all talk big and say big things. But when the time comes to open up the wallet and let some cash go towards creating and or maintaining their upgraded DR plans, very little cash will come out and the cash that does will require us worker bees to jump through various hoops so that the boss can claim he is compliant while we know that even the smallest issue will cause the entire structure to collapse.
Some companies are already very HA capable. Most are not. Most say that they are so their stock holders dont complain, but they are simply unprepared for what every nature or humanity throws at them. And it all comes down to the mighty dollar.
Im sorry. Its wrong and its sad. But that is the way that it is and the way it will stay.
What are NYC's notorious electricity problems? Outages in the city itself are much less frequent than in the surrounding suburbs,.
I believe this issue was identified during the great power outage a few years back. Any data center operator claiming they just learned this is either in CYA mode, is an idiot or is not experienced enough to be in their current position. Period.
I think anyone running generator is facing a fuel issue I do not care if it's a DC or just a house hold generator that cannot get fuel in the NY, NJ or PA area.
http://www.thetechnologygeek.org
I spent some years over there, mostly in the Philadelphia area, and that entire section of East Coast had pretty regular power outages (sections of it, that is... not the whole thing at any one time, of course).
Compare that to here in the (relatively speaking) West, where electrical outages in any one place happen maybe about once every 5 years, and outages that last more than an hour or so might happen roughly once every 20 years or so.
I am aware that there are storm issues, etc. I'm not trying to say it's anybody's fault. But at the same time, it's true. Compared to much of the United States, the East Coast has some major power issues.
> "Who knew that the most critical element of operating a data center in New York City was ensuring a steady supply of diesel fuel? Anyone who paid attention to what happened to data centers when Katrina hit New Orleans. Check the Clay Shirky interview linked to in this post: http://cis471.blogspot.com/2012/10/the-internet-damage-caused-by-hurricane.html