New York Data Centers Battle Floods, Utility Outages
miller60 writes "At least three data center buildings in lower Manhattan are struggling with power problems amid widespread flooding and utility outages caused by Hurricane Sandy. Flooded basements at two sites took out diesel fuel pumps, leaving them unable to refuel generators on higher levels. One of these was Datagram, which knocked out Buzzfeed and the Gawker network of sites. At 111 8th Avenue, some tenants lost power when Equinix briefly experienced generator problems."
The NY Times has a running list of Sandy-related problems, including 5,700 more flight cancellations, 6 million people without power, rising water levels at a nuclear plant, official disaster declarations from President Obama, and a death toll of 38. On the upside, and despite the high water levels, the Nuclear Energy Institute was quick to point out that all 34 nuclear facilities in Sandy's path made it through without problems.
If we're really lucky, it'll take out all the high frequency traders systems for a few days and we can have an actual market without parasites.
Nah, who am I kidding. If that actually happened they'd keep Wall Street closed.
I still have more fans than freaks. WTF is wrong with you people?
Why aren't there more datacenters in Kansas, Nebraska, North Dakota, etc.? Surely the threat from Tornados could be mitigated and the electrical infrastructure built out more cheaply than the losses due to coastal disasters, no?
If you do what you always did, you get what you always got.
The city's collective-arrogance towards the rest of the world makes it really tough to pity the celebrities, financial crooks, government-thugs, hipsters and other undesirables.
The best thing we can hope for is maybe Anderson Cooper will wash away.
HMS Bounty. Never forget.
0 = 1 + e^(Alt something)
Not a sysadmin myself, but isn't it poor planning to not have your sites mirrored in a remote location for disaster recovery in situations like this?
I was promised a NUKULAR OPOCALIPSE yesterday, and here we are with satellite images that don't even show the entire East Coast as a glowing radioactive wasteland.
I think that this complete lack of NUKULAR meltdown is 100% absolute irrefutable proof of two key concepts:
1. Capitalism is an abject failure and we need U.N. control of everything and everyone right now.
2. NUKULAR power is obviously far too dangerous and should be banned right now before somebody doesn't get killed again JUST LIKE FUKUSHIMA.
AntiFA: An abbreviation for Anti First Amendment.
I thought it was standard procedure to teach servers to swim these days.
How about not putting mission critical equipment susceptible to water damage in the one place all water will go.
Your hair look like poop, Bob! - Wanker.
This is as strong as a case forever to stop paying attention to the media.
Sounds like you already stopped paying attention. Had you even checked the one link in the article, or even read the summary, you'd know it was a catastophy, and New York will not function "perfectly tomorrow like nothing ever happened."
Schools closed. Subways closed 4-5 days. 38 people dead. Market closed. Fire in Queens destroys dozens of homes. Power outages for millions. 7% of the US population in fact without power. Tunnels flooded (subway and car). NYU Tisch hospital evacuated due to flood related generator failures, including premature babies on ventilators. Just a small summary, of just one city.
I'm a former NYer. Have spoken to many friends and family. None expect normal life tomorrow. Some have considerable property damage. None lost a life, thankfully. I live in South Florida, incidentally, and rather well understand how damaging hurricanes can be. Wilma damaged my car, and cancelled my wedding day as the roof caved in on the place we were getting married in that Friday.
This is a rare strong case to NOT stop paying attention to the media.
They were idiots for going out to sea. Even if it was marginally safer for the boat to be at sea than in port for the storm, it was safer for the crew to be in port, or in hotel rooms 100 miles inland. Those people died due to a combination of greedy ownership that valued the ship more than the crew's safety and their own stupidity in not refusing. All this for a reproduction used in movie shoots. A senseless waste.
I still have more fans than freaks. WTF is wrong with you people?
It's like the old joke about trailer parks attracting tornadoes. There seems to be some sort of physical law that says that if a data center is going to be set up, it will be in a basement, and in a low-lying area.
Proverbs 21:19
Pretty sure the 38 deaths is for the entire region hit, not just NYC. But that said, 38 deaths over two days might be a decline in NYC's death rate over the same time period.
That's o.k. the guy is a first rate ass-hole. What do you expect from someone who posts anonymously.
I don't think you're doing that right.
You must not have seen the pictures of the new LaGuardia seaport.
"What were they thinking building a city between two bodies of water near the ocean so close to sea level"?
"Maybe we should just move the city to further inland, where its safer and we won't have to go through this over and over again...and keep wasting money"
I know I've heard that before..hmmmm.
Seems like the should have known not to build so close to the water, and if they did...not to put things that react badly with salt water underground, eh?
Sorry if it sounds like sour grapes...and I do feel horrible for the people that flooded and lost things, I know first hand how that feels from Katrina.
But I do get a bit uneasy..seeing how differently things are treated during the storms and afterwards....depending on where in the US you are situated.
I mean, this storm, while large in breadth....was a weak Cat. 1 when it made landfall. I could see it being worse, if it had happened and turned into a blizzard over NJ and NYC as was a worst case scenario.
But c'mon...if you have property ON the freakin shoreline, beach front houses, guess what...you're gonna get damaged with a hurricane or other strong storm.
Aside from the areas right near the water..I didn't see all that much damage. Sure, people are going to be without power a couple or more weeks in some areas. Those of us that live in the Gulf south take this as normal a couple times a year...that's what you get for living close to the ocean.
Again....not to make light of anyone's loss, flooding is very difficult to deal with...it sucks.
But this was a weak storm, and did about normal damage as happens to places with a storm like this hitting a coastal area.
I feel bad for those that will next have to deal with FEMA.....then again, maybe their a bit better by now...but I still have less than fond memories for them.
Anyway...people in those areas up there....you'd better get used to these storms hitting more often.
These things go in cycles..and ya'll have been lucky the past decades. I believe back in the 50's a number of storms made landfall up there...but has been so long, that people forget.
Last year, Irene and this year Sandy....hope you're better prepared for next year....if nothing else, you're a dolt if you don't purchase flood insurance. It is DIRT cheap....most everyone would be well served buying it if you ever have even occasional flash floods in your area....
Light travels faster than sound. This is why some people appear bright until you hear them speak.........
idiot... troll somewhere else
If that movie taught us one thing, it was to make sure critical infrastructure was in the physical middle of the country, away from the massive tidal waves that would be generated by impacting asteroids.
And to think they thought they had any hope.
Nature just shit in his cereal. HARD.
After this huge mess, disbanding it would be horrible.
There'll likely be worse storms around the corner, this winter certainly ain't looking up to be a nice one.
It is more a thought of the fact that it is becoming more unpredictable that is worrying. The climate is chaotic as hell right now.
Who knows what kinds of hell it might be in 5 and then 10 years from now.
Think I might go build a mountain city in Scotland. Yes. Everybody secretly wants to live in a cave anyway. Caves are just awesome.
You don't like caves? Well, you are WRONG.
Callcentric apparently had a single datacenter in NYC with no backup power generator. Lots of discussion here.
typo - meant to type EPA
I don't think you got the joke.
****
Service Temporarily Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Apache/2.2.14 (Ubuntu) Server at xkcd.com Port 80
****
No 'might' about it.
Death rate of NYC is around 1,000/week. or 140 per day.
There are two types of people in the world: Those who crave closure
Everyone depending on mass transportation...a little flooding and the city comes to a stand still?
Hell, we get hit with Cat 1 level storms all the time in the Gulf south area...it is a major PITA, and I do feel for people that got flooded, it is horrible...but it *IS* part of living near the coast of an ocean.
I hope everyone had flood insurance, it is dirt cheap.
But really aside from the expected shoreline damage..this wasn't that bad of a storm. It appeared the media had to try to go out of their way to try to make areas where they were reporting look worse than it was. I mean, reporters not on the coast...barely had wind blowing, a little rain in the background....it wasn't THAT bad of a storm.
But it does show the drawback of everyone proclaiming the good points of living urban...at least in less urban places, the storm doesn't put everything at a stand still...everyone in more 'normal' cities and areas, can still get in their own car or truck and move and go as needed to repair things, resupply and yes...get back to work and normal life.
Seems the urban style, and 99% dependence on mass, underground transportation is proving to be a nagging single point of failure, no?
This was not that bad of a storm. If it hit NYC and other cities there worse than it should...maybe it should be a wake up call for them...to be better prepared. These storms come in cycles and this is likely going to be more of a routine occurrence in the next decade or so.
it isn't like anyone will call for us to "move NYC to a safer area" like we've heard when talk of rebuilding flooded and hurricane damaged cities happened in the not so distant past...even on Slashdot.
Light travels faster than sound. This is why some people appear bright until you hear them speak.........
38 dead from the storm, not total.
As I understand it, it's the availability of fiber. DC needs a fuck ton of fiber, and a big city is more likely to have it already present then some podunk town.
Did you realize that fiber connects things? And as a result there is a "fuck-ton" of fiber not just in a big city but anywhere the major fiber trunks go, which is all across the country?
What did you think all that traffic did when it left the city? Get printed out and put on a Greyhound?
"There is more worth loving than we have strength to love." - Brian Jay Stanley
that flooding ruins backup generators, pumps, fuel storage. I hope that disaster mitigation plans are reviewed.
I also hate people who judge negatively from hindsight, but disaster planning is about considering the most probable of the improbable. Flooding looms most threatening and probable of the improbable.
Perhaps putting all the backup infrastructure on a higher floor makes it harder to maintain, access, and/ or protect from mischief/ terrorism. However plain old flooding seems to be an issue time and time again in disaster scenarios and really needs highest priority in disaster plans.
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
I'm not even white, I'm off-white. We're a new race, we will prevail! - Mitch Hedberg
http://en.wikipedia.org/wiki/Manhattan,_Kansas
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
Is that president Obama declared the situation a disaster area.
Thank you captain obvious.
Huh. Works for me.
"Be better prepared" - than what? The storm surge that flooded lower Manhattan beat the previous record by 2 feet, and that record was set almost 200 years ago. The fact that there were only 38 people killed in the entire region shows just how well prepared they were. Nobody was drowned in the subways, because they stopped the subways before they got flooded. Nobody was stuck in elevators, because they turned off the elevators before the power was shut down.
What is it with all these people saying 'it was not that bad a storm'. It was that bad a storm for the area. It was record flooding. From what I understand, Category 5 hurricanes are 'not that bad a storm' compared to the storms on Jupiter - pretty meaningless comparison, isn't it?
Eh, religion probbably killed more than 38 yesterday
Remember the data center in the movie "Wall Street"? That was the real life data center of Telerate Systems -- on the 103rd floor of the World Trade Center.
They never did get flooded.
Yes, because a single fiber cable in some backwater town with zero redundancy
Why do you think there is zero redundancy?
Even if you had only one carrier in a town that had a fiber run go through (very unlikely), if the cable gets cut on one side of town the fiber provider can just run all the traffic around the nationwide loop the other way until it gets where it needs to go - and believe me a cut will be fixed VERY quickly.
A town where "a single fiber" dead ends is obviously not a good location, but there are many towns across the U.S. with bundles of fiber going through.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Weird, it works just fine for me.
Reminds me of one of xkcd's funniest comics:
http://xkcd.com/404/
Tic-Tac-Toe, Global Thermonuclear War, and relationships all have the same winning move.
Nobody was drowned in the subways, because they stopped the subways before they got flooded. Nobody was stuck in elevators, because they turned off the elevators before the power was shut down.
You sound pretty sure. Nobody swept the subways, they just shut off the trains. Homeless that live down there most certainly died, likely after wondering why the trains weren't running, and since no one will miss them, their numbers will never be counted.
What happened to the generator underneath Grand Central that the Alphas could not even stop?
u mad bro?
The war of the previous ones that attacked New York gave Bush another mandate. Now Obama to get reelected must declare war on weather, explain that they have weapons of mass destruction, and nuke clouds, that will teach them who they are dealing with.
Sandy's wrath of ghod on the banksters.
Notice it didn't hit when OWS was in town.
First Tampa, now this. Oh the inhumanity!
If the media didn't portray this as the " END OF TEH WORLD ! " then I may pay more attention to it.
The bottom line is, while tragic, the damage is pretty much what is expected from a Category One Storm. Could have been a lot worse. There are four more categories of stronger storms to contend with . . .
Lots of rain, storm surge, high winds for a while, fire, flooding, no power for several days to months ( YES months. Some places here after IKE had no power for MONTHS ), the usual. At least it didn't spawn a dozen tornadoes over the city for you. Nothing like dealing with a hurricane AND the friggin tornadoes they usually spawn on top of it all. ( Hurricane Alicia - 1983 I think it was )
Lots of lessons are going to be learned. ( Like not putting emergency generators and / or their fuel tanks in the basements. Realizing your home insurance doesn't include Flood insurance. Etc. )
The only folks that are shocked at the damage are those that have never been through this. They may have seen it on TV in past years when Florida, Texas or Louisiana got hammered by similar storms and didn't think that much of it. Now, at least, this regions generation has a healthy respect for what a hurricane is capable of. If another one comes along, they'll be much better prepared to deal with it.
The Navy did the same with ships from Norfolk. It is far safer to be out at sea than tied up dockside.
Unless you misjudge the size, speed and track of the storm. Now add in a generator that craps out, and cannot pump out the bilge anymore.
When they left port, the projected storm track was much different than what actually happened.
The navy has much bigger, more modern ships. There's also national security risks if the navy boats are damaged. No such problems for a movie prop.
And like I said- it may be safer for the boat. It isn't safer for the crew. This wasn't a surprise storm, this had been forecast for a week. Tie up the boat, stay in a hotel (preferably about 100 miles or so inland), and then repair the boat as needed. The health and safety of the crew is far more important than the ship. The owners deserve to be sued into oblivion for even asking. It is NOT acceptable to risk 16 lives to save money on repairs. Hell, take it to a dry dock if you're that scared.
I still have more fans than freaks. WTF is wrong with you people?
One dead; one missing. Never forget.
0 = 1 + e^(Alt something)
The biggest data center in the path of Sandy was Bloomberg's huge mile-long data center at Washington Street And Houston, Right next to the Hudson River!
When I worked for Bloomberg, we took a tour through that data center. It was not on the ground floor but it would be seriously suffering for power and how were those Bloomberg Terminal signals supposed to get out of that island called Manhattan.
But Bloomberg will keep it secret if anything happens. If he can.
1 dead, 1 missing, and 13 came damn close and needed rescue choppers during a hurricane, endangering both themselves and the people who had to save them. Its inexcusable and criminally negligent. It probably doesn't deserve quite the amount of media attention its getting, but it was amazingly stupid.
I still have more fans than freaks. WTF is wrong with you people?
It took off a 1000sq ft section of roof in our Seacaucus Data Center and let the rain in -and the magic smoke out....
will be several days to recover, but we have other Data Centers in Mass, NY and Atlanta and Denver which can still cover that half of the country....
-I'm just sayin'
All Linux VPS located in New York City, NY VPS are currently offline. They are located in Internap's LGA6 facility in 111 8th Avenue.
Please be advised that Internap's LGA11 facility is experiencing significant flooding in the sub-basement of the 75 Broad Street building as a result of Hurricane Sandy. The flooding has submerged and destroyed the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level.
Thankfully, our NYC server nodes are not directly located in LGA11's facility, but rather LGA6. The cause of this temporary outage is that LGA6 routes certain parts of the network's backbones through LGA11 which is currently offline as explained above. URPad's downtown NYC facility, located at 111 8th Avenue, is currently experiencing a network-only outage. The datacenter is not located in the storm surge zone, and is not suffering from any flooding. All URPad hardware and assets are safe and remain powered on. Engineers are aware of the network outage and all efforts will be made to restore network connectivity as soon as possible.
Internap & URPad will continue to work hard to assess the situation and our recovery plans, and will communicate those plans as soon as possible. Thank you for your patience and understanding during this crisis. Please trust that we are doing everything we can to bring your services back online as soon as possible.
So it's impossible for a nuclear meltdown to happen?
Because unless it isn't, then your rant was unfounded (as unfounded as what you think the warnings were about).
before we can tell all those people along the jersey and CT shores, ya wanna live on the ocean, get your own darn flood insurance, we the taxpayers are tired of picking up the tab so you can have a great view ?
This is a category 5 reductio absurdum event.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
Marriage? That was a sign from god!
Within the next 80 years 7 billion deaths are expected.
"Nuclear Energy Institute was quick to point out that all 34 nuclear facilities in Sandy's path made it through without problems."
I by that for a dollar!!!!!!
"The NY Times has a running list of Sandy-related problems, including ... official disaster declarations from President Obama"
Ooookay.
They better have made copies of the servers in case of flooding. Million dollar HFT algos and matching algorithms are in those servers. Here is the simple stock exchange I made. I am going to integrate live market data soon http://sigmatrader.com/trade.php
Think I might go build a mountain city in Scotland
Just to warn you then, the weather here has been more shite than usual lately. Maybe better than other places in the world but the recent decade has been much worse than the previous three. I can't comment on weather prior to that on account of being too young to remember and ultimately not even existing.
They were idiots for going out to sea.
Are you saying every boat that goes out to sea is an idiot? They didn't go out to sea to get away from the storm, the were already out to sea. They were travelling from one side of the country to the other. Apparently they decided to go around the storm, when they had equipment malfunctions.
They didn't jump in their boat, and say "we'll be safer at sea"
I live in PA and we got hit hard people for 1 to 2 weeks expected with no power or phones. We have lost over 16 power stations. I am a ham radio operator and we have been helping people who do not have phones and no way of communicating. We got hit hard and we are on out skirts so I can only imagine what NY and NJ going threw.
http://www.thetechnologygeek.org
Why does the USA have these disasters each time?
Now it's New York.
Their biggest financial center TOTALLY disabled and destroyed for an estimated 50-100 billion dollars.
No real protection against water.
Our 2nd Maasvlakte -an industrial/harbour area- has 18 meters of levees.
What height of protection did Manhattan have?
Do we need to send aid to build some protection against the water for our former colony?
I hope everyone had flood insurance, it is dirt cheap.
Dirt cheap, unless you're in a flood plain ... then it's very expensive.
They were idiots for going out to sea. Even if it was marginally safer for the boat to be at sea than in port for the storm, it was safer for the crew to be in port, or in hotel rooms 100 miles inland. Those people died due to a combination of greedy ownership that valued the ship more than the crew's safety and their own stupidity in not refusing. All this for a reproduction used in movie shoots. A senseless waste.
With an attitude like that, you'd never have any crab or probably fish for that matter. Some jobs are simply more dangerous than others, and sailing even in good weather is one of those. I suspect they knew what they were getting into way before they were even hired, let alone asked to sail out. They left with plenty of time and when the forecasts painted a much more favorable situation. For that matter, I know some sailors on tall ships here on the West coast, and if you think you could have kept them from sailing out n their ship, you don't know sailors. Pretty sure, if given the chance to do it again knowing what they know now, they'd still take the ship out, just with a backup generator.
Anyone living in the low lying costal areas like NYC should be well aware that storm surges will wipe their little assets out. Building a data center in such a place is absolutely foolhardy. Living there is stupid enough.