Con Ed Says NYC Datacenters Should Get Power Saturday

Nerval's Lobster writes "The local utility serving most of the New York City area, Con Edison, reported that it should begin supplying utility power to midtown and lower Manhattan by Saturday evening, returning the island's data centers and citizens to some semblance of normalcy. In the past few days, data center managers have been forced to add fuel logistics to their list of responsibilities, as most Manhattan data centers have been subsisting on generator power. That should come to an end, for the most part, when utility power is restored. In a possibly worrying note, Verizon warned late on Nov. 1 that its services to business customers could be impacted due to lack of fuel."

Comments

Fuel logistics

[hawguy]

In the past few days, data center managers have been forced to add fuel logistics to their list of responsibilities, as most Manhattan data centers have been subsisting on generator power.

Any datacenter manager that doesn't already have fuel logistics in their disaster plan is in the wrong line of work. Few inner city datacenters have a week or more of fuel on-site - most have only days of fuel, and they count on fuel contracts from suppliers to keep them running. And the supplier may not be able to honor the contract in a disaster.

Suburban and rural datacenters have the space (and less conflict with fire codes since the fuel is not stored in or near an office building) to keep weeks of fuel on hand. The last datacenter that I colocated in had 2 weeks of fuel on-site, and had another week of fuel in a trailer that can be trucked in from their other facility 60 miles away if the roads are passable. They had a spare generator that can be trucked in from that other facility as well. (and this facility could send fuel and a generator to that facility if needed)

Re:Fuel logistics

[hawguy]

Any datacenter manager that doesn't already have fuel logistics in their disaster plan is in the wrong line of work.

I don't think any data center manager had a line item in the disaster recovery plan that included having all transportation access cut to the entire island due to flooding of the tunnel and closure of the bridges, for over a week. Everyone is having a problem getting fuel into the city; even mission-critical services like emergency services, hospitals, and telecommunications facilities.

What kind of disaster plan is it that doesn't account for a likely disaster? I can guarantee that every sizable datacenter had exactly this scenario in the DR plans. You don't run a $10M facility in a coastal city on an island without including flooding in your DR plan. This was a 100 year event, so it was definitely on everyone's horizon. Having it in the plan doesn't mean that you have a good solution - living in SF means Earthquakes are a big part of our DR plans, but we have no expectation that our facility will survive a 7.5 earthquake intact, or even that our employees will be motivated to come to work when they are worried about their own survival.

As to your comment that "suburban and rural datacenters have the space", sure... but where's the fiber optic cable hookups and the telecommunications infrastructure located? I'll give you a hint: Not in a barn. Those data centers are located downtown because that's where everything else is.

Just follow the train lines to find out where the major telecommunication lines are -- I have access to more carriers down on the Peninsula outside of San Francisco than I do downtown. You may be surprised at how much bandwidth runs through Colorado and even Missouri.

Not only that, but most of the data centers on the island are there because that's where Wall St. is, and milliseconds matter when it comes to high volume trading and financial transactions.

Sure, latency is a reason to be close to NYC, but I don't think any of the exchanges even have datacenters in the city anymore,they are all across the river. I know NASDAQ has a backup facility in Virginia.

Commercial real estate is at a premium in New York. Actually, all real estate is, leading to the old joke that when a New Yorker hears someone has died, the first question they ask is, "Is their apartment for rent?"

I'm not sure what your point is? Datacenters have to be built in the city because that's where the carriers are, but commercial real estate is expensive so don't build your datacenter in the city?

I think it's more likely to assume you've made an error in your reasoning, writing opinions from the comfort of an armchair, than people being paid over six figures who's job depends on balancing everything out exactly and to the nearest penny an hour.

You're obviously not in NYC if you think "over six figures" means someone is highly paid. Part of my job is planning our IT DR strategy. Fortunately, our Facilities dept is in charge of the generator so I don't need to worry about fuel contracts or keeping it running, but I do need to make sure our data is safe no matter what happens to the building and that we can continue to operate as a business. The only "disaster" that we plan on riding out on the building generator is a localized power outage when we know we'll be able to get fuel once our 3 - 5 days of fuel runs out. If there's a widespread power outage or disaster, our plan is to transition to the remote site since we know we may not be able to keep the generator fueled.

Re:Fuel logistics

[kriston]

The government is not "screw[ing] over the little guy" as you so eloquently put it. The non-road diesel fuel is dyed specifically because that fuel's tariff does not carry road tax. Road tax pays for the roads. This is what the situation is really about. So many violators were using non-road fuel that they had to take steps.

All licensed truck drivers implicitly understand that you do not put non-road/farm fuel into a road vehicle if you intend to keep that vehicle in revenue service. It's intuitively obvious to everyone, but it's too bad your drivers were either ignorant of the law or chose not to follow it.

Re:Fuel logistics

[hawguy]

People who vote for big government get everything that comes with it... like dyes added to products for no reason other than to enable government to control you (by regulating how you use things and by taxing things with complex lobbyist-influenced tax codes).

The Fuel tax is one of the more fair taxes - it taxes you (roughly) based on your usage of the resource it's supposed to be paying for. The thing I like least about fuel taxes is that they are not high enough to pay for road maintenance.

Roving looters who cannot be stopped by the disarmed citizens

If very citizen had a gun, then those packs of roving looters would be well armed, so there would be wild-west style shootouts in the street between the good guys and the bad guys. But in a disaster, the lines are blurred and it's not always clear who the good guy is and who the bad guy is. Is the guy walking down the street with a few bottles of water for his elderly mother a good guy or a bad guy when he refuses to share it with the young mother and child begging for it. What if the mother had a gun and took the water by force? Or what if the man had a gun and shot the mother when she tried to take the water by force?

Lack of fuel (nobody has their own reserves. Most would not bother but there are plenty of people and businesses who would if they were not going to be taxed-and-regulated to death for trying)

People are bemoaning the lack of fresh water and food. Both are legal to stockpile both by citizens and private businesses, yet there seem to be few businesses willing to stock up on thousands of bottles of water just in case disaster strikes. Why do you think they'd be more likely to stock up on a perishable and expensive to store fuel?

Lack of sufficient private help (Americans used to have a huge number of private charitable orgs and "mutual aid" orgs that average people used to join that rushed in to help after disasters, but now nobody bothers because they've been taught to pay taxes and then count on government)

The Red Cross and other private, charitable organizations always offer disaster relief. But some disasters require more manpower, machinery, legal law enforcement powers, or weapons than can be reasonably mustered by an organization funded by donations, so that's where the US government can step in. How many $10M helicopters and $100M heavy lift aircraft do you think the Red Cross should own and maintain in order to shuttle people and supplies to a disaster zone?

Hopefully this very sad event will finally hammer into peoples' heads what Katrina should have (but apparently failed to) ... that in the end, when the going gets tough, you are responsible for being prepared for disasters and taking care of your family first, your neighbors next, and the larger community if you can; no government jerk from a far-away city will care as much as you do.

That's definitely true and a good lesson to be learned from any disaster - be prepared to be on your own for 3 - 7 days. But few city dwellers can provide enough food, water, heat, medications (some need refrigeration), and sanitation (with water and possibly sewage not working in a big highrise) to stick it out for more than week or so in a disaster.

If the people and businesses of NY and NJ were not burdened by crazy regulations and threats of lawsuits they'd probably get things back up and runninga on their own twice as fast as they are going to (locals would commit acts of common sense on behalf of themselves and their neighbors).

Not all regulations are driven by government greed or thirst for power, many regulations are safety based. If you try to fill the back f your pickup with twenty 5 gallon buckets of fuel and try to sell it to the highest bidder on the streets, you're (hopefully) going to get stopped on the street. Likewise, if you set up your own s

Re:Get Some Priorities!

[CastrTroy]

Depends what's running in the datacenter. Banks, grocery stores, public utilities, repair contractors all need datacenters to get their jobs done. Also, many people work in data centers, or have jobs which rely on the datacenters being up. For them, the datacenter does feed their family

Re:Get Some Priorities!

[hawguy]

Dozens are dead, billions of dollars of property destroyed and businesses decimated, millions still in darkness and cold, and you are talking about data centers??

Data centers cannot feed my family or heat my cold, flooded house.

  Where is your decency sir? It's a long week and weekend for the millions affected and the first responders. The glibness of this post is shocking and disgusting.

This is Slashdot, there are plenty of other sites that cover human misery and suffering, this site is for geek news, and geeks care about datacenters.

Datacenters can't feed your family or heat your cold, flooded house (unless you live next to a datacenter that recovers waste heat for residential heating), but the datacenter can help your utility get power back online, it can help your local merchant process credit card transactions and use their POS system so you can buy replacement goods, it can let you post your family's status so your mom can stop worrying about you, it can help your bank conduct online transactions efficiently to let you receive your insurance money, and of course, you're using multiple datacenters right now to read and post to Slashdot.

If you think datacenters are unimportant, try going a week without using any good or service that wasn't produced or delivered to you without the aid of datacenters.

Re:Get Some Priorities!

[hawguy]

As little as I like agreeing with ACs, this was my exact thought upon reading TFS. I know this is News For Nerds, but let's not pretend getting data centers back up is more important than rebuilding an area that's been severely decimated (and not in the Roman 10% way).

Take a look at this:

https://www.google.com/search?tbm=nws&sclient=psy-ab&q=+new+york+city+recovery+hurricane+sandy&oq=+new+york+city+recovery+hurricane+sandy

Over 200,000 articles and 3600 news sources covering NYC's recovery efforts. Surely there's enough space left on the Internet for a News for Nerds site to cover news for nerds?