People who vote for big government get everything that comes with it... like dyes added to products for no reason other than to enable government to control you (by regulating how you use things and by taxing things with complex lobbyist-influenced tax codes).
The Fuel tax is one of the more fair taxes - it taxes you (roughly) based on your usage of the resource it's supposed to be paying for. The thing I like least about fuel taxes is that they are not high enough to pay for road maintenance.
Roving looters who cannot be stopped by the disarmed citizens
If very citizen had a gun, then those packs of roving looters would be well armed, so there would be wild-west style shootouts in the street between the good guys and the bad guys. But in a disaster, the lines are blurred and it's not always clear who the good guy is and who the bad guy is. Is the guy walking down the street with a few bottles of water for his elderly mother a good guy or a bad guy when he refuses to share it with the young mother and child begging for it. What if the mother had a gun and took the water by force? Or what if the man had a gun and shot the mother when she tried to take the water by force?
Lack of fuel (nobody has their own reserves. Most would not bother but there are plenty of people and businesses who would if they were not going to be taxed-and-regulated to death for trying)
People are bemoaning the lack of fresh water and food. Both are legal to stockpile both by citizens and private businesses, yet there seem to be few businesses willing to stock up on thousands of bottles of water just in case disaster strikes. Why do you think they'd be more likely to stock up on a perishable and expensive to store fuel?
Lack of sufficient private help (Americans used to have a huge number of private charitable orgs and "mutual aid" orgs that average people used to join that rushed in to help after disasters, but now nobody bothers because they've been taught to pay taxes and then count on government)
The Red Cross and other private, charitable organizations always offer disaster relief. But some disasters require more manpower, machinery, legal law enforcement powers, or weapons than can be reasonably mustered by an organization funded by donations, so that's where the US government can step in. How many $10M helicopters and $100M heavy lift aircraft do you think the Red Cross should own and maintain in order to shuttle people and supplies to a disaster zone?
Hopefully this very sad event will finally hammer into peoples' heads what Katrina should have (but apparently failed to)... that in the end, when the going gets tough, you are responsible for being prepared for disasters and taking care of your family first, your neighbors next, and the larger community if you can; no government jerk from a far-away city will care as much as you do.
That's definitely true and a good lesson to be learned from any disaster - be prepared to be on your own for 3 - 7 days. But few city dwellers can provide enough food, water, heat, medications (some need refrigeration), and sanitation (with water and possibly sewage not working in a big highrise) to stick it out for more than week or so in a disaster.
If the people and businesses of NY and NJ were not burdened by crazy regulations and threats of lawsuits they'd probably get things back up and runninga on their own twice as fast as they are going to (locals would commit acts of common sense on behalf of themselves and their neighbors).
Not all regulations are driven by government greed or thirst for power, many regulations are safety based. If you try to fill the back f your pickup with twenty 5 gallon buckets of fuel and try to sell it to the highest bidder on the streets, you're (hopefully) going to get stopped on the street. Likewise, if you set up your own s
I'm well aware that bandwidth runs through rural areas, but planting a data center in a place conveniently located on top of one that also bisects a town large enough to host said data center is somewhat rare. When Google announced it needed a data center, only a dozen small towns in the whole country met the requirements. So it's not as common as you think.
Sounds like there are at least a dozen.
Sure, latency is a reason to be close to NYC, but I don't think any of the exchanges even have datacenters in the city anymore,they are all across the river. I know NASDAQ has a backup facility in Virginia.
I seem to recall there were two of them located in the Twin Towers. You know, until the day they were exploded. So don't "think" me anything -- either know, or keep quiet.
So you're spouting 10 year old information for a datacenter that you already know doesn't exist and you're calling me uninformed?
The data centers that house their operations are all across the Hudson River, in Carteret (Nasdaq), Mahwah (NYSE), Secaucus (Direct Edge) and Weehawken (BATS), N.J.
I'm not sure what your point is? Datacenters have to be built in the city because that's where the carriers are, but commercial real estate is expensive so don't build your datacenter in the city?
Where do you plan on locating the several thousand gallons of diesel fuel needed to run a 2 or 4MW data center for over a week, in downtown Manhattan? It's expensive enough just finding space for the servers.
In the basement, where datacenters in NY typically locate their fuel? As I pointed out in your earlier message, a week's worth of fuel for your 2MW generator would consume around 300 sq feet of 10 foot high tank space - or about the space of 2 parking spaces in the parking garage.
You're obviously not in NYC if you think "over six figures" means someone is highly paid. Part of my job is planning our IT DR strategy. Fortunately, our Facilities dept is in charge of the generator so I don't need to worry about fuel contracts or keeping it running, but I do need to make sure our data is safe no matter what happens to the building and that we can continue to operate as a business.
I didn't say it was highly paid, I was just suggesting they were probably being paid more than you.
No doubt - a datacenter manager for a large facility in NYC makes more money than me, but I'm not sure why that's relevant.
And given some of the comments I've read in your previous post, it's clear to me that whatever disaster recovery plans you are involved with or in charge of is nowhere near the scale of the data centers under discussion.
Actually, my datacenter (well, 3 of them in 3 different buildings next to each other) is surprisingly close to the size of what you're talking about - a modern datacenter with 2MW of generating capacity may only be 10,000 sq feet, which is not terribly large these days. However, It's not very dense, we have a lot of sparsely populated cabinets of non-computer gear, our total power consumption not including cooling is around 100KVA.
But since billion dollar fiber links are being laid to cut latency by milliseconds primarily for algorithmic trading, if you really need a datacenter in NYC, $83M/year could be worth it.
And that, in a nut shell is why, you don't really want to use "The Cloud". Keep your data within reach.
if you live in NY and keep your data in reach, it's likely that you would have been affected by this storm.
If you kept it at 2 different cloud providers (or 2 physically separated regions of the same cloud provider), you would have been fine.
The cloud is as safe as you want it to be, but don't assume that storing data in a single cloud provider's facility is any safer than storing it at your own facility unless you're paying to have your data replicated somewhere else.
Any datacenter manager that doesn't already have fuel logistics in their disaster plan is in the wrong line of work.
I don't think any data center manager had a line item in the disaster recovery plan that included having all transportation access cut to the entire island due to flooding of the tunnel and closure of the bridges, for over a week. Everyone is having a problem getting fuel into the city; even mission-critical services like emergency services, hospitals, and telecommunications facilities.
What kind of disaster plan is it that doesn't account for a likely disaster? I can guarantee that every sizable datacenter had exactly this scenario in the DR plans. You don't run a $10M facility in a coastal city on an island without including flooding in your DR plan. This was a 100 year event, so it was definitely on everyone's horizon. Having it in the plan doesn't mean that you have a good solution - living in SF means Earthquakes are a big part of our DR plans, but we have no expectation that our facility will survive a 7.5 earthquake intact, or even that our employees will be motivated to come to work when they are worried about their own survival.
As to your comment that "suburban and rural datacenters have the space", sure... but where's the fiber optic cable hookups and the telecommunications infrastructure located? I'll give you a hint: Not in a barn. Those data centers are located downtown because that's where everything else is.
Just follow the train lines to find out where the major telecommunication lines are -- I have access to more carriers down on the Peninsula outside of San Francisco than I do downtown. You may be surprised at how much bandwidth runs through Colorado and even Missouri.
Not only that, but most of the data centers on the island are there because that's where Wall St. is, and milliseconds matter when it comes to high volume trading and financial transactions.
Sure, latency is a reason to be close to NYC, but I don't think any of the exchanges even have datacenters in the city anymore,they are all across the river. I know NASDAQ has a backup facility in Virginia.
Commercial real estate is at a premium in New York. Actually, all real estate is, leading to the old joke that when a New Yorker hears someone has died, the first question they ask is, "Is their apartment for rent?"
I'm not sure what your point is? Datacenters have to be built in the city because that's where the carriers are, but commercial real estate is expensive so don't build your datacenter in the city?
I think it's more likely to assume you've made an error in your reasoning, writing opinions from the comfort of an armchair, than people being paid over six figures who's job depends on balancing everything out exactly and to the nearest penny an hour.
You're obviously not in NYC if you think "over six figures" means someone is highly paid. Part of my job is planning our IT DR strategy. Fortunately, our Facilities dept is in charge of the generator so I don't need to worry about fuel contracts or keeping it running, but I do need to make sure our data is safe no matter what happens to the building and that we can continue to operate as a business. The only "disaster" that we plan on riding out on the building generator is a localized power outage when we know we'll be able to get fuel once our 3 - 5 days of fuel runs out. If there's a widespread power outage or disaster, our plan is to transition to the remote site since we know we may not be able to keep the generator fueled.
The last datacenter that I colocated in had 2 weeks of fuel on-site, and had another week of fuel in a trailer that can be trucked in from their other facility 60 miles away if the roads are passable.
How often do they rotate/burn through that supply?
As has been mentioned in a few articles that I've read, diesel fuel doesn't last forever, and so if you tried to use it after it's been sitting too long you may have problems. Hopefully as part of regular maintenance / testing they burn through it of the course of a year and replenish it with fresh stuff.
I don't remember the details on their rotation schedule, but I imagine that having their own fuel trailers made it easier to swap out fuel with some other site that needed it. Maybe they sold 9 month old fuel to an industrial site at a discount.
I do remember that they had their generator vendor bring in a load bank large enough to let them do full-load testing every month. And once a quarter they did a live cutover from utility power to generator. This seemed a little risky at first, but it makes sense - would you rather find out during a disaster that some component of the emergency power system wasn't working because it was not adequately tested, or would you rather find out when the generator vendor and electricians are standing by and utility power is available to fail back from the generator.
That facility weathered several power outages without a glitch, but the longest I can remember was around 48 hours.
Well the good news is that few drivers in the USA can use the diesel that's needed to feed datacenter generators in their cars, so hopefully the trucks will get through.
As little as I like agreeing with ACs, this was my exact thought upon reading TFS. I know this is News For Nerds, but let's not pretend getting data centers back up is more important than rebuilding an area that's been severely decimated (and not in the Roman 10% way).
Over 200,000 articles and 3600 news sources covering NYC's recovery efforts. Surely there's enough space left on the Internet for a News for Nerds site to cover news for nerds?
Dozens are dead, billions of dollars of property destroyed and businesses decimated, millions still in darkness and cold, and you are talking about data centers??
Data centers cannot feed my family or heat my cold, flooded house.
Where is your decency sir? It's a long week and weekend for the millions affected and the first responders. The glibness of this post is shocking and disgusting.
This is Slashdot, there are plenty of other sites that cover human misery and suffering, this site is for geek news, and geeks care about datacenters.
Datacenters can't feed your family or heat your cold, flooded house (unless you live next to a datacenter that recovers waste heat for residential heating), but the datacenter can help your utility get power back online, it can help your local merchant process credit card transactions and use their POS system so you can buy replacement goods, it can let you post your family's status so your mom can stop worrying about you, it can help your bank conduct online transactions efficiently to let you receive your insurance money, and of course, you're using multiple datacenters right now to read and post to Slashdot.
If you think datacenters are unimportant, try going a week without using any good or service that wasn't produced or delivered to you without the aid of datacenters.
In the past few days, data center managers have been forced to add fuel logistics to their list of responsibilities, as most Manhattan data centers have been subsisting on generator power.
Any datacenter manager that doesn't already have fuel logistics in their disaster plan is in the wrong line of work. Few inner city datacenters have a week or more of fuel on-site - most have only days of fuel, and they count on fuel contracts from suppliers to keep them running. And the supplier may not be able to honor the contract in a disaster.
Suburban and rural datacenters have the space (and less conflict with fire codes since the fuel is not stored in or near an office building) to keep weeks of fuel on hand. The last datacenter that I colocated in had 2 weeks of fuel on-site, and had another week of fuel in a trailer that can be trucked in from their other facility 60 miles away if the roads are passable. They had a spare generator that can be trucked in from that other facility as well. (and this facility could send fuel and a generator to that facility if needed)
Anyone that's ever worked with touchscreens before knows that those things need frequent recalibration
Maybe in 1995 when touch screens were new technology.
My smartphone has no way to calibrate it because it's not necessary. I can believe that calibration-free touch screens are more expensive than cheap ones that need constant calibration.
I don't recall my bank's ATM screen ever being out of calibration either. Maybe it was calibrated at install time, but it doesn't seem to slip out of calibration.
Which ignores the point that regardless of the touch screen calibration, it's trivial to design a UI that gives immediate feedback that the correct selection was picked.
Aside from a simple check mark next to the chosen candidate's name, how about inverting the color of the entire name field making it clear which one has been chosen?
I've heard this before, but I've never seen the reasoning - why do the generators need full overhauls after running for a few days? Aren't these the same engines that are in trucks/construction equipment and run for thousands of overalls without an overhaul?
When i worked at some remote construction sites, they had 1MW+ generators that went for over a year of nearly continuous use between overhauls. As I recall, they did oil changes and other routine maintenance monthly.
An emergency power generator might be run at a higher load than a continuous duty generator, but I can't imagine any large diesel generator needing an overhaul after a few hundred hours of operation.
A major disaster is the time when you really need your hospital to function properly. So spending some money on disaster preparedness is prudent.
Yes, we keep large amounts of emergency supplies of consumables on site : medical gases, medications, food, linens, antiseptics, needles, lines, etc.
Telling injured hurricane victims that show up in the E.D. "Sorry, dude, but this storm really screwed us up, man. We can't treat you." isn't acceptable.
Sure, I understand that a hospital needs to function for some time during/after a disaster, but 45 days of supplies seems like expensive overkill - do you have millions of gallons of water on site and your own treatment plant on-site since it's not likely that your municipal plant will withstand a 30+ day power outage.
I just googled to source the numbers to get a ballpark figure. So yeah, undoubtedly they're off -- people spend months preparing reports for their data center on fuel consumption, storage costs, location, etc. I spent 20 minutes. But putting the numbers together shows that even if you lowball all the numbers by 50%, storage cost is still massively eclipsing fuel cost.
Sure, I understand the fuel usage numbers were an estimate, but your calculations weren't off by 50% - you were claiming that it would take an entire 264x264 foot office floor (70,000 square feet) to store a week's worth of fuel. I was pointing out that it only takes about 300 square feet - you were off by a factor of 233, not by 50%. Instead of an unaffordable $83M in rent and an entire office floor to store a week's worth of fuel, it only takes an 18 foot by 18 foot room and $400K/year. That's a huge difference.
Your key mistakes were in dividing cubic inches by 12 instead of by 12*12*12 (or 1728) to convert from in^3 to ft^3, and again where you assumed that 67,000 ft^3 of fuel would need 69,000 square feet of office space to store, ignoring the 12 foot ceilings that you noted earlier.
If you did devote an entire 70,000 square foot 10 foot high office floor to fuel storage, you'd have 700,000 ft^3 of fuel storage, or 5,000,000 gallons, enough to power the 2MW generator for 1400 days. Of course, it would weigh 35M lbs, so the building would need to be purpose built (or retrofitted) to support the weight.
"The only time you can justify spending that kind of cash is if you're supporting critical infrastructure like phones, hospitals, and emergency services."
Which is why I can't understand why some hospitals in New York City had to evacuate due to power loss. I heard that one hospital evacuated the goddamn NICU.
...
Whoever was responsible for disaster planning for these hospitals ought to be facing criminal charges.
Hospitals, even publicly funded ones, don't have unlimited funds to implement disaster recovery plans. If it comes down to reducing services during the 99.9% of the time when there is no disaster versus spending money to ride out the 0.1% of the time when there's a major disaster, spending the money to help more people now isn't necessarily worse than planning to evacuate during an uncommon disaster.
I'd question the need for a hospital to have 45 days of fuel on-site... keeping the lights on does little good when you run out of drugs, antiseptics, fresh water, bandages, oxygen, and all of the other supplies you need to keep the facility running.
Do you keep a 45 day supply of all of the rest of your consumables on-site? My only experience is with a small rural/regional hospital and they had 3 days of fuel on-site, and 4 -7 days of most consumables.
If you only need 1MW of power, that means you have 80,000 gallons of fuel in storage - at 7mpg that means your delivery trucks are traveling over 500,000 miles every 6 months. What kind of deliveries are you making that puts a million miles on your fleet each year?
I would have thought it was obvious, but the middle of the city is where the telecommunications infrastructure is. It doesn't exist in a barn a hundred miles from any major city. And fuel has a shelf life. Diesel will slowly oxidize over time, and so the time you can keep it in the tank is about 12 months. You'll burn about 72 gallons an hour per megawatt (as a rough average). So a 2 megawatt data center will need about 3,500 gallons of diesel per day. A gallon takes up 231 cubic inches of space, so a single day's worth of fuel would need a tank with a capacity of 67,375 cubic feet. The average height of a floor in a skyscraper is 12.5 feet. In New York city, the average city block is 264 feet. That means that even if you filled an entire floor of a skyscraper with nothing but diesel fuel, you'd still get less than a week's worth of fuel guys. At a rate of perhaps $100 per month per square foot... you're talking about $83 million a year for a floor of an outlying area just to store that diesel fuel. Mind you, near Wall St., that price is probably going to be double or triple. The price of fuel is peanuts compared to this; 7 days of fuel for a 2MW plant would cost you only $94,000, plus transport costs.
While I agree with your point in general, your math is off. I can see our own 1MW generator and 3-day fuel tank from my office, which is no where near the size of tank you calculated -- it should consume around 1/4 of a city block using your figures.
A week of fuel (24,000 gallons) is 3200 cubic feet, with a 10 foot high tank, that's only 320 square feet, at $100/month that's still a pricey $384,000/year. But since that 2MW of power is enough to power 1000 servers (assuming 1000 watts/server, half the power goes to cooling), that's only $384 per server per year or $32/month per server.
Each server rack consumes around 6 square feet of space (allowing room in front and behind the rack), so that 6 square feet would cost $7200/year, or $171/server/year if you put 42 servers in a rack. ($14/server/month). So spending $32/month to provide a week of backup power isn't that out of line if you really want a server close to your office.
But $100 sounds pretty high for unfinished office space in Manhattan even in a class-A building. If you put your datacenter into a cheaper class-B or class-C building - rent would be closer to $40/sq ft.
If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York. Now, of course, you generally want a backup site that is more than 100 miles away, in the event of actual disasters, but you definitely want your primary facilities as close to your users as you can get.
Sure, there will always be datacenters in NYC, but that doesn't change the fact that instead of putting all your money into trying to build a datacenter impervious to all hazards, you're better off having a second site far from your primary site. Real-world constraints mean that you can't build that perfectly impervious datacenter when you're subject to real estate prices, building codes, fire codes, and Murphy's Law -- there will always be a disaster that the facility can't handle.
Everyone already knew that the on-site fuel supply is the limiting factor of power availability in a disaster. Even fuel delivery contracts mean nothing in a disaster or wide-spread outages - hospitals, EMS and other government services will trump the fuel delivery contract, if a hospital needs fuel, they are going to get the fuel that's been "guaranteed" for your datacenter.
There's no reason to spend big $$ creating a flood proof, earthquake proof, tornado proof, airplane crash proof datacenter in the middle of a city when you can have a disaster recovery site 1000 miles away that's not subject to the same type of disaster. (except maybe an asteroid strike, but there are few datacenters on the moon). No matter how disaster-proof you make your datacenter, mother nature (or man) will always find a way to create a disaster you didn't plan for -- even if that "disaster" is a typo in a router configuration file that takes down the network, or a contractor accidentally shorting out the emergency power cutoff switch wiring when bolting a rack to the wall.
If the whole purpose of the backup system is to provide electrical power, then you'd think they'd consider the power sources needed for it.
We generally assume that the purpose of a backup system is to actually operate when the main system fails. Providing assurance that it will is an engineering job that requires enumerating the possible failure modes and providing contingencies for them. The customer should presumably have that assessment done, then decide which of the contingencies that they'll pay for (hopefully based on some cost/benefit/likelihood balancing strategy).
In this case, either they didn't do the right assessment, or else they didn't think this mode of failure was likely. Either way, you hope they learn something.
It's not that they didn't account for having power to the fuel pump, they didn't account for having the basement, the fuel tank, fuel pump and associated electrical panels to be submerged in water.
After a certain point, you stop designing redundancy and resilience into your current facility and use the money to build a completely separate, geographically distant facility. When building in an existing office building, it may not be possible to design in basic safeguards that you'd have in a purpose built datacenter. In a suburban datacenter, the backup generators and fuel tanks will be outside on skids, raised several feet above ground level - in a city office building, you rarely have that luxury and local building and fire codes further restrict what you can do.
The biggest takeaway is to not count on a single datacenter since there's always some disaster that can take it down.
If we can figure out how to put a pump into a well full of water deep underground, it should not be too difficult to figure out how to do something similar with a fuel pump, relatively speaking. To have not done so is a fail.
There's a lot that *could* be done, but usually the price tag precludes *everything* from being done. If the electric panel that feeds your underwater pump is underwater itself, then the pump isn't going to do any good.
The tanks are required to be below ground by the fire code. I think the better question is why put big data centers on a low lying coastline/island and/or a city with a giant target painted on it by every anti-american, anti-establishment, anti-whatever whacko the world over. Data centers belong in places with low risk of natural disaster, war, terrorist attack, riot or really anything that brings the police out.
Because some companies want to be physically close to their datacenter. Virtualization and cloud computing is changing this... slowly.
Since when has Diesel been highly flammable? You can actually use it to put fires out. It takes quite some heat to get it going so really poses less risk than you average stationary cupboard.
Until it leaks out, saturates wood framing and other building materials, then comes into contact with an ignition source (like a pilot light, or candles that the residents are using since the power is out).
Then this hard to ignite fuel quickly turns an office into an inferno.
I can knock a candle over in my stationary cupboard and as long as I pick it up quickly, I wouldn't expect a fire. Soak that same cabinet and knock a candle over and it's a different story. Kerosene (very similar to diesel) makes a good fire starter to help get a wood fire started.
As somebody else already said, human power. There's also the option of tapping into the generator. I don't know the specifics, but if it takes less than a gallon of fuel to lift a gallon of fuel up the side of the building, you could just tap into the generator you're trying to refuel. The other option is to have a second smaller generator on the ground that powers just the lift/winch. This could be easily refueled. Seems like bucket brigade was the first thing that came to mind, and once they had that going, people stopped thinking of better ideas.
Or maybe they've seen enough Bugs Bunny cartoons to know what happens when a makeshift winch fails - whatever it's carrying falls to the ground in a big splat, flattening whoever it falls onto.
Hoisting a 30 pound gas can 200 feet in the air during a disaster is not the time for amateurs to jury rig something together.
Slashdot Mirror
People who vote for big government get everything that comes with it... like dyes added to products for no reason other than to enable government to control you (by regulating how you use things and by taxing things with complex lobbyist-influenced tax codes).
The Fuel tax is one of the more fair taxes - it taxes you (roughly) based on your usage of the resource it's supposed to be paying for. The thing I like least about fuel taxes is that they are not high enough to pay for road maintenance.
Roving looters who cannot be stopped by the disarmed citizens
If very citizen had a gun, then those packs of roving looters would be well armed, so there would be wild-west style shootouts in the street between the good guys and the bad guys. But in a disaster, the lines are blurred and it's not always clear who the good guy is and who the bad guy is. Is the guy walking down the street with a few bottles of water for his elderly mother a good guy or a bad guy when he refuses to share it with the young mother and child begging for it. What if the mother had a gun and took the water by force? Or what if the man had a gun and shot the mother when she tried to take the water by force?
Lack of fuel (nobody has their own reserves. Most would not bother but there are plenty of people and businesses who would if they were not going to be taxed-and-regulated to death for trying)
People are bemoaning the lack of fresh water and food. Both are legal to stockpile both by citizens and private businesses, yet there seem to be few businesses willing to stock up on thousands of bottles of water just in case disaster strikes. Why do you think they'd be more likely to stock up on a perishable and expensive to store fuel?
Lack of sufficient private help (Americans used to have a huge number of private charitable orgs and "mutual aid" orgs that average people used to join that rushed in to help after disasters, but now nobody bothers because they've been taught to pay taxes and then count on government)
The Red Cross and other private, charitable organizations always offer disaster relief. But some disasters require more manpower, machinery, legal law enforcement powers, or weapons than can be reasonably mustered by an organization funded by donations, so that's where the US government can step in. How many $10M helicopters and $100M heavy lift aircraft do you think the Red Cross should own and maintain in order to shuttle people and supplies to a disaster zone?
Hopefully this very sad event will finally hammer into peoples' heads what Katrina should have (but apparently failed to) ... that in the end, when the going gets tough, you are responsible for being prepared for disasters and taking care of your family first, your neighbors next, and the larger community if you can; no government jerk from a far-away city will care as much as you do.
That's definitely true and a good lesson to be learned from any disaster - be prepared to be on your own for 3 - 7 days. But few city dwellers can provide enough food, water, heat, medications (some need refrigeration), and sanitation (with water and possibly sewage not working in a big highrise) to stick it out for more than week or so in a disaster.
If the people and businesses of NY and NJ were not burdened by crazy regulations and threats of lawsuits they'd probably get things back up and runninga on their own twice as fast as they are going to (locals would commit acts of common sense on behalf of themselves and their neighbors).
Not all regulations are driven by government greed or thirst for power, many regulations are safety based. If you try to fill the back f your pickup with twenty 5 gallon buckets of fuel and try to sell it to the highest bidder on the streets, you're (hopefully) going to get stopped on the street. Likewise, if you set up your own s
I'm well aware that bandwidth runs through rural areas, but planting a data center in a place conveniently located on top of one that also bisects a town large enough to host said data center is somewhat rare. When Google announced it needed a data center, only a dozen small towns in the whole country met the requirements. So it's not as common as you think.
Sounds like there are at least a dozen.
Sure, latency is a reason to be close to NYC, but I don't think any of the exchanges even have datacenters in the city anymore,they are all across the river. I know NASDAQ has a backup facility in Virginia.
I seem to recall there were two of them located in the Twin Towers. You know, until the day they were exploded. So don't "think" me anything -- either know, or keep quiet.
So you're spouting 10 year old information for a datacenter that you already know doesn't exist and you're calling me uninformed?
http://www.tradersmagazine.com/news/nyse-levitt-wrong-about-backup-readiness-110474-1.html
No U.S. exchanges actually operate in lower Manhattan any longer, even though the New York Stock Exchange maintains a trading floor at 11 Wall Street.
The data centers that house their operations are all across the Hudson River, in Carteret (Nasdaq), Mahwah (NYSE), Secaucus (Direct Edge) and Weehawken (BATS), N.J.
I'm not sure what your point is? Datacenters have to be built in the city because that's where the carriers are, but commercial real estate is expensive so don't build your datacenter in the city?
Where do you plan on locating the several thousand gallons of diesel fuel needed to run a 2 or 4MW data center for over a week, in downtown Manhattan? It's expensive enough just finding space for the servers.
In the basement, where datacenters in NY typically locate their fuel? As I pointed out in your earlier message, a week's worth of fuel for your 2MW generator would consume around 300 sq feet of 10 foot high tank space - or about the space of 2 parking spaces in the parking garage.
You're obviously not in NYC if you think "over six figures" means someone is highly paid. Part of my job is planning our IT DR strategy. Fortunately, our Facilities dept is in charge of the generator so I don't need to worry about fuel contracts or keeping it running, but I do need to make sure our data is safe no matter what happens to the building and that we can continue to operate as a business.
I didn't say it was highly paid, I was just suggesting they were probably being paid more than you.
No doubt - a datacenter manager for a large facility in NYC makes more money than me, but I'm not sure why that's relevant.
And given some of the comments I've read in your previous post, it's clear to me that whatever disaster recovery plans you are involved with or in charge of is nowhere near the scale of the data centers under discussion.
Actually, my datacenter (well, 3 of them in 3 different buildings next to each other) is surprisingly close to the size of what you're talking about - a modern datacenter with 2MW of generating capacity may only be 10,000 sq feet, which is not terribly large these days. However, It's not very dense, we have a lot of sparsely populated cabinets of non-computer gear, our total power consumption not including cooling is around 100KVA.
a couple of times a century is definitely a likely disaster. Would you not plan for an event with a 2% chance of occuring every year?
Not if it cost me $86 million a year, and my total operating budget was $200 million.
Where does $86M come from? The $83M number you gave in an earlier thread was based on faulty math and the true number is closer to $400K:
http://slashdot.org/comments.pl?sid=3224821&cid=41849463
But since billion dollar fiber links are being laid to cut latency by milliseconds primarily for algorithmic trading, if you really need a datacenter in NYC, $83M/year could be worth it.
And that, in a nut shell is why, you don't really want to use "The Cloud".
Keep your data within reach.
if you live in NY and keep your data in reach, it's likely that you would have been affected by this storm.
If you kept it at 2 different cloud providers (or 2 physically separated regions of the same cloud provider), you would have been fine.
The cloud is as safe as you want it to be, but don't assume that storing data in a single cloud provider's facility is any safer than storing it at your own facility unless you're paying to have your data replicated somewhere else.
Any datacenter manager that doesn't already have fuel logistics in their disaster plan is in the wrong line of work.
I don't think any data center manager had a line item in the disaster recovery plan that included having all transportation access cut to the entire island due to flooding of the tunnel and closure of the bridges, for over a week. Everyone is having a problem getting fuel into the city; even mission-critical services like emergency services, hospitals, and telecommunications facilities.
What kind of disaster plan is it that doesn't account for a likely disaster? I can guarantee that every sizable datacenter had exactly this scenario in the DR plans. You don't run a $10M facility in a coastal city on an island without including flooding in your DR plan. This was a 100 year event, so it was definitely on everyone's horizon. Having it in the plan doesn't mean that you have a good solution - living in SF means Earthquakes are a big part of our DR plans, but we have no expectation that our facility will survive a 7.5 earthquake intact, or even that our employees will be motivated to come to work when they are worried about their own survival.
As to your comment that "suburban and rural datacenters have the space", sure... but where's the fiber optic cable hookups and the telecommunications infrastructure located? I'll give you a hint: Not in a barn. Those data centers are located downtown because that's where everything else is.
Just follow the train lines to find out where the major telecommunication lines are -- I have access to more carriers down on the Peninsula outside of San Francisco than I do downtown. You may be surprised at how much bandwidth runs through Colorado and even Missouri.
Not only that, but most of the data centers on the island are there because that's where Wall St. is, and milliseconds matter when it comes to high volume trading and financial transactions.
Sure, latency is a reason to be close to NYC, but I don't think any of the exchanges even have datacenters in the city anymore,they are all across the river. I know NASDAQ has a backup facility in Virginia.
Commercial real estate is at a premium in New York. Actually, all real estate is, leading to the old joke that when a New Yorker hears someone has died, the first question they ask is, "Is their apartment for rent?"
I'm not sure what your point is? Datacenters have to be built in the city because that's where the carriers are, but commercial real estate is expensive so don't build your datacenter in the city?
I think it's more likely to assume you've made an error in your reasoning, writing opinions from the comfort of an armchair, than people being paid over six figures who's job depends on balancing everything out exactly and to the nearest penny an hour.
You're obviously not in NYC if you think "over six figures" means someone is highly paid. Part of my job is planning our IT DR strategy. Fortunately, our Facilities dept is in charge of the generator so I don't need to worry about fuel contracts or keeping it running, but I do need to make sure our data is safe no matter what happens to the building and that we can continue to operate as a business. The only "disaster" that we plan on riding out on the building generator is a localized power outage when we know we'll be able to get fuel once our 3 - 5 days of fuel runs out. If there's a widespread power outage or disaster, our plan is to transition to the remote site since we know we may not be able to keep the generator fueled.
The last datacenter that I colocated in had 2 weeks of fuel on-site, and had another week of fuel in a trailer that can be trucked in from their other facility 60 miles away if the roads are passable.
How often do they rotate/burn through that supply?
As has been mentioned in a few articles that I've read, diesel fuel doesn't last forever, and so if you tried to use it after it's been sitting too long you may have problems. Hopefully as part of regular maintenance / testing they burn through it of the course of a year and replenish it with fresh stuff.
I don't remember the details on their rotation schedule, but I imagine that having their own fuel trailers made it easier to swap out fuel with some other site that needed it. Maybe they sold 9 month old fuel to an industrial site at a discount.
I do remember that they had their generator vendor bring in a load bank large enough to let them do full-load testing every month. And once a quarter they did a live cutover from utility power to generator. This seemed a little risky at first, but it makes sense - would you rather find out during a disaster that some component of the emergency power system wasn't working because it was not adequately tested, or would you rather find out when the generator vendor and electricians are standing by and utility power is available to fail back from the generator.
That facility weathered several power outages without a glitch, but the longest I can remember was around 48 hours.
Well, given that folks are already drawing firearms in gas lines, http://www.cbsnews.com/8301-201_162-57544187/new-yorker-sean-bailey-accused-of-pulling-gun-in-gas-line/ , part of the plan had better be to contract Mad Max to bring in a tanker of precious juice past The Humungous, Wez, and their pals.
Well the good news is that few drivers in the USA can use the diesel that's needed to feed datacenter generators in their cars, so hopefully the trucks will get through.
As little as I like agreeing with ACs, this was my exact thought upon reading TFS. I know this is News For Nerds, but let's not pretend getting data centers back up is more important than rebuilding an area that's been severely decimated (and not in the Roman 10% way).
Take a look at this:
https://www.google.com/search?tbm=nws&sclient=psy-ab&q=+new+york+city+recovery+hurricane+sandy&oq=+new+york+city+recovery+hurricane+sandy
Over 200,000 articles and 3600 news sources covering NYC's recovery efforts. Surely there's enough space left on the Internet for a News for Nerds site to cover news for nerds?
Dozens are dead, billions of dollars of property destroyed and businesses decimated, millions still in darkness and cold, and you are talking about data centers??
Data centers cannot feed my family or heat my cold, flooded house.
Where is your decency sir? It's a long week and weekend for the millions affected and the first responders. The glibness of this post is shocking and disgusting.
This is Slashdot, there are plenty of other sites that cover human misery and suffering, this site is for geek news, and geeks care about datacenters.
Datacenters can't feed your family or heat your cold, flooded house (unless you live next to a datacenter that recovers waste heat for residential heating), but the datacenter can help your utility get power back online, it can help your local merchant process credit card transactions and use their POS system so you can buy replacement goods, it can let you post your family's status so your mom can stop worrying about you, it can help your bank conduct online transactions efficiently to let you receive your insurance money, and of course, you're using multiple datacenters right now to read and post to Slashdot.
If you think datacenters are unimportant, try going a week without using any good or service that wasn't produced or delivered to you without the aid of datacenters.
In the past few days, data center managers have been forced to add fuel logistics to their list of responsibilities, as most Manhattan data centers have been subsisting on generator power.
Any datacenter manager that doesn't already have fuel logistics in their disaster plan is in the wrong line of work. Few inner city datacenters have a week or more of fuel on-site - most have only days of fuel, and they count on fuel contracts from suppliers to keep them running. And the supplier may not be able to honor the contract in a disaster.
Suburban and rural datacenters have the space (and less conflict with fire codes since the fuel is not stored in or near an office building) to keep weeks of fuel on hand. The last datacenter that I colocated in had 2 weeks of fuel on-site, and had another week of fuel in a trailer that can be trucked in from their other facility 60 miles away if the roads are passable. They had a spare generator that can be trucked in from that other facility as well. (and this facility could send fuel and a generator to that facility if needed)
Anyone that's ever worked with touchscreens before knows that those things need frequent recalibration
Maybe in 1995 when touch screens were new technology.
My smartphone has no way to calibrate it because it's not necessary. I can believe that calibration-free touch screens are more expensive than cheap ones that need constant calibration.
I don't recall my bank's ATM screen ever being out of calibration either. Maybe it was calibrated at install time, but it doesn't seem to slip out of calibration.
Which ignores the point that regardless of the touch screen calibration, it's trivial to design a UI that gives immediate feedback that the correct selection was picked.
Aside from a simple check mark next to the chosen candidate's name, how about inverting the color of the entire name field making it clear which one has been chosen?
I've heard this before, but I've never seen the reasoning - why do the generators need full overhauls after running for a few days? Aren't these the same engines that are in trucks/construction equipment and run for thousands of overalls without an overhaul?
When i worked at some remote construction sites, they had 1MW+ generators that went for over a year of nearly continuous use between overhauls. As I recall, they did oil changes and other routine maintenance monthly.
An emergency power generator might be run at a higher load than a continuous duty generator, but I can't imagine any large diesel generator needing an overhaul after a few hundred hours of operation.
A major disaster is the time when you really need your hospital to function properly. So spending some money on disaster preparedness is prudent.
Yes, we keep large amounts of emergency supplies of consumables on site : medical gases, medications, food, linens, antiseptics, needles, lines, etc.
Telling injured hurricane victims that show up in the E.D. "Sorry, dude, but this storm really screwed us up, man. We can't treat you." isn't acceptable.
Sure, I understand that a hospital needs to function for some time during/after a disaster, but 45 days of supplies seems like expensive overkill - do you have millions of gallons of water on site and your own treatment plant on-site since it's not likely that your municipal plant will withstand a 30+ day power outage.
What hospital is this?
I just googled to source the numbers to get a ballpark figure. So yeah, undoubtedly they're off -- people spend months preparing reports for their data center on fuel consumption, storage costs, location, etc. I spent 20 minutes. But putting the numbers together shows that even if you lowball all the numbers by 50%, storage cost is still massively eclipsing fuel cost.
Sure, I understand the fuel usage numbers were an estimate, but your calculations weren't off by 50% - you were claiming that it would take an entire 264x264 foot office floor (70,000 square feet) to store a week's worth of fuel. I was pointing out that it only takes about 300 square feet - you were off by a factor of 233, not by 50%. Instead of an unaffordable $83M in rent and an entire office floor to store a week's worth of fuel, it only takes an 18 foot by 18 foot room and $400K/year. That's a huge difference.
Your key mistakes were in dividing cubic inches by 12 instead of by 12*12*12 (or 1728) to convert from in^3 to ft^3, and again where you assumed that 67,000 ft^3 of fuel would need 69,000 square feet of office space to store, ignoring the 12 foot ceilings that you noted earlier.
If you did devote an entire 70,000 square foot 10 foot high office floor to fuel storage, you'd have 700,000 ft^3 of fuel storage, or 5,000,000 gallons, enough to power the 2MW generator for 1400 days. Of course, it would weigh 35M lbs, so the building would need to be purpose built (or retrofitted) to support the weight.
"The only time you can justify spending that kind of cash is if you're supporting critical infrastructure like phones, hospitals, and emergency services."
Which is why I can't understand why some hospitals in New York City had to evacuate due to power loss. I heard that one hospital evacuated the goddamn NICU.
...
Whoever was responsible for disaster planning for these hospitals ought to be facing criminal charges.
Hospitals, even publicly funded ones, don't have unlimited funds to implement disaster recovery plans. If it comes down to reducing services during the 99.9% of the time when there is no disaster versus spending money to ride out the 0.1% of the time when there's a major disaster, spending the money to help more people now isn't necessarily worse than planning to evacuate during an uncommon disaster.
I'd question the need for a hospital to have 45 days of fuel on-site... keeping the lights on does little good when you run out of drugs, antiseptics, fresh water, bandages, oxygen, and all of the other supplies you need to keep the facility running.
Do you keep a 45 day supply of all of the rest of your consumables on-site? My only experience is with a small rural/regional hospital and they had 3 days of fuel on-site, and 4 -7 days of most consumables.
If you only need 1MW of power, that means you have 80,000 gallons of fuel in storage - at 7mpg that means your delivery trucks are traveling over 500,000 miles every 6 months. What kind of deliveries are you making that puts a million miles on your fleet each year?
I would have thought it was obvious, but the middle of the city is where the telecommunications infrastructure is. It doesn't exist in a barn a hundred miles from any major city. And fuel has a shelf life. Diesel will slowly oxidize over time, and so the time you can keep it in the tank is about 12 months. You'll burn about 72 gallons an hour per megawatt (as a rough average). So a 2 megawatt data center will need about 3,500 gallons of diesel per day. A gallon takes up 231 cubic inches of space, so a single day's worth of fuel would need a tank with a capacity of 67,375 cubic feet. The average height of a floor in a skyscraper is 12.5 feet. In New York city, the average city block is 264 feet. That means that even if you filled an entire floor of a skyscraper with nothing but diesel fuel, you'd still get less than a week's worth of fuel guys. At a rate of perhaps $100 per month per square foot... you're talking about $83 million a year for a floor of an outlying area just to store that diesel fuel. Mind you, near Wall St., that price is probably going to be double or triple. The price of fuel is peanuts compared to this; 7 days of fuel for a 2MW plant would cost you only $94,000, plus transport costs.
While I agree with your point in general, your math is off. I can see our own 1MW generator and 3-day fuel tank from my office, which is no where near the size of tank you calculated -- it should consume around 1/4 of a city block using your figures.
A week of fuel (24,000 gallons) is 3200 cubic feet, with a 10 foot high tank, that's only 320 square feet, at $100/month that's still a pricey $384,000/year. But since that 2MW of power is enough to power 1000 servers (assuming 1000 watts/server, half the power goes to cooling), that's only $384 per server per year or $32/month per server.
Each server rack consumes around 6 square feet of space (allowing room in front and behind the rack), so that 6 square feet would cost $7200/year, or $171/server/year if you put 42 servers in a rack. ($14/server/month). So spending $32/month to provide a week of backup power isn't that out of line if you really want a server close to your office.
But $100 sounds pretty high for unfinished office space in Manhattan even in a class-A building. If you put your datacenter into a cheaper class-B or class-C building - rent would be closer to $40/sq ft.
Exactly. This is why your monday-quarterback "have a disaster recovery site 1000 miles away" is just as totally frigging useless.
You've left straw all over the place. Sweep up and sit down.
Care to elaborate? How is geographically diversity not significantly better than no geographical diversity?
If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York. Now, of course, you generally want a backup site that is more than 100 miles away, in the event of actual disasters, but you definitely want your primary facilities as close to your users as you can get.
Sure, there will always be datacenters in NYC, but that doesn't change the fact that instead of putting all your money into trying to build a datacenter impervious to all hazards, you're better off having a second site far from your primary site. Real-world constraints mean that you can't build that perfectly impervious datacenter when you're subject to real estate prices, building codes, fire codes, and Murphy's Law -- there will always be a disaster that the facility can't handle.
Everyone already knew that the on-site fuel supply is the limiting factor of power availability in a disaster. Even fuel delivery contracts mean nothing in a disaster or wide-spread outages - hospitals, EMS and other government services will trump the fuel delivery contract, if a hospital needs fuel, they are going to get the fuel that's been "guaranteed" for your datacenter.
There's no reason to spend big $$ creating a flood proof, earthquake proof, tornado proof, airplane crash proof datacenter in the middle of a city when you can have a disaster recovery site 1000 miles away that's not subject to the same type of disaster. (except maybe an asteroid strike, but there are few datacenters on the moon). No matter how disaster-proof you make your datacenter, mother nature (or man) will always find a way to create a disaster you didn't plan for -- even if that "disaster" is a typo in a router configuration file that takes down the network, or a contractor accidentally shorting out the emergency power cutoff switch wiring when bolting a rack to the wall.
Can't think of a better pen than the Pilot V5
I'll second that - the Palm Pilot V was great for taking notes. Once you learn Graffiti.
If the whole purpose of the backup system is to provide electrical power, then you'd think they'd consider the power
sources needed for it.
We generally assume that the purpose of a backup system is to actually operate when the main system fails.
Providing assurance that it will is an engineering job that requires enumerating the possible failure modes and
providing contingencies for them. The customer should presumably have that assessment done, then decide which
of the contingencies that they'll pay for (hopefully based on some cost/benefit/likelihood balancing strategy).
In this case, either they didn't do the right assessment, or else they didn't think this mode of failure was likely.
Either way, you hope they learn something.
It's not that they didn't account for having power to the fuel pump, they didn't account for having the basement, the fuel tank, fuel pump and associated electrical panels to be submerged in water.
After a certain point, you stop designing redundancy and resilience into your current facility and use the money to build a completely separate, geographically distant facility. When building in an existing office building, it may not be possible to design in basic safeguards that you'd have in a purpose built datacenter. In a suburban datacenter, the backup generators and fuel tanks will be outside on skids, raised several feet above ground level - in a city office building, you rarely have that luxury and local building and fire codes further restrict what you can do.
The biggest takeaway is to not count on a single datacenter since there's always some disaster that can take it down.
If we can figure out how to put a pump into a well full of water deep underground, it should not be too difficult to figure out how to do something similar with a fuel pump, relatively speaking. To have not done so is a fail.
There's a lot that *could* be done, but usually the price tag precludes *everything* from being done. If the electric panel that feeds your underwater pump is underwater itself, then the pump isn't going to do any good.
The tanks are required to be below ground by the fire code. I think the better question is why put big data centers on a low lying coastline/island and/or a city with a giant target painted on it by every anti-american, anti-establishment, anti-whatever whacko the world over. Data centers belong in places with low risk of natural disaster, war, terrorist attack, riot or really anything that brings the police out.
Because some companies want to be physically close to their datacenter. Virtualization and cloud computing is changing this... slowly.
Since when has Diesel been highly flammable? You can actually use it to put fires out. It takes quite some heat to get it going so really poses less risk than you average stationary cupboard.
Until it leaks out, saturates wood framing and other building materials, then comes into contact with an ignition source (like a pilot light, or candles that the residents are using since the power is out).
Then this hard to ignite fuel quickly turns an office into an inferno.
I can knock a candle over in my stationary cupboard and as long as I pick it up quickly, I wouldn't expect a fire. Soak that same cabinet and knock a candle over and it's a different story. Kerosene (very similar to diesel) makes a good fire starter to help get a wood fire started.
As somebody else already said, human power. There's also the option of tapping into the generator. I don't know the specifics, but if it takes less than a gallon of fuel to lift a gallon of fuel up the side of the building, you could just tap into the generator you're trying to refuel. The other option is to have a second smaller generator on the ground that powers just the lift/winch. This could be easily refueled. Seems like bucket brigade was the first thing that came to mind, and once they had that going, people stopped thinking of better ideas.
Or maybe they've seen enough Bugs Bunny cartoons to know what happens when a makeshift winch fails - whatever it's carrying falls to the ground in a big splat, flattening whoever it falls onto.
Hoisting a 30 pound gas can 200 feet in the air during a disaster is not the time for amateurs to jury rig something together.