NSA's New Utah Data Center Suffering Meltdowns
linuxwrangler writes "NSA's new Utah data-center has been suffering numerous power-surges that have caused as much as $100,000 damage per event. The root cause is 'not yet sufficiently understood' but is suspected to relate to the site's 'inability to simultaneously run computers and keep them cool.' Frustrating the analysis and repair are 'incomplete information about the design of the electrical system' and the fact that "regular quality controls in design and construction were bypassed in an effort to fast track the Utah project."" Ars Technica has a short article, too, as does ITworld.
maybe this is okay.
I hope the whole place melts down or any type of natural disaster. These tools will be used against the evil and the good alike.
Hmm...so your equipment is randomly failing...you don't say?
Save yourself some trouble and stop spying on your own people then.
Ha Ha
Respect our right to privacy. Stop the snooping. I am upset and ashamed that the people of Utah have allowed this spy center to be built in their state. I had higher expectations of them. They should have cherished our individual liberties, upheld the Constitution, and told the Federal government take their spy center elsewhere.
It probably lacks certification from the Department of Redundancy Department anyway...
You set out to snoop on the minutiae of the lives of tens of millions of innocent people. Then your data center melts down... Perhaps God/Angels had something to do with this? =)
Why did the chicken cross the road? Because Elon Musk put an AI chip in its head.
The only thing that will save us from the massive dragnet of the NSA is apparently the incompetence of the NSA.
The submission had one article, the editors linked to two more.
ALL THREE ARTICLES REFERENCE & LINK TO THE WALL STREET JOURNAL
Is it so hard to include a link to the source of this story? /. does this far too often and I hope to see better in the future
http://online.wsj.com/article/SB10001424052702304441404579119490744478398.html
(Google Cache just in case
[Fuck Beta]
o0t!
They probably used a power budget similar to the public Facebook datacenter data but then decided to run their machines on Windows Azure.
I have noticed that power consumption of my computers is significantly higher when running Windows - and the laptops have seriously reduced battery life, even while doing nothing.
No sig. Move along - nothing to see here.
The more likely cause is too much porn overheating the CPUs.
Someone explain to me why the holy bloody fuck these enemies of the American people haven't taken an involuntary 8-day vacation along with the rest of the noncritical federal government?
afterall google revealed a good amount on how they go about building their data centers and keeping it cool. But then again, contractors...
------ Curiosity killed the cat. {satisfaction brought it back | it didn't die ignorant | lack of it is killing mankind
Secret budgets. No bid contracts. Endless graft. Pork.
The NSA is a money pit. Welfare for private business who perpetuate endless war for a buck.
You get what you give, you give the world shit, you get shit...
payback is a bitch...
If they fast-tracked the project, they probably didn't have an electrical engineer do a load analysis. I wouldn't be surprised they are tripping panel main breakers, but not individual load breakers. This tends to give a nice inductive kickback to the system, which can cascade into tripping superficially unrelated circuits.
"I have sworn upon the altar of God, eternal hostility against every form of tyranny over the mind of man.
-- some godless pinko and human trafficer
give me apathy or give me cheetos
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
. . .eats huge amounts of power, not large amounts of water for cooling.
And thus, power requirements go up, pushing the limits of your provisioned electrical infrastructure.
And extremely-high-capacity circuit breakers tend to be explody when they fail. My guess: someone used some REALLY bad assumptions for electrical infrastructure planning. . .
I, for one, definitely trust an outfit that can't size a bloody datacenter power distribution system to build those magic technical safeguards that are allegedly allowing a spying operation of unprecedented size to occur with no abuses (And that's no bullshit!)
I wonder if we could convince them to switch to a utility that conducts background checks on electrons before sending them to the customer? That would clearly help...
'incomplete information about the design of the electrical system'
Well, duh, it's secret.
"I believe in Karma. That means I can do bad things to people all day long and I assume they deserve it." : Dogbert
> Frustrating the analysis and repair are 'incomplete information about the design of the electrical system'
Why don't they just snoop the needed information off the contractor's network?
SSIA but form wouldn't accept a subject-only comment
Cry me a river, and all that.
Python: 'And then suddenly you have a language which says "we're all stuck with whatever the whiniest coder wants".'
afterall google revealed a good amount on how they go about building their data centers and keeping it cool. But then again, contractors...
Rule #1 of government spending: why by one, when you can have two at twice the price?
Rule #2 of government spending: a penny saved is a spending oversight.
Just theorizing, but I'd venture to say:
1) Army Corp project and design
2) Lowest Bidders (if not for the prime, at least for the sub contractors)
3) Multiple Change Orders
4) Multiple on the Fly revisions
5) GOTO 1 Repeat till deadline.
fuck off and die, and btw, god hates you, study the equations of electromagnetism (the Dirac equation among others) and you'l see for yourself if your brain is any larger than an ant
HA HA
The Blowback!
"Flyin' in just a sweet place,
Never been known to fail..."
No the movie, but the folk legend kind http://en.wikipedia.org/wiki/Gremlin.
Or could it be a computer bug? http://en.wikipedia.org/wiki/File:H96566k.jpg
This isn't the result of incompetence - rather the result of trying to racing to finish the thing before any more opposition builds up that may stop the project. Wile E. Coyote trying to run on air, knowing it's impossible, but trying to get to the cliff before gravity notices the flagrant violation.
When that monster is done - and it seems that they are turning it on *right now*, this week - human history is done as we understand it. We will all behave as though someone is watching and recording us, because they will be.
Scientology is going to *love* this - one stop shopping for all its spying needs. The NSA just last week asked permission for private corporations to access their new trove of data, Because Terrorism. The Unification Church and Scientology will be first in line with front corporations to drink deep of this wonderful new integrated terrorism enabling center - terrorism because bad guys like Scientology will be able to terrorize people with fresh, holistic super-knowledge not only of who they are, what they say, what they read and where they've been, but also of everyone their enemies ever talk to, email, walk next to, text or write to. That center isn't about just metadata, it's the *actual phone conversations* that will be recorded. Don't ever piss off the powerful, 'cause they can nail you and anyone who ever contacts you until you give up. Blackmail, extortion, we-know-where-you-kids-are... anything. And the coolest part is that it will all be secret! Persecutors with behind stage access to the NSA superboxes and analytic tools won't even be logged in any real sense. Political opposition, nullified, instantly. The possibilities for our brave new world owners are limitless.
Maybe they shouldn't have built it in Utah?
More likely they didn't account in the power budget for the seven secret sub-basements and the underground vacutrain for the reptoids to commute from the Denver International Airport.
honestly, it's like the right hand doesn't know what the left talon is doing these days.
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
My guess: someone used some REALLY bad assumptions for electrical infrastructure planning. . .
Hey, don't be too hard on the electrical engineers - James Clapper told them that the power requirements would be really low.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
Enjoy the serenade, NSA
An enigma, wrapped in a riddle, shrouded in bacon and cheese
People with even one brain cell functioning know the biggest problem in the USA is how government projects have to be spread across EVERY State, no matter how inefficient of inappropriate this is. So, in a nation that spreads from some of the hottest to some of the coldest places in that part of the world, you build your data centres in the hottest place, and then complain when cooling them is an issue. Well let me tell you straight- the rest of the world applauds such American stupidity.
As the USA becomes a vile, evil, genocidal cancer spreading across my planet, the ONLY thing us decent Humans assume can give Humanity any hope for the future is the notorious inherent corruption present in the 'UNITED' STATES. The full surveillance programs of NSA are simply another means to growing the US war-machine unchecked, until the time that war machine is fully unleashed across the globe. The real rulers of the USA do NOT know exactly how they will start their World War, but they do know that so long as the US war machine grows massively every year, and the US sheeple are are converted into worshippers of this war-machine, and aggressive war itself, at some point they WILL find the straw that breaks the camel's back.
So we decent people take small comfort at the missteps like this Utah story, or the revelations of Bill Gates' work with the NSA found in the Xbox One, or the failure of Obama to holocaust hundreds of thousands of Humans in Syria, but we do no doubt that one day the monsters will have their last straw.
This is what happens when you go with the lowest bidder.
...the NSA tries to tail -f the [censored] planet!
; )
Uh, Linux geek since 1999.
This sounds like someone was in such a great hurry to get their shiny new toy that they bypassed a lot of the steps they should have followed.
And, somehow I doubt there's a lot of sympathy for the NSA here on Slashdot.
Lost at C:>. Found at C.
From: "The Edge Of Darkness" http://www.imdb.com/title/tt0090424
"Craven: The word azure is a police intelligence term. It means the room is bugged or under some sort of electronic surveillance"
A perfect name for a cloud computing product.
Here's hoping it stays down until we can reign in the NSA.
Works for Switch in Las Vegas. Cold in winter and cools off at night so 70% of annual hours they can pull in ambient air through filters. Evaporative cooling, whether direct or to cool the hot-side of a refrigerated system, works best in dry climates but it's only used to improve efficiency as they can run fine with air-cooling albeit at much higher power costs.
I'm still surprised at the number of places that think cooling is optional. We had equipment in a Sacramento data-center that had plenty of backup electricity for servers but couldn't run the AC in a power outage. The SLA only had provisions for exceeding 80-degrees for more than something like 90 or 120 minutes. *Ahem*, cold-comfort when a dense data-center can blow through 100 in minutes without AC.
UC Berkeley had a widespread power outage about a week ago. The main campus data center had power but, you guessed it, couldn't run cooling and had to "gracefully" shut down most of the core systems while watching the center breach 100F.
But I agree with your base assumption - really bad planning and/or execution on the power systems.
~~~~~~~
"You are not remembered for doing what is expected of you." - Atul Chitnis
Seems NSA has also pissed off some UFOs and they are doing their magic. Thanks!
to a nicer data center...
is probably the main cause
I'm sure it won't last, nothing this good ever does, but let's enjoy it while we can.
This Space Intentionally Left Blank
I'd laugh except I bought it.
"Just get it done" strikes again.
Typical incompetent Government Contractors and Engineers or Administration.
IF you cant keep them cool then the guy that designed the HVAC is a moron. It is not hard to calculate the required heat generating load unless some moron NSA higher up changed the number of systems the place was supposed to hold after construction started. Then it's that guys fault and he needs to be waterboarded for 30 days just to teach him a lesson.
Honestly a data center is not rocket science anymore, it is so well documented in what is needed that even a 1st year engineering student can easily design a proper one with proper HVAC needs.
Do not look at laser with remaining good eye.
Couldn't happen to nicer people.
Prove anything by multiplying Huge Number times Tiny Number
No, knowing govt projects. Nothing labelled and changes made on the fly that are not documented. One building they had 48 breakers and NONE labeled.
Do not look at laser with remaining good eye.
so we now have an official price tag on your stolen personal information.
by TheSpoom (715771) Uncaring Linux user here. I have nothing to add to this but please continue. *munches popcorn*
I don't think you can plan everything to the fourth decimal. At some point, design gives way to monitoring - you need to measure environmentals, be able to correlate it to load, and have an emergency plan if you're going out of bounds. There is simply no excuse for running equipment until it physically fails.
So maybe the universe does have a sense of humor - the NSA failed miserably because they didn't put enough importance on monitoring.
Spell check - there's an app for that.
A friend of mine is researching power surges in the local town.
Most building codes under-specify the gauge of the neutral/return wire. For illustration, if you have three phases each rated for N amps, there is one shared neutral/return wire rated at N amps going out. At the end of the runs all phases are connected to the shared neutral line.
This is due to the nature of 3-phase electricity: the phases will tend to cancel out, so in a perfect setup you would need no neutral/return at all. Of course, the load on each phase won't exactly balance, and the load can vary as people connect/disconnect appliances, so you still need the neutral line in practice.
(Not true for house wiring, which has one or two phases coming in. Each phase has a return with the same gauge as the supply.)
This was fine when appliances were (generally) resistive loads, but nowadays switch-mode power supplies are common. When you do some math, it turns out that this type of load appears equivalent to 120 Hz power coming together at the neutral/return junction. Since 120 Hz [equivalent] power does not cancel out, the power in the return wire can be 3x as large as the building codes allow.
I've got a book explaining all this. Typically the neutral line will heat up and catch fire, breaking the circuit. Once that happens the various phases are connected without a neutral, playing hob with whatever is on those circuits and making occasional high-power ground loops and other unexpected behaviours.
They might be catching on to my sabotage to their facility. Nothing to see here, move along. Not short circuiting anything. It's not like if you supply 480V to a hard drive directly and you won't have any problems. Also, is it just me or does one hundred thousand feet of data center space seem pretty small to hold 5 zettabytes in data storage. I tried to build the same out of BackBlaze storage pods and came up with 180TB of space in a 4U pod, with 1.8PB per rack, that would still take a lot more than they are showing. I suppose they could be building into the ground, but wouldn't that show up on their calculations for square feet?
Place something witty here
HA...HA (ala Nelson)
On one hand the NSA is publicly embarrassing itself with costly malfunctions On the other hand, it's all happening on my dime
Did they check for jellyfish in the intakes?
Chaos maximizes locally around me.
After being stuffed with all the information in the world, the network is starting to gain self-consciousness.
Isn't it terrible when bad things happen to absolute fascists?
It is a sad sad day when one of the three giant NSA centers is down, due to somebody forgetting to do checksums on the loaded programs in their power systems.
Sad sad sad.
Yes, I said three.
-- Tigger warning: This post may contain tiggers! --
There Ain't No Such Thing As A Free Lurk -- NSA
The land was cheap
So they did it fast, but not right, and now they are paying more than they would have to do it properly in the first place?
This is an old story. You know what they say about those who do not learn from history, right?
So if you (NSA/whomever) haven't learned yet, then it is unlikely you will start learning now. I'm going to learn from history and suggest that you aren't going to learn this time either. And maybe you can learn from my learning, and show yourself better than a zero-order player of games.
I'm surprised cold fjord isn't all over this, given the anti-NSA sentiment expressed.
Or is he not working because of the government shutdown?
Wasn't there some discussion about how the NSA couldn't really store "zettabytes" of data at this site as claimed?
Maybe they really do have all that capacity. Them transcapacitors are power-hungry, I hear.
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
Wonder if their servers were infected with one of those bitcoin mining botnets ?
Would you like to cover a 'news' event yourself?
What happens when big brother says you can't because you don't have the proper credentials?
Check out some New York City rules in reverse order:
Applicants also must submit one or more articles, commentaries, books, photographs, videos, films or audios published or broadcast within the twenty–four (24) months immediately preceding the Press Card application, sufficient to show that the applicant covered in person six (6) or more events occurring on separate days .
Applicants must be a member of the media who covers, in person, emergency, spot or breaking news events and/or public events of a non-emergency nature, where police, fire lines or other restrictions, limitations, or barriers established by the City of New York have been set up for security or crowd control purposes, within the City of New York; or covers, in person, events sponsored by the City of New York which are open to members of the press.
First-time applicants should contact the Press Credentials office (above) before completing their application
No brain, no pain.
Why did they build a computer room in Utah. Alaska would be a much better place. Cheaper cooling.
There must be some clear objective to fast-track such a massive installation to skip certain of the build quality checks. This must exist under the guise of some approach not previously discussed since this is a new datacentre.
Have gnu, will travel.
Quite. The basic concept of conservation of energy seems to have passed a lot of people by.
Quite. The basic concept of conservation of energy seems to have passed a lot of people by.
Most basic concepts pass most people by, which is why life is so hard for the rest of us and the world is in the lovely state that it's in.
I reckon some subversive put some red pinko-commie electrons in the power circuits to undermine the NSA.
Stick Men
raped in the ass again!!! Taxpayers rejoice!
Oh, don't mind that. Echelon and Helios are just attaining sentience.
The problem is assuming that it could ever be seen as sarcasm even by those so well equipped. This is due to how often these arguments are made seriously far outnumbering the times they are made with sarcasm. Notice it's tagged "insightful" and not funny. That is the state of the world we find ourselves in with regard to gun ownership and the debate surrounding thereof I'm sorry to say, even here on /. Other topics where I'd avoid sarcasm would include abortion, global climate change and evolution. I know, I'm such a killjoy, but there are other paths to humor. I challenge you to find them.
It's probably Thor expressing his disapproval of the NSA.
http://en.wikipedia.org/wiki/District_of_Columbia_v._Heller
http://en.wikipedia.org/wiki/McDonald_v._Chicago
We definitely need anti-monopoly laws on our media-the news and the pseudo-news kind. I almost want to say that no person or group should own more than 5% or something like that.
Just wait for next week story: "The NSA can't fucking bring up their servers, whole datacenter crashes in a chaotic storm of network packets"
Panelists will include Philip Zimmerman (PGP + Silent Circle), Nico Sell (Wickr), and Casey Openheim (Disconnect). The question being asked is, is privacy possible today?
The web site for the event is : http://www.vlab.org/article.html?aid=480
Naturally, the panel will look at technologies like data mining (and anti-data mining) and encryption. VLAB is interested in business opportunities for startups with disruptive technologies, the panel will be seeing what's possible.
VLAB usually looks for major disruptive trends, events, and technologies. The Snowden leaks (and also the Google, Facebook, etc. reactions) may have started something significant, and sparked an awareness.
In any case, it should be an interesting event. If you're in the area, you might want to check it out. (full disclosure - I'm the event chair for the privacy event).
Best Regards,
Tony
Sounds like a good opportunity to try those wine powered processors.