Facebook Suffers Actual Cloud In Oregon Datacenter

An anonymous reader writes "The Register carries the funniest, most topical IT story of the year: 'Facebook's first data center ran into problems of a distinctly ironic nature when a literal cloud formed in the IT room and started to rain on servers. Though Facebook has previously hinted at this via references to a 'humidity event' within its first data center in Prineville, Oregon, the social network's infrastructure king Jay Parikh told The Reg on Thursday that, for a few minutes in Summer, 2011, Facebook's data center contained two clouds: one powered the social network, the other poured water on it.'"

Comments

Obligitory

[Anonymous Coward]

And nothing of value was lost.

Re:Obligitory

[Anonymous Coward]

And nothing of value was lost.

Various US intelligence agencies and their chums in other countries beg to differ.

Where are the Pics?

[anthony_greer]

I dont se any pics in the linked article, Someone has to have pictures of this if it happened...

Re:Where are the Pics?

[Wild Wizard]

This is one of those RFA to get to the RA type stories.

The link next to the quote is the one you want :-
http://www.opencompute.org/2011/11/17/learning-lessons-at-the-prineville-data-center/

Re:Where are the Pics?

[anthony_greer]

saw that and I think the issue is that the sudden humidity change caused condensation, not terribly uncommon if prompt action isnt taken upon AC failure in a humid climate...I don't see a "Cloud in the room"

Hype to sell newspapers...and link bait...

Re:Where are the Pics?

[aXis100]

That is one of the funniest things I have read for ages!

Power supplies are dealing with 700+ watts at maybe 90% efficiency, and you describe them as "pretty cool"? Bollocks! Capacitors and Inductors have current flowing and series equivalent resistance, which certainly generates heat. They *should* certainly be hotter than ambient, which is the most important factor in condensation.

If I saw condensation on powered electronics components I would be looking for the ladder to climb out of the pool.

Obligatory

[identity0]

Welcome to Oregon, it rains a lot.

Re:Obligatory

[Ol Biscuitbarrel]

Sure, if you think 10.4 inches yearly average is a lot. East side of the state's actually quite arid; the west side is quite soggy in the Coast Range and seaside but the Willamette Valley where most of the population lives isn't exceptionally rainy, it's that it's subject to never-ending spells of overcast weather; other parts of the country actually have higher annual precipitation.

Re:Obligatory

[donaldm]

Welcome to Oregon, it rains a lot.

From the Article

This resulted in cold aisle supply temperature exceeding 80F and relative humidity exceeding 95%. The Open Compute servers that are deployed within the data center reacted to these extreme changes. Numerous servers were rebooted and few were automatically shut down due to power supply unit failure.

WTF 80 deg F (approx 27 deg C) is quite warm in a Data-centre especially in a "cold aisle" and 95% humidity is criminal.

Facebook learned from the mistakes, and now designs its servers with a seal around their power supply, or as Parikh calls it, "a rubber raincoat."

When designing a Data-centre you have to plan for a certain temperature range that the equipment you have inside operates optimally. In addition you have to keep the humidity within manufacture recommended limits since too low results in static electricity and too high well you could get condensation on the electrical equipment. Rubber seals may protect power supplies although I don't think this will completely protect them, but what about the rest of the equipment such as the electronics, connectors and what about backup systems which are very susceptible to temperature and humidity just to name a few.

My score for this design is zero out of ten and ten out of ten for LOL, welcome to cloud computing :)

Re:Obligatory

[Anonymous Coward]

WTF 80 deg F (approx 27 deg C) is quite warm in a Data-centre especially in a "cold aisle" and 95% humidity is criminal.

You're used to classic datacentres, where the goal was "shove as much cold air into them as possible", i.e. "the lower the temperature the better". It all depends on how the datacentre was built, how its cooling system is/was engineered, and an almost indefinite number of variables. References for you to read (not skim) -- the study in the PDF will probably interest you the most:

http://www.datacenterknowledge.com/archives/2011/03/10/energy-efficiency-guide-data-center-temperature/
http://www.geek.com/chips/googles-most-efficient-data-center-runs-at-95-degrees-1478473/
http://blog.schneider-electric.com/datacenter/2013/05/06/getting-comfortable-with-elevated-data-center-temperatures/
http://www.cs.toronto.edu/~nosayba/temperature_cam.pdf (PDF)
http://www.dummies.com/how-to/content/data-center-temperature-and-humidity-range-recomme.html

TL;DR -- 80F is not "quite warm" for a datacentre designed/built within the past 10-11 years.

Maybe Berndnaut Smilde snuck in:

[Tablizer]

http://www.mnn.com/lifestyle/arts-culture/stories/artist-creates-beautiful-indoor-clouds

Streamed Hams

[Anonymous Coward]

Superintendant Chalmers: A rain storm? At this time of year? At this time of day? In this part of the country? Localized entirely within your datacentre?

happened before

[crutchy]

in nasa's vehicle assembly building

There are at least 3 clouds then.

[VortexCortex]

The first cloud would be the humidity and condensation sort. The second cloud would be the online service itself. The third cloud, would be the open Internet between the endpoints in a network graph.

What do all these clouds have in common? They're dangerous. The less clouds in your diagram the more you know about your network architecture, latency, and data integrity. The less clouds the better! When a packet goes into the shroud of the cloud in the diagram there's a much higher chance we'll never see it again. This cloud is the one where we must encrypt our data and protect against spoofing and hacking and all forms of data manipulation and latency. The receiving end must be very careful to sanitize the inputs and verify the requests vigorously all because the packet has encountered the cloud. Likewise if we want to interact with an online "cloud" service, we shift the name packet to "our stuff" our login credentials and even bank account info, we have to worry about availability and bandwidth caps when streaming, and unwanted prying eyes from folks we may not desire to have looking, everything becomes far more risky because our stuff touched the cloud service; Far more risky than physically going to the bank or visiting a friend in person would not be subject to. If someone hacks the ATM, the entire bank doesn't lose everyone's credentials. As for the mist filled variety of cloud: It can not only get wet, but if you have a big enough cloud, it can strike you with lightning. We must have surge protections and battery backups against this cloud too.

When I hear people talking about embracing the "cloud" I cringe. "To The Cloud!", in my mind means, "Danger Will Robinson!"

Re:As somebody working on building energy topics

[myowntrueself]

That this happens shows me that they realy optimize their air conditioning for energy consumption.

Traditionally the approach would have been: "Dont think, cool down and re-heat the air constantly to dehumidify it sufficiently". So traditionally you do this dumb with a lot of energy, even if its not needed at all times. What we probably see there is that some control could not (predict or) handle some drop in the inner load (electrial power) in the data center.

The Facebook Oregon datacenter doesn't 'have' air conditioning.

The building is an 'air conditioner'. Its an experimental design...

No, really, WTF?

[faedle]

Ok, I work for a data center operator. In Central Oregon.

Our data center is so damn dry that most of the time in the summer we're getting alerts about the humidity being too low. How did Facebook fuck this one up?

It's clouds

[rastos1]

It's clouds all the way ... up?

Load balancing

[gmuslera]

Both clouds were leaking and pissing off users. Facebook must have real sysadmins.

Re:As somebody working on building energy topics

[faedle]

Not just the people, but yes, static electricity is the primary concern. Also, I'm told by the people that manage these sorts of things that a "too dry" environment also makes air cooling less effective. Something to do with the fact that a little bit of moisture actually allows the air to carry more heat than if it was 100% dry.

Re:Long story in short ...

[Mr. Freeman]

No, you are (mostly) wrong.

What happened was that the system malfunctioned which led to hot and humid air being circulated throughout the system. This normally would not cause condensation. However, all of the equipment was previously cold (because the system was working normally before it failed). The hot and humid air came into contact with cold components (various components in the power supply and computer casing). This caused condensation (because the hot and humid air contacted the cold components, cooled down, and had to ditch some of its water due to thermodynamics). The condensation on components of the power supply was then blown into various PCB components when the maintenance staff increased the airflow in the datacenter in an attempt to bring the temperature down. The condensation hitting the PCB caused what you would expect when dumping water onto electronics: stuff shorted out and failed.

You're correct insofar as that humidity was not properly controlled, but this was hardly a case of idiots at the controls or at the drawing board. This was an unanticipated failure mode that was experienced because of a faulty control sequence being generated.