Slashdot Mirror


Facebook Suffers Actual Cloud In Oregon Datacenter

An anonymous reader writes "The Register carries the funniest, most topical IT story of the year: 'Facebook's first data center ran into problems of a distinctly ironic nature when a literal cloud formed in the IT room and started to rain on servers. Though Facebook has previously hinted at this via references to a 'humidity event' within its first data center in Prineville, Oregon, the social network's infrastructure king Jay Parikh told The Reg on Thursday that, for a few minutes in Summer, 2011, Facebook's data center contained two clouds: one powered the social network, the other poured water on it.'"

13 of 83 comments (clear)

  1. Obligitory by Anonymous Coward · · Score: 5, Insightful

    And nothing of value was lost.

    1. Re:Obligitory by Anonymous Coward · · Score: 5, Funny

      And nothing of value was lost.

      Various US intelligence agencies and their chums in other countries beg to differ.

  2. Where are the Pics? by anthony_greer · · Score: 5, Insightful

    I dont se any pics in the linked article, Someone has to have pictures of this if it happened...

    1. Re:Where are the Pics? by Wild+Wizard · · Score: 3, Informative

      This is one of those RFA to get to the RA type stories.

      The link next to the quote is the one you want :-
      http://www.opencompute.org/2011/11/17/learning-lessons-at-the-prineville-data-center/

    2. Re:Where are the Pics? by anthony_greer · · Score: 5, Insightful

      saw that and I think the issue is that the sudden humidity change caused condensation, not terribly uncommon if prompt action isnt taken upon AC failure in a humid climate...I don't see a "Cloud in the room"

      Hype to sell newspapers...and link bait...

  3. Obligatory by identity0 · · Score: 5, Funny

    Welcome to Oregon, it rains a lot.

    1. Re:Obligatory by Ol+Biscuitbarrel · · Score: 4, Informative

      Sure, if you think 10.4 inches yearly average is a lot. East side of the state's actually quite arid; the west side is quite soggy in the Coast Range and seaside but the Willamette Valley where most of the population lives isn't exceptionally rainy, it's that it's subject to never-ending spells of overcast weather; other parts of the country actually have higher annual precipitation.

    2. Re:Obligatory by Anonymous Coward · · Score: 3, Informative

      WTF 80 deg F (approx 27 deg C) is quite warm in a Data-centre especially in a "cold aisle" and 95% humidity is criminal.

      You're used to classic datacentres, where the goal was "shove as much cold air into them as possible", i.e. "the lower the temperature the better". It all depends on how the datacentre was built, how its cooling system is/was engineered, and an almost indefinite number of variables. References for you to read (not skim) -- the study in the PDF will probably interest you the most:

      http://www.datacenterknowledge.com/archives/2011/03/10/energy-efficiency-guide-data-center-temperature/
      http://www.geek.com/chips/googles-most-efficient-data-center-runs-at-95-degrees-1478473/
      http://blog.schneider-electric.com/datacenter/2013/05/06/getting-comfortable-with-elevated-data-center-temperatures/
      http://www.cs.toronto.edu/~nosayba/temperature_cam.pdf (PDF)
      http://www.dummies.com/how-to/content/data-center-temperature-and-humidity-range-recomme.html

      TL;DR -- 80F is not "quite warm" for a datacentre designed/built within the past 10-11 years.

  4. There are at least 3 clouds then. by VortexCortex · · Score: 3, Interesting

    The first cloud would be the humidity and condensation sort. The second cloud would be the online service itself. The third cloud, would be the open Internet between the endpoints in a network graph.

    What do all these clouds have in common? They're dangerous. The less clouds in your diagram the more you know about your network architecture, latency, and data integrity. The less clouds the better! When a packet goes into the shroud of the cloud in the diagram there's a much higher chance we'll never see it again. This cloud is the one where we must encrypt our data and protect against spoofing and hacking and all forms of data manipulation and latency. The receiving end must be very careful to sanitize the inputs and verify the requests vigorously all because the packet has encountered the cloud. Likewise if we want to interact with an online "cloud" service, we shift the name packet to "our stuff" our login credentials and even bank account info, we have to worry about availability and bandwidth caps when streaming, and unwanted prying eyes from folks we may not desire to have looking, everything becomes far more risky because our stuff touched the cloud service; Far more risky than physically going to the bank or visiting a friend in person would not be subject to. If someone hacks the ATM, the entire bank doesn't lose everyone's credentials. As for the mist filled variety of cloud: It can not only get wet, but if you have a big enough cloud, it can strike you with lightning. We must have surge protections and battery backups against this cloud too.

    When I hear people talking about embracing the "cloud" I cringe. "To The Cloud!", in my mind means, "Danger Will Robinson!"

  5. Re:As somebody working on building energy topics by myowntrueself · · Score: 3, Interesting

    That this happens shows me that they realy optimize their air conditioning for energy consumption.

    Traditionally the approach would have been: "Dont think, cool down and re-heat the air constantly to dehumidify it sufficiently". So traditionally you do this dumb with a lot of energy, even if its not needed at all times. What we probably see there is that some control could not (predict or) handle some drop in the inner load (electrial power) in the data center.

    The Facebook Oregon datacenter doesn't 'have' air conditioning.

    The building is an 'air conditioner'. Its an experimental design...

    --
    In the free world the media isn't government run; the government is media run.
  6. It's clouds by rastos1 · · Score: 4, Funny

    It's clouds all the way ... up?

  7. Load balancing by gmuslera · · Score: 4, Funny

    Both clouds were leaking and pissing off users. Facebook must have real sysadmins.