Slashdot Mirror

← Back to Stories (view on slashdot.org)

Facebook Suffers Actual Cloud In Oregon Datacenter

Posted by timothy on from the but-the-cloud-is-the-computer dept.

An anonymous reader writes "The Register carries the funniest, most topical IT story of the year: 'Facebook's first data center ran into problems of a distinctly ironic nature when a literal cloud formed in the IT room and started to rain on servers. Though Facebook has previously hinted at this via references to a 'humidity event' within its first data center in Prineville, Oregon, the social network's infrastructure king Jay Parikh told The Reg on Thursday that, for a few minutes in Summer, 2011, Facebook's data center contained two clouds: one powered the social network, the other poured water on it.'"

83 comments

  1. Obligitory by Anonymous Coward · · Score: 5, Insightful

    And nothing of value was lost.

    1. Re:Obligitory by K.+S.+Kyosuke · · Score: 1

      I'm sure that won't prevent Facebook from putting the people responsible behind a grid.

      --
      Ezekiel 23:20
    2. Re:Obligitory by Anonymous Coward · · Score: 5, Funny

      And nothing of value was lost.

      Various US intelligence agencies and their chums in other countries beg to differ.

    3. Re:Obligitory by geogob · · Score: 0

      I'm pretty sure those servers and power supplies were valuable, regardless of the data they host (which is also of value to someone).

    4. Re:Obligitory by Coren22 · · Score: 1

      Some insurance company will pay for them I am sure...and some AC tech might lose his job...

      --
      APK likes to ask for responses to the same things over and over. Maybe he just likes the responses?
    5. Re:Obligitory by Dr+Floppy · · Score: 1

      And nothing of value was lost.

      Oh how absolutely true

  2. Where are the Pics? by anthony_greer · · Score: 5, Insightful

    I dont se any pics in the linked article, Someone has to have pictures of this if it happened...

    1. Re:Where are the Pics? by Wild+Wizard · · Score: 3, Informative

      This is one of those RFA to get to the RA type stories.

      The link next to the quote is the one you want :-
      http://www.opencompute.org/2011/11/17/learning-lessons-at-the-prineville-data-center/

    2. Re:Where are the Pics? by anthony_greer · · Score: 5, Insightful

      saw that and I think the issue is that the sudden humidity change caused condensation, not terribly uncommon if prompt action isnt taken upon AC failure in a humid climate...I don't see a "Cloud in the room"

      Hype to sell newspapers...and link bait...

    3. Re:Where are the Pics? by Anonymous Coward · · Score: 0, Insightful

      Pics or GTFO

    4. Re:Where are the Pics? by Anonymous Coward · · Score: 0

      666 failure? Are you sure your looking the right direction?

    5. Re:Where are the Pics? by Tablizer · · Score: 0, Offtopic

      Here ya go:

      http://hansenent.net/sitebuilder/images/gray_square-354x242.jpg

      --
      Table-ized A.I.
    6. Re:Where are the Pics? by Anonymous Coward · · Score: 1

      Prineville is high desert and is very dry.

    7. Re:Where are the Pics? by Anonymous Coward · · Score: 1

      there are - those photos are just private

    8. Re:Where are the Pics? by sjames · · Score: 1

      Given that it was a condensing environment to the point that even the power supplies (typically hotter than the incoming air) got wet enough to short out, yes a cloud formed in the data center. There would have to have been actual droplets of water in the air.

    9. Re:Where are the Pics? by Anonymous Coward · · Score: 0

      Meanwhile back in reality (and the source article if you read it) you'll find no reference to a physical cloud forming. What happened was the same thing as a cold glass on a hot humid day. The water vapor (not droplets) condensed against cooler surfaces and PSUs are pretty cool components except where the various power transistors are located. In the photos you can clearly see condensation forming on the chassis, capacitors, and inductors, components which don't produce meaningful, if any, heat.

    10. Re:Where are the Pics? by sjames · · Score: 1

      Actually, those pics look a LOT like what things look like when you're actually inside a cloud. I'm guessing you haven't ever been.

    11. Re:Where are the Pics? by aXis100 · · Score: 2

      That is one of the funniest things I have read for ages!

      Power supplies are dealing with 700+ watts at maybe 90% efficiency, and you describe them as "pretty cool"? Bollocks! Capacitors and Inductors have current flowing and series equivalent resistance, which certainly generates heat. They *should* certainly be hotter than ambient, which is the most important factor in condensation.

      If I saw condensation on powered electronics components I would be looking for the ladder to climb out of the pool.

    12. Re:Where are the Pics? by antdude · · Score: 1

      Or better, videos. :)

      --
      Ant(Dude) @ Quality Foraged Links (AQFL.net) & The Ant Farm (antfarm.ma.cx / antfarm.home.dhs.org).
    13. Re:Where are the Pics? by Anonymous Coward · · Score: 0

      Someone call the doctor! Gramps woke up and is talking! It's a miracle!

  3. Obligatory by identity0 · · Score: 5, Funny

    Welcome to Oregon, it rains a lot.

    1. Re:Obligatory by Ol+Biscuitbarrel · · Score: 4, Informative

      Sure, if you think 10.4 inches yearly average is a lot. East side of the state's actually quite arid; the west side is quite soggy in the Coast Range and seaside but the Willamette Valley where most of the population lives isn't exceptionally rainy, it's that it's subject to never-ending spells of overcast weather; other parts of the country actually have higher annual precipitation.

    2. Re:Obligatory by Anonymous Coward · · Score: 0

      I'll still take Oregon over DC any day of the week...and yes, I am a native of DC.

    3. Re:Obligatory by donaldm · · Score: 2, Interesting

      Welcome to Oregon, it rains a lot.

      From the Article

      This resulted in cold aisle supply temperature exceeding 80F and relative humidity exceeding 95%. The Open Compute servers that are deployed within the data center reacted to these extreme changes. Numerous servers were rebooted and few were automatically shut down due to power supply unit failure.

      WTF 80 deg F (approx 27 deg C) is quite warm in a Data-centre especially in a "cold aisle" and 95% humidity is criminal.

      Facebook learned from the mistakes, and now designs its servers with a seal around their power supply, or as Parikh calls it, "a rubber raincoat."

      When designing a Data-centre you have to plan for a certain temperature range that the equipment you have inside operates optimally. In addition you have to keep the humidity within manufacture recommended limits since too low results in static electricity and too high well you could get condensation on the electrical equipment. Rubber seals may protect power supplies although I don't think this will completely protect them, but what about the rest of the equipment such as the electronics, connectors and what about backup systems which are very susceptible to temperature and humidity just to name a few.

      My score for this design is zero out of ten and ten out of ten for LOL, welcome to cloud computing :)

      --
      There ain't no such thing as proprietary standards only proprietary formats. Standards are by definition open.
    4. Re:Obligatory by myowntrueself · · Score: 1

      Do you actually know how the Facebook Oregon datacenter works?

      It doesn't *have* HVAC. The building *is* HVAC: The entire building is one very very large HVAC unit.

      HVAC units have 'efficiency of scale'; as you build them larger they get more efficient. Thats what Facebook was aiming at. Unfortunately its a relatively new development in building and datacenter design and clearly has some bugs to work out!

      --
      In the free world the media isn't government run; the government is media run.
    5. Re:Obligatory by Anonymous Coward · · Score: 3, Informative

      WTF 80 deg F (approx 27 deg C) is quite warm in a Data-centre especially in a "cold aisle" and 95% humidity is criminal.

      You're used to classic datacentres, where the goal was "shove as much cold air into them as possible", i.e. "the lower the temperature the better". It all depends on how the datacentre was built, how its cooling system is/was engineered, and an almost indefinite number of variables. References for you to read (not skim) -- the study in the PDF will probably interest you the most:

      http://www.datacenterknowledge.com/archives/2011/03/10/energy-efficiency-guide-data-center-temperature/
      http://www.geek.com/chips/googles-most-efficient-data-center-runs-at-95-degrees-1478473/
      http://blog.schneider-electric.com/datacenter/2013/05/06/getting-comfortable-with-elevated-data-center-temperatures/
      http://www.cs.toronto.edu/~nosayba/temperature_cam.pdf (PDF)
      http://www.dummies.com/how-to/content/data-center-temperature-and-humidity-range-recomme.html

      TL;DR -- 80F is not "quite warm" for a datacentre designed/built within the past 10-11 years.

    6. Re:Obligatory by Runaway1956 · · Score: 1

      "and yes, I am a native of DC."

      I've never met anyone who was willing to admit that! Oh, wait - Anonymous Coward? I still haven't met anyone who is willing to admit that he is a native of the District of Columbia.

      --
      "Windows is like the faint smell of piss in a subway: it's there, and there's nothing you can do about it." - Charlie Br
    7. Re: Obligatory by DigiShaman · · Score: 1

      80f does cause the server fans to work harder via higher RPMs though. The higher your temps, the less margin of error you have to make corrections in a DC.

      --
      Life is not for the lazy.
    8. Re:Obligatory by Anonymous Coward · · Score: 0

      But that's PERFECT!!!!!! An AC in DC!!!! He's clearly either highly charged up about this, or he's on the Highway to Hell. Pick one.

    9. Re:Obligatory by Anonymous Coward · · Score: 1

      http://upload.wikimedia.org/wikipedia/commons/8/80/Oregon_Average_Annual_Precipitation_(1961-1990)_Map.pngAnother citation

      Clearly OP is from western oregon, where it is very rainy indeed.

    10. Re: Obligatory by Anonymous Coward · · Score: 0

      80f does cause the server fans to work harder via higher RPMs though. The higher your temps, the less margin of error you have to make corrections in a DC.

      (Same AC person here...)

      Your logic is correct, but you're making some very erroneous assumptions about server-grade hardware and do not appear to have a good grasp of familiarity with hardware monitoring. What you say might be true for desktop systems, but is often not for server systems. I'll explain:

      The types of fans used in actual server-grade hardware do not have on-die thermistors for temperature detection in them (meaning they will not throttle the incoming voltage used to drive the motor themselves). Most chassis fans are driven either off of a 4-pin Molex that goes right to the PSU (and many of these fans are rated at 10,000rpm or 15,000rpm), while some are driven off of a 3 or 4-pin header on the mainboard linked to a Super I/O chip that also offers H/W monitoring capability (Nuvoton/Winbond, Maxim LM75, ITE IT8712F, ADM1028, etc.), which is nice since then you can actually monitor the fans through software, usually via SMBus but sometimes through classic LPC (classic ISA ports via x86 inb/outb instructions).

      In the case of the latter, the Super I/O chip is configurable in many different ways. For servers, the stock defaults in the BIOS are always "Full speed", meaning chassis temperature has no bearing on fan voltage/RPMs. If the BIOS (which configures the Super I/O chip pre-POST) is set to something that causes the chip to honour that in its calculation formula, then the server administrator needs to have his/her ass kicked. The only time you change from "Full speed" to something else is if your server is in an environment where noise is a concern (i.e. a 2-post telco rack sitting next to your desk). We're talking about actual enterprise-grade datacenters here, so that shouldn't be the case. The fans stay at a constant RPM regardless of workload.

      This same advice applies to CPU cooling fans as well, and in fact some thinner chassis (1U) systems lack a CPU fan entirely, instead relying on nothing but a large "notched" or "sliced" copper or aluminium heatsink (with several 15,000rpm fans blowing air through/across it).

      How do I know all this to be true? Because I write open-source software that monitors and manages the H/W monitoring capabilities of a large number of Nuvoton/Winbond Super I/O chips, specifically for server-class motherboards, and also a system administrator (23 years since I started learning the ropes, 18 years in the field/professionally) who worked in datacenters for about 8 of those years.

    11. Re: Obligatory by sjames · · Score: 1

      Most of the servers I have seen default to throttling the fans based on temperature. That included 1U systems where the fans are controlled by CPU temperature. Perhaps some enterprises run the fans full speed, but others *INSIST* on throttling in the datacenter. I prefer to run them at full throttle, but do acknowledge that in an im[perfect world, after a few years there WILL be a few dead fans being actively ignored.

      In a cluster, you can actually hear it when a job is submitted, it sounds a bit like a jet plane prepping for takeoff.

      80F is a bit warm, 75-78 is more reasonable. The old school likes 65-70F.

      As fopr appeal to authority, I have written the software that configures those Superio chips (right after getting the main memory and the busses running).

    12. Re:Obligatory by faedle · · Score: 1

      Only on the western third. Prineville is, in fact, on the northwestern edge of the Great Basin Desert, and in fact gets about 11 inches a year of rainfall.

      By comparison, Phoenix, Arizona, gets 9.

    13. Re: Obligatory by OdinOdin_ · · Score: 1

      I concur...

      Re GPP: " Nuvoton/Winbond Super I/O chips" these sound like cheap chips, I don't think many of my servers uses these. The servers under high temp and high CPU load do indeed have sensors to throttle/control fans to increase/decrease airflow, they also have a fail-safe mechanism that is the sensor breaks the falls run at full speed. On top of all this the server unit has a motherboard base thermal shutdown that turns the whole server off if the ambient temperature is over some limit.

      I have had this happen before in just such a scenario where the D/C aircon failed and the redundant capacity was under-rated for the area being cooled. The room temperature rose so the point that it increased the set-points of the equipments ambient temperature and other parts to trigger a shutdown to protect itself.

    14. Re:Obligatory by Anonymous Coward · · Score: 0

      Not in Prineville.

    15. Re: Obligatory by DigiShaman · · Score: 1

      You're right about server grade hardware being more robust and intelligently managed to that of a standard desktop. However, in this context, I was speaking of servers. I'm not sure what kind of units you're talking about, but I work exclusively with Dell PowerEdge units. Take a PE R710 (IU) for example. It's maximum warning threshold is set for 42C (108F) with a failure of 47C (117F). The RPMs will throttle on their own (factory default) based on ambient temp per sensor readings on the IPMI bus. Though it should be noted that excessive high temps will shorten the lives HDDs and in some cases cause a premature SMART failure of the drives. And we haven't even begun looking into the theoretical reduction in MTBF. Perhaps with someone as large as Google and Amazon, the savings in energy is great that it offsets the cost to swap hardware often. I wouldn't know. But for a small to medium business that requires their own off site servers, the risk is not acceptable to me.

      --
      Life is not for the lazy.
  4. Lightning in that particular server room by Anonymous Coward · · Score: 1

    as well and Ill start believing in a just $deity

    1. Re:Lightning in that particular server room by maxwell+demon · · Score: 1

      There were some power units failing. I can imagine that this happened with sparks, which are sort of small lightning.

      --
      The Tao of math: The numbers you can count are not the real numbers.
  5. Maybe Berndnaut Smilde snuck in: by Tablizer · · Score: 5, Interesting

    http://www.mnn.com/lifestyle/arts-culture/stories/artist-creates-beautiful-indoor-clouds

    --
    Table-ized A.I.
    1. Re:Maybe Berndnaut Smilde snuck in: by TheLink · · Score: 1

      If you take it to the extreme you get a security system:
      https://www.youtube.com/watch?v=cOgKti335tQ
      https://www.youtube.com/watch?v=cAPw_xbTJzk
      https://www.youtube.com/watch?v=vVf3s6PtFkw

      --
  6. Tech Support by Tablizer · · Score: 1

    Caller: "There's a cloud in our cloud, come immediately!"

    Support: "Speak loud, I cannot hear you."

    Caller: "No, it's cloudy in our cloud."

    Support: "Yes, speak loud."

    Caller: "Yes, there is a big cloud."

    Support: "Yes, you must speak loud, that's what I said."

    Caller: "You must have a cloud also, nothing's making sense. Let's try this: bring some sun."

    Support: "Come soon?"

    Caller: "Yeah, that too. Come soon with sun."

    Support: "I can't hear you, my connection is cloudy."

    --
    Table-ized A.I.
  7. Long story in short ... by Anonymous Coward · · Score: 0

    Traditional air conditioners in data centres regulate temperature and humidity. Energy efficient air conditioners used in trendy data centres take care of the temperature, but ignore the humidity. This created condensation. The condensation was bad for operating electronics. Things went wrong.

    Moral: hire more competent engineers, preferably ones who understand physics.

    1. Re:Long story in short ... by Mr.+Freeman · · Score: 2

      No, you are (mostly) wrong.

      What happened was that the system malfunctioned which led to hot and humid air being circulated throughout the system. This normally would not cause condensation. However, all of the equipment was previously cold (because the system was working normally before it failed). The hot and humid air came into contact with cold components (various components in the power supply and computer casing). This caused condensation (because the hot and humid air contacted the cold components, cooled down, and had to ditch some of its water due to thermodynamics). The condensation on components of the power supply was then blown into various PCB components when the maintenance staff increased the airflow in the datacenter in an attempt to bring the temperature down. The condensation hitting the PCB caused what you would expect when dumping water onto electronics: stuff shorted out and failed.

      You're correct insofar as that humidity was not properly controlled, but this was hardly a case of idiots at the controls or at the drawing board. This was an unanticipated failure mode that was experienced because of a faulty control sequence being generated.

      --
      -1 disagree is not a modifier for a reason. -1 troll, flaimbait, redundant, overrated are NOT acceptable substitutes.
  8. TFA has an even better one by Anonymous Coward · · Score: 0

    "a few were automatically shut down due to power supply unit failure."

    I'm not a modern IT guy -- possibly there's actually something that can be described that way? Till then, that's easily the best PR facepalm I've heard this week.

  9. Streamed Hams by Anonymous Coward · · Score: 2, Funny

    Superintendant Chalmers: A rain storm? At this time of year? At this time of day? In this part of the country? Localized entirely within your datacentre?

    1. Re:Streamed Hams by crutchy · · Score: 1

      it's supernintendo chalmers... you insensitive clod!

    2. Re: Streamed Hams by Colourspace · · Score: 1

      Can I see it? .... no. One of my favourite Simpsons sketches ever..

  10. happened before by crutchy · · Score: 2

    in nasa's vehicle assembly building

  11. Sorry for this but by Anonymous Coward · · Score: 0

    Yo dawg, we heard you like clouds, so we put a cloud in your cloud.

  12. There are at least 3 clouds then. by VortexCortex · · Score: 3, Interesting

    The first cloud would be the humidity and condensation sort. The second cloud would be the online service itself. The third cloud, would be the open Internet between the endpoints in a network graph.

    What do all these clouds have in common? They're dangerous. The less clouds in your diagram the more you know about your network architecture, latency, and data integrity. The less clouds the better! When a packet goes into the shroud of the cloud in the diagram there's a much higher chance we'll never see it again. This cloud is the one where we must encrypt our data and protect against spoofing and hacking and all forms of data manipulation and latency. The receiving end must be very careful to sanitize the inputs and verify the requests vigorously all because the packet has encountered the cloud. Likewise if we want to interact with an online "cloud" service, we shift the name packet to "our stuff" our login credentials and even bank account info, we have to worry about availability and bandwidth caps when streaming, and unwanted prying eyes from folks we may not desire to have looking, everything becomes far more risky because our stuff touched the cloud service; Far more risky than physically going to the bank or visiting a friend in person would not be subject to. If someone hacks the ATM, the entire bank doesn't lose everyone's credentials. As for the mist filled variety of cloud: It can not only get wet, but if you have a big enough cloud, it can strike you with lightning. We must have surge protections and battery backups against this cloud too.

    When I hear people talking about embracing the "cloud" I cringe. "To The Cloud!", in my mind means, "Danger Will Robinson!"

  13. As somebody working on building energy topics by drolli · · Score: 1

    That this happens shows me that they realy optimize their air conditioning for energy consumption.

    Traditionally the approach would have been: "Dont think, cool down and re-heat the air constantly to dehumidify it sufficiently". So traditionally you do this dumb with a lot of energy, even if its not needed at all times. What we probably see there is that some control could not (predict or) handle some drop in the inner load (electrial power) in the data center.

    1. Re:As somebody working on building energy topics by myowntrueself · · Score: 3, Interesting

      That this happens shows me that they realy optimize their air conditioning for energy consumption.

      Traditionally the approach would have been: "Dont think, cool down and re-heat the air constantly to dehumidify it sufficiently". So traditionally you do this dumb with a lot of energy, even if its not needed at all times. What we probably see there is that some control could not (predict or) handle some drop in the inner load (electrial power) in the data center.

      The Facebook Oregon datacenter doesn't 'have' air conditioning.

      The building is an 'air conditioner'. Its an experimental design...

      --
      In the free world the media isn't government run; the government is media run.
    2. Re:As somebody working on building energy topics by drolli · · Score: 1

      Right. I said "air conditioniong" i did not say "air conditioner".
       

    3. Re:As somebody working on building energy topics by jbengt · · Score: 1

      Traditionally the approach would have been: "Dont think, cool down and re-heat the air constantly to dehumidify it sufficiently"

      Actually, traditionally, the cooling and the reheat would each be cycled/modulated by the thermostat/humidistat, not run constantly
      The mistake in this case was not accounting for changes in temperature and humidity, including the fact that the dewpoint temperautre of the air can change much more rapidly than the temperature of the solid objects in the room. It really was a boneheaded mistake, one that is hard to understand given the unusual nature of the HVAC for this facility; you would think that a lot of careful thought about the controls would have taken place during the design of something out-of-the ordinary.
      I have designed around simlar potential problems on much more mundane designs before.

    4. Re:As somebody working on building energy topics by faedle · · Score: 1

      I work for another data center operator in Central Oregon. We also use ambient air cooling. I don't know if we use the same system Facebook does, to be honest.

      Central Oregon is the northwestern edge of the Great Basin desert. Summers here are bone dry. Our data center gets so dry we actually have the opposite problem: it gets TOO dry.

    5. Re:As somebody working on building energy topics by Runaway1956 · · Score: 1

      Alright - what happens in a data center when it gets "TOO dry"? I would assume that people entering the building would become static electricity hazards. It would become essential that anyone handling or touching equipment must use a grounding strap. Anything else?

      --
      "Windows is like the faint smell of piss in a subway: it's there, and there's nothing you can do about it." - Charlie Br
    6. Re:As somebody working on building energy topics by jones_supa · · Score: 1

      Would hard drives work in very low humidity? Rubber mounting components (if any) in the servers might start crumbling down.

    7. Re:As somebody working on building energy topics by faedle · · Score: 2

      Not just the people, but yes, static electricity is the primary concern. Also, I'm told by the people that manage these sorts of things that a "too dry" environment also makes air cooling less effective. Something to do with the fact that a little bit of moisture actually allows the air to carry more heat than if it was 100% dry.

    8. Re:As somebody working on building energy topics by Runaway1956 · · Score: 1

      Ahhh - never thought of that. Makes sense to anyone who has ever had cold survival training. A humid atmosphere leeches warmth from a human body much faster than a dry atmosphere, all other things being equal.

      --
      "Windows is like the faint smell of piss in a subway: it's there, and there's nothing you can do about it." - Charlie Br
    9. Re:As somebody working on building energy topics by Anonymous Coward · · Score: 0

      That this happens shows me that they realy optimize their air conditioning for energy consumption.

      Traditionally the approach would have been: "Dont think, cool down and re-heat the air constantly to dehumidify it sufficiently". So traditionally you do this dumb with a lot of energy, even if its not needed at all times. What we probably see there is that some control could not (predict or) handle some drop in the inner load (electrial power) in the data center.

      The Facebook Oregon datacenter doesn't 'have' air conditioning.

      The building is an 'air conditioner'. Its an experimental design...

      The 'building' is an air conditioner.

      Fixed that for you.

  14. Heat exchange by Reliable+Windmill · · Score: 1

    I'm thinking there has to have been great heat exchange in a system like this.

    --
    Signature intentionally left blank.
  15. I severely doubt by wisnoskij · · Score: 1

    That the roof was high enough to actually allow the formation of clouds. Also, I believe you need dust in the air to form clouds, and I would think that their would be a lot less than normal in a server room. Intense condensation on the roof causing rain is another thing.

    --
    Troll is not a replacement for I disagree.
    1. Re:I severely doubt by Anonymous Coward · · Score: 0

      The only building I know of that's big enough to form clouds inside it is NASA's Vehicle Assembly Building.

      -- hendrik

    2. Re:I severely doubt by Bearhouse · · Score: 1

      There are others...

      http://en.wikipedia.org/wiki/Airship_hangar

  16. Re: facebook learned fom its mistakes... by Anonymous Coward · · Score: 0

    no, it didn't.

  17. No, really, WTF? by faedle · · Score: 2

    Ok, I work for a data center operator. In Central Oregon.

    Our data center is so damn dry that most of the time in the summer we're getting alerts about the humidity being too low. How did Facebook fuck this one up?

    1. Re:No, really, WTF? by Anonymous Coward · · Score: 0

      Two words: "swamp coolers".

  18. Phone Conversation by FeatureSpace · · Score: 1

    Phone conversation between two data center techs:

    Tech 1: "There's a cloud in the Facebook datacenter!"

    Tech 2: "So? Facebook is built on cloud technology!"

    Tech 1: "No I mean a real cloud!"

    Tech 2: "Facebook is built on a server cloud architecture. It IS a real cloud you idiot!"

    Tech 1: "There is a real cloud with real rain in the data center you geek retard! Its shutting down the servers!"

    Tech 2: "Servers shutting down? Maybe the rainfall service is flooding the network with raindrop packets? That would be an emergency! Facebook's cloud is overloaded!"

    Tech 1: "WTF are you talking about? There's real water in the servers. The water is causing electrical faults. Humidity is really high."

    Tech 2: "There's a humidity problem? Probably the flood of raindrop packets interfered with the environmental control service.

    Tech 1: "You might be on to something. Maybe that's what caused the cloud in the first place?"

    Tech 2: "Caused the cloud? The environmental controls have nothing to do with the Facebook cloud!"

    Tech 1: "You idiot! There is a real meteorological cloud in the data center, complete with real rain! Everything is getting wet! I'm not talking about the Facebook cloud!"

    Tech 2: "So what's the problem? Just adjust the environmental controls and reduce the humidity. At least the Facebook's cloud and raindrop service are ok. You really had me worried about Facebook's cloud for a second."

    Tech 1: "The entire data center is powering off you flaming moron!"

    Tech 2: "Wow that's a real emergency! Facebook can't operate without its cloud!"

    Tech 1: "Right. So back to the real cloud in the data center. I'm looking at it right now. Its unreal! You should see it!"

    Tech 2: "Great! Now that the power is back I'll VPN in and check on the cloud. Are you seeing a lot of raindrop packets on the network?"

  19. It's clouds by rastos1 · · Score: 4, Funny

    It's clouds all the way ... up?

  20. I warned them ... by PPH · · Score: 1

    ... they shouldn't have hired that Joe Btfsplk guy for IT support.

    --
    Have gnu, will travel.
  21. Load balancing by gmuslera · · Score: 4, Funny

    Both clouds were leaking and pissing off users. Facebook must have real sysadmins.

    1. Re:Load balancing by ImprovOmega · · Score: 1

      A better example

  22. New entry to fortune database by Anonymous Coward · · Score: 0

    $fortune bofh-excuses
    "Humidity event" in datacenter

  23. You know your datacenter is too big... by alanshot · · Score: 1

    ...When it develops its own atmospheric systems. (that include the water cycle)

  24. Re:first post by bkcallahan · · Score: 1

    Yup, that's Oregon.

  25. Raining in Datacenters by Anonymous Coward · · Score: 0

    This actually happened to me as I was working in a containerized datacenter it was -2 outside and probably around 45-55 degrees internally.....

  26. This actually happened to me around 1990 by azav · · Score: 1

    In the computer section of the UMass Dartmouth Library back around 1990, we had a really humid day one spring/summer inside the Library. There is a part of the library that is open several stories tall and glass windows let the light shine in. We actually had a few drops fall within the library that day. Luckily, no computers were harmed in this event. Super strange to be in the middle of one of these events, that's for sure.

    --
    - Zav - Imagine a Beowulf cluster of insensitive clods...
  27. may i call bullshit on this one? by Anonymous Coward · · Score: 0

    the condom on the power supply is also suspect

  28. And data recovery was ..... by dcpking · · Score: 1

    And I'm sure that they were happy to be able to ask their friends in the NSA for a backup copy of all their data for restoration :)