Slashdot Mirror


Loud Sound From Fire Alarm System Shuts Down Nasdaq's Scandinavian Data Center (bleepingcomputer.com)

Catalin Cimpanu, writing for BleepingComputer: A loud sound emitted by a fire alarm system has destroyed the hard drives of a Swedish data center, downing Nasdaq operations across Northern Europe. The incident took place in the early hours of Wednesday, April 19, and was caused by a gas-based fire alarm system that are typically deployed in data centers because of their ability to put out fires without destroying non-burnt equipment. These systems work by releasing inert gas at high speeds, a mechanism usually accompanied by a loud whistle-like sound. With non-calibrated systems, this sound can get very loud, a big no-no in data centers, where loud sounds are known to affect performance, shut down, or even destroy hard drives.

16 of 114 comments (clear)

  1. Total failure by Daetrin · · Score: 2

    "caused by a gas-based fire alarm system that are typically deployed in data centers because of their ability to put out fires without destroying non-burnt equipment. [...] With non-calibrated systems, this sound can get very loud [and destroy-non burnt-equipment]"

    You had one job! (Directed at both the system itself and whoever was supposed to be in charge of calibrating it.)

    ...actually, that person may have zero job now.

    --
    This Space Intentionally Left Blank
  2. Re:Monosyllabic Much? by Nidi62 · · Score: 3

    >> this sound can get very loud, a big no-no Jeezzez Crisco. What did we do now to get SlashDot editors talking down to us with one syllable at a time?

    Would you prefer "the aforementioned audible emanation attained a decibel level detrimental to the proper operation of the installed data storage mechanisms"?

    --
    The only thing necessary for evil to triumph is for it to be pitted against a slightly greater evil
  3. Re:SSDs by Anonymous Coward · · Score: 3, Funny

    I herd SSDs are silent and mostly immune to vibration.

    How can you herd them if they are silent?

  4. Fire SUPPRESSION system not fire ALARM system by jtara · · Score: 4, Informative

    It's a fire SUPPRESSION system, not a fire ALARM system.

    At least this time, it's the article (on Bleeping Computer) that is wrong, not the summary on /.

    It's apparent the author of the article didn't bother to read the article (on Motherboard) that she cites about a similar incident in Romania at ING Bank. It clearly states that incident resulted from a "fire extinguishing test".

    The sound BTW comes from the release of the gas, not some alarm. In both cases, the pressure was set too high. It was basically a - very loud - over 130db - hissing sound!

    The second article cites a study about the effect of this sound done by Siemens. Siemens has a vested interest. I guess neither of these data centers were equipped with the Sinorix Silent Nozzle.

    1. Re:Fire SUPPRESSION system not fire ALARM system by Szeraax · · Score: 2

      Obligatory story about new "clean agent" fire suppression systems (hint: nitrogen + water):

      Tuesday was a regular day for me. I am working remotely and live almost 2 hours away from work. At 11:02am, this chat began between my boss and me:

              Boss: Please monitor all systems. Alarm system fired in server room

              Me: I can't get on firewall

              Me: I'm off vpn

              Me: You still have internet?

              Me: Do you mean the fire alarm went off?

              Me: Or other alarm

              Me: Still can't get on vpn

              Me: I was hoping to be able to get on via X2 or X3, but I still can't get on VPN via any WAN interfaces.

              Me: $TaskServer seems to be working; just got an email generated by it. so $SMTPServer must also working

              Boss (8 agonizing minutes later): Fire

      Turns out... our data center fire alarm was getting tested today! And the company that installed it had one of their techs come out to do the test.

      My boss was in the server room escorting him when it happened. Normally they disengage the solenoid from the suppression system when testing, but he must not have done it right. They heard a weird hissing noise... like something was charging.

      Now, since our server room is NOT sealed, we can't use systems that remove oxygen from the air. We bought a new fancy suppression system that will basically atomize a gallon of water for a 12'x12' space distributed by a tank of pressurized nitrogen. This distributed water is safe to breathe, safe to suck through fans and all equipment, and totally kills fire.

      Well guess what? the Cisco ASR 1000 router that provides our primary internet (that we all go through to get to firewall for VPN) didn't like all the moisture and rebooted itself (maybe it shorted? dunno). Hence, it took like 15 minutes for me to be able to get back onto VPN. It also was making weird fan noises.

      $Boss described it as a very thick fog that cleared after a minute or two. Like... can't see 6" from your face initially. Needless to say, he isn't very happy with the alarm company at the moment. Hopefully none of our equipment will suffer from the test.

      In the aftermath, I noticed that some yahoo (hopefully not me) had disabled incoming VPN on the X2 and X3 backup ISPs. So that's why I couldn't get in on those other interfaces...

  5. Odd by Virtucon · · Score: 3, Informative

    When my Ex-Wife let out loud, obnoxious sounds my hard drives didn't self-destruct. My mind did, but not my hard drives.

    --
    Harrison's Postulate - "For every action there is an equal and opposite criticism"
  6. Poor Planning? by bestweasel · · Score: 3, Informative

    Another site without replication, redundancy and automatic failover, similar to when British Airways' entire computer system collapsed because of a power surge when someone turned it off and on again.

    It's not possible to test all scenarios ("Right, now we're going to see how our network would cope if someone took an axe to that cabinet. Derek, when you're ready") but someone should have planned for a data centre failure. Is this lack of resilience common in big networks?

  7. Didn't we learn not to yell at hard drives? by Anonymous Coward · · Score: 2, Insightful

    No seriously, I can't find the video atm but many years ago there was a guy from (Sun?) showing how susceptible they are to sound vibrations. His test literally was yelling at them while watching dtrace.

    Update: Found it. Was Brendan Gregg - https://www.youtube.com/watch?v=tDacjrSCeq4

    * (headphone warning)

  8. Re:SSDs by mschaffer · · Score: 4, Funny

    You herd them the usual way---with dogs. The dogs can see them and smell them, so they don't need to hear them.

  9. Re:I had no idea this could happen by darkain · · Score: 4, Informative
  10. Re:Monosyllabic Much? by Richard+Stalin · · Score: 3, Interesting

    “It's a beautiful thing, the destruction of words. Of course the great wastage is in the verbs and adjectives, but there are hundreds of nouns that can be got rid of as well. It isn't only the synonyms; there are also the antonyms. After all, what justification is there for a word which is simply the opposite of some other words? A word contains its opposite in itself. Take 'good,' for instance. If you have a word like 'good,' what need is there for a word like 'bad'? 'Ungood' will do just as well--better, because it's an exact opposite, which the other is not. Or again, if you want a stronger version of 'good,' what sense is there in having a whole string of vague useless words like 'excellent' and 'splendid' and all the rest of them? 'Plusgood' covers the meaning, or 'doubleplusgood' if you want something stronger still...In the end the whole notion of goodness and badness will be covered by only six words--in reality, only one word. Don't you see the beauty of that, Winston?”

  11. Re:screaming by mschaffer · · Score: 2
  12. If you've ever wondered what it sounded like.. by Alpha232 · · Score: 2

    This sounds like an Inergen type system... the two failings are the particulate matter it stirs up and the vibration caused by the sounds of a freight train rolling through...

    https://youtu.be/yM80eBR_b2w

  13. Re:SSDs by cellocgw · · Score: 3, Funny

    I herd SSDs are silent and mostly immune to vibration.

    How can you herd them if they are silent?

    Just gather them up in a coral. A sheep dog can help with that.

    That won't work either. All the coral is dying.

    --
    https://app.box.com/WitthoftResume Code: https://github.com/cellocgw
  14. Re:Well known for ages now by Strider- · · Score: 3, Insightful

    For most common materials to burn, you need an oxygen content of 12% or higher. On the other hand, a human can maintain consciousness down to something like 5%. As such, inerting systems are designed to drop the oxygen content to something like 8%. Too low for combustion to take place, but high enough not to kill the occupants. Nitrogen type inerting systems will actually often include a small amount of CO2 in the gas mix; this causes any remaining occupants to breath harder, thus allowing them to work better in a low oxygen atmosphere.

    reference: I worked on a small power plant with a two compartment FM-200 fire suppression system. One of the things that the design Engineers needed was an accurate measurement of the room volume so that they could calibrate the amount of agent in the tanks appropriately. However, just in case, before an agent dump there is a 30 second siren and several large "Cancel dump" mushroom switches.

    --
    ...si hoc legere nimium eruditionis habes...
  15. Can confirm by RabidMonkey · · Score: 2

    At a previous employer, we lost an entire row of servers in a DC after a water leak (somehow) triggered the suppression system. The 'explosion' was strong enough to knock the doors off cabinets, bend 2 cabinets, and cause a couple hundreds drives to be dead. Thankfully our service was spread out far enough to survive the loss of a row for a few week while we waited for all new disks to arrive from IBM.

    The pictures were crazy, it looks like a bomb went off.

    --
    We emerge from our mother's womb an unformatted diskette; our culture formats us. - Douglas Coupland