Loud Sound From Fire Alarm System Shuts Down Nasdaq's Scandinavian Data Center (bleepingcomputer.com)

Catalin Cimpanu, writing for BleepingComputer: A loud sound emitted by a fire alarm system has destroyed the hard drives of a Swedish data center, downing Nasdaq operations across Northern Europe. The incident took place in the early hours of Wednesday, April 19, and was caused by a gas-based fire alarm system that are typically deployed in data centers because of their ability to put out fires without destroying non-burnt equipment. These systems work by releasing inert gas at high speeds, a mechanism usually accompanied by a loud whistle-like sound. With non-calibrated systems, this sound can get very loud, a big no-no in data centers, where loud sounds are known to affect performance, shut down, or even destroy hard drives.

Comments

Monosyllabic Much?

[xxxJonBoyxxx]

>> this sound can get very loud, a big no-no

Jeezzez Crisco. What did we do now to get SlashDot editors talking down to us with one syllable at a time?

Re:Monosyllabic Much?

[Nidi62]

>> this sound can get very loud, a big no-no Jeezzez Crisco. What did we do now to get SlashDot editors talking down to us with one syllable at a time?

Would you prefer "the aforementioned audible emanation attained a decibel level detrimental to the proper operation of the installed data storage mechanisms"?

Re:Monosyllabic Much?

[frank_adrian314159]

Quite.

Re:Monosyllabic Much?

[Anonymous Coward]

Yes actually. We're not in kindergarten. Ideally the writing level would at least be appropriate for a 12th grade audience.

Re: Monosyllabic Much?

[haonlladdis]

I was told once that most news sources are actually written for a seventh grade reading level.

Re:Monosyllabic Much?

[forkfail]

Whelp, there go your mod privileges for life.

Pointing out the... ah... youthful and inexperienced nature of our current crop of editors, let alone their lack of technical prowess, blatant political agenda, or patronizing manner of address to the folks who have been here just a wee bit longer than they have has consequences beyond being called an insensitive clod.

How those who themselves are in need of a serious LARTing are now running this place is beyond me.

Re:Monosyllabic Much?

[Nidi62]

Yes actually. We're not in kindergarten. Ideally the writing level would at least be appropriate for a 12th grade audience.

Anyone can usually memorize more complicated synonyms of everyday words in order to try and sound smarter. However, a much more effective demonstration of intelligence would be showing the ability to understand what level of communication is appropriate and necessary for both the subject and target of the communication. In effect: Keep It (as)Simple (as possible) Stupid.

Re:Monosyllabic Much?

[suutar]

that's a quote from TFA

Re:Monosyllabic Much?

[Richard Stalin]

“It's a beautiful thing, the destruction of words. Of course the great wastage is in the verbs and adjectives, but there are hundreds of nouns that can be got rid of as well. It isn't only the synonyms; there are also the antonyms. After all, what justification is there for a word which is simply the opposite of some other words? A word contains its opposite in itself. Take 'good,' for instance. If you have a word like 'good,' what need is there for a word like 'bad'? 'Ungood' will do just as well--better, because it's an exact opposite, which the other is not. Or again, if you want a stronger version of 'good,' what sense is there in having a whole string of vague useless words like 'excellent' and 'splendid' and all the rest of them? 'Plusgood' covers the meaning, or 'doubleplusgood' if you want something stronger still...In the end the whole notion of goodness and badness will be covered by only six words--in reality, only one word. Don't you see the beauty of that, Winston?”

Re: Monosyllabic Much?

[rickb928]

I wrote tech bulletins and advisories at the 8th grade level. on advice of the vendors.

Troubleshooting and diagnosis guides and general documentation I wrote at the 5th grade level, by request, again by those with long experience writing to technical audiences. Not because they are deficient, or marginally illiterate, but because they pay too little attention to detail, are hurried, and don't really care enough. And to limit the vocabulary to the most commonly shared set. Being clever doesn't serve your audience in these situations, even if they are as clever as you.

Re:Monosyllabic Much?

[rickb928]

This is wrong, by the way, demonstrably wrong.

'Ungood' isn't the exact opposite of 'good'. It may be better defined as the absence of good, which could be neutral or 'bad'.

You started it, not including the sarcasm flag is your fault.

Re:Monosyllabic Much?

[forkfail]

Please tell me that you yourself left off the /sarcasm flag in your response here, that you understood the reference being made.

Re:Monosyllabic Much?

[rickb928]

How would you know?

Re:Monosyllabic Much?

[sjames]

Nobody wants subtle nuance in a tech manual, save it for poetry and novels. Otherwise you just read like an 8th grader trying to sound sophisticated.

Re:Monosyllabic Much?

[Pascoea]

Exactly. That's the key to technical writing, use exactly as many words as are necessary to accurately get a point across, and no more.

Total failure

[Daetrin]

"caused by a gas-based fire alarm system that are typically deployed in data centers because of their ability to put out fires without destroying non-burnt equipment. [...] With non-calibrated systems, this sound can get very loud [and destroy-non burnt-equipment]"

You had one job! (Directed at both the system itself and whoever was supposed to be in charge of calibrating it.)

...actually, that person may have zero job now.

Re:Total failure

[painandgreed]

You had one job! (Directed at both the system itself and whoever was supposed to be in charge of calibrating it.) ...actually, that person may have zero job now.

They were probably a contractor that offered to calibrate the system but some manager decided he didn't want it coming out of their budget.

Re:Total failure

[aaarrrgggh]

This is actually a common failure mode for FM200 systems (fire PROTECTION, not fire ALARM). The last I had heard about it, research was suggesting that it was the sudden pressure change, but I hadn't been following it closely since.

A couple bank data centers were hit by it (HSBC was one) in 2016.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:Happy Friday From A Crashing Market!

[Richard Stalin]

There's a kernel of truth to this. I know a few guys who work for the NYSE, and it's the same there. One of them told me to watch a few particular derivatives before they take down the system and watch them after they bring the system back online. Three out of four times this is to prevent a catastrophe and had nothing to do with a true hardware/software failure. But it does keep a '29 stock market crash from happening.

Re:SSDs

[Anonymous Coward]

I herd SSDs are silent and mostly immune to vibration.

How can you herd them if they are silent?

Fire SUPPRESSION system not fire ALARM system

[jtara]

It's a fire SUPPRESSION system, not a fire ALARM system.

At least this time, it's the article (on Bleeping Computer) that is wrong, not the summary on /.

It's apparent the author of the article didn't bother to read the article (on Motherboard) that she cites about a similar incident in Romania at ING Bank. It clearly states that incident resulted from a "fire extinguishing test".

The sound BTW comes from the release of the gas, not some alarm. In both cases, the pressure was set too high. It was basically a - very loud - over 130db - hissing sound!

The second article cites a study about the effect of this sound done by Siemens. Siemens has a vested interest. I guess neither of these data centers were equipped with the Sinorix Silent Nozzle.

Re:Fire SUPPRESSION system not fire ALARM system

[Critical Facilities]

Thank you. Moreover, I would bet that it wasn't the sound, but the particulates that did it. Whether this was Halon, or FM-200, or some other "clean agent", if you've ever seen a Data Center after one of those goes off, it's covered in powder and dust. The idea that they're truly "clean" is complete crap. Sure, they won't soak your gear in water, but it's not this lovely, clean, unobtrusive gas like the sales folks tell you it is.

Re:Fire SUPPRESSION system not fire ALARM system

[snugge]

Hard disks don't die from dust. They *do* die from too loud noises, though.

Seen that first hand after an incident just like this one.

Re:Fire SUPPRESSION system not fire ALARM system

[bobbied]

Oh sure the GAS is clean and unobtrusive... The problem is the speed it is released and where it blows.... I've seen what ends up under the raised floors or in the rafters in a data center after a few years in operation and it's not pretty. Imagine all this dirt, dust and trash suddenly flying around.

House keeping! Clean up in the server room!

Then there is the whole, fire alarm pulled, kill the power to limit the damage! Use case. Modern computers do NOT take kindly to having the power yanked out from under them and few take the time to do an orderly shutdown when a fire was reported.

Re:Fire SUPPRESSION system not fire ALARM system

[Khyber]

"Moreover, I would bet that it wasn't the sound, but the particulates that did it."

All spinning rust hard drives are fully-sealed against particulates. SSDs don't need that, and the stuff used in electronics fire control tends to be non-conductive.

Re:Fire SUPPRESSION system not fire ALARM system

[Megol]

Do they still use halon? Thought it was reserved for military and critical civilian uses only.

Re:Fire SUPPRESSION system not fire ALARM system

[Critical Facilities]

Perhaps the drives themselves don't, but the servers or DASD boxes they're sitting in are very susceptible to dust. I realize the article says the hard drives failed, but further down they talk about needing to replace the servers. I'm asserting that it was more likely the dirt and debris that ultimately caused the problem, not the sound.

Re:Fire SUPPRESSION system not fire ALARM system

[Szeraax]

Obligatory story about new "clean agent" fire suppression systems (hint: nitrogen + water):

Tuesday was a regular day for me. I am working remotely and live almost 2 hours away from work. At 11:02am, this chat began between my boss and me:

        Boss: Please monitor all systems. Alarm system fired in server room

        Me: I can't get on firewall

        Me: I'm off vpn

        Me: You still have internet?

        Me: Do you mean the fire alarm went off?

        Me: Or other alarm

        Me: Still can't get on vpn

        Me: I was hoping to be able to get on via X2 or X3, but I still can't get on VPN via any WAN interfaces.

        Me: $TaskServer seems to be working; just got an email generated by it. so $SMTPServer must also working

        Boss (8 agonizing minutes later): Fire

Turns out... our data center fire alarm was getting tested today! And the company that installed it had one of their techs come out to do the test.

My boss was in the server room escorting him when it happened. Normally they disengage the solenoid from the suppression system when testing, but he must not have done it right. They heard a weird hissing noise... like something was charging.

Now, since our server room is NOT sealed, we can't use systems that remove oxygen from the air. We bought a new fancy suppression system that will basically atomize a gallon of water for a 12'x12' space distributed by a tank of pressurized nitrogen. This distributed water is safe to breathe, safe to suck through fans and all equipment, and totally kills fire.

Well guess what? the Cisco ASR 1000 router that provides our primary internet (that we all go through to get to firewall for VPN) didn't like all the moisture and rebooted itself (maybe it shorted? dunno). Hence, it took like 15 minutes for me to be able to get back onto VPN. It also was making weird fan noises.

$Boss described it as a very thick fog that cleared after a minute or two. Like... can't see 6" from your face initially. Needless to say, he isn't very happy with the alarm company at the moment. Hopefully none of our equipment will suffer from the test.

In the aftermath, I noticed that some yahoo (hopefully not me) had disabled incoming VPN on the X2 and X3 backup ISPs. So that's why I couldn't get in on those other interfaces...

Re:Fire SUPPRESSION system not fire ALARM system

[rickb928]

There are so many reasons to not use water.

Unless it's very pure, you will deliver minerals, salts, with predictable consequences, not all immediate.

Water will also dissolve and/or deliver contaminants throughout equipment. Again, not good, and the impacts not necessarily immediate.

As a solvent, water is different than many used in manufacturing electronic equipment, so what wasn't a problem before exposure may become later. Even high humidity can cause this.

Pure water should be safe, but it's all the other contaminants that it dissolves and delivers that are the problem. And datacenters are often not very clean, despite looking so.

Re:Fire SUPPRESSION system not fire ALARM system

[Szeraax]

It is just regular filtered, distilled water. The high humidity is what caused our WAN router to reboot and freak out. We still haven't had any extra issues due to it, but that's not to say that it WONT happen in the future from this one event. We were not pleased with the alarm company.

Re:Fire SUPPRESSION system not fire ALARM system

[painandgreed]

The sound BTW comes from the release of the gas, not some alarm. In both cases, the pressure was set too high. It was basically a - very loud - over 130db - hissing sound!

Ya, but if you don't pay for calibration and extra nozzles, you get a fire extinguishing system AND an alarm for less cost!

Odd

[Virtucon]

When my Ex-Wife let out loud, obnoxious sounds my hard drives didn't self-destruct. My mind did, but not my hard drives.

Re:Odd

[hraponssi]

Sounds like you got away, and I like the sound of that.

Re:Odd

[antdude]

What about your dongle? Not the computer hardwares! Your body's. :P

Well known for ages now

[green1]

These sorts of systems, have been well known for ages now to be a high risk for damaging equipment, most of them release with such force that they can do a lot of harm. As a result, many data centers, and other similar businesses have moved away from them to avoid exactly this sort of incident. A popular alternative right now is water mist, despite what people immediately think, these have actually been proven to be safe around electronics when properly designed and deployed.

Re:Well known for ages now

[Anonymous Coward]

These type of systems work by displacing of reacting or otherwise removing the oxygen from the room - starving the fire. I am pretty sure most businesses have moved away from them because of the liability of killing all the humans that don't make it out of the room before the gas releases.

Re:Well known for ages now

[Strider-]

For most common materials to burn, you need an oxygen content of 12% or higher. On the other hand, a human can maintain consciousness down to something like 5%. As such, inerting systems are designed to drop the oxygen content to something like 8%. Too low for combustion to take place, but high enough not to kill the occupants. Nitrogen type inerting systems will actually often include a small amount of CO2 in the gas mix; this causes any remaining occupants to breath harder, thus allowing them to work better in a low oxygen atmosphere.

reference: I worked on a small power plant with a two compartment FM-200 fire suppression system. One of the things that the design Engineers needed was an accurate measurement of the room volume so that they could calibrate the amount of agent in the tanks appropriately. However, just in case, before an agent dump there is a 30 second siren and several large "Cancel dump" mushroom switches.

Poor Planning?

[bestweasel]

Another site without replication, redundancy and automatic failover, similar to when British Airways' entire computer system collapsed because of a power surge when someone turned it off and on again.

It's not possible to test all scenarios ("Right, now we're going to see how our network would cope if someone took an axe to that cabinet. Derek, when you're ready") but someone should have planned for a data centre failure. Is this lack of resilience common in big networks?

Re:Poor Planning?

[bobbied]

Is this lack of resilience common in big networks?

Yes, it is. Resilience is conceptually easy to understand, but historically exceptionally hard to implement because it requires rigorous controls on such mundane things as where you plugged that server and network switch in. Because it's so hard to maintain, supposedly resilient systems often are not really. Somebody moved some power cable, switched a network port to another switch or forgot to update the standby system and their data backup scripts... Then the unthinkable happens and you discover some simple mistake that renders all that effort and money spent pointless.

Re:Poor Planning?

[rickb928]

Yup, as simple as unplugging a power cord because 'everything's redundant, we can cover this'.

You can do that once only.

Re:Poor Planning?

[F.Ultra]

Well they have a secondary site at which they opened trading hours later, the question that is left for Nasdaq to answer is why it took them so long to open trading on the secondary site when they and us the members take part in regularly disaster exercises where everyone switches to use the secondary site within seconds.

Re:Poor Planning?

[F.Ultra]

And it's somewhat more complicated here since also every member of the exchange have to switch over their systems to use the secondary site, and due to their rules they cannot open unless all members have switched. Also there had already been trades between members right before and right at the moment the servers was destroyed so there where some cleaning up to do as well (all the trades had to be cancelled).

Re:Poor Planning?

[sjames]

One of my favorites was a diesel generator with a small local tank and a larger tank installed well behind the building. Never a problem during the monthly tests where they actually failed over to the generator.

Actual power failure happens, generator picks up the load just fine. An hour later, the generator shuts down. Turns out the transfer pump that fed the small local tank from the large tank was connected to grid power only.

Didn't we learn not to yell at hard drives?

[Anonymous Coward]

No seriously, I can't find the video atm but many years ago there was a guy from (Sun?) showing how susceptible they are to sound vibrations. His test literally was yelling at them while watching dtrace.

Update: Found it. Was Brendan Gregg - https://www.youtube.com/watch?v=tDacjrSCeq4

* (headphone warning)

Re:Didn't we learn not to yell at hard drives?

[nanoflower]

Ah, yes. I remember that video. I was quite surprised that drives could be impacted that easily just by the human voice.

Reminds me of an old Usenet legend...

[Anonymous Coward]

Dunno if it's true or not, but I've always found it amusing nonetheless:

https://www.hactrn.net/sra/vaxen.html

Not fire alarm

[makerfixer]

Pro tip: fire alarms make noise, fire protection systems put out fires.... this was the latter. A clean agent release system.

Re:Not fire alarm

[omnichad]

But it was set up wrong, so it made noise. So it's also an alarm by your definition.

Re: Not fire alarm

[makerfixer]

My definition of an alarm is a temporal pattern set up to alert occupants. Not the sound of rushing gas.

Actually, it's the opposite

[Anonymous Coward]

Reporters and bloggers cannot even get the facts right let alone write at a higher level.
Fire alarms make noise to warn people.
Fire suppression systems extinguish fires.
Fire protection systems often incorporate suppression and alarm systems.

Re:SSDs

[mschaffer]

You herd them the usual way---with dogs. The dogs can see them and smell them, so they don't need to hear them.

Re:I had no idea this could happen

[darkain]

From a decade ago: https://www.youtube.com/watch?...

Re:screaming

[mschaffer]

It's here: https://www.youtube.com/watch?...

Re:I had no idea this could happen

[rickb928]

I guess I'll be wearing active hearing protection to the data center from now on, eh?

Re:SSDs

[rickb928]

"How can you herd them if they are silent?"

You're doing it wrong. You're supposed to cluster

Oh, wait...

If you've ever wondered what it sounded like..

[Alpha232]

This sounds like an Inergen type system... the two failings are the particulate matter it stirs up and the vibration caused by the sounds of a freight train rolling through...

https://youtu.be/yM80eBR_b2w

Re: SSDs

[BKX]

Most electronics use switching supplies for the base power rails. Switching supplies have capacitors, some of which may be charged and discharged at a frequency in the audible range. Those capacitors can be quite loud. You can even use them as speakers. Nowadays most switching supplies use higher frequencies to cut down on audible noise, but noisy caps are still a thing. Furthermore, there are almost always caps that smooth out the AC input. 50/60hz is audible, so those caps will always hum.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:SSDs

[webnut77]

I herd SSDs are silent and mostly immune to vibration.

How can you herd them if they are silent?

Just gather them up in a coral. A sheep dog can help with that.

A keep those doggies rolling. Rawhide!

Wednesday, April 19?

[GungaDan]

Something's not quite right about that. For the year 2018, at least.

Re:SSDs

[cellocgw]

I herd SSDs are silent and mostly immune to vibration.

How can you herd them if they are silent?

Just gather them up in a coral. A sheep dog can help with that.

That won't work either. All the coral is dying.

Re:SSDs

[UnknownSoldier]

Allegedly (*) the same way you herd cats -- with great difficulty. =P

(*) I've never met anyone who has actually done (**) this yet everyone is magically an "armchar expert".

(**) I don't understand what the big deal is -- you just wait till everyone is hung and then string them along with food. How hard (***) could it be? :-)

(***) Famous last words.

Re:SSDs

[UnknownSoldier]

Whoops, Freudian (?) slip.

s/hung/hungry

Re:SSDs

[webnut77]

I herd SSDs are silent and mostly immune to vibration.

How can you herd them if they are silent?

Just gather them up in a coral. A sheep dog can help with that.

That won't work either. All the coral is dying.

Really? What color? Blue, I hope. I have a bolt of cotton cloth I'd like dyed blue.
How do I contact the coral?

"Suppression" is the word you were looking for

[Pascoea]

A fire alarm system isn't designed to extinguish a fire, a suppression system is. I hate to be pedantic, but there is a difference. The device in your home that disturbs fajita night is an alarm system...

Re:I had no idea this could happen

[Joce640k]

I guess I shouldn't burp near my NAS then

You jest, but, there's people who can burp at 100dB.

https://encrypted.google.com/s...

ie. This guy can burp at a heavy metal concert and the people around him will tell him to STFU because they can't hear the music for all the burping.

Re:I had no idea this could happen

[Joce640k]

I meant 110dB, dammit.

Sometimes it happens on purpose

[magarity]

Did anyone bother to check if there was an actual fire or are they too busy kvetching about the noise?

Can confirm

[RabidMonkey]

At a previous employer, we lost an entire row of servers in a DC after a water leak (somehow) triggered the suppression system. The 'explosion' was strong enough to knock the doors off cabinets, bend 2 cabinets, and cause a couple hundreds drives to be dead. Thankfully our service was spread out far enough to survive the loss of a row for a few week while we waited for all new disks to arrive from IBM.

The pictures were crazy, it looks like a bomb went off.