Slashdot Mirror
Server Room Smells Can Be an Early Warning
Barence writes "As embarrassing as it may seem, an eggy smell in a server room needn't mean broaching the delicate subject of hygiene with a colleague. It can actually be a signal that something is about to go wrong with your server setup, as this consultant discovered after days of assuming questionable personal habits were to blame. The culprit? An expiring UPS device, sending out its own unique warning signal."
Amazing how many dying UPS devices must be hidden in my boss's office.
something smell fishy in here?
Unique Pheromone Signals?
Hygiene with the UPS.
Does this mean I can use my father-in-law as a UPS?
Sulfur Dioxide. Ventilate, replace or recondition battery. If the egg smell is strong and you quit smelling it, that's olifactory fatigue and lethal levels of the gas exist.
An alternative to blaming the dog. "Wasn't me, honey, it was the computer."
Two comments...
1. Windows in the server room?
2. No-one noticed the UPS with all its error lights on?
This is funny, I routinely smell my servers and my UPS at the fans where the air come out of them to make sure nothing overheats but I never thought about mentioning that to anybody ;-))
hehe...
Everything I write is lies, read between the lines.
I remember some old AMD slot style CPU's would smell like waffles when they burned out. Good times.
"Can be an early warning?"
"CAN be?"
Like all IT administrators who've actually worked with server hardware, I have a heightened sense of smell, but only specifically for the smell of burning plastic. It's not a mere warning, it's an instant alarm that'll have every IT person in the room sniffing the power supplies.
We IT people, we're like bloodhounds or something. I can smell burned plastic from across the street. I've been set off by welders at a car mechanic a block away. I've been set off by an invisibly tiny bit of cheese someone dropped into a toaster oven once... three floors down from the server room. Had me in a right panic.
IT is all fun and games until the servers literally melt into slag. There's no repair CD for that -- and we all know that the backup tapes, while wonderful for backing up, aren't so good at the actual restoring bit. That's why they're called backup tapes, not restore tapes, see?
> As the season has changed, for the first time in something like five months, I've opened a window in the server room I've been working in.
Then the smell was gone, but there was this loud beeping sound. After doing another Google search I found out from a security consultant's blog that it was the break-in alarm on the window.
Another crazy day at the office.
lucm, indeed.
of things about to go wrong in a big way...
http://slashdot.org/~GuyFawkes/journal
Server Room Smoke can be an Early warning, also.
As embarrasing as it may seem, a cloud of smoke in the server room needn't mean broaching the delicate subject of kicking the marijuana habit with a colleague. It can actually be a signal that something is about to go wrong with your server setup.
as this consultant discovered after days of assuming questionable personal habits were to blame. The culprit? A server whose board was in the process of being about to friggin explode!
I've never physically been inside a data center, but I'd have thought that the locales would have really good ventilation, that would simply shut close (or rely on gas weight and gravity) if the halon system or equivalent would need turning on. The ventilation is in fact so bad, there can be a gas buildup so severe you need to (according to posters above me) go in with hazmat gear?
Emotions! In your brain!
From the clasic BOFH :)
"The admin gene," the PFY explains. "The ability to recognise things that users don't. A slight flicker of lighting, a whiff of hot component in the air, a fractional change in the pitch of a cooling fan - all of which the garden variety user misses in the headlong rush to read their email."
http://www.theregister.co.uk/2008/07/04/bofh_2008_episode_24/
I just noticed that I also suffer from synasthesia.
APC UPS's have a tendency to cook their batteries as they get near the end of their lifetime. The results can be horrifying... bulging batteries, and if allowed to go on long enough, yes, even "sealed" lead acid batteries will rupture and you'll get the lovely sulfur smell.
I recently pulled these APC batteries out of an APC Smart-UPS 1400, which had to be disassembled (including the removal/replacement of rivets) in order to get the batteries out.
http://img221.imageshack.us/img221/171/imageyv.jpg
Computers run on smoke... when the smoke comes out, they stop running.
If they're like some of the IT departments I've seen, they might be working by some rule from upper management that they need to justify their existence by writing internal invoices for everything they do. It tends to result in them doing nothing until you tell them to, so they can bill you for it. The UPS could have not only the error lights on, but a binking "RED ALLERT" sign and the accompanying acoustic blare, and verily be on fire and billowing smoke, and nobody would touch it until you fill the proper form requesting them to put it out.
Because, yes, that's another thing I've noticed that a lot of departments love, IT including: inventing bureaucracy and paperwork to discourage and delay actually having anything to do. You may need to fill in a 5 page form and draw powerpoint diagrams as to why you want the UPS doused and what are the architecture implications of that. And if you're unlucky a few meetings too, to convince some Mordac The Information Services Preventer why he should move his ass and turn that UPS off, and why his suggested workarounds (in which he'd not have to do anything) aren't quite solving the problem.
A polar bear is a cartesian bear after a coordinate transform.
This is why I prefer to build my new server rooms with individually cooled racks - each rack having its own AC-circulation - as well as using centralized water cooling for its efficiency and reliability. Circulating all your cooling air around the server room is simply a bad idea. When you have 1 kilometer of rack space on a single building floor, one source of contaminant, be it chemical or metal particles, will get into all the enclosures in the hall and cost you everything. And BTW UPS maintenance is something that modern IT management, especially outsourced services, have forgotten. Any veteran admin knows you need to estimate the end-of-life for their electronics AND replace them BEFORE they fail - just like AC-filters - If allow those to fail, they will have already done some damage! There's no "RAID" for burning electronics or blocked cooling air!
www.tribalnetworks.org - helping tribal people around the world to own their own means of high-tech communications
I trained my bloodhound to find decaying human bodies underwater by smelling the offgassing while he hangs his head over the side of a boat. I could train my bloodhoud to detect smells in my server room and take him to work everyday. Unfortunately, even after it is fixed he could still smell it after a month or two with the smell being in a closed room. In the outdoors he can track a human even after 10 days have passed, his record is 16 days.
Listen...do you smell something?
Why does the UPS not have a fail safe that kills it when the battery goes bad to stop a fire?
We've recently had two UPSs expire in the last couple of months. We were talking about it and we have a UPS fail more often than we have a power outage. If the UPS fails more often than a power outage, why do we even use UPSs?
Anyone else notice that old HDDs smell strongly of body odor? It's like bacteria love the warmth and go through X generations until there's a constant "eww" smell to them. It seems to happen more with 10-15k RPM SCSI drives in 1U servers where they don't get as much ventilation, but I've noticed it with IDE and SATA in desktops too.
That's a damned good question. Is there a UPS engineer in the house??
~REZ~ #43301. Who'd fake being me anyway?
You can test lead-acid batteries by measuring the resistance between the terminals. Most upses only look for voltage from the battery. This means that the battery casing can split open, leak acid, before the batter grounds out or goes kaboom, or whatnot. I had one battery in a 64 battery cabinet leak out and ground out the batteries.... The bloody APC Silcon (now discontinued) ups didn't throw an alarm at all, all we noticed that there were no lights all of a sudden when the power flickered. It was apparently a firmware bug or some sort because APC swapped out the firmware IC chips when they replaced the batteries.
Generally, with larger ups systems, you tend to have a quarterly battery inspection where a tech comes out, takes voltage and ohms readings for each battery in the unit, and visually inspects the batteries in the cabinet.
Service Status Last Check Duration Attempt Status Information
UPS1 OK 03-21-2010 13:01:44 2d 1h 3s 1/3
UPS2 OK 03-21-2010 13:01:44 5d 4h 1s 1/3 Hey it wasn't ME
UPS3 OK 03-21-2010 13:01:44 2d 1h 3s 1/3
UPS4 Gassy 03-21-2010 13:01:44 5d 4h 1s 1/3
#DeleteChrome
UPS system should automatically alert via SMTP or SNMP when a battery faults. The batteries are hot swappable too, so there would be no outage required to replace it. If Murphy strikes and you get a power outage whilst swapping out the UPS batteries, having dual PSUs in your servers powered off dual UPS units would cover you.
It gripped her hand gently. 'Regret is for humans,' it said.
If your server room stinks, likely Ed Peterlin is sleeping in there.
Nearly there, but all the symptoms and the reactions you are describing is the result of Hydrogen Sulphide exposure and not Suplhur Dioxide.
It becomes stinky at around 5-10ppm.
It becomes highly irritant to the repository system at around 20ppm
It paralyses your sense of smell at 30ppm. At this point you have about 5 min or so to get the hell out before you pass out.
It becomes lethal (death, and not simply passing out) at around 100ppm.
To those wondering how you're supposed to smell it above 30ppm, if a sudden whaft of pungent smell so potent it nearly knocks you off your feet suddenly disappears, then get the hell out.
Think of it as if a million geeks farted in unison, and then nothing.
You could also just buy a H2S detector. I wear a Dragger H2S detector around work.
Their design engineers must be breathing in too much sulphuric acid fumes methinks.
Any UPS that costs more than $400 should have full battery monitoring, including temperature, internal resistance and moisture on the bottom of the compartment. The cost of doing that is perhaps $5 in parts. Anything over $1000 can as well have chemical detectors -- you surely can get them for $5 each in quantity.
The fact that APC doesn't routinely do it the right way is why I laugh very hard into the receiver every time I get a sales call from them. They are idiots, and I have no qualms telling it to anyone who listens.
Now the fact that no other "household name" company I know of does it right is a whole another story.
A successful API design takes a mixture of software design and pedagogy.
Well, the UPS part was more like satire. But I've seen the attitude I've described about everything else. Hard drive space, database configuration, MQ queues, you name it. There are some departments out there who'll let you drive against the wall without even knowing, because they monitor nothing, do nothing except what you explicitly request so they can bill you (e.g., DBAs who require that the _programmers_ tell _them_ what tuning parameters to set for the database), created a lot of unneeded bureaucracy, and occasionally yes need to be coaxed to even bloody do their job at all.
And those aren't even the biggest WTF's I've seen so far.
And for some of those there are no easy failsafes. Or none that would actually help.
A polar bear is a cartesian bear after a coordinate transform.
Perhaps it does, but it failed. What it needs is a device that shuts down the UPS if the device that shuts down the UPS when the battery fails fails.
'Smart' UPS'es from various vendors properly attached to a server or network SNMP trap will tell you when its batteries are going bad (opr the telltale signs of the same). I think you guys need to step up from the $99 Costco UPS'es into something a little more reliable. Home consumer grade electronics have NO place in a commercial environment really (they can be pressed into service in a pinch).
Please return to reality and stop waving numbers about just to win an argument, what is important is actually WHAT THE NUMBERS MEAN and not clueless numerology.
Obviously tape and hard disks are used in different ways - so with tape reliability is not is you can run the same tape continously for years, it's if you can write it, store it and then read it years later. Hard disks are not so reliable by that measure. You can not rely on them. In this case that is what reliability actually MEANS even if you try to win an argument by saying that is longevity instead.
The last comment has things reversed as you would know if you've had the misfortune of multiple failures with optical media or had actaully read anything about the experiences of others.
For many clients I've seen (and I work in IT outsourcing company), anything what costs more than $5 causes a month of paper ping-pong, 'cause you tell them "This is critical, your server will go down if you not replace this", they: "Oh, okay, replace it!", you: "It costs $50", they: "Oh noes! Too much!!! Can do something other?". No bitch, your single HDD is failing and I no IT Wizard to magically repair it.