Slashdot Mirror
Server Room Smells Can Be an Early Warning
Barence writes "As embarrassing as it may seem, an eggy smell in a server room needn't mean broaching the delicate subject of hygiene with a colleague. It can actually be a signal that something is about to go wrong with your server setup, as this consultant discovered after days of assuming questionable personal habits were to blame. The culprit? An expiring UPS device, sending out its own unique warning signal."
Sulfur Dioxide. Ventilate, replace or recondition battery. If the egg smell is strong and you quit smelling it, that's olifactory fatigue and lethal levels of the gas exist.
Uhhhmmm - it isn't just computers. If I notice an odd smell when I walk through the plant, I investigate. Our plant makes plastic products, and 2/3 of the time, the odd smell is just overheated plastic. But, the other 1/3 finds a problem of one sort, or another. Overheating oils are bad news, overheating capacitors are more bad news - actually, ANYTHING hot enough to give off an odor is bad news. Three weeks ago, we had a machine that was kicking our asses - the mold wouldn't open either manually, or in automatic. 4 of us went over that machine from one end to the other, multiple times. Ohmeters and voltmeters said that everything was just fine, believe it or not. Finally, I caught a whiff of something funky, opened up a solenoid from which the odor seemed to be coming, and found that half of the windings were burnt and shorting.
The sense of smell is a valuable tool in troubleshooting and maintenance, unless you ignore it.
"Windows is like the faint smell of piss in a subway: it's there, and there's nothing you can do about it." - Charlie Br
I've never physically been inside a data center, but I'd have thought that the locales would have really good ventilation, that would simply shut close (or rely on gas weight and gravity) if the halon system or equivalent would need turning on. The ventilation is in fact so bad, there can be a gas buildup so severe you need to (according to posters above me) go in with hazmat gear?
Emotions! In your brain!
From the clasic BOFH :)
"The admin gene," the PFY explains. "The ability to recognise things that users don't. A slight flicker of lighting, a whiff of hot component in the air, a fractional change in the pitch of a cooling fan - all of which the garden variety user misses in the headlong rush to read their email."
http://www.theregister.co.uk/2008/07/04/bofh_2008_episode_24/
Tapes are traditionally used because they are much cheaper, not because they are more reliable (which they are not)
In fact, hard drives are orders of magnitude more reliable than tapes.
I've actually yet to meet anyone whose ever done a successful restore of any significant amount of data from tape, beyond test restores of some files, without running into problems that prevented the restore from succeeding properly.
Tapes do not last very long, wear and tear is huge, and organizations generally like to re-use old tapes, which makes matters even worse.
Having worked with various standalone and both older and newer rackmount models, I would have to say that one of the most frustrating bits about APC's designs are that their battery compartments seem to be engineered to assume that batteries never bulge.
On something like a standalone 1400, any bulging of the batteries tends to increase their width, jamming the batteries against the sides like Comic Book Guy trying to get through a turnstile.
On the older 3U 1400's, the front of the chassis is cut-to-battery-size, and being constrained spacewise front-to-back, bulging usually happens widthwise across the batteries, forcing them bigger than the opening. This means that even if you remove the chassis top, you can't just get in there with a crowbar and forcibly eject them out the front chassis opening, you still need to unscrew the inner battery compartment partition (and maybe even remove it).
This newer 2U 1400, though, wow, what a pain... the chassis is an artfully bent sheet of steel. If you cannot get the drawer to slide out of the battery compartment, the inner battery compartment partition is actually riveted to the chassis, so you have to drill out the rivets, and even after you get the partition removed, you only have one option for which battery to remove first, and they're taped down, so you have to use something like a crowbar to get them out, which is just vaguely scary because all the wiring for the battery tray surrounds the batteries in all the points where you need to exert force to break the tape adhesive.
It's almost like they want it to be frustrating as all hell (maybe so you just replace the whole UPS?) while looking like they've tried to make it new and easy with their wonderful battery tray.
Why does the UPS not have a fail safe that kills it when the battery goes bad to stop a fire?
Far too many people rely on performance metrics and alarms. You're one of the ones who actually pays some attention :P
Any time you enter the DC you should take stock:
1. What do you hear? Perhaps an alarm through all the server noise? Unusually loud fans/ACUs? Anything unusually quiet? Other noises? (I 'predicted' an ACU failure because I heard the fan belt rubbing on something lightly shizz-shizz-shizz-shizz-shizz...)
2. What do you smell? This article basically points this out. Could be leaking ACU coolant. Batteries dying. Burning server. Overloaded circuit, etc.
3. What do you see? Yea, stupid I know but - does that corner of the room appear slightly dimmer? Better go check it out, a rack might be down and you haven't noticed yet.
4. What do you feel? Vibrations through the floor? Could be an ACU about to pop a fan belt or blow a compressor.
5. What do you feel further? Unusually dry or humid air? Temperatures etc.
In short, you should be using every sense except taste and direct tactile feel. Anything shorter and you just aren't paying full attention.
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...