Slashdot Mirror
The Most Dangerous Server Rooms
Ymerej writes "The Register has an article on dangerous server rooms. Have you seen worse?" Perhaps The Register would like a picture of my desk if they really want to be scared.
← Back to Stories (view on slashdot.org)
One of my favourites was actually sitting inside a motor control center, with all sorts of high voltage DC motor starters right behind the main computer terminal. Don't lean back.
ed
As a long time Customer Engineeer for a major manufacture, I've had my share of basement computer rooms and all the subsequent flooding thereof. That being said, my favourite customer had a mainframe in the basement of his 100 year old private house. He was running a time share (remember those) system over 300 baud modems. To enter the place one had to walk beside his wifes pottery kiln. However once inside a really excellent air conditioned room there was the beast, a brand new HP3000 series 2 with 500Mb of disk. He was my favourite because on my first after hours preventive maintenance call - in fact it was the first time I had ever seen the place - his wife called out after a couple of hours.." Phil, dinner is ready !"
Semper ubi sub ubi
I was in a large shared server facility that still used a Halon system, which when released, fills the area suffocating the fire (and any living creatures in the area as well).
Anyway, one day we were working in our cage, when we heard a warning alarm, and saw all the employees running for their lives. Not knowing that the alarm meant the Halon system was about to be activiate, we joined in anyway and ran for the emergency exits.
It turned out the fire alarms were set off by accident by someone drilling and creating dust, and luckily the people on-site disabled the fire supression system before it went off.
_______
2B1ASK1
At my last job we had a network of about 600 local users. Our server room had two racks of equiment on the building's ups, so the racks plugged right into the wall.
:-)
One day we had a broom leaning up against the wall next to two of our cabinet's. Someone bumped the broom, which fell in the long arc that brooms do when they fall along a wall when leaning. One the way, it happened to unplug our two cabinets from the wall. So much for uptime. The place when quiet and we all just stared at each other for a few seconds. This is in an envirnment where downtime isn't really tolerated at all.
Our task the next weekend?
We took a whole package of zip ties, loosened up the plug wall plate, zip tied the plugs around the back of the outlet wall plate with an ungodly amount of zip ties, and screwed the wall plate tight again.
Our version of 120volt twist locks.
Was interesting to hear what people would ask after seeing it for the first time.
Not quite the server room from hell, but the story's on topic.
-Pete
Soccer Goal Plans
to ensure it could withstand the island's regular earthquakes
... yeah ... we just had a 6.2 earthquake and we're rebuilding the server room", then he gave a kinda "is he serious?/boy am I embarassed" pause and gave me his professional opinion: "Oh, okay, that might be the problem." And then the fucking dial-up users. They're on the TV saying We know there has been an earthquake, please do not call the police to tell them there has been an earthquake, try to use the phone for emergencies only!. And our users are trying to get on freaking AOL instant messenger via dial-up. I was half expecting them to say "Oh yeah, I tried to dial in and it doesn't work. The phone works fine because I just called the police to tell them there was an earthquake."
Has anyone else been in a non-earthquake-prone place and then had an earthquake? Here at work our server room was completely unprepared for an earthquake. Some of the machines came off the racks, some of the whole racks fell, our T1s got damaged or disconnected somewhere in the process. The whole disaster showed us how stupid some users can be. First, the T1 provider calls from Boston or something, "Duh, we show that your T1 lines are down, blah blah blah..", "Uh
Anyway, we bolted down all the racks, shame on us for not doing it in the first place.
I work in industrial maintenance and the most interesting electrical problems happen when the weather changes.
120VAC isn't too bad. Connections soaked in water might survive for a while until the corrosion finally breaks it down, melts the wirenuts, etc. Getting shocked by it isn't enough to blow your fingertips off. 240 volts is usually just 110 volts split into two phases, so it doesn't present any worse of a problem. 480 volts is another story...
480 volts gets interesting when the humidity rises and gets absorbed by the dust surrounding breakers and other switching components. Often, it will flash across the phases, vaporizing the debris, and mysteriously tripping the breaker. No one will figure this out until they happen to take a close look at the wiring, and the humidy from their breath will illuminate the brightest flash they have ever seen in their lives.
I've been in two datacenters where there were regularly spaced 'emergency air supply' masks, right next to the regularly spaced fire extinguishers.
Not sure how well they would work in a room full of smoke and halon, but I suppose you could run from station to station, and then somehow pry open the massive firedoor...
"Can of worms? The can is open... the worms are everywhere."
I remember that - it got sealed into a wall.
BTW, the author of the magic switch story is Guy Steele.
Robots are everywhere, and they eat old people's medicine for fuel.
Okay, I looked at it. Heck, for the sake of argument, let's say that I'll believe Google started business made out of Legos.
Can you tell me why the heck there are always blurs of one type or another shielding the front of the box from view?
I work for a network consulting company, so I've seen some pretty funny stuff in the last few years. Here are some right off the top of my head:
One company didn't order a rack mount kit for their KVM switch (some Belkin model), so they duct taped it to the main monitor. No subtle tape loops under the KVM..... they wrapped the tape three or four times around the KVM and the monitor.
Another company was remodeling their server room but neglected to move the servers somewhere else. There was an inch and a half of drywall and sawdust on top of all the network equipment and servers. The circuit boards looked like it had snowed on them.
I'm doing an audit on some systems. I see a motherboard sitting in a cardboard tray (the kind you get when you purchase a 24-pack of Coke from Costco), along with a hard drive, floppy, power supply and network card. No case. No cooling. Turns out it was their PDC and print server. That's quality craftsmanship.
This isn't about server rooms, per-se, but I did some work for a national pizza chain. They had modems at a central site that were supposed to make a phone call to the stores to print out order tickets. We were sent to figure out why they weren't printing. At one site, the printer was on the floor next to the prep counter where they add the toppings. Someone had spilled a good quart of marinara sauce into the printer. They gave the outside of the printer a good once over, but the inside was just nasty.
We were sent out to troubleshoot a voice-over-IP problem at a garden nursery. We arrive on site and lo and behold, there was a dead rat on top of the router. It didn't have anything to do with the problem, but it sure was unexpected.
I love when people don't properly plan their electrical power consumption in their server rooms. I walked into some company's server room, plugged in my laptop to the rack mounted power strip, turned it on, and blew the breaker for two racks of servers.
I watched a wireless network installer gob Liquid Nails onto the back of an Aironet access point and stick it to the ceiling. I hope they never want to upgrade that particular access point.
Any other good stories?
A couple of weeks ago I came accross a couple of photograps of actual telephone exchanges in the streets of Beirut. You just wouldn't believe it, and it took me a few seconds to understand the picture. there were so many wires that you can hardly see the box behind -- kinda like Johnny Mnemonic, except with 10x more wires, and 2 or 3 handsets plugged in seemingly random (or probably not) outlets. I'll post again if I can find it back.
I've got one, heard from an old-timer friend of mine who used to be a field rep (and, at other times, an AIX kernel coder) for some of IBM's big iron.
The situation: The client's systems are crashing, on a regular basis, for no understandable reason. No remote diagnostics work, so they send out my friend.
He gets to the server room, and keeps thinking he's seeing things out the corner of his eye. He tells everyone to leave the room, and turns out the lights. The room glows.
The server room at this place was sitting under a huge radar system. (He had some additional explanation -- used to be a physics major -- but I didn't entirely follow it). They moved the equipment (a substantial undertaking!) and the problem went away.
I've worked tons of places pullin wire...
:)
I worked at mothers cookies in oakland CA for a spell. You think malaysia is dangerous?
I would pull off the 880 by the colliseum,mc arthur blvd? It takes you through a very derilict section of oakland factories.
The entire complex is surrounded by bobwire. If you drive down the street, less than a block, you are in one of the worst neighborhoods in oakland. I had a friend who lived in the neighborhood, he wasn't shittin me either about the danger. I went to his house twice to hang out and that was enough for me.
The server room was cool, about 40degrees all the time so you wore a jacket when you went in there. But pullin wire....Ohhh my god!
I had to run a fiber line from the main building to some office in the back of the bakerery. Now before you get the picture of little mothers running around with cookie sheets and kenmoore ovens you have to understand.... That is not what a huge production cookie plant is.
Imaging a HUGE fricken warehouse with conveyer belts of cookies going everywhere, machinery whiiring and cookies going into boxes and filled with creme and those animal cookies with the dots, all in this HUGE room about the size of a football field.
One end was the mixing end, where they had these mixing machines the size of my garage. Into those would go 50 gallon oil drums of butter, lard. Huge bags of flour being loaded by forklift, ect.
Now at the time, wireless hadn't really made it mainstream. So my mangers convienced mothers fiber would be good since it would provide the best ROI. They were sold and I was sent out to work.
Now the factory was built from steel girders covered with that tin roofing, the stuff that looks like a ruffles potato chip. I got up there to where the top girders are and before my eyes was the most treachorous wire run I ever saw.
Remember what I said about 50 gallon oil drums of lard? Well, when the cookies baked, the lard would vaporize and rise to the ceiling then settle on the steel girders. Over the years a 1/4" layer of lard had deposited 70 feet up in the rafters.
I put my finger in the goop to see how slippery it was.
No friction.
I called the office and told my boss. Later he called one of his friends to subcontract the work out too.
*Disclaimer* Despite the lard, mothers cookies makes a great product, and was an awesome place to work. If you ever get the chance to work there, jump on it, you won't regret it (or the 50cents a bag price for employees
Heh, I've got a 480v story for you. I work in IT for a manufacturing company. We had an apparently faulty air conditioner take out most of the plant, which is no small feat.
First, a little background info. It's an old building. The electrical is a mess of old and new circuits, some three phase 480 delta, some 240/120 single phase, and one major branch circuit even with a high leg. The former maintenence manager was of the mind to "get things working", rather than "get things right". After he was fired, I started helping out a little with the maintenence staff when emergency things came up, since I know a little about electronics, mostly to help them with computerized and digital control systems.
Anyway, from the street, there are 5000 amp fuses, huge suckers, then 1200 amp fuses on a few main branches. From there to a 1200 amp panel breaker for a major section of the plant (the one that the server room is on), along with most of the manufacturing. In that 1200 amp panel there is a 250 amp 3 phase breaker for the air conditioner. This is all 480v delta 3 phase.
Somehow, that air conditioner breaker failed. It vaporized part of the busbar, tripped the 1200 amp panel breaker, and blew the 1200 amp fuse for one of the phases, leave us down a phase. For the benefit of those who do not know, 3 phase motors running with one dropped phase tend to burn up... quickly! Ideally there is an thermal overload circuit to shut them down before that happens, but that doesn't always work. So bang.. the lights are off, and motors start to burn up in various places around the plant.
Once the maintenence staff figures out what is going on, that we are down a phase, they throw the mains on the service entrance panel for that 1200 amp branch. All seems to be good. We just need to replace that 1200 amp fuse and the faulty breaker right?
Heh. Well, it happened that we didn't have any spare 1200 amp fuses. A 1200 amp fuse isn't something you can run down to your hardware store and get. We send an employee to the next town where a store has exactly three of them in stock. $400 each. We tell him to buy all three. He comes back with one.
We replace the fuse, and the maintenence staff replaces the breaker. Upon reenergizing the circuit everything seems fine... until they go to put the protective metal cover back on.
The panel literally explodes. I wasn't in the room at the time thankfully, but the guys that were there say is was bright, loud, and scary. Apparently what happened was pieces of molten busbar had dripped near the bottom breaker in the panel and were close to shorting out the phases. The slight movement caused by putting the cover back on jarred the chunks of metal and shorted out the phases.
So... the 1200 amp panel breaker trips.... but not fast enough to save the upstream 1200 amp fuses near the service entrance. The ones we didn't have spares for. Again. And now it blew all three of them. And the store only had two in stock.
So we get back on the phone. We find another store that has two in stock, so we send an employee out to get all five, from both stores. He gets it right this time.
We finally replace the three fuses, triple check EVERYTHING, and throw all the breakers back on. We had sent all the employees home hours before... they couldn't do anything without power. But we are finally up... nearly 6 hours later.
Needless to say, some things have changed as a result of this, and it really underscored why the former maintenence manager was fired. We called the electrical engineering firm that had most recently surveyed our power systems, and had them run some more short circuit computer simulations, things like that.
Upon reading their report, I learn that our service panel has a ground fault interrupter, but it was turned all the way up to 1500 amps to prevent nusience trips, after it tripped several times due to our really bad "normal" phase imbalances.
Things are definitely improving, and we are much safer now then we ever were. It goes to show how one bad maintenence manager with a reign of terror, and a long tenure, can really screw things up though. I compare it to a programmer that never comments their code, and uses lots of goto statements, only the stakes are much higher.
I've had enough abrasive sigs. Kittens are cute and fuzzy.
Ouch. Every place I worked at had 14.4KV branched out to several substations fused at 90A, which was good for at least 1.6MW, which branched out into even smaller substations. A failure at one point was rarely noticed elsewhere. Except for the occasional exploding capacitor on the pole outside. Worked great for years with few surprises...
One day I would find out why the 14.4KV fiberglass-epoxy reinforced fuses had mufflers installed on them. Remember that electrical current resists change. If the circuit breaks, the magnetic field surrounding the current collapses and increases the voltage until it goes *somewhere.* Each substation transformer was the magnetic equivalent to a ten foot tall capacitor. Well, if the fuse blows, the the remaining energy in the transformer's magnetic field immediately collapses (the magnetic equivalent of a ten foot tall capacitor) and detonates the fuse filament. This muffler vents this energy harmlessly into the substation as heat without blowing the panels off.
One day, when turning back on the power from vacation, we would find one of these fuses didn't have its muffler installed... And we would learn how things would *seem* to work on two phases.
We would try to install more fuses without the muffler on that phase. The magnetic field was strong enough to pop the fuse out of its holder when the switch was thrown. A wire tie solved the problem while parts were being ordered.
Moral of the story: if you work on high voltage equipment, always leave it as you found it.