Slashdot Mirror
The Most Dangerous Server Rooms
Ymerej writes "The Register has an article on dangerous server rooms. Have you seen worse?" Perhaps The Register would like a picture of my desk if they really want to be scared.
← Back to Stories (view on slashdot.org)
One of my favourites was actually sitting inside a motor control center, with all sorts of high voltage DC motor starters right behind the main computer terminal. Don't lean back.
ed
...but one Xserve could probably replace the whole mess in some of those pictures.
As a long time Customer Engineeer for a major manufacture, I've had my share of basement computer rooms and all the subsequent flooding thereof. That being said, my favourite customer had a mainframe in the basement of his 100 year old private house. He was running a time share (remember those) system over 300 baud modems. To enter the place one had to walk beside his wifes pottery kiln. However once inside a really excellent air conditioned room there was the beast, a brand new HP3000 series 2 with 500Mb of disk. He was my favourite because on my first after hours preventive maintenance call - in fact it was the first time I had ever seen the place - his wife called out after a couple of hours.." Phil, dinner is ready !"
Semper ubi sub ubi
I was in a large shared server facility that still used a Halon system, which when released, fills the area suffocating the fire (and any living creatures in the area as well).
Anyway, one day we were working in our cage, when we heard a warning alarm, and saw all the employees running for their lives. Not knowing that the alarm meant the Halon system was about to be activiate, we joined in anyway and ran for the emergency exits.
It turned out the fire alarms were set off by accident by someone drilling and creating dust, and luckily the people on-site disabled the fire supression system before it went off.
_______
2B1ASK1
is going on here. That photo on the main page this article links to is the same one that George Ziemann has on his site from the Ebay Vs. Musician article earlier.
I don't see any indication that this is supposed to be his server room. So who's lying?
It's all going according to
At my last job we had a network of about 600 local users. Our server room had two racks of equiment on the building's ups, so the racks plugged right into the wall.
:-)
One day we had a broom leaning up against the wall next to two of our cabinet's. Someone bumped the broom, which fell in the long arc that brooms do when they fall along a wall when leaning. One the way, it happened to unplug our two cabinets from the wall. So much for uptime. The place when quiet and we all just stared at each other for a few seconds. This is in an envirnment where downtime isn't really tolerated at all.
Our task the next weekend?
We took a whole package of zip ties, loosened up the plug wall plate, zip tied the plugs around the back of the outlet wall plate with an ungodly amount of zip ties, and screwed the wall plate tight again.
Our version of 120volt twist locks.
Was interesting to hear what people would ask after seeing it for the first time.
Not quite the server room from hell, but the story's on topic.
-Pete
Soccer Goal Plans
to ensure it could withstand the island's regular earthquakes
... yeah ... we just had a 6.2 earthquake and we're rebuilding the server room", then he gave a kinda "is he serious?/boy am I embarassed" pause and gave me his professional opinion: "Oh, okay, that might be the problem." And then the fucking dial-up users. They're on the TV saying We know there has been an earthquake, please do not call the police to tell them there has been an earthquake, try to use the phone for emergencies only!. And our users are trying to get on freaking AOL instant messenger via dial-up. I was half expecting them to say "Oh yeah, I tried to dial in and it doesn't work. The phone works fine because I just called the police to tell them there was an earthquake."
Has anyone else been in a non-earthquake-prone place and then had an earthquake? Here at work our server room was completely unprepared for an earthquake. Some of the machines came off the racks, some of the whole racks fell, our T1s got damaged or disconnected somewhere in the process. The whole disaster showed us how stupid some users can be. First, the T1 provider calls from Boston or something, "Duh, we show that your T1 lines are down, blah blah blah..", "Uh
Anyway, we bolted down all the racks, shame on us for not doing it in the first place.
Think you're safe when you've got your computer room all tidy and clean?
Forget it. We had a problem with the airconditioning at a medium size company in Delft which had it's heat-outlets on the top of the roof.
These outlets were not so well protected to cold as was shown two years ago when, after a freezingly cold weekend we came into the server room and it was really boiling hot. Problem was the huge ventilators on the roof were stuck frozen.
It was the only time we had all windows and doors open in the middle of winter. But this cold could well have started a fire.
sig not found
I get a call from the folks on the 5th floor saying there's water coming out from under the door (they don't have access to the room). Being the onsite telephony person, I rush up wondering what the hell is going on. Sure enough, water has soaked the carpet around the door. Opening the door I see as much as 5cm (2in) of water sitting around the base of the switch and various servers connected to the switch. All in all, probably around $300K worth of equipment, and I don't dare go in, because there are power cords lying in the water. Finally get the power turned off, the water out, clean everything up. All in all, costed us a couple hundred dollars for some new cables, one monitor, and various odds and ends.
Apparently what happened was someone on the next floor up was in a bathroom, turned on a faucet and forgot about it. The water managed to move about 5m (15ft) down the hall before deciding to pour out into our telephony room!
I work in industrial maintenance and the most interesting electrical problems happen when the weather changes.
120VAC isn't too bad. Connections soaked in water might survive for a while until the corrosion finally breaks it down, melts the wirenuts, etc. Getting shocked by it isn't enough to blow your fingertips off. 240 volts is usually just 110 volts split into two phases, so it doesn't present any worse of a problem. 480 volts is another story...
480 volts gets interesting when the humidity rises and gets absorbed by the dust surrounding breakers and other switching components. Often, it will flash across the phases, vaporizing the debris, and mysteriously tripping the breaker. No one will figure this out until they happen to take a close look at the wiring, and the humidy from their breath will illuminate the brightest flash they have ever seen in their lives.
Some of the more complex and heafty NOC's I've heard about actually have halon, or similarly based fire prevention systems for the entire room. Auto sealing doors and the works. The old "you have 20 seconds to exit the room" plot device brought to life.
I've been in two datacenters where there were regularly spaced 'emergency air supply' masks, right next to the regularly spaced fire extinguishers.
Not sure how well they would work in a room full of smoke and halon, but I suppose you could run from station to station, and then somehow pry open the massive firedoor...
"Can of worms? The can is open... the worms are everywhere."
I remember that - it got sealed into a wall.
BTW, the author of the magic switch story is Guy Steele.
Robots are everywhere, and they eat old people's medicine for fuel.
Okay, I looked at it. Heck, for the sake of argument, let's say that I'll believe Google started business made out of Legos.
Can you tell me why the heck there are always blurs of one type or another shielding the front of the box from view?
I work for a network consulting company, so I've seen some pretty funny stuff in the last few years. Here are some right off the top of my head:
One company didn't order a rack mount kit for their KVM switch (some Belkin model), so they duct taped it to the main monitor. No subtle tape loops under the KVM..... they wrapped the tape three or four times around the KVM and the monitor.
Another company was remodeling their server room but neglected to move the servers somewhere else. There was an inch and a half of drywall and sawdust on top of all the network equipment and servers. The circuit boards looked like it had snowed on them.
I'm doing an audit on some systems. I see a motherboard sitting in a cardboard tray (the kind you get when you purchase a 24-pack of Coke from Costco), along with a hard drive, floppy, power supply and network card. No case. No cooling. Turns out it was their PDC and print server. That's quality craftsmanship.
This isn't about server rooms, per-se, but I did some work for a national pizza chain. They had modems at a central site that were supposed to make a phone call to the stores to print out order tickets. We were sent to figure out why they weren't printing. At one site, the printer was on the floor next to the prep counter where they add the toppings. Someone had spilled a good quart of marinara sauce into the printer. They gave the outside of the printer a good once over, but the inside was just nasty.
We were sent out to troubleshoot a voice-over-IP problem at a garden nursery. We arrive on site and lo and behold, there was a dead rat on top of the router. It didn't have anything to do with the problem, but it sure was unexpected.
I love when people don't properly plan their electrical power consumption in their server rooms. I walked into some company's server room, plugged in my laptop to the rack mounted power strip, turned it on, and blew the breaker for two racks of servers.
I watched a wireless network installer gob Liquid Nails onto the back of an Aironet access point and stick it to the ceiling. I hope they never want to upgrade that particular access point.
Any other good stories?
These are actually pretty common. I remember the university's data center was like this, there were 3 large red buttons that would temporarily stop the halon release so that you could get out or do something to the servers before the room was flooded. My current employers datacenter sadly had only high temp overhead sprinklers, by the time they go off its more about saving the building as the servers are already slag.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
A couple of weeks ago I came accross a couple of photograps of actual telephone exchanges in the streets of Beirut. You just wouldn't believe it, and it took me a few seconds to understand the picture. there were so many wires that you can hardly see the box behind -- kinda like Johnny Mnemonic, except with 10x more wires, and 2 or 3 handsets plugged in seemingly random (or probably not) outlets. I'll post again if I can find it back.
I've got one, heard from an old-timer friend of mine who used to be a field rep (and, at other times, an AIX kernel coder) for some of IBM's big iron.
The situation: The client's systems are crashing, on a regular basis, for no understandable reason. No remote diagnostics work, so they send out my friend.
He gets to the server room, and keeps thinking he's seeing things out the corner of his eye. He tells everyone to leave the room, and turns out the lights. The room glows.
The server room at this place was sitting under a huge radar system. (He had some additional explanation -- used to be a physics major -- but I didn't entirely follow it). They moved the equipment (a substantial undertaking!) and the problem went away.
in college i was doing some work in our computer center pulling cable under a presurized floor. it was your standard room of ancient large blue IBM 72" tall units. arranged in classic grid.
then a maintence crew working on something in an attached room did something to start a small fire.
half my body is under the floor -- head first. trying to unknot some cable.
i didnt know anything about the halion (sp?) gas until the rescue worker explained it to me.
i should have sued, but was worried about pizza money from my job.
members are seeing something, your seeing an ad
How about this?
-no fire suppression system. Period. White Plains City Fire Marshall threw a shit fit, threatened to shut the entire place down, but then conveniently forgot about it.
-could not hear the BUILDING fire alarms from inside the room (there were fire alarm annunciators inside the server room for the building system, but they did not work)
Our landlord was a sleazeball who cut every corner and did as little as possible...and probably paid off the City of White Plains, NY fire inspector. The elevators were death traps, too, but what did the landlord care? His office was on the first floor, with a parking space right next to the door- handicapped spaces 100 feet away, and blocked off.
Google was a wiring nightmare, with their older "non-boxed" boxes. I think you even had to move one rack in order to reach another (in one of their cages there). Google has since cleaned up. Then there was Ebay, with the water collecting garbage cans at the end of each row - SC3 cooling was so poor that Ebay had a water cooled system installed (IIRCTICBW).
I've worked tons of places pullin wire...
:)
I worked at mothers cookies in oakland CA for a spell. You think malaysia is dangerous?
I would pull off the 880 by the colliseum,mc arthur blvd? It takes you through a very derilict section of oakland factories.
The entire complex is surrounded by bobwire. If you drive down the street, less than a block, you are in one of the worst neighborhoods in oakland. I had a friend who lived in the neighborhood, he wasn't shittin me either about the danger. I went to his house twice to hang out and that was enough for me.
The server room was cool, about 40degrees all the time so you wore a jacket when you went in there. But pullin wire....Ohhh my god!
I had to run a fiber line from the main building to some office in the back of the bakerery. Now before you get the picture of little mothers running around with cookie sheets and kenmoore ovens you have to understand.... That is not what a huge production cookie plant is.
Imaging a HUGE fricken warehouse with conveyer belts of cookies going everywhere, machinery whiiring and cookies going into boxes and filled with creme and those animal cookies with the dots, all in this HUGE room about the size of a football field.
One end was the mixing end, where they had these mixing machines the size of my garage. Into those would go 50 gallon oil drums of butter, lard. Huge bags of flour being loaded by forklift, ect.
Now at the time, wireless hadn't really made it mainstream. So my mangers convienced mothers fiber would be good since it would provide the best ROI. They were sold and I was sent out to work.
Now the factory was built from steel girders covered with that tin roofing, the stuff that looks like a ruffles potato chip. I got up there to where the top girders are and before my eyes was the most treachorous wire run I ever saw.
Remember what I said about 50 gallon oil drums of lard? Well, when the cookies baked, the lard would vaporize and rise to the ceiling then settle on the steel girders. Over the years a 1/4" layer of lard had deposited 70 feet up in the rafters.
I put my finger in the goop to see how slippery it was.
No friction.
I called the office and told my boss. Later he called one of his friends to subcontract the work out too.
*Disclaimer* Despite the lard, mothers cookies makes a great product, and was an awesome place to work. If you ever get the chance to work there, jump on it, you won't regret it (or the 50cents a bag price for employees
Heh, I've got a 480v story for you. I work in IT for a manufacturing company. We had an apparently faulty air conditioner take out most of the plant, which is no small feat.
First, a little background info. It's an old building. The electrical is a mess of old and new circuits, some three phase 480 delta, some 240/120 single phase, and one major branch circuit even with a high leg. The former maintenence manager was of the mind to "get things working", rather than "get things right". After he was fired, I started helping out a little with the maintenence staff when emergency things came up, since I know a little about electronics, mostly to help them with computerized and digital control systems.
Anyway, from the street, there are 5000 amp fuses, huge suckers, then 1200 amp fuses on a few main branches. From there to a 1200 amp panel breaker for a major section of the plant (the one that the server room is on), along with most of the manufacturing. In that 1200 amp panel there is a 250 amp 3 phase breaker for the air conditioner. This is all 480v delta 3 phase.
Somehow, that air conditioner breaker failed. It vaporized part of the busbar, tripped the 1200 amp panel breaker, and blew the 1200 amp fuse for one of the phases, leave us down a phase. For the benefit of those who do not know, 3 phase motors running with one dropped phase tend to burn up... quickly! Ideally there is an thermal overload circuit to shut them down before that happens, but that doesn't always work. So bang.. the lights are off, and motors start to burn up in various places around the plant.
Once the maintenence staff figures out what is going on, that we are down a phase, they throw the mains on the service entrance panel for that 1200 amp branch. All seems to be good. We just need to replace that 1200 amp fuse and the faulty breaker right?
Heh. Well, it happened that we didn't have any spare 1200 amp fuses. A 1200 amp fuse isn't something you can run down to your hardware store and get. We send an employee to the next town where a store has exactly three of them in stock. $400 each. We tell him to buy all three. He comes back with one.
We replace the fuse, and the maintenence staff replaces the breaker. Upon reenergizing the circuit everything seems fine... until they go to put the protective metal cover back on.
The panel literally explodes. I wasn't in the room at the time thankfully, but the guys that were there say is was bright, loud, and scary. Apparently what happened was pieces of molten busbar had dripped near the bottom breaker in the panel and were close to shorting out the phases. The slight movement caused by putting the cover back on jarred the chunks of metal and shorted out the phases.
So... the 1200 amp panel breaker trips.... but not fast enough to save the upstream 1200 amp fuses near the service entrance. The ones we didn't have spares for. Again. And now it blew all three of them. And the store only had two in stock.
So we get back on the phone. We find another store that has two in stock, so we send an employee out to get all five, from both stores. He gets it right this time.
We finally replace the three fuses, triple check EVERYTHING, and throw all the breakers back on. We had sent all the employees home hours before... they couldn't do anything without power. But we are finally up... nearly 6 hours later.
Needless to say, some things have changed as a result of this, and it really underscored why the former maintenence manager was fired. We called the electrical engineering firm that had most recently surveyed our power systems, and had them run some more short circuit computer simulations, things like that.
Upon reading their report, I learn that our service panel has a ground fault interrupter, but it was turned all the way up to 1500 amps to prevent nusience trips, after it tripped several times due to our really bad "normal" phase imbalances.
Things are definitely improving, and we are much safer now then we ever were. It goes to show how one bad maintenence manager with a reign of terror, and a long tenure, can really screw things up though. I compare it to a programmer that never comments their code, and uses lots of goto statements, only the stakes are much higher.
I've had enough abrasive sigs. Kittens are cute and fuzzy.
Ouch. Every place I worked at had 14.4KV branched out to several substations fused at 90A, which was good for at least 1.6MW, which branched out into even smaller substations. A failure at one point was rarely noticed elsewhere. Except for the occasional exploding capacitor on the pole outside. Worked great for years with few surprises...
One day I would find out why the 14.4KV fiberglass-epoxy reinforced fuses had mufflers installed on them. Remember that electrical current resists change. If the circuit breaks, the magnetic field surrounding the current collapses and increases the voltage until it goes *somewhere.* Each substation transformer was the magnetic equivalent to a ten foot tall capacitor. Well, if the fuse blows, the the remaining energy in the transformer's magnetic field immediately collapses (the magnetic equivalent of a ten foot tall capacitor) and detonates the fuse filament. This muffler vents this energy harmlessly into the substation as heat without blowing the panels off.
One day, when turning back on the power from vacation, we would find one of these fuses didn't have its muffler installed... And we would learn how things would *seem* to work on two phases.
We would try to install more fuses without the muffler on that phase. The magnetic field was strong enough to pop the fuse out of its holder when the switch was thrown. A wire tie solved the problem while parts were being ordered.
Moral of the story: if you work on high voltage equipment, always leave it as you found it.
http://images.e-gerbil.net/ghetto Most of these have not been referenced by any links posted thus far.
I read that one on Computer Stupidities a while ago.
"Programming is like sex - one mistake and you'll have to support it for the rest of your life."
I work at a large truck company in the Northwest. While pulling up the raised floor in the data center one day I found a fax server that the IT department had "lost" more than three years ago. The damn thing still worked and they could telnet into it but no one new where it was. It sat under the floor beneith a conference table with a coil of hundred pair sitting on top of it merrily chugging along. The fax gate had never gone down!
Basically the same story repeated.
I ran a test lab that was basically full of new PCs and PC servers. Except over in the corner is this IBM PS/2 Tower machine that happens to be running Windows NT 3.1 Beta Version
(NT 4.0 had just shipped, so this thing was *old*.)
Furthermore, the only thing keeping the thing running is a piece of plastic shoved into the power button holding it down.
One day, the plastic bit falls out and wirrrr the thing powers down. Within minutes 3 guys I'd never seen before in my life run into the room and we get the plastic shoved back in.
Still not quite sure what that thing was doing, but mad props to Dave Cutler for that version of NT.
I used to work at a glass company in the greater Seattle area (name withheld to protect them). Their server room had wiring on the walls inside a half baked closet (yes it was hot, too). Now, rather than just running Cat-5A or Cat-5B they had both. The wiring in the snakes was Cat-5B and the rest was Cat-5A. Lots of fun when you have to repunch or patch on the fly.
But that is never enough is it? They also ran serial connections through the snakes without using any serial concentrators along the way. All of the wiring was thrown together as was needed with documentation a distant dream rather than a reality.
Knowing that this system was in serious trouble (RAID level 0 with no way to reinstall without calling an out of state vendor to bail us out), and knowing that a migration was looming about a year away (from AIX to Windows and I was happy about it), I proceeded to map the darn thing. Took a week of climbing through broken glass (remember where this was), scaling ladders to find runs that went nowhere, and finding that old runs were ran along the outside of the office (well, before they expanded the office that is).
After making a diagram of where everything went and checking everything twice, my manager started ripping out wires before I could get there. Fortunately he didn't pull anything vital and we remained up and running, but I really wanted to deck him for that.
In the server room I pulled out over 500 feet (I measured it as I didn't believe it) of dead wire. We had only three computers in there but major runs ran down into the electrical room below.
Imagine standing on an electrical transformer while installing a new hub and really hoping that you don't slip onto the trash laden floor. That was fun!
I didn't mention that the server room was directly above the electrical room and that above the server room was an ancient air conditioning unit that would dump its condensation on the server did I? Moving the server protected it from the 'rain', buckets prevented the water from pooling, and the cleaning crew would remove the acoustical tiling when they would rot and fall down.
Ask me if I miss the place.
"Giving money and power to governments is like giving whiskey and car keys to teenage boys." - P.J. O'Rourke
...was one I inherited when I was working for one of the railway companies in the UK. The network at the time was all 10base2 and there was a multiport repeater on every floor, each connected to the main repeater up in the server room by an individual length of thinwire that ran up a central riser.
Some time before I started work there the cable running down to the seventh floor had failed (probably hungry rodents) and my predecessor had come up with a cunning workaround rather than going to the hassle of laying a proper replacement. He had run a thinwire cable out of the window of the server room, down the outside of the building and in through a window on the seventh floor (I really don't want to know how) which was then run along the ceiling using a whole load of bent paperclips rammed into the polystyrene ceiling tiles, and then into the comms cabinet that housed the repeater.
I was told by one of the staff there that this temporary solution had been in place for months, with only occasional outages. Then again, given the fact that the server room had no racks, shelves or airconditioning and the servers were just piled on top of each other with random assortments of keyboards and monitors dotted around, nothing there surprised me.
"Are you being weird, or sarcastic?" said Emma. I said I didn't know because I get the two feelings mixed up.
Finally, the demo was over and we thought we were safe. One Saturday morning at 5AM, after a huge blizzard in NYC, I got a call from building security that there had been a massive flood originating from the residences in an area directly above our data room. Turns out the owner had installed forced-water heaters in the apartments and hadn't insulated the pipes, which had frozen and burst. Although there was about 1/4" accumulation on the floor in the data room, the leak there had missed the nearest cabinet by about five inches. The adjacent NOC room was completely destroyed, but at least it was just monitoring equipment and not production servers and telephones. We somehow got everything ready in time for the market open Monday morning.
By the time we found out the owner planned to build a parking garage in the building's basement, he had already begun demolishing what was left of a restaurant and storage rooms that used to be down there. Keep in mind this is a 100-year old building that had telecom infrastructure for most of the 100 years. When we went down to see what was going on, we found they had demolished suspended ceilings and framed walls that were in some places the only things holding up bundles of 1000-pair copper cables. They had demolished a cinderblock wall to which the 1000-pair splices had been attached. The fiber to Verizon's frame room had only been installed in innerduct, and was lying in the mud and waste. One of our WorldCom 25-pairs (Internet, 800 service, market data, NASDAQ) was running at eye level straight through the middle of the demo area, where forklift drivers had to lift it out of the way each time they passed. The owner had started a war with the building unions, and a couple of suspicious fires had started down there. Oh, and there was loose asbestos in part of the site, so the telcos refused to work until the owner had it abated and environmentally certified.
For the six months it took us to get the owner to clean up the asbestos, for the telcos to install new access away from the demolition, and for us to install steel conduit for our service, we faced the notion that we might be out of business at any point. We did finally get the telecom infrastructure hardened, but the real benefit of this set of circumstances was that they forced us to implemented a real business continuity plan and build an offsite recovery capability in northern Jersey. Although our motivation was due to the landlord's recklessness, the Jersey capability saved our firm during the time we were shut out of our building after 9/11.