Slashdot Mirror
The Most Dangerous Server Rooms
Ymerej writes "The Register has an article on dangerous server rooms. Have you seen worse?" Perhaps The Register would like a picture of my desk if they really want to be scared.
← Back to Stories (view on slashdot.org)
. . .where the city rats are bigger than the IT guys.
And they carry card keys.
-bpl
Now that's how you ensure job security!
- tristan
My server room is so bad not even the rats will go in there
For example, statistics show that people who work in server room almost never catch any STDs. I wonder why that is...
Actually, this is the third article about the subject.
Teenagers these days don't have as much sex as they want each other to think they do.
One of my favourites was actually sitting inside a motor control center, with all sorts of high voltage DC motor starters right behind the main computer terminal. Don't lean back.
ed
It's all about removing floor tiles and then forgetting to set up warning cones. The clearance between our tiles and the concrete floor underneath is a good 4 or 5 feet; I would not want to fall stiff-legged into that.
Interesting side note: apparently finding high-priced Cisco gear not connected to anything is not that uncommon. I've also heard horror stories of guy that traced a cable my hand(toner was on the fritz) that looped 4 times around the data center but wasn't hooked up to shit, on either end. Took him an entire afternoon.
Finally, math books without any of that base 6 crap in them.
Check it out here.
;-)
This is by far to custom case I've ever seen.
Look a bit dangerous though
I'm a slob; I admit it. Even if I didn't, it would be obvious from the way I dress and the state of my desk. But at my job, I am cleaning up the spaghetti mess in the ceiling and trying to lay the wire cleanly from patch panels - switches.
Documenting connections has a real payoff in troubleshooting. But doing stuff on aesthetic grounds is a harder sell. I have a gut sense that a clean layout is important even if you know the destination both ends of a wire whose middle runs through a snarl. Here's what I came up with:
my version of the community policing broken windows theory.
It's psychologically harder to do slipshod, shoddy work if everything around you has been done well. And it's hard to do a proper job if everything else is slipshod. As a matter of housemate politics, it's easier to leave the nth dirty dish in the sink than the first. You are only adding an increment, not changing state.
Doing the Right Thing is contagious. At least, it is among folks I care to work with. Doing the Wrong Thing is catching, too. Morale is higher and people challenge themselves more at a shop that is run well.
That's how I pitched it, and my boss bought it.
You want scary:
Our server room has wobbly floors and tower cases stacked 2 high sitting in the middle of a 16'x16' room, with the more important servers sitting on the top.
We also got a new air conditioner that has an electronic switch and we have problems with brown outs. So in the middle of the summer, the power goes out and the A/C doesn't come back on, usually on a weekend too.
One more thing: brace yourself:
We use Windows servers with IIS!!!!!!!!
As a long time Customer Engineeer for a major manufacture, I've had my share of basement computer rooms and all the subsequent flooding thereof. That being said, my favourite customer had a mainframe in the basement of his 100 year old private house. He was running a time share (remember those) system over 300 baud modems. To enter the place one had to walk beside his wifes pottery kiln. However once inside a really excellent air conditioned room there was the beast, a brand new HP3000 series 2 with 500Mb of disk. He was my favourite because on my first after hours preventive maintenance call - in fact it was the first time I had ever seen the place - his wife called out after a couple of hours.." Phil, dinner is ready !"
Semper ubi sub ubi
I was in a large shared server facility that still used a Halon system, which when released, fills the area suffocating the fire (and any living creatures in the area as well).
Anyway, one day we were working in our cage, when we heard a warning alarm, and saw all the employees running for their lives. Not knowing that the alarm meant the Halon system was about to be activiate, we joined in anyway and ran for the emergency exits.
It turned out the fire alarms were set off by accident by someone drilling and creating dust, and luckily the people on-site disabled the fire supression system before it went off.
_______
2B1ASK1
In rural Indiana, you don't always have space to have a whole room devoted to servers and network equipment, ya know?! But I was still surprised when I visited my former ISPs local point of presence - in one of their employee's one and only bathroom at his house. Photo here. Do some laundry, take a dump, watch some network traffic go by. Uh-huh.
...looks like some new cybernetic monster I imagine will make an appearance in Doom 3.
At my last job we had a network of about 600 local users. Our server room had two racks of equiment on the building's ups, so the racks plugged right into the wall.
:-)
One day we had a broom leaning up against the wall next to two of our cabinet's. Someone bumped the broom, which fell in the long arc that brooms do when they fall along a wall when leaning. One the way, it happened to unplug our two cabinets from the wall. So much for uptime. The place when quiet and we all just stared at each other for a few seconds. This is in an envirnment where downtime isn't really tolerated at all.
Our task the next weekend?
We took a whole package of zip ties, loosened up the plug wall plate, zip tied the plugs around the back of the outlet wall plate with an ungodly amount of zip ties, and screwed the wall plate tight again.
Our version of 120volt twist locks.
Was interesting to hear what people would ask after seeing it for the first time.
Not quite the server room from hell, but the story's on topic.
-Pete
Soccer Goal Plans
You want dangerous, well the only fire prevention we have for our 23 servers, is one halon extinguisher. So if the room goes up, we can save it, as long as we don't want any oxygen...
Xaotik Designs
... when my foot broke through one of those floor tiles in the server room. Funny, someone told me they were high resistence, that must not be exactly the case ;)
That doesn't surprise me. I have a strange feeling many of you have been caught doing this in front of your pc.
So much for my Delta Force keyboard layout ;)
"I can't give you a brain, so I'll give you a diploma" - The Great Oz (blatently stolen sig)
I bet he's the guy that invented tcpdump.
We launched a website with publicity on a live primetime TV show about the internet (in the UK), while the server (singular) was still running under my desk. It was a little while before we moved it out of there and, amazingly, I never accidentally shut down the site with my knee.
Of course, when we did, eventually, move it into a server room, the aircon subsequently broke down and, being an underfunded dotcom, nobody wanted to spring for repairs. We lost at least one server that way (thankfully not a live-facing one).
Our main wiring closet looks a little like the 'not so bad examples' of wiring closets in those pictures - though nothing like the more extreme ones.
The problem is, once the thing gets into that kind of mess, you rarely have the chance to bring down the entire network to repatch all the cables and cable tie them into some kind of order.
Not only that, but if you have loads of trunks and VLAN's configured, putting it all back in the right order can be a total ballache!
"Hey! Unless this is a nude love-in, get the hell off my property!!"
to ensure it could withstand the island's regular earthquakes
... yeah ... we just had a 6.2 earthquake and we're rebuilding the server room", then he gave a kinda "is he serious?/boy am I embarassed" pause and gave me his professional opinion: "Oh, okay, that might be the problem." And then the fucking dial-up users. They're on the TV saying We know there has been an earthquake, please do not call the police to tell them there has been an earthquake, try to use the phone for emergencies only!. And our users are trying to get on freaking AOL instant messenger via dial-up. I was half expecting them to say "Oh yeah, I tried to dial in and it doesn't work. The phone works fine because I just called the police to tell them there was an earthquake."
Has anyone else been in a non-earthquake-prone place and then had an earthquake? Here at work our server room was completely unprepared for an earthquake. Some of the machines came off the racks, some of the whole racks fell, our T1s got damaged or disconnected somewhere in the process. The whole disaster showed us how stupid some users can be. First, the T1 provider calls from Boston or something, "Duh, we show that your T1 lines are down, blah blah blah..", "Uh
Anyway, we bolted down all the racks, shame on us for not doing it in the first place.
I get a call from the folks on the 5th floor saying there's water coming out from under the door (they don't have access to the room). Being the onsite telephony person, I rush up wondering what the hell is going on. Sure enough, water has soaked the carpet around the door. Opening the door I see as much as 5cm (2in) of water sitting around the base of the switch and various servers connected to the switch. All in all, probably around $300K worth of equipment, and I don't dare go in, because there are power cords lying in the water. Finally get the power turned off, the water out, clean everything up. All in all, costed us a couple hundred dollars for some new cables, one monitor, and various odds and ends.
Apparently what happened was someone on the next floor up was in a bathroom, turned on a faucet and forgot about it. The water managed to move about 5m (15ft) down the hall before deciding to pour out into our telephony room!
I work in industrial maintenance and the most interesting electrical problems happen when the weather changes.
120VAC isn't too bad. Connections soaked in water might survive for a while until the corrosion finally breaks it down, melts the wirenuts, etc. Getting shocked by it isn't enough to blow your fingertips off. 240 volts is usually just 110 volts split into two phases, so it doesn't present any worse of a problem. 480 volts is another story...
480 volts gets interesting when the humidity rises and gets absorbed by the dust surrounding breakers and other switching components. Often, it will flash across the phases, vaporizing the debris, and mysteriously tripping the breaker. No one will figure this out until they happen to take a close look at the wiring, and the humidy from their breath will illuminate the brightest flash they have ever seen in their lives.
I'll have to drag my camera up into my attic to take a picture of the mess up there one day. I've 3 machines that run 24/7 up there, covered in dust, various other crap and spiders (and boy do those spiders grow big up there!)
I put them up there as I got sick of the noise, then ran ethernet cabling down to sockets around the place. My main web server (see sig) has been happily running up there for about 1 1/2 years now - barring the odd upgrade. It's not pleasant to open up a case full of dust and cobwebs, BTW!
Code, Hardware, stuff like that.
Some of the more complex and heafty NOC's I've heard about actually have halon, or similarly based fire prevention systems for the entire room. Auto sealing doors and the works. The old "you have 20 seconds to exit the room" plot device brought to life.
Well, that's sorta the whole point of halon (and thus, why isn't used anymore.)
Those are nothing. NOTHING, I tell you!
I've got a set of routers located in a crawlspace where the only way to get to them is to walk across boards spanning small metal beams that were put in to hold a suspended false ceiling. One missed step and you'll drop right through the ceiling, AND IT'S A 2 STORY DROP! Once I dropped a power pack while replacing it and nearly killed a gal working below. Power pack exploded like a bomb when it hit.
We recently had a "security audit" where they recommended we should mount those routers in a locked cabinet for increased security. Not a mention about the boards, lack of handrail, safety net, etc. Heck, who needs a locked cabinet? Just remove one of the boards and NO ONE can get to those routers, not even the people who are supposed to maintain them!
Back when we used thinnet one of the managers didn't like stringing new coax through the building whenever we remodeled or moved people, so he had us cut all the coaxes to length PLUS 25 FEET! He figured if someone moved we could pull back the excess and save time. The cables all terminated in what came to be called the spaghetti room, from the coils of coax all over the floor. We had to step over all the coaxes to get to the routers and hubs. Eventually, the coaxes got damaged from all the abuse and had to be cut off to length anyway, but for several years it was a serious tripping hazard for anyone who entered that room.
Beta sux! Join the Slashcott! http://hardware.slashdot.org/comments.pl?sid=4760465&cid=46173047
I've been in two datacenters where there were regularly spaced 'emergency air supply' masks, right next to the regularly spaced fire extinguishers.
Not sure how well they would work in a room full of smoke and halon, but I suppose you could run from station to station, and then somehow pry open the massive firedoor...
"Can of worms? The can is open... the worms are everywhere."
My personal favorite was building a small network out in a field. We set up our four machines [286's] in the dirt, got our power from a generator being towed by a five-ton and wired together on a 10Base2 network. For the first day or so the only shelter we had for the machines was a tarp that we pulled over them when it started to rain.
Lying on the ground, underneath the leaky tarp, hoping that I did not get electrocuted, or if I did that I would not be held accountable for the damaged equipment [trust me, this was not my idea], I decided that re-enlistment was not a great idea.
[former] USMC geek
That's right you read that correctly.
I once visited a client who had his server racks in an old lockerroom shower. This would not have been so bad except when one of my co-workers leaned against the wall and hit the valve we discovered that the pipes hadn't been capped by just had the shower heads removed....that's right three full racks of equipment in a live shower. =)
I was told this story from a reliable source...
An HP technician (yup they have at least one) was restoring the data to a customer's fileserver but the backup software was asking for tape#2. The customer only ever had one backup tape that they recalled, so they were quite perplexed until the security guard entered the server room...
Apparently every morning around 3am when he made his rounds he found the backup server screen blinking "insert next tape" -- The security guard proudly said that he was pushing in the tape for at least six months now...
From excellent karma to terible karma with a single +5 funny post...
I work for a network consulting company, so I've seen some pretty funny stuff in the last few years. Here are some right off the top of my head:
One company didn't order a rack mount kit for their KVM switch (some Belkin model), so they duct taped it to the main monitor. No subtle tape loops under the KVM..... they wrapped the tape three or four times around the KVM and the monitor.
Another company was remodeling their server room but neglected to move the servers somewhere else. There was an inch and a half of drywall and sawdust on top of all the network equipment and servers. The circuit boards looked like it had snowed on them.
I'm doing an audit on some systems. I see a motherboard sitting in a cardboard tray (the kind you get when you purchase a 24-pack of Coke from Costco), along with a hard drive, floppy, power supply and network card. No case. No cooling. Turns out it was their PDC and print server. That's quality craftsmanship.
This isn't about server rooms, per-se, but I did some work for a national pizza chain. They had modems at a central site that were supposed to make a phone call to the stores to print out order tickets. We were sent to figure out why they weren't printing. At one site, the printer was on the floor next to the prep counter where they add the toppings. Someone had spilled a good quart of marinara sauce into the printer. They gave the outside of the printer a good once over, but the inside was just nasty.
We were sent out to troubleshoot a voice-over-IP problem at a garden nursery. We arrive on site and lo and behold, there was a dead rat on top of the router. It didn't have anything to do with the problem, but it sure was unexpected.
I love when people don't properly plan their electrical power consumption in their server rooms. I walked into some company's server room, plugged in my laptop to the rack mounted power strip, turned it on, and blew the breaker for two racks of servers.
I watched a wireless network installer gob Liquid Nails onto the back of an Aironet access point and stick it to the ceiling. I hope they never want to upgrade that particular access point.
Any other good stories?
A couple of weeks ago I came accross a couple of photograps of actual telephone exchanges in the streets of Beirut. You just wouldn't believe it, and it took me a few seconds to understand the picture. there were so many wires that you can hardly see the box behind -- kinda like Johnny Mnemonic, except with 10x more wires, and 2 or 3 handsets plugged in seemingly random (or probably not) outlets. I'll post again if I can find it back.
I've got one, heard from an old-timer friend of mine who used to be a field rep (and, at other times, an AIX kernel coder) for some of IBM's big iron.
The situation: The client's systems are crashing, on a regular basis, for no understandable reason. No remote diagnostics work, so they send out my friend.
He gets to the server room, and keeps thinking he's seeing things out the corner of his eye. He tells everyone to leave the room, and turns out the lights. The room glows.
The server room at this place was sitting under a huge radar system. (He had some additional explanation -- used to be a physics major -- but I didn't entirely follow it). They moved the equipment (a substantial undertaking!) and the problem went away.
I have always thought that how organized a companies wiring is - is a direct reflection of the staff that works there.
I know that when i moved into one company in Redwood City - the network wsa a nightmare. We had rooms that looke like that - but over the next 2 years we replaced almost every wire on that network - and demanded budget for proper closet setups - and got it.
We eliminated all those closets that looked like that, and learned one hell of a lot in the process.
I think that if your closets look like that - you are asking for fire - and it shows just how lazy you really are. No arguments of "I'm too busy" allowed - it just means your a lazy slob period.
Actually, it isn't used any more because it was found to be ozone-depleting and banned by international agreement.
p ://palimpsest.stanford.edu/waac/wn/wn15/wn15-2 /wn15-208.html
http://www.haifire.com/press/halon_rep.htm
htt
Several of the ozone-friendly replacements (I forget their unfriendly alphanumeric designations) still work by sucking all of the oxygen out ofthe room.
I'm sure that some of you have worked in large server rooms with a big red emergency power shutoff button on the wall...
:)
At my old university, one of these server rooms was emptied as new, smaller hardware came available and the room was no longer needed. They turned this room into an office for a student organization... leaving the large red button, but taking the "Emergency Shutoff" sign.
This unlabeled button sat neglected on the wall of this little office for about 7 or 8 years until one day a curious student just had to find out what the button did...
The network for all of the engineering schools at this university of 36,000 students went down for most of a day..
The best part is that the button is still in the little office with the students, and it is still unlabeled yet fully functional... They did hide it behind a file cabinet, though
There are 10 types of people in this world, those who can count in binary and those who can't.
I was in a large shared server facility that still used a Halon system, which when released, fills the area suffocating the fire (and any living creatures in the area as well).
Look -- geeks sue for the wrong kind of keyboard -- ergonomics = safety after all -- do you really think there is just this dangerous gas ready to be released where you work and you have exactly 20 seconds to escape before you die?
Taken from the first site that came up on Google...
"Three things must come together at the same time to start a fire. The first ingredient is fuel (anything that can burn), the second is oxygen (normal breathing air is ample) and the last is an ignition source (high heat can cause a fire even without a spark or open flame). Traditionally, to stop a fire you need to remove one side of the triangle - the ignition, the fuel or the oxygen. Halon adds a fourth dimension to fire fighting - breaking the chain reaction. It stops the fuel, the ignition and the oxygen from dancing together by chemically reacting with them. Many people believe that Halon displaces the air out of the area it is dispensed in. Wrong! Even for the toughest hazards, less than an 8% concentration by volume is required. There is still plenty of air to use in the evacuation process. (Emphasis added by me)
I'm not an expert, but urban myths bug me...
I would have to say that explosives are the most abused technology in all of history.
I've worked tons of places pullin wire...
:)
I worked at mothers cookies in oakland CA for a spell. You think malaysia is dangerous?
I would pull off the 880 by the colliseum,mc arthur blvd? It takes you through a very derilict section of oakland factories.
The entire complex is surrounded by bobwire. If you drive down the street, less than a block, you are in one of the worst neighborhoods in oakland. I had a friend who lived in the neighborhood, he wasn't shittin me either about the danger. I went to his house twice to hang out and that was enough for me.
The server room was cool, about 40degrees all the time so you wore a jacket when you went in there. But pullin wire....Ohhh my god!
I had to run a fiber line from the main building to some office in the back of the bakerery. Now before you get the picture of little mothers running around with cookie sheets and kenmoore ovens you have to understand.... That is not what a huge production cookie plant is.
Imaging a HUGE fricken warehouse with conveyer belts of cookies going everywhere, machinery whiiring and cookies going into boxes and filled with creme and those animal cookies with the dots, all in this HUGE room about the size of a football field.
One end was the mixing end, where they had these mixing machines the size of my garage. Into those would go 50 gallon oil drums of butter, lard. Huge bags of flour being loaded by forklift, ect.
Now at the time, wireless hadn't really made it mainstream. So my mangers convienced mothers fiber would be good since it would provide the best ROI. They were sold and I was sent out to work.
Now the factory was built from steel girders covered with that tin roofing, the stuff that looks like a ruffles potato chip. I got up there to where the top girders are and before my eyes was the most treachorous wire run I ever saw.
Remember what I said about 50 gallon oil drums of lard? Well, when the cookies baked, the lard would vaporize and rise to the ceiling then settle on the steel girders. Over the years a 1/4" layer of lard had deposited 70 feet up in the rafters.
I put my finger in the goop to see how slippery it was.
No friction.
I called the office and told my boss. Later he called one of his friends to subcontract the work out too.
*Disclaimer* Despite the lard, mothers cookies makes a great product, and was an awesome place to work. If you ever get the chance to work there, jump on it, you won't regret it (or the 50cents a bag price for employees
would someone please flush the mail queue?
oh no, the logs are getting backed up!
I have no special gift, I am only passionately curious. --Albert Einstein
Well, that's sorta the whole point of halon (and thus, why isn't used anymore.)
Having thoroughly studied the subject, umm, no, you're wrong.
Halon is no longer used because:
a)it was poisonous
b)it was ozone depleting
The number one cause of death in server room fires is toxic smoke inhalation from burning plastic/paint/wire insulation/circuit board components/etc. Why? People go in to rescue backup tapes, and are rather quickly incapacitated when they inhale the smallest amount of toxic fumes. Folks- NEVER ENTER A BURNING ROOM WITH A FIRE. PERIOD. Backup tapes are NEVER more important than your life. People think "oh, I'll hold my breath." They don't think about how far it may be, or they may get disoriented in the smoke..and they think "oh, just a little breath, i need to breathe." That gets you coughing. Then you die. Smoke inhalation is actually(I believe) the number one cause of death in ALL fires.
Let's not even start about what opening a door into a hot, mostly sealed room letting in a ton of oxygen does.
My "Blue Book" (computer security, mostly focuses on physical) says to NEVER store backup tapes in the same room as server equipment, PERIOD, because the temptation to rescue them is too great.
CO2 is the agent that suffocates you...in that case, yes, the point is to remove as much oxygen from the room, and yes, you will suffocate very quickly.
FM200 is the most common substitute and is fairly harmless, but works on pushing oxygen from the room, and because it is much heavier than air, requires very good sealing of the room in order to work properly. It is next to useless on fires high up in the room unless the system is very well designed.
An alternative is INTERGEN, which is 100% harmless and is mixed specifically to make your body naturally breathe faster, which is a good thing if smoke isn't really a problem and you just need to keep from suffocating. It also works mostly on rapid temperature change and chemically interfering with combustion(not by removing oxygen), and its weight is much closer to natural air, so it doesn't sink.
Thus, the server room doesn't have to be sealed(Halon and FM200 both require it), but the amount of extinguishing gas is greater and piping is slightly different.
In server rooms of sufficient size that use Halon, CO2, etc... SCBA units are usually recommended if not required. Interrupt switches and delays are very common as well, although both are designed mostly with reducing false discharges, which can cost $5-10k per 1,000 square feet.
More and more commonly, dry-pipe water systems are being used, particularly in 24x7x365 managed buildings. The idea is that if the fire is small, an alarm goes off and a countdown starts. On site staff can take appropriate action(cut power, run in with a fire extinguisher, that sort of thing) or, if it's bad, let the system come on. They're usually zoned triggered and fogging nozzles are used to minimize equipment damage.
Unfortunately, for your average 1,000 sq ft room, you're looking at about $30k to $50k, not including modifications to the room to get it sealed. Dry pipe water is less expensive, but one false alarm could cost you 10 times the system's cost in equipment alone.
Heh, I've got a 480v story for you. I work in IT for a manufacturing company. We had an apparently faulty air conditioner take out most of the plant, which is no small feat.
First, a little background info. It's an old building. The electrical is a mess of old and new circuits, some three phase 480 delta, some 240/120 single phase, and one major branch circuit even with a high leg. The former maintenence manager was of the mind to "get things working", rather than "get things right". After he was fired, I started helping out a little with the maintenence staff when emergency things came up, since I know a little about electronics, mostly to help them with computerized and digital control systems.
Anyway, from the street, there are 5000 amp fuses, huge suckers, then 1200 amp fuses on a few main branches. From there to a 1200 amp panel breaker for a major section of the plant (the one that the server room is on), along with most of the manufacturing. In that 1200 amp panel there is a 250 amp 3 phase breaker for the air conditioner. This is all 480v delta 3 phase.
Somehow, that air conditioner breaker failed. It vaporized part of the busbar, tripped the 1200 amp panel breaker, and blew the 1200 amp fuse for one of the phases, leave us down a phase. For the benefit of those who do not know, 3 phase motors running with one dropped phase tend to burn up... quickly! Ideally there is an thermal overload circuit to shut them down before that happens, but that doesn't always work. So bang.. the lights are off, and motors start to burn up in various places around the plant.
Once the maintenence staff figures out what is going on, that we are down a phase, they throw the mains on the service entrance panel for that 1200 amp branch. All seems to be good. We just need to replace that 1200 amp fuse and the faulty breaker right?
Heh. Well, it happened that we didn't have any spare 1200 amp fuses. A 1200 amp fuse isn't something you can run down to your hardware store and get. We send an employee to the next town where a store has exactly three of them in stock. $400 each. We tell him to buy all three. He comes back with one.
We replace the fuse, and the maintenence staff replaces the breaker. Upon reenergizing the circuit everything seems fine... until they go to put the protective metal cover back on.
The panel literally explodes. I wasn't in the room at the time thankfully, but the guys that were there say is was bright, loud, and scary. Apparently what happened was pieces of molten busbar had dripped near the bottom breaker in the panel and were close to shorting out the phases. The slight movement caused by putting the cover back on jarred the chunks of metal and shorted out the phases.
So... the 1200 amp panel breaker trips.... but not fast enough to save the upstream 1200 amp fuses near the service entrance. The ones we didn't have spares for. Again. And now it blew all three of them. And the store only had two in stock.
So we get back on the phone. We find another store that has two in stock, so we send an employee out to get all five, from both stores. He gets it right this time.
We finally replace the three fuses, triple check EVERYTHING, and throw all the breakers back on. We had sent all the employees home hours before... they couldn't do anything without power. But we are finally up... nearly 6 hours later.
Needless to say, some things have changed as a result of this, and it really underscored why the former maintenence manager was fired. We called the electrical engineering firm that had most recently surveyed our power systems, and had them run some more short circuit computer simulations, things like that.
Upon reading their report, I learn that our service panel has a ground fault interrupter, but it was turned all the way up to 1500 amps to prevent nusience trips, after it tripped several times due to our really bad "normal" phase imbalances.
Things are definitely improving, and we are much safer now then we ever were. It goes to show how one bad maintenence manager with a reign of terror, and a long tenure, can really screw things up though. I compare it to a programmer that never comments their code, and uses lots of goto statements, only the stakes are much higher.
I've had enough abrasive sigs. Kittens are cute and fuzzy.
As a telecom contractor, I know that wiring closets have the lowest priority in terms of cleanup or the "make-it-look-pretty-the-boss-is-coming" effect. You should see some of the ones from the old Ma Bell days buried deep in the hearts of old office buildings. Yikes!
I used to consult for a client whose server room was in the women's restroom. It had the largest unallocated space of any room in the building, so they stuffed in two low-boy cabinets full of DEC gear right next to the ladies' crapper.
I had to remember to knock before rebooting.
You kids these days just have it too fscking soft, I tell you...
That is all.
http://www.tfb.net/~nicl/images/?image=EthernetKil ler.jpg
That'll make any machine room the most dangerous.
Not strictly rats-nest, but it follows from the previous.
This goes back to the sixties, when computers had twitching reels of mag tape and paper tape was king.
The company had a regular overnight run. A control tape was put into the high speed tape reader, all the relevant mag tapes mounted, and the computer got on with its six hour job (about 20 second job by todays standards). Originally there was an operator on duty, but he blatantly had nothing to do, so they decided they could do without him.
But as soon as the operator disappeared, the job started failing at dead of night.
OK, bring back the operator - he can fix the problem and restart the phase which went wrong.
But, as soon as the operator came back, the problem went away. And this was the pattern - if they watched the system, it worked perfectly. But left alone, it invariably failed.
So an engineer decided to sit there and not touch anything. He told the operator to go away, as if he was't there. Which he did, turning the lights out and leaving our hero in the dark - except for the glow of the high-speed tape reader, which shines a strong light through the holes in the punched tape onto photocells. And as he watches, a moth appears and flies through the pool of light, confusing the tape reader and aborting the job.
At the old company that I used to work for. They had no server room yet so the servers just sat in the corner. One day I was working late and the cleaning staff came in. The first thing that they did was to plug the vaccuum cleaner into the UPS that the main server was hooked up to.
It seems they had always been doing that!
134340: I am not a number. I am a free planet!
Ouch. Every place I worked at had 14.4KV branched out to several substations fused at 90A, which was good for at least 1.6MW, which branched out into even smaller substations. A failure at one point was rarely noticed elsewhere. Except for the occasional exploding capacitor on the pole outside. Worked great for years with few surprises...
One day I would find out why the 14.4KV fiberglass-epoxy reinforced fuses had mufflers installed on them. Remember that electrical current resists change. If the circuit breaks, the magnetic field surrounding the current collapses and increases the voltage until it goes *somewhere.* Each substation transformer was the magnetic equivalent to a ten foot tall capacitor. Well, if the fuse blows, the the remaining energy in the transformer's magnetic field immediately collapses (the magnetic equivalent of a ten foot tall capacitor) and detonates the fuse filament. This muffler vents this energy harmlessly into the substation as heat without blowing the panels off.
One day, when turning back on the power from vacation, we would find one of these fuses didn't have its muffler installed... And we would learn how things would *seem* to work on two phases.
We would try to install more fuses without the muffler on that phase. The magnetic field was strong enough to pop the fuse out of its holder when the switch was thrown. A wire tie solved the problem while parts were being ordered.
Moral of the story: if you work on high voltage equipment, always leave it as you found it.
When our old NEC mainframe came out we had a few tiles with 18"x10" holes formerly used for cabling in our raised floor. We still don't have exactly enough tiles, but furniture is arranged better now, a year later, so that these holes are strategically covered by desks, shelves or other equipment.
Better than accidently wheeling your chair over that duct tape patch! I kid you not.
Actually my college still has them, and they work.
My boss was telling me about how a guy working in the AC system kicked up some dust and it triggered the Halon system. A voice came on to announce they had 15 seconds to get out of the room before it would be deployed. My boss of course hits the button and stops the countdown. But he lets ago, apperantly you have to hold the button until someone can come by and turn off the system. So he and the network admin go diving out of the room just as the halon is released.
There are other labs on campus that have Halon warning labels on them also, and I wouldn't dare try to check if its true.
Back in '86 I had a top-of-the-line Corona IBM-PC clone (cost me nearly $5,000 then). It had those big full-height floppy drives (two!) and was a very well-built, sturdy unit.
I was working as a computer hardware technician at the time, and I had recently bought a bunch of 256k memory chips. I brought my computer to work to show it off to the guys, and also install the memory where I had a nice anti-static station.
So there I was with all my buddies, showing off my toy. I open the case of my computer, ready to wow them, and at least a pound of dog kibbles spills out of the case. Dog kibbles are strewn all over the computer motherboard. We all kind of stood there for a moment, dumbfounded.
Eventually, I discovered the cause. My house was infested with mice, this I had known. But what I didn't know was that, in the middle of the night, mice will steal dog kibbles from the dog dish, and hide them in little places they can get to later. Apparently, they had been climbing in through the full-height floppy drives and storing the kibbles.
Interestingly, it never seemed to affect the computer!
If I see ANYONE near the server room with a camera this week, I'll personally remove your jimmies and make them part of megapod 3.
You have been warned.
(we're in the middle of a rebuild, so it's major chaos before restoration to order)
-- There is no sig line, only Zuul.
It is OK to smoke cigarettes in the server room at Philip Morris. They keep ashtrays there for the sysadmins.
FreeSpeech.org