Slashdot Mirror
The Most Dangerous Server Rooms
Ymerej writes "The Register has an article on dangerous server rooms. Have you seen worse?" Perhaps The Register would like a picture of my desk if they really want to be scared.
← Back to Stories (view on slashdot.org)
. . .where the city rats are bigger than the IT guys.
And they carry card keys.
-bpl
Now that's how you ensure job security!
- tristan
calling application programmers rats is insulting to rats, Pls refer to them by their proper designation...(L)Users :)
errr....umm...*whooosh* *whoosh* Is this thing on ?
My server room is so bad not even the rats will go in there
For example, statistics show that people who work in server room almost never catch any STDs. I wonder why that is...
Actually, this is the third article about the subject.
Teenagers these days don't have as much sex as they want each other to think they do.
One of my favourites was actually sitting inside a motor control center, with all sorts of high voltage DC motor starters right behind the main computer terminal. Don't lean back.
ed
It's all about removing floor tiles and then forgetting to set up warning cones. The clearance between our tiles and the concrete floor underneath is a good 4 or 5 feet; I would not want to fall stiff-legged into that.
Interesting side note: apparently finding high-priced Cisco gear not connected to anything is not that uncommon. I've also heard horror stories of guy that traced a cable my hand(toner was on the fritz) that looped 4 times around the data center but wasn't hooked up to shit, on either end. Took him an entire afternoon.
Finally, math books without any of that base 6 crap in them.
Check it out here.
;-)
This is by far to custom case I've ever seen.
Look a bit dangerous though
I'm a slob; I admit it. Even if I didn't, it would be obvious from the way I dress and the state of my desk. But at my job, I am cleaning up the spaghetti mess in the ceiling and trying to lay the wire cleanly from patch panels - switches.
Documenting connections has a real payoff in troubleshooting. But doing stuff on aesthetic grounds is a harder sell. I have a gut sense that a clean layout is important even if you know the destination both ends of a wire whose middle runs through a snarl. Here's what I came up with:
my version of the community policing broken windows theory.
It's psychologically harder to do slipshod, shoddy work if everything around you has been done well. And it's hard to do a proper job if everything else is slipshod. As a matter of housemate politics, it's easier to leave the nth dirty dish in the sink than the first. You are only adding an increment, not changing state.
Doing the Right Thing is contagious. At least, it is among folks I care to work with. Doing the Wrong Thing is catching, too. Morale is higher and people challenge themselves more at a shop that is run well.
That's how I pitched it, and my boss bought it.
You want scary:
Our server room has wobbly floors and tower cases stacked 2 high sitting in the middle of a 16'x16' room, with the more important servers sitting on the top.
We also got a new air conditioner that has an electronic switch and we have problems with brown outs. So in the middle of the summer, the power goes out and the A/C doesn't come back on, usually on a weekend too.
One more thing: brace yourself:
We use Windows servers with IIS!!!!!!!!
about 3 years ago, I was pulling cable from 1 room to another.. while standing on a ladder pulling the cable out of the cieling, the cieling collapsed on me! About half the tiles from the room fell on to the floor, and a florescent light hit me in the back on its way down.. that's pretty dangerous!
As a long time Customer Engineeer for a major manufacture, I've had my share of basement computer rooms and all the subsequent flooding thereof. That being said, my favourite customer had a mainframe in the basement of his 100 year old private house. He was running a time share (remember those) system over 300 baud modems. To enter the place one had to walk beside his wifes pottery kiln. However once inside a really excellent air conditioned room there was the beast, a brand new HP3000 series 2 with 500Mb of disk. He was my favourite because on my first after hours preventive maintenance call - in fact it was the first time I had ever seen the place - his wife called out after a couple of hours.." Phil, dinner is ready !"
Semper ubi sub ubi
Check this out:
;)
www.theregister.co.uk/media/926.jpg
That's a HELL of a case mod
IGB: More fun than eating oatmeal!
I was in a large shared server facility that still used a Halon system, which when released, fills the area suffocating the fire (and any living creatures in the area as well).
Anyway, one day we were working in our cage, when we heard a warning alarm, and saw all the employees running for their lives. Not knowing that the alarm meant the Halon system was about to be activiate, we joined in anyway and ran for the emergency exits.
It turned out the fire alarms were set off by accident by someone drilling and creating dust, and luckily the people on-site disabled the fire supression system before it went off.
_______
2B1ASK1
In rural Indiana, you don't always have space to have a whole room devoted to servers and network equipment, ya know?! But I was still surprised when I visited my former ISPs local point of presence - in one of their employee's one and only bathroom at his house. Photo here. Do some laundry, take a dump, watch some network traffic go by. Uh-huh.
is going on here. That photo on the main page this article links to is the same one that George Ziemann has on his site from the Ebay Vs. Musician article earlier.
I don't see any indication that this is supposed to be his server room. So who's lying?
It's all going according to
...looks like some new cybernetic monster I imagine will make an appearance in Doom 3.
At my last job we had a network of about 600 local users. Our server room had two racks of equiment on the building's ups, so the racks plugged right into the wall.
:-)
One day we had a broom leaning up against the wall next to two of our cabinet's. Someone bumped the broom, which fell in the long arc that brooms do when they fall along a wall when leaning. One the way, it happened to unplug our two cabinets from the wall. So much for uptime. The place when quiet and we all just stared at each other for a few seconds. This is in an envirnment where downtime isn't really tolerated at all.
Our task the next weekend?
We took a whole package of zip ties, loosened up the plug wall plate, zip tied the plugs around the back of the outlet wall plate with an ungodly amount of zip ties, and screwed the wall plate tight again.
Our version of 120volt twist locks.
Was interesting to hear what people would ask after seeing it for the first time.
Not quite the server room from hell, but the story's on topic.
-Pete
Soccer Goal Plans
That first picture looks like a cross between something out of a Terry Gilliam movie, a Borg Cube and Tetsuo the Iron Man. The ones at the bottom are just plain bizarre.
These aren't servers, they're representative works of art- kind of like a city. Granted, these servers will die the same flaming death that Chicago did in the great fire, but it's still kind of neat.
That's ok, Jesus likes me anyway.
You want dangerous, well the only fire prevention we have for our 23 servers, is one halon extinguisher. So if the room goes up, we can save it, as long as we don't want any oxygen...
Xaotik Designs
I'm messy by nature, but I know that as long as I'm the only person who understands the mess, I'm indispensable
And I got a new job yesterday... I pity my successor
I don't read your sig, why do you read mine?
... when my foot broke through one of those floor tiles in the server room. Funny, someone told me they were high resistence, that must not be exactly the case ;)
That doesn't surprise me. I have a strange feeling many of you have been caught doing this in front of your pc.
So much for my Delta Force keyboard layout ;)
"I can't give you a brain, so I'll give you a diploma" - The Great Oz (blatently stolen sig)
For those with more disposable income, as long as they keep the Cherry 2000 model out of the water, everything should be fine.
I bet he's the guy that invented tcpdump.
We launched a website with publicity on a live primetime TV show about the internet (in the UK), while the server (singular) was still running under my desk. It was a little while before we moved it out of there and, amazingly, I never accidentally shut down the site with my knee.
Of course, when we did, eventually, move it into a server room, the aircon subsequently broke down and, being an underfunded dotcom, nobody wanted to spring for repairs. We lost at least one server that way (thankfully not a live-facing one).
Our main wiring closet looks a little like the 'not so bad examples' of wiring closets in those pictures - though nothing like the more extreme ones.
The problem is, once the thing gets into that kind of mess, you rarely have the chance to bring down the entire network to repatch all the cables and cable tie them into some kind of order.
Not only that, but if you have loads of trunks and VLAN's configured, putting it all back in the right order can be a total ballache!
"Hey! Unless this is a nude love-in, get the hell off my property!!"
to ensure it could withstand the island's regular earthquakes
... yeah ... we just had a 6.2 earthquake and we're rebuilding the server room", then he gave a kinda "is he serious?/boy am I embarassed" pause and gave me his professional opinion: "Oh, okay, that might be the problem." And then the fucking dial-up users. They're on the TV saying We know there has been an earthquake, please do not call the police to tell them there has been an earthquake, try to use the phone for emergencies only!. And our users are trying to get on freaking AOL instant messenger via dial-up. I was half expecting them to say "Oh yeah, I tried to dial in and it doesn't work. The phone works fine because I just called the police to tell them there was an earthquake."
Has anyone else been in a non-earthquake-prone place and then had an earthquake? Here at work our server room was completely unprepared for an earthquake. Some of the machines came off the racks, some of the whole racks fell, our T1s got damaged or disconnected somewhere in the process. The whole disaster showed us how stupid some users can be. First, the T1 provider calls from Boston or something, "Duh, we show that your T1 lines are down, blah blah blah..", "Uh
Anyway, we bolted down all the racks, shame on us for not doing it in the first place.
Statistics also show that "server room affairs" by non technical personal, are responsible for more than half of office place STD transmission.
The "photo copy room trysts" follow in a close second place, while the "boss's office boinkings" are becoming more uncommon as more CEOs are of the female gender.
Saskboy's blog is good. 9 out of 10 dentists agree.
Think you're safe when you've got your computer room all tidy and clean?
Forget it. We had a problem with the airconditioning at a medium size company in Delft which had it's heat-outlets on the top of the roof.
These outlets were not so well protected to cold as was shown two years ago when, after a freezingly cold weekend we came into the server room and it was really boiling hot. Problem was the huge ventilators on the roof were stuck frozen.
It was the only time we had all windows and doors open in the middle of winter. But this cold could well have started a fire.
sig not found
I get a call from the folks on the 5th floor saying there's water coming out from under the door (they don't have access to the room). Being the onsite telephony person, I rush up wondering what the hell is going on. Sure enough, water has soaked the carpet around the door. Opening the door I see as much as 5cm (2in) of water sitting around the base of the switch and various servers connected to the switch. All in all, probably around $300K worth of equipment, and I don't dare go in, because there are power cords lying in the water. Finally get the power turned off, the water out, clean everything up. All in all, costed us a couple hundred dollars for some new cables, one monitor, and various odds and ends.
Apparently what happened was someone on the next floor up was in a bathroom, turned on a faucet and forgot about it. The water managed to move about 5m (15ft) down the hall before deciding to pour out into our telephony room!
I work in industrial maintenance and the most interesting electrical problems happen when the weather changes.
120VAC isn't too bad. Connections soaked in water might survive for a while until the corrosion finally breaks it down, melts the wirenuts, etc. Getting shocked by it isn't enough to blow your fingertips off. 240 volts is usually just 110 volts split into two phases, so it doesn't present any worse of a problem. 480 volts is another story...
480 volts gets interesting when the humidity rises and gets absorbed by the dust surrounding breakers and other switching components. Often, it will flash across the phases, vaporizing the debris, and mysteriously tripping the breaker. No one will figure this out until they happen to take a close look at the wiring, and the humidy from their breath will illuminate the brightest flash they have ever seen in their lives.
I'll have to drag my camera up into my attic to take a picture of the mess up there one day. I've 3 machines that run 24/7 up there, covered in dust, various other crap and spiders (and boy do those spiders grow big up there!)
I put them up there as I got sick of the noise, then ran ethernet cabling down to sockets around the place. My main web server (see sig) has been happily running up there for about 1 1/2 years now - barring the odd upgrade. It's not pleasant to open up a case full of dust and cobwebs, BTW!
Code, Hardware, stuff like that.
Some of the more complex and heafty NOC's I've heard about actually have halon, or similarly based fire prevention systems for the entire room. Auto sealing doors and the works. The old "you have 20 seconds to exit the room" plot device brought to life.
Well, that's sorta the whole point of halon (and thus, why isn't used anymore.)
Those are nothing. NOTHING, I tell you!
I've got a set of routers located in a crawlspace where the only way to get to them is to walk across boards spanning small metal beams that were put in to hold a suspended false ceiling. One missed step and you'll drop right through the ceiling, AND IT'S A 2 STORY DROP! Once I dropped a power pack while replacing it and nearly killed a gal working below. Power pack exploded like a bomb when it hit.
We recently had a "security audit" where they recommended we should mount those routers in a locked cabinet for increased security. Not a mention about the boards, lack of handrail, safety net, etc. Heck, who needs a locked cabinet? Just remove one of the boards and NO ONE can get to those routers, not even the people who are supposed to maintain them!
Back when we used thinnet one of the managers didn't like stringing new coax through the building whenever we remodeled or moved people, so he had us cut all the coaxes to length PLUS 25 FEET! He figured if someone moved we could pull back the excess and save time. The cables all terminated in what came to be called the spaghetti room, from the coils of coax all over the floor. We had to step over all the coaxes to get to the routers and hubs. Eventually, the coaxes got damaged from all the abuse and had to be cut off to length anyway, but for several years it was a serious tripping hazard for anyone who entered that room.
Beta sux! Join the Slashcott! http://hardware.slashdot.org/comments.pl?sid=4760465&cid=46173047
I've been in two datacenters where there were regularly spaced 'emergency air supply' masks, right next to the regularly spaced fire extinguishers.
Not sure how well they would work in a room full of smoke and halon, but I suppose you could run from station to station, and then somehow pry open the massive firedoor...
"Can of worms? The can is open... the worms are everywhere."
My personal favorite was building a small network out in a field. We set up our four machines [286's] in the dirt, got our power from a generator being towed by a five-ton and wired together on a 10Base2 network. For the first day or so the only shelter we had for the machines was a tarp that we pulled over them when it started to rain.
Lying on the ground, underneath the leaky tarp, hoping that I did not get electrocuted, or if I did that I would not be held accountable for the damaged equipment [trust me, this was not my idea], I decided that re-enlistment was not a great idea.
[former] USMC geek
I just knew there was a punchline there that I was totally missing out on. Damn. Thanks for that. :)
If our IT depatment let out infrastructure degrade to anything close to that I would terminate the lot. These are representations of unprofessional half-assed setups. There is no justifiable reason for these examples. A small investment of time and an operational plan prevents this. These managers should be hung.
If we don't fight for ourselves no one will.
That's right you read that correctly.
I once visited a client who had his server racks in an old lockerroom shower. This would not have been so bad except when one of my co-workers leaned against the wall and hit the valve we discovered that the pipes hadn't been capped by just had the shower heads removed....that's right three full racks of equipment in a live shower. =)
I was told this story from a reliable source...
An HP technician (yup they have at least one) was restoring the data to a customer's fileserver but the backup software was asking for tape#2. The customer only ever had one backup tape that they recalled, so they were quite perplexed until the security guard entered the server room...
Apparently every morning around 3am when he made his rounds he found the backup server screen blinking "insert next tape" -- The security guard proudly said that he was pushing in the tape for at least six months now...
From excellent karma to terible karma with a single +5 funny post...
I work for a network consulting company, so I've seen some pretty funny stuff in the last few years. Here are some right off the top of my head:
One company didn't order a rack mount kit for their KVM switch (some Belkin model), so they duct taped it to the main monitor. No subtle tape loops under the KVM..... they wrapped the tape three or four times around the KVM and the monitor.
Another company was remodeling their server room but neglected to move the servers somewhere else. There was an inch and a half of drywall and sawdust on top of all the network equipment and servers. The circuit boards looked like it had snowed on them.
I'm doing an audit on some systems. I see a motherboard sitting in a cardboard tray (the kind you get when you purchase a 24-pack of Coke from Costco), along with a hard drive, floppy, power supply and network card. No case. No cooling. Turns out it was their PDC and print server. That's quality craftsmanship.
This isn't about server rooms, per-se, but I did some work for a national pizza chain. They had modems at a central site that were supposed to make a phone call to the stores to print out order tickets. We were sent to figure out why they weren't printing. At one site, the printer was on the floor next to the prep counter where they add the toppings. Someone had spilled a good quart of marinara sauce into the printer. They gave the outside of the printer a good once over, but the inside was just nasty.
We were sent out to troubleshoot a voice-over-IP problem at a garden nursery. We arrive on site and lo and behold, there was a dead rat on top of the router. It didn't have anything to do with the problem, but it sure was unexpected.
I love when people don't properly plan their electrical power consumption in their server rooms. I walked into some company's server room, plugged in my laptop to the rack mounted power strip, turned it on, and blew the breaker for two racks of servers.
I watched a wireless network installer gob Liquid Nails onto the back of an Aironet access point and stick it to the ceiling. I hope they never want to upgrade that particular access point.
Any other good stories?
These are actually pretty common. I remember the university's data center was like this, there were 3 large red buttons that would temporarily stop the halon release so that you could get out or do something to the servers before the room was flooded. My current employers datacenter sadly had only high temp overhead sprinklers, by the time they go off its more about saving the building as the servers are already slag.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
This brings back memories.
I remember working at an isp way back when and the server room was so bad that you could basically lean on the rats nest of wiring like it was a makeshift hammock.
Walking around behind the racks meant being completely aware of which line you put tension on, lest it knock some connector not fastened down out of place.
A couple of weeks ago I came accross a couple of photograps of actual telephone exchanges in the streets of Beirut. You just wouldn't believe it, and it took me a few seconds to understand the picture. there were so many wires that you can hardly see the box behind -- kinda like Johnny Mnemonic, except with 10x more wires, and 2 or 3 handsets plugged in seemingly random (or probably not) outlets. I'll post again if I can find it back.
I would love to see what this person could do with Christmas tree lights
I've got one, heard from an old-timer friend of mine who used to be a field rep (and, at other times, an AIX kernel coder) for some of IBM's big iron.
The situation: The client's systems are crashing, on a regular basis, for no understandable reason. No remote diagnostics work, so they send out my friend.
He gets to the server room, and keeps thinking he's seeing things out the corner of his eye. He tells everyone to leave the room, and turns out the lights. The room glows.
The server room at this place was sitting under a huge radar system. (He had some additional explanation -- used to be a physics major -- but I didn't entirely follow it). They moved the equipment (a substantial undertaking!) and the problem went away.
I have always thought that how organized a companies wiring is - is a direct reflection of the staff that works there.
I know that when i moved into one company in Redwood City - the network wsa a nightmare. We had rooms that looke like that - but over the next 2 years we replaced almost every wire on that network - and demanded budget for proper closet setups - and got it.
We eliminated all those closets that looked like that, and learned one hell of a lot in the process.
I think that if your closets look like that - you are asking for fire - and it shows just how lazy you really are. No arguments of "I'm too busy" allowed - it just means your a lazy slob period.
Who gets paid to produce messes like these, and how much?
Can you imagine what it would look like if the same methodology were applied to coding? Maybe it is...and if so, it's no wonder bad things happen (bugs, project failures, etc).
Check this out. Just happened today!
in college i was doing some work in our computer center pulling cable under a presurized floor. it was your standard room of ancient large blue IBM 72" tall units. arranged in classic grid.
then a maintence crew working on something in an attached room did something to start a small fire.
half my body is under the floor -- head first. trying to unknot some cable.
i didnt know anything about the halion (sp?) gas until the rescue worker explained it to me.
i should have sued, but was worried about pizza money from my job.
members are seeing something, your seeing an ad
Taiwanese typhoon? pfffft. Come on - lets get some webcams in these dangerous server rooms and then /. em for some good old fashioned geek fireworks.
Yeah - you know you want to.
__ Someday, but not this morning, I'll finally learn to use the preview button.
then accountants with key cards could not be rats, just simply bean-counters, (another demographic group to toy with MWAHAHAHA! )
:)
What's your (l)username?...ok...**clickety click** Why thank you for increasing our IT budget by 1 billion$...now i'm afraid you'll have to go. Do me a favor and check if the pover is on, just stick a piece of metal in the mains... You won't feel a thing
Live for the present, learn from the past, and dream of the future!
...or the electrician dropping a wrench inside a power room at one of Bell Canada's largest CO's. The short and resulting fire (and halon response) took out about 25% of their capacity in the entire city of Toronto for hours.
I had lots of fun (was at a wireless carrier at the time) chatting with the switch techs about how they'd distribute our traffic over the remaining circuits.
We just built a new server room and the raised floor tiles are in and of themselves dangerous -- sheet metal covering concrete, 2 ft x 2ft. They must weight 40 lbs each, I wonder who the first person to drop one on their feet will be. I've already taken out some chunks of the newly painted wallboard in the course of swapping solid tiles for ones with grommeted holes.
The installer told me it was a good flooring system, but I wish it wasn't concrete. It takes a special coring machine to put round holes, square cuts require a wet-cut bandsaw.
They're both in the billing office. I tried making them lethal so no one would touch them, espcially considering the boss's kids seem to think the 2000 server should be used for chatting up their friends via AIM and has no other purpose. Unfortunatly, reguardless of the mass of cables running in to the hub next to it and the fact I attempt to hid any interface devices they still seem to muck with it whenever they have a half day from school. I think I'll just set up a mine field around it and call it a day.
Luckily they've never found a use for the SCO box other than looking at it with a wrinkled nose. I do too, mind you, but for entirely different reasons I would guess.
Washington State has a significant history of major earthquakes. It is part of the pacific ring of fire. See the USGS site here But really, you have to ask yourself, what are you going to do when Mt Ranier blows and your server farrm is buried under 1 Million tons of liquid hot magma?
Actually, it isn't used any more because it was found to be ozone-depleting and banned by international agreement.
p ://palimpsest.stanford.edu/waac/wn/wn15/wn15-2 /wn15-208.html
http://www.haifire.com/press/halon_rep.htm
htt
Several of the ozone-friendly replacements (I forget their unfriendly alphanumeric designations) still work by sucking all of the oxygen out ofthe room.
i used to work in a server room like this..i remember talking to the guy that replaced the Halon and maintaned the system.."Ok so..ive got a certain number of seconds to get out...or i won't be able to breathe...right..no problem"
A friend of mine used to be a contractor for Navistar at their engine block casting plant. One of the computers on the factory floor was so close to a stream of molten metal that he had a hard time sitting at the computer. He could take only so much heat. It would also pop and shoot sparks at him. He had a large set of shirts that were already ruined that he wore to work. They had little holes burned in them from hot things.
Similarly, I worked for a computer rental company in central florida years ago. We had an original IBM PC at a phosphorous surface strip-mining facility. The parking lot had showers for your car that you had to go through when you left. The black bezels on the front of the case were bleached to a nice light off-white grey by chemicals floating in the air.
Disconnect your television. Do your own research. Draw your own conclusions. They're probably lying. Don't be a sheep.
Actually, at my first IT job the office was overcrowded and they didn't have a place to put me, so they set me up with a desk in the server room. It's not so bad. When people can't access the room, it makes it harder for them to bother you. You can play games without fear of being watched. And you get used to it being 60 degrees all the time after a while. I can now stand outside in the dead of winter with a t-shirt and shorts on and I'm A-OK.
Particularly given that Rats are in the order Rodentia, and Kangaroos are in the order Diprotodontia.
I guess they are lawyers though...
Lots of software is the equivilant of those pictures. The only problem is that you can't see that from the pretty pictures on the box cover. Instead you see it in crashes and strange behavior. Two things cause it, first unskilled careless workers, second stressed workers in unrealistic situations. The second seems to be more previlant. They guy who wired that first mess probably knew that its better to label everything and run it down the side of the rack to the gutter in the floor (notice the one rack is accually nicely wired under the mess on top). Instead you have the manager who gives you 1 day to wire up 100 computers, or 1 week to add some big feature to the code base. The result is scrambling like mad and a "just plug it in and make it work don't make it pretty" attitude.
I haven't seen a mess of wires like that since the Borg took Picard!
Borg Cube
Live web cams
IT nerd in bar: Hey, baby, you like dangerous guys?
Hot Chick: Yeah, I guess so.
IT nerd in bar: Sometimes I take my pocket protector out of my pocket!
Hot Chick: Uh-huh...
IT nerd in bar: Sometimes I take my mint condition Megatron action figure to LAN parties!
Hot Chick: Uh-huh...
IT nerd in bar: And sometimes I let my server room get really messy so that it's a hazard to my life!
Hot Chick: Wow, that is dangerous. I'm really turned on...hop on me right now...
"It's a tarp!" -- Dyslexic Admiral Ackbar
I'm sure that some of you have worked in large server rooms with a big red emergency power shutoff button on the wall...
:)
At my old university, one of these server rooms was emptied as new, smaller hardware came available and the room was no longer needed. They turned this room into an office for a student organization... leaving the large red button, but taking the "Emergency Shutoff" sign.
This unlabeled button sat neglected on the wall of this little office for about 7 or 8 years until one day a curious student just had to find out what the button did...
The network for all of the engineering schools at this university of 36,000 students went down for most of a day..
The best part is that the button is still in the little office with the students, and it is still unlabeled yet fully functional... They did hide it behind a file cabinet, though
There are 10 types of people in this world, those who can count in binary and those who can't.
I was in a large shared server facility that still used a Halon system, which when released, fills the area suffocating the fire (and any living creatures in the area as well).
Look -- geeks sue for the wrong kind of keyboard -- ergonomics = safety after all -- do you really think there is just this dangerous gas ready to be released where you work and you have exactly 20 seconds to escape before you die?
Taken from the first site that came up on Google...
"Three things must come together at the same time to start a fire. The first ingredient is fuel (anything that can burn), the second is oxygen (normal breathing air is ample) and the last is an ignition source (high heat can cause a fire even without a spark or open flame). Traditionally, to stop a fire you need to remove one side of the triangle - the ignition, the fuel or the oxygen. Halon adds a fourth dimension to fire fighting - breaking the chain reaction. It stops the fuel, the ignition and the oxygen from dancing together by chemically reacting with them. Many people believe that Halon displaces the air out of the area it is dispensed in. Wrong! Even for the toughest hazards, less than an 8% concentration by volume is required. There is still plenty of air to use in the evacuation process. (Emphasis added by me)
I'm not an expert, but urban myths bug me...
I would have to say that explosives are the most abused technology in all of history.
I've worked tons of places pullin wire...
:)
I worked at mothers cookies in oakland CA for a spell. You think malaysia is dangerous?
I would pull off the 880 by the colliseum,mc arthur blvd? It takes you through a very derilict section of oakland factories.
The entire complex is surrounded by bobwire. If you drive down the street, less than a block, you are in one of the worst neighborhoods in oakland. I had a friend who lived in the neighborhood, he wasn't shittin me either about the danger. I went to his house twice to hang out and that was enough for me.
The server room was cool, about 40degrees all the time so you wore a jacket when you went in there. But pullin wire....Ohhh my god!
I had to run a fiber line from the main building to some office in the back of the bakerery. Now before you get the picture of little mothers running around with cookie sheets and kenmoore ovens you have to understand.... That is not what a huge production cookie plant is.
Imaging a HUGE fricken warehouse with conveyer belts of cookies going everywhere, machinery whiiring and cookies going into boxes and filled with creme and those animal cookies with the dots, all in this HUGE room about the size of a football field.
One end was the mixing end, where they had these mixing machines the size of my garage. Into those would go 50 gallon oil drums of butter, lard. Huge bags of flour being loaded by forklift, ect.
Now at the time, wireless hadn't really made it mainstream. So my mangers convienced mothers fiber would be good since it would provide the best ROI. They were sold and I was sent out to work.
Now the factory was built from steel girders covered with that tin roofing, the stuff that looks like a ruffles potato chip. I got up there to where the top girders are and before my eyes was the most treachorous wire run I ever saw.
Remember what I said about 50 gallon oil drums of lard? Well, when the cookies baked, the lard would vaporize and rise to the ceiling then settle on the steel girders. Over the years a 1/4" layer of lard had deposited 70 feet up in the rafters.
I put my finger in the goop to see how slippery it was.
No friction.
I called the office and told my boss. Later he called one of his friends to subcontract the work out too.
*Disclaimer* Despite the lard, mothers cookies makes a great product, and was an awesome place to work. If you ever get the chance to work there, jump on it, you won't regret it (or the 50cents a bag price for employees
I think the main point is that when the Halon goes off, it's more than likely to burst your eardrums. At least that's how it worked in a friend's Uni's server room.
:)
I do remember they did a fire test, trying to set off the fire extinguishers with a burning wastebasket. Nothing happened, and they moved the wastebasket closer and closer to the detector. The alarm finally went off when the flames were licking at the damn thing
- Oliver
The right to bear arms is only slightly less stupid than the right to arm bears...
would someone please flush the mail queue?
oh no, the logs are getting backed up!
I have no special gift, I am only passionately curious. --Albert Einstein
Well, that's sorta the whole point of halon (and thus, why isn't used anymore.)
Having thoroughly studied the subject, umm, no, you're wrong.
Halon is no longer used because:
a)it was poisonous
b)it was ozone depleting
The number one cause of death in server room fires is toxic smoke inhalation from burning plastic/paint/wire insulation/circuit board components/etc. Why? People go in to rescue backup tapes, and are rather quickly incapacitated when they inhale the smallest amount of toxic fumes. Folks- NEVER ENTER A BURNING ROOM WITH A FIRE. PERIOD. Backup tapes are NEVER more important than your life. People think "oh, I'll hold my breath." They don't think about how far it may be, or they may get disoriented in the smoke..and they think "oh, just a little breath, i need to breathe." That gets you coughing. Then you die. Smoke inhalation is actually(I believe) the number one cause of death in ALL fires.
Let's not even start about what opening a door into a hot, mostly sealed room letting in a ton of oxygen does.
My "Blue Book" (computer security, mostly focuses on physical) says to NEVER store backup tapes in the same room as server equipment, PERIOD, because the temptation to rescue them is too great.
CO2 is the agent that suffocates you...in that case, yes, the point is to remove as much oxygen from the room, and yes, you will suffocate very quickly.
FM200 is the most common substitute and is fairly harmless, but works on pushing oxygen from the room, and because it is much heavier than air, requires very good sealing of the room in order to work properly. It is next to useless on fires high up in the room unless the system is very well designed.
An alternative is INTERGEN, which is 100% harmless and is mixed specifically to make your body naturally breathe faster, which is a good thing if smoke isn't really a problem and you just need to keep from suffocating. It also works mostly on rapid temperature change and chemically interfering with combustion(not by removing oxygen), and its weight is much closer to natural air, so it doesn't sink.
Thus, the server room doesn't have to be sealed(Halon and FM200 both require it), but the amount of extinguishing gas is greater and piping is slightly different.
In server rooms of sufficient size that use Halon, CO2, etc... SCBA units are usually recommended if not required. Interrupt switches and delays are very common as well, although both are designed mostly with reducing false discharges, which can cost $5-10k per 1,000 square feet.
More and more commonly, dry-pipe water systems are being used, particularly in 24x7x365 managed buildings. The idea is that if the fire is small, an alarm goes off and a countdown starts. On site staff can take appropriate action(cut power, run in with a fire extinguisher, that sort of thing) or, if it's bad, let the system come on. They're usually zoned triggered and fogging nozzles are used to minimize equipment damage.
Unfortunately, for your average 1,000 sq ft room, you're looking at about $30k to $50k, not including modifications to the room to get it sealed. Dry pipe water is less expensive, but one false alarm could cost you 10 times the system's cost in equipment alone.
Think about it. If the data for "tape#2" was overwriting the data on the only tape...
Hate to ruin an otherwise good joke.
Heh, I've got a 480v story for you. I work in IT for a manufacturing company. We had an apparently faulty air conditioner take out most of the plant, which is no small feat.
First, a little background info. It's an old building. The electrical is a mess of old and new circuits, some three phase 480 delta, some 240/120 single phase, and one major branch circuit even with a high leg. The former maintenence manager was of the mind to "get things working", rather than "get things right". After he was fired, I started helping out a little with the maintenence staff when emergency things came up, since I know a little about electronics, mostly to help them with computerized and digital control systems.
Anyway, from the street, there are 5000 amp fuses, huge suckers, then 1200 amp fuses on a few main branches. From there to a 1200 amp panel breaker for a major section of the plant (the one that the server room is on), along with most of the manufacturing. In that 1200 amp panel there is a 250 amp 3 phase breaker for the air conditioner. This is all 480v delta 3 phase.
Somehow, that air conditioner breaker failed. It vaporized part of the busbar, tripped the 1200 amp panel breaker, and blew the 1200 amp fuse for one of the phases, leave us down a phase. For the benefit of those who do not know, 3 phase motors running with one dropped phase tend to burn up... quickly! Ideally there is an thermal overload circuit to shut them down before that happens, but that doesn't always work. So bang.. the lights are off, and motors start to burn up in various places around the plant.
Once the maintenence staff figures out what is going on, that we are down a phase, they throw the mains on the service entrance panel for that 1200 amp branch. All seems to be good. We just need to replace that 1200 amp fuse and the faulty breaker right?
Heh. Well, it happened that we didn't have any spare 1200 amp fuses. A 1200 amp fuse isn't something you can run down to your hardware store and get. We send an employee to the next town where a store has exactly three of them in stock. $400 each. We tell him to buy all three. He comes back with one.
We replace the fuse, and the maintenence staff replaces the breaker. Upon reenergizing the circuit everything seems fine... until they go to put the protective metal cover back on.
The panel literally explodes. I wasn't in the room at the time thankfully, but the guys that were there say is was bright, loud, and scary. Apparently what happened was pieces of molten busbar had dripped near the bottom breaker in the panel and were close to shorting out the phases. The slight movement caused by putting the cover back on jarred the chunks of metal and shorted out the phases.
So... the 1200 amp panel breaker trips.... but not fast enough to save the upstream 1200 amp fuses near the service entrance. The ones we didn't have spares for. Again. And now it blew all three of them. And the store only had two in stock.
So we get back on the phone. We find another store that has two in stock, so we send an employee out to get all five, from both stores. He gets it right this time.
We finally replace the three fuses, triple check EVERYTHING, and throw all the breakers back on. We had sent all the employees home hours before... they couldn't do anything without power. But we are finally up... nearly 6 hours later.
Needless to say, some things have changed as a result of this, and it really underscored why the former maintenence manager was fired. We called the electrical engineering firm that had most recently surveyed our power systems, and had them run some more short circuit computer simulations, things like that.
Upon reading their report, I learn that our service panel has a ground fault interrupter, but it was turned all the way up to 1500 amps to prevent nusience trips, after it tripped several times due to our really bad "normal" phase imbalances.
Things are definitely improving, and we are much safer now then we ever were. It goes to show how one bad maintenence manager with a reign of terror, and a long tenure, can really screw things up though. I compare it to a programmer that never comments their code, and uses lots of goto statements, only the stakes are much higher.
I've had enough abrasive sigs. Kittens are cute and fuzzy.
As a telecom contractor, I know that wiring closets have the lowest priority in terms of cleanup or the "make-it-look-pretty-the-boss-is-coming" effect. You should see some of the ones from the old Ma Bell days buried deep in the hearts of old office buildings. Yikes!
I used to consult for a client whose server room was in the women's restroom. It had the largest unallocated space of any room in the building, so they stuffed in two low-boy cabinets full of DEC gear right next to the ladies' crapper.
I had to remember to knock before rebooting.
You kids these days just have it too fscking soft, I tell you...
That is all.
http://www.tfb.net/~nicl/images/?image=EthernetKil ler.jpg
That'll make any machine room the most dangerous.
I thought that was the capivara (not sure of the english spelling) which seems to have been the inspiration for the ROUS in The Princess Bride. I saw a team of firefighters spend a good 90 minutes trying to get one out of an open sewer one day in Brazil. That was one tough giant rat.
Lasers Controlled Games!
Not strictly rats-nest, but it follows from the previous.
This goes back to the sixties, when computers had twitching reels of mag tape and paper tape was king.
The company had a regular overnight run. A control tape was put into the high speed tape reader, all the relevant mag tapes mounted, and the computer got on with its six hour job (about 20 second job by todays standards). Originally there was an operator on duty, but he blatantly had nothing to do, so they decided they could do without him.
But as soon as the operator disappeared, the job started failing at dead of night.
OK, bring back the operator - he can fix the problem and restart the phase which went wrong.
But, as soon as the operator came back, the problem went away. And this was the pattern - if they watched the system, it worked perfectly. But left alone, it invariably failed.
So an engineer decided to sit there and not touch anything. He told the operator to go away, as if he was't there. Which he did, turning the lights out and leaving our hero in the dark - except for the glow of the high-speed tape reader, which shines a strong light through the holes in the punched tape onto photocells. And as he watches, a moth appears and flies through the pool of light, confusing the tape reader and aborting the job.
At the old company that I used to work for. They had no server room yet so the servers just sat in the corner. One day I was working late and the cleaning staff came in. The first thing that they did was to plug the vaccuum cleaner into the UPS that the main server was hooked up to.
It seems they had always been doing that!
134340: I am not a number. I am a free planet!
Ouch. Every place I worked at had 14.4KV branched out to several substations fused at 90A, which was good for at least 1.6MW, which branched out into even smaller substations. A failure at one point was rarely noticed elsewhere. Except for the occasional exploding capacitor on the pole outside. Worked great for years with few surprises...
One day I would find out why the 14.4KV fiberglass-epoxy reinforced fuses had mufflers installed on them. Remember that electrical current resists change. If the circuit breaks, the magnetic field surrounding the current collapses and increases the voltage until it goes *somewhere.* Each substation transformer was the magnetic equivalent to a ten foot tall capacitor. Well, if the fuse blows, the the remaining energy in the transformer's magnetic field immediately collapses (the magnetic equivalent of a ten foot tall capacitor) and detonates the fuse filament. This muffler vents this energy harmlessly into the substation as heat without blowing the panels off.
One day, when turning back on the power from vacation, we would find one of these fuses didn't have its muffler installed... And we would learn how things would *seem* to work on two phases.
We would try to install more fuses without the muffler on that phase. The magnetic field was strong enough to pop the fuse out of its holder when the switch was thrown. A wire tie solved the problem while parts were being ordered.
Moral of the story: if you work on high voltage equipment, always leave it as you found it.
When our old NEC mainframe came out we had a few tiles with 18"x10" holes formerly used for cabling in our raised floor. We still don't have exactly enough tiles, but furniture is arranged better now, a year later, so that these holes are strategically covered by desks, shelves or other equipment.
Better than accidently wheeling your chair over that duct tape patch! I kid you not.
It was plenty of fun - I had no responsibility in the matter and it was easy for the tech to slosh traffic around. We were playing 'watch the lights go red' on one wireless network at a time.
Actually my college still has them, and they work.
My boss was telling me about how a guy working in the AC system kicked up some dust and it triggered the Halon system. A voice came on to announce they had 15 seconds to get out of the room before it would be deployed. My boss of course hits the button and stops the countdown. But he lets ago, apperantly you have to hold the button until someone can come by and turn off the system. So he and the network admin go diving out of the room just as the halon is released.
There are other labs on campus that have Halon warning labels on them also, and I wouldn't dare try to check if its true.
http://images.e-gerbil.net/ghetto Most of these have not been referenced by any links posted thus far.
...
"I'm sorry, but comfortable is the last thing I want in my server room. I want it unbearably cold, and noisy. I want items scattered dangerously around the floor. I want random floor tiles to be missing. I want a very old sandwich of undetermined origin sitting half-eaten in the corner. I want the first thought of any person that enters my server room to be "Dear $DEITY, I must get out of this place IMMEDIATELY!"
Time flies like an arrow, fruit flies like a banana.
Heard a great one while working at a steel plant.
A few years before I started, they were having some issues with a server apparently rebooting every evening, around 11pm. They checked everything they could think of, and found no problems with they system. After a few weeks, they just couldn't nail down the problem, so one of the guys stayed late to see what happened.
Around 11pm, in walks one of the night cleaners. She reaches down, unplugs the server from the UPS, and plugs in the vacuum sweeper. She cleans the carpet, unplugs the sweeper, plugs back in the server, and leaves.
We suggested that she use a wall outlet, and our server problems were fixed.
Back in '86 I had a top-of-the-line Corona IBM-PC clone (cost me nearly $5,000 then). It had those big full-height floppy drives (two!) and was a very well-built, sturdy unit.
I was working as a computer hardware technician at the time, and I had recently bought a bunch of 256k memory chips. I brought my computer to work to show it off to the guys, and also install the memory where I had a nice anti-static station.
So there I was with all my buddies, showing off my toy. I open the case of my computer, ready to wow them, and at least a pound of dog kibbles spills out of the case. Dog kibbles are strewn all over the computer motherboard. We all kind of stood there for a moment, dumbfounded.
Eventually, I discovered the cause. My house was infested with mice, this I had known. But what I didn't know was that, in the middle of the night, mice will steal dog kibbles from the dog dish, and hide them in little places they can get to later. Apparently, they had been climbing in through the full-height floppy drives and storing the kibbles.
Interestingly, it never seemed to affect the computer!
If I see ANYONE near the server room with a camera this week, I'll personally remove your jimmies and make them part of megapod 3.
You have been warned.
(we're in the middle of a rebuild, so it's major chaos before restoration to order)
-- There is no sig line, only Zuul.
It is OK to smoke cigarettes in the server room at Philip Morris. They keep ashtrays there for the sysadmins.
FreeSpeech.org
Halon is not especially toxic (except to the Ozone layer). It is very difficult to breath but it is just about possible. I agree with you about smoke inhalation though. Server rooms have large quantities of plastics (especially as cable insulation). When burnt thius gives off all kinds of nasty crud, like cyanide and so on.
CO2 is bad when it goes off because the room fogs and you can't see the exit.
As your site says 8% is required to fight a fire, we can assume the average deployment you might not be able to run out if you hang around. Also, you might expect an electrical fire to produce the required temperatures to let those baddies out.
So yes, do leave the room. As others have pointed out, smoke inhalation will get you before any of the above does. So, screw the data, screw your stuff, screw everything and leave. When a fire alarm goes off, calmly exit the building and let trained responers do their job.
Friends don't help friends install M$ junk.
Halon Escape looks like fun.
Friends don't help friends install M$ junk.
>this might be one of the cruelest ways to enable
>job insuarance...
Once upon a time, there was a network.
One of the patch closets had a very confusing, completely lunatic wiring layout, i.e. what was labelled as jack 80 upstairs actually corresponded to jack 53 downstairs, etc. etc. (At some point a couple of trunks had been crushed or something, so a few wires had been cut. There was no money to buy more CAT5 trunk cable, so things had been sort of jury-rigged.)
The config was fairly stable, it was quite rare for anyone to deal with that mess.
There was a single copy of the "map" stapled to the patch panel wall to help people to figure out which connection went where.
OK, getting to the point, a rather serious personality conflict developed between the guy who managed the network team and his boss.
The fight raged for months, and the network team could see the writing on the wall. The only copy of the map was moved to the network manager's briefcase, and went home with him every night. (Of course he let his own team use it. No-one else even realized what a clusterfuck that closet was.)
A few months later, the entire network team had moved on to greener pastures, and the wiring moved on with the last fellow out, a bitter little souvenir.
What?! What are you talking about? My boss, she and I...
Err, crap, can't talk about that, I signed an NDA about that sort of stuff.
// file: mice.h
#include "frickin_lasers.h"
More...
Fleas, Crabs - pubic lice, scabies. And that is just from the Dell servers. The Compaq servers are the really dirty sluts.
Saskboy's blog is good. 9 out of 10 dentists agree.
IT nerd in bar: Sometime I post on slashdot without previewing. Hot Chick: Oh wow! Let me call my twin sister so we can get together for a weekend in a cabin.
MMO Quests are like orgasms:
You may solo them, I prefer them in a group.
A number of years ago a local bank and a local newspaper upgraded from the stoneage and discarded their Burroughs systems.. Being that I was the one that installed the PC's and token ring (yuk) I just HAD to snag the beasts.
:-)
I got all my friends and a parade of pickup trucks and we got all the iron into my home, with much bner to help us.
They took up ALL of the free space in my house. Good thing I was single then! A 2,200sq ft. house jammed with terminals, CPU's and disk drives where you couldn't walk through it all.
The discs were freaking HUGE, as in the size of trash can lids and held a whopping 5 megabytes.
It booted up from a punch tape and that was the code to begin the disc IO routines so it could boot up a little more and do try to do something useful. NO CRT's on this system! Teletype's only! A REAL MAN'S COMPUTER! YEAH!!
Well, we had to be dumb and power it up. The power cord for the CPU alone was 1.5" in diameter!
I powered up the CPU and disc drive (sounded like an airport in there) and the lights dimmed down pretty low. That should have been a warning sign. But Nooooooooooooooo! Crank it up baby, MORE POWER!
We loaded the paper tape and the discs spun up and all my buds flipped on the terminals. POW!!!
Total blackness. It blew my fuses (yes, old screw in type) and set my fuse box on fire. We had to call the FD to come put it out because I didn't have a fire extinguisher at home and you can spray water on an electrical fire.
The FD wrote me up a nice little fine for violating numerous safety codes and the city inspector cited me for operating a business in my home after seeing all the iron.
So much for my own private mainframe. We gutted the units and now I have hundreds of HUGE capacitors for a future project that will involve Tesla coils, water, and flux capacitors..
I don't even dare mention my current computer room. I really do not have any real idea how many computers I have now. I can see 7 running right now on the hub. I have stacks of stuff that would give any normal tech nightmares. After 25+ years at being a tech you tend to accumulate a few spare parts... Arrrrgh!!
If losing a phase in a three phase systems burns electical motors - why the heck do people use three phase systems for powering motors?
I am really green on electrical stuff, but isn't it possible to convert three phase into one phase?
Stop the brainwash
I used to work at a glass company in the greater Seattle area (name withheld to protect them). Their server room had wiring on the walls inside a half baked closet (yes it was hot, too). Now, rather than just running Cat-5A or Cat-5B they had both. The wiring in the snakes was Cat-5B and the rest was Cat-5A. Lots of fun when you have to repunch or patch on the fly.
But that is never enough is it? They also ran serial connections through the snakes without using any serial concentrators along the way. All of the wiring was thrown together as was needed with documentation a distant dream rather than a reality.
Knowing that this system was in serious trouble (RAID level 0 with no way to reinstall without calling an out of state vendor to bail us out), and knowing that a migration was looming about a year away (from AIX to Windows and I was happy about it), I proceeded to map the darn thing. Took a week of climbing through broken glass (remember where this was), scaling ladders to find runs that went nowhere, and finding that old runs were ran along the outside of the office (well, before they expanded the office that is).
After making a diagram of where everything went and checking everything twice, my manager started ripping out wires before I could get there. Fortunately he didn't pull anything vital and we remained up and running, but I really wanted to deck him for that.
In the server room I pulled out over 500 feet (I measured it as I didn't believe it) of dead wire. We had only three computers in there but major runs ran down into the electrical room below.
Imagine standing on an electrical transformer while installing a new hub and really hoping that you don't slip onto the trash laden floor. That was fun!
I didn't mention that the server room was directly above the electrical room and that above the server room was an ancient air conditioning unit that would dump its condensation on the server did I? Moving the server protected it from the 'rain', buckets prevented the water from pooling, and the cleaning crew would remove the acoustical tiling when they would rot and fall down.
Ask me if I miss the place.
"Giving money and power to governments is like giving whiskey and car keys to teenage boys." - P.J. O'Rourke
Why do a lot of server rooms have raised floors?
Is it for the cables, air-con?
The Head of Security(tm) gave fire safety lessons to every employee. So we went behind the plant down the street and learned how to use a
If you aren't part of the solution, there is good money to be made prolonging the problem
I work right next door to two of our larger data centers in Michigan. (These are for an unnamed, VERY large IT provider / slavedriver) Each of the raised floor areas in these data centers are about the size of two football fields. We, too, have the SCBA (self-contained breathing apparatus') spaced out as well. I'm not sure what kind of system is used for extinguishing fires in the place, but I don't believe it is water based. None the less, if the alarms started going off in there, you can bet your ass that the first thing I would be looking for is one of the columns with a SCBA mounted on it. I'd rather take a few seconds and grab one of these than try to run to an egress on whatever breath I'm holding.
Three phase motors are more efficient. Consider single phase or split phase electrical current, where every cycle has a dead spot while alternating polarity. During this time, the motor has no torque. The motor can be thought of as pulsing the load. Now look at the redundancy provided by three phases and our happy motor has no dropout periods of magnetic excitement. And all phases of rotation have a strong pull. This is good.
Same can be said for three phase generators. The alternator in your car is three phase for this very good reason. If your car alternator was only single, or split phase, your car stereo would pick up some serious electrical noise. The lower parts of the three phase waveform are minimal compared to the zero crossing point, or dead spot, of only one phase.
Also, three phase motors can be reversed. This useful feature can be done by "reversing" two phase wires. The three phase waveform actually has a direction and you can rotate a motor in the opposite direction by swaping two wires.
Converting three phase into one phase is easy. Just only use any two wires and that will be your single phase relative to each other. Or if the three phase is in the Y configuration, pick any phase and the neutral wire. That's how you can get 277 volts off 480 volts. Any two phases will get you 480, while one phase and neutral will get you 277 volts. Touching one wire will give you the nasty shock of 277 volts and possibly burned. Touching two phases will give you the full 480 volts and quite possibly blow your fingertips off. It just depends on how you take your power off the transformer windings.
And if anyone asks why use voltages so high, its because of safety reasons. Higher voltages mean less current, smaller wires, less chance of overheating and fire. The added requirement of metal sheilding is cheap compared to much thicker wire and the possibility of loose connections of unusually bulky high amperage circuits. A rather small, flexible 480 volt wire can take the place of a very heavy, difficult to place 120 volt wire. If we were to supply our building with just 120 volts, rather than 14.4KV, the copper bus bars would be insanely huge, and could explode if thermal tension created loose connections. The 14.4KV allowed us to use small, flexible, underground cables with easy to install connections. Higher voltages require extra safety guards to protect against the environment, but its worth it.
My one-time employer had a room full of servers and test machines.
To keep them running 24/7, the machines were on an internal generator, not the standard power lines.
The air conditioners were on the standard power lines.
One Friday night, the local power company had a blackout.
What was found Monday morning is left as an exercise for the reader.
...was one I inherited when I was working for one of the railway companies in the UK. The network at the time was all 10base2 and there was a multiport repeater on every floor, each connected to the main repeater up in the server room by an individual length of thinwire that ran up a central riser.
Some time before I started work there the cable running down to the seventh floor had failed (probably hungry rodents) and my predecessor had come up with a cunning workaround rather than going to the hassle of laying a proper replacement. He had run a thinwire cable out of the window of the server room, down the outside of the building and in through a window on the seventh floor (I really don't want to know how) which was then run along the ceiling using a whole load of bent paperclips rammed into the polystyrene ceiling tiles, and then into the comms cabinet that housed the repeater.
I was told by one of the staff there that this temporary solution had been in place for months, with only occasional outages. Then again, given the fact that the server room had no racks, shelves or airconditioning and the servers were just piled on top of each other with random assortments of keyboards and monitors dotted around, nothing there surprised me.
"Are you being weird, or sarcastic?" said Emma. I said I didn't know because I get the two feelings mixed up.
At the General Motors Design Building in Warren, MI, if you go in many of the the studios and offices the ceilings are brown. Why? Up until sometime in the 80s (I believe, might have been the 70s) employees were allowed to smoke at their desks. To this day the buildings still have a mild lingering smoke smell.
My manager also tells stories of how when she hired into the company (not GM) she could smoke at her desk but she couldn't wear pants.
What a better workplace now...
Finally, the demo was over and we thought we were safe. One Saturday morning at 5AM, after a huge blizzard in NYC, I got a call from building security that there had been a massive flood originating from the residences in an area directly above our data room. Turns out the owner had installed forced-water heaters in the apartments and hadn't insulated the pipes, which had frozen and burst. Although there was about 1/4" accumulation on the floor in the data room, the leak there had missed the nearest cabinet by about five inches. The adjacent NOC room was completely destroyed, but at least it was just monitoring equipment and not production servers and telephones. We somehow got everything ready in time for the market open Monday morning.
By the time we found out the owner planned to build a parking garage in the building's basement, he had already begun demolishing what was left of a restaurant and storage rooms that used to be down there. Keep in mind this is a 100-year old building that had telecom infrastructure for most of the 100 years. When we went down to see what was going on, we found they had demolished suspended ceilings and framed walls that were in some places the only things holding up bundles of 1000-pair copper cables. They had demolished a cinderblock wall to which the 1000-pair splices had been attached. The fiber to Verizon's frame room had only been installed in innerduct, and was lying in the mud and waste. One of our WorldCom 25-pairs (Internet, 800 service, market data, NASDAQ) was running at eye level straight through the middle of the demo area, where forklift drivers had to lift it out of the way each time they passed. The owner had started a war with the building unions, and a couple of suspicious fires had started down there. Oh, and there was loose asbestos in part of the site, so the telcos refused to work until the owner had it abated and environmentally certified.
For the six months it took us to get the owner to clean up the asbestos, for the telcos to install new access away from the demolition, and for us to install steel conduit for our service, we faced the notion that we might be out of business at any point. We did finally get the telecom infrastructure hardened, but the real benefit of this set of circumstances was that they forced us to implemented a real business continuity plan and build an offsite recovery capability in northern Jersey. Although our motivation was due to the landlord's recklessness, the Jersey capability saved our firm during the time we were shut out of our building after 9/11.
Leave the fire by the door, and then enter the room. Much safer.
Shop as usual. And avoid panic buying.
You know, dihydrogen oxide is pretty dangerous in high concentrations too.
Don't you mean DHMO, or dihydrogen monoxide? (Do Google search for DHMO.)
I'll see your senator, and I'll raise you two judges.
Not sure if this counts as datacentre, but we had a brand new generator installed at a place where I worked once. The thing was the size of a shipping container (you know, those huge metal boxes) and contained a Volvo marine diesel engine, a 400 litre tank, 4 batteries (for the starter) intake filters the size of beer barrels and on the roof, a silencer the size of an oil drum with a 8 inch outlet.
:P
The project to get this installed had run for months, with legal difficulties, environmental people, the simple logistics of getting the thing craned over the building as there was no access etc etc. All this meant it had cost a packet - but we had to have a generator to satisfy the banking authorities.
So, the fateful day arrives and it's fueled up, and plumbed into the building power supply. The "dashboard" for the engine is in the building plant room and you can't see the generator from there. We switch the bypass circuit in and fire it up for a test run.
Now, this engine had never been run since it was signed off at the factory, so it contained a lot of protective oil coatings which added to the already substatntial smoke that would normally exit the tailpipe of a 7 litre turbo diesel. After the rev counter hits a steady 1500 rpm , I go round to look at it running.
Yep - there it is chugging away, with smoke like black toothpaste falling out of the exhaust. Within seconds, I couldn't see my own hand, let alone the generator or the building beyond.
After 2 minutes or so, I reckon we've seen enough for now, so I go back to the plant room.... to discover that it has filled the building with toxic fumes, the fire alarm has gone off, and the entire staff are down the bottom of the car park behind 2 fire engines....
Bit of a design flaw that - turns out the air intake for the building AC is right next to the generator exhaust... (not my doing I hasten to add - it was all perpetrated before my arrival). Needless to say, the generator was never switched out of bypass - we couldn't take the risk that it might actually start - it might keep the computer systems running, but would kill all the users... hmm wait a second..
$100,000 worth of useless metal!
Our server room is small enough that, if the buzzer goes, you hold your breath and run for the door. There's a 'slap switch' which you can hit with your palm and get out.
The absolute first service we set up on anything is ssh. Then we administer *everything* remotely.
dave
um - no.
It means that we learned a lot about the network we had - a lot about how to properly wire it - and a lot about budgets, planning, roll out schedules etc....
Sure - people make mistakes, but anytime anyone says they learned a lot it does not always translate to "I fucked up a lot"
I worked for a company called Maximum Charisma Studios last year before they went chapter 7. We had a Cisco 3548XL 1U height switch on a bracketed wood shelf too small for the switch, and all of the patch cables dangling below it because there was no way to secure them.
I asked the boss several times to buy me a $200 aluminum relay rack so that I could mount it down and take care of all of our cables, eliminate the possibility of the thing falling off.
About a month later I was working in the server room trying to move some rack servers around and the damn switch fell off of the shelf, flipped over, and hit me on the head.
The damage to my head was minimal, but it hurt. It could have been serious had the metal corner hit me on the head instead of the flat part.
The damage to the switch was pretty bad. The IEC 320 power port was bent and damaged and ten of the RJ45 ports had their retaining clips ripped out, meaning that plugs would no longer clip in. The network was down for about 30 minutes while I replaced it with another switch that we just happened to have at the time.
Finally after the incident did the boss allow me to buy the relay rack. On the same day that I was about to install the thing we had the "You are all fired!" meeting with the CEO.
The only good part about it all is that I got to keep the Cisco switch. It still has 38 good ports, and I was able to repair the bent chassis and solder in a new IEC 320 plug. A surface mount micro fuse also blew up that controlled the blower in the switch, so I just shorted it with a wire. It works, and a I have a Cisco switch now.
And that is my story.
I am curious about data center safety requirements. I think that it is three feet width between all relay racks and data cabinets, but I would be interested in official documents if anyone knows. Thanks in advance.
I've actually been in a data centre when the halon system was triggered. Everyone got out before the actual release.
That's barbed wire. :-) Bob doesn't have his own standard yet.
I can't remember where exactly I heard this from. It was either one of the techs at either of Cabletron cert classes I took or it was one of my peer institutions. It seems that during a school break, some construction work was done in a building. On one of the lower floors of the building, not in an area under construction, was a rack with a Cabletron MMAC8. It was a nice rack with a door and everything. Apparently the construction folks were poring concrete on the floor above the rack. There was a small hole in the floor and concrete kept seeping out it. The contractors kept adding more concrete until it stopped seeping out that hole. Unfortunately the hole was directly above the rack. The netadmins came back to campus once the break was over. Eventually one of them noticed that the rack looked a bit odd. The opened the door only to find that the MMAC8 had been encased in concrete. it was still working fine, chugging away. The fiber lines were sticking out the front and everything. Talk about drastic measures in physical security!
Damn. This'll undo two moderations, but I can't let this one lie.
do you really think there is just this dangerous gas ready to be released where you work
Son, anywhere I work is a place where there is dangerous gas just waiting to be released, and it usually doesn't have to wait very long. Fortunately, the alarm is loud enough that fatalaties are rare, just the occasional nausea and vomiting. There are NO cases of long-term exposure.
Breathing high concentrations of CO2 can cause distress because the ph of the blood stream drops due to the presence of Carbonic acid which can't be eliminated as CO2 by the lungs.
Actually, the best system is called HiFog. It is used for server rooms, but was originally designed for ships. It is basically a water misting system. The main benefit that it offers over FM200 et al is that it can activate and actually scrub the air free of the smoke.
Usually, a fire starts as smoke, rather than flames. If you can put out the smoke, it can avoid the fire in the whole place.
BTW, most cities require that you have a dry pipe backup system, in case the "fancy" things don't work. The goal of anything else is really just to try and prevent equipment damage...
Did work for a company that had outgrown its server room, so they decided to have a server in each cube that could support it. How do you decide if you can add a server? Well, if the breaker trips, try another cube...