Slashdot Mirror
The Most Dangerous Server Rooms
Ymerej writes "The Register has an article on dangerous server rooms. Have you seen worse?" Perhaps The Register would like a picture of my desk if they really want to be scared.
← Back to Stories (view on slashdot.org)
. . .where the city rats are bigger than the IT guys.
And they carry card keys.
-bpl
Now that's how you ensure job security!
- tristan
calling application programmers rats is insulting to rats, Pls refer to them by their proper designation...(L)Users :)
errr....umm...*whooosh* *whoosh* Is this thing on ?
My server room is so bad not even the rats will go in there
For example, statistics show that people who work in server room almost never catch any STDs. I wonder why that is...
It's all about removing floor tiles and then forgetting to set up warning cones. The clearance between our tiles and the concrete floor underneath is a good 4 or 5 feet; I would not want to fall stiff-legged into that.
Interesting side note: apparently finding high-priced Cisco gear not connected to anything is not that uncommon. I've also heard horror stories of guy that traced a cable my hand(toner was on the fritz) that looped 4 times around the data center but wasn't hooked up to shit, on either end. Took him an entire afternoon.
Finally, math books without any of that base 6 crap in them.
Check it out here.
;-)
This is by far to custom case I've ever seen.
Look a bit dangerous though
You want scary:
Our server room has wobbly floors and tower cases stacked 2 high sitting in the middle of a 16'x16' room, with the more important servers sitting on the top.
We also got a new air conditioner that has an electronic switch and we have problems with brown outs. So in the middle of the summer, the power goes out and the A/C doesn't come back on, usually on a weekend too.
One more thing: brace yourself:
We use Windows servers with IIS!!!!!!!!
about 3 years ago, I was pulling cable from 1 room to another.. while standing on a ladder pulling the cable out of the cieling, the cieling collapsed on me! About half the tiles from the room fell on to the floor, and a florescent light hit me in the back on its way down.. that's pretty dangerous!
Check this out:
;)
www.theregister.co.uk/media/926.jpg
That's a HELL of a case mod
IGB: More fun than eating oatmeal!
In rural Indiana, you don't always have space to have a whole room devoted to servers and network equipment, ya know?! But I was still surprised when I visited my former ISPs local point of presence - in one of their employee's one and only bathroom at his house. Photo here. Do some laundry, take a dump, watch some network traffic go by. Uh-huh.
...looks like some new cybernetic monster I imagine will make an appearance in Doom 3.
That first picture looks like a cross between something out of a Terry Gilliam movie, a Borg Cube and Tetsuo the Iron Man. The ones at the bottom are just plain bizarre.
These aren't servers, they're representative works of art- kind of like a city. Granted, these servers will die the same flaming death that Chicago did in the great fire, but it's still kind of neat.
That's ok, Jesus likes me anyway.
You want dangerous, well the only fire prevention we have for our 23 servers, is one halon extinguisher. So if the room goes up, we can save it, as long as we don't want any oxygen...
Xaotik Designs
I'm messy by nature, but I know that as long as I'm the only person who understands the mess, I'm indispensable
And I got a new job yesterday... I pity my successor
I don't read your sig, why do you read mine?
... when my foot broke through one of those floor tiles in the server room. Funny, someone told me they were high resistence, that must not be exactly the case ;)
That doesn't surprise me. I have a strange feeling many of you have been caught doing this in front of your pc.
So much for my Delta Force keyboard layout ;)
"I can't give you a brain, so I'll give you a diploma" - The Great Oz (blatently stolen sig)
I bet he's the guy that invented tcpdump.
We launched a website with publicity on a live primetime TV show about the internet (in the UK), while the server (singular) was still running under my desk. It was a little while before we moved it out of there and, amazingly, I never accidentally shut down the site with my knee.
Of course, when we did, eventually, move it into a server room, the aircon subsequently broke down and, being an underfunded dotcom, nobody wanted to spring for repairs. We lost at least one server that way (thankfully not a live-facing one).
These messes are on par with an average recording studios. At least in the studio you can see sparks leap from the microphone at the guitarist.
Statistics also show that "server room affairs" by non technical personal, are responsible for more than half of office place STD transmission.
The "photo copy room trysts" follow in a close second place, while the "boss's office boinkings" are becoming more uncommon as more CEOs are of the female gender.
Saskboy's blog is good. 9 out of 10 dentists agree.
Those are nothing. NOTHING, I tell you!
I've got a set of routers located in a crawlspace where the only way to get to them is to walk across boards spanning small metal beams that were put in to hold a suspended false ceiling. One missed step and you'll drop right through the ceiling, AND IT'S A 2 STORY DROP! Once I dropped a power pack while replacing it and nearly killed a gal working below. Power pack exploded like a bomb when it hit.
We recently had a "security audit" where they recommended we should mount those routers in a locked cabinet for increased security. Not a mention about the boards, lack of handrail, safety net, etc. Heck, who needs a locked cabinet? Just remove one of the boards and NO ONE can get to those routers, not even the people who are supposed to maintain them!
Back when we used thinnet one of the managers didn't like stringing new coax through the building whenever we remodeled or moved people, so he had us cut all the coaxes to length PLUS 25 FEET! He figured if someone moved we could pull back the excess and save time. The cables all terminated in what came to be called the spaghetti room, from the coils of coax all over the floor. We had to step over all the coaxes to get to the routers and hubs. Eventually, the coaxes got damaged from all the abuse and had to be cut off to length anyway, but for several years it was a serious tripping hazard for anyone who entered that room.
Beta sux! Join the Slashcott! http://hardware.slashdot.org/comments.pl?sid=4760465&cid=46173047
My personal favorite was building a small network out in a field. We set up our four machines [286's] in the dirt, got our power from a generator being towed by a five-ton and wired together on a 10Base2 network. For the first day or so the only shelter we had for the machines was a tarp that we pulled over them when it started to rain.
Lying on the ground, underneath the leaky tarp, hoping that I did not get electrocuted, or if I did that I would not be held accountable for the damaged equipment [trust me, this was not my idea], I decided that re-enlistment was not a great idea.
[former] USMC geek
I just knew there was a punchline there that I was totally missing out on. Damn. Thanks for that. :)
That's right you read that correctly.
I once visited a client who had his server racks in an old lockerroom shower. This would not have been so bad except when one of my co-workers leaned against the wall and hit the valve we discovered that the pipes hadn't been capped by just had the shower heads removed....that's right three full racks of equipment in a live shower. =)
I was told this story from a reliable source...
An HP technician (yup they have at least one) was restoring the data to a customer's fileserver but the backup software was asking for tape#2. The customer only ever had one backup tape that they recalled, so they were quite perplexed until the security guard entered the server room...
Apparently every morning around 3am when he made his rounds he found the backup server screen blinking "insert next tape" -- The security guard proudly said that he was pushing in the tape for at least six months now...
From excellent karma to terible karma with a single +5 funny post...
As a long time Customer Engineeer for a major manufacture
I'll take a blonde, 6 feet tall, please. Make her lippy, but only to a point. Oh, and she has to know how to cook. Nothing special, just the basics. Thanks.
Finally, math books without any of that base 6 crap in them.
This brings back memories.
I remember working at an isp way back when and the server room was so bad that you could basically lean on the rats nest of wiring like it was a makeshift hammock.
Walking around behind the racks meant being completely aware of which line you put tension on, lest it knock some connector not fastened down out of place.
I would love to see what this person could do with Christmas tree lights
I have a kind of similar story. Our UPS and generator are attached to three sets of strobe lights and alarms, one on each entry door and one inside the datacenter. Well one day I go into the datacenter because the UPS alarm is going off making all sorts of racket. I try to clear the alarm and figure out what is going on, well while hitting the button to clear the alarm my finger slips across the surface mount switch to the one under it. Problem is the switch under it is the OFF button! Who makes a UPS with a surface mount off button let alone one not protected by a flip guard? Well I have to say that I now HATE the sound of silence, at least with respect to datacenters. Worst part about the experience, the alarm was actually just a notification to perform routine maintenance, clean the air filter.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Taiwanese typhoon? pfffft. Come on - lets get some webcams in these dangerous server rooms and then /. em for some good old fashioned geek fireworks.
Yeah - you know you want to.
__ Someday, but not this morning, I'll finally learn to use the preview button.
then accountants with key cards could not be rats, just simply bean-counters, (another demographic group to toy with MWAHAHAHA! )
:)
What's your (l)username?...ok...**clickety click** Why thank you for increasing our IT budget by 1 billion$...now i'm afraid you'll have to go. Do me a favor and check if the pover is on, just stick a piece of metal in the mains... You won't feel a thing
Live for the present, learn from the past, and dream of the future!
They're both in the billing office. I tried making them lethal so no one would touch them, espcially considering the boss's kids seem to think the 2000 server should be used for chatting up their friends via AIM and has no other purpose. Unfortunatly, reguardless of the mass of cables running in to the hub next to it and the fact I attempt to hid any interface devices they still seem to muck with it whenever they have a half day from school. I think I'll just set up a mine field around it and call it a day.
Luckily they've never found a use for the SCO box other than looking at it with a wrinkled nose. I do too, mind you, but for entirely different reasons I would guess.
I love when people don't properly plan their electrical power consumption in their server rooms. I walked into some company's server room, plugged in my laptop to the rack mounted power strip, turned it on, and blew the breaker for two racks of servers.
I dunno, it sounds like they planned their power consumption PERFECTLY.
"And like that
How loud can spiders actually get?
kinda makes me want to start redesigning my company's server room to get into the register.
this might be one of the cruelest ways to enable job insuarance...
i used to work in a server room like this..i remember talking to the guy that replaced the Halon and maintaned the system.."Ok so..ive got a certain number of seconds to get out...or i won't be able to breathe...right..no problem"
I got a call from a small publishing company to do some work on their machines (they just bought some new Macs). So I looked around, and found the network cables, and the printer, and...
"Um, where's your print server?"
"We don't have one."
"Yeah, you do, all of your machines are talking to it, it's here somewhere."
"I've been here seven years, and we don't have a server."
I traced the cables into a closet. That's blocked off by a workstation/desk. After some convincing, I managed to get them to let me move the desk, and I got into the closet. Where I found a 1987-vintage Mac II, happily munching along as a print server. Hooked into an old phone company-style UPS. Covered in a solid inch of dust and debris. And running without anyone noticing it for at least seven years...
IT nerd in bar: Hey, baby, you like dangerous guys?
Hot Chick: Yeah, I guess so.
IT nerd in bar: Sometimes I take my pocket protector out of my pocket!
Hot Chick: Uh-huh...
IT nerd in bar: Sometimes I take my mint condition Megatron action figure to LAN parties!
Hot Chick: Uh-huh...
IT nerd in bar: And sometimes I let my server room get really messy so that it's a hazard to my life!
Hot Chick: Wow, that is dangerous. I'm really turned on...hop on me right now...
"It's a tarp!" -- Dyslexic Admiral Ackbar
I'm sure that some of you have worked in large server rooms with a big red emergency power shutoff button on the wall...
:)
At my old university, one of these server rooms was emptied as new, smaller hardware came available and the room was no longer needed. They turned this room into an office for a student organization... leaving the large red button, but taking the "Emergency Shutoff" sign.
This unlabeled button sat neglected on the wall of this little office for about 7 or 8 years until one day a curious student just had to find out what the button did...
The network for all of the engineering schools at this university of 36,000 students went down for most of a day..
The best part is that the button is still in the little office with the students, and it is still unlabeled yet fully functional... They did hide it behind a file cabinet, though
There are 10 types of people in this world, those who can count in binary and those who can't.
would someone please flush the mail queue?
oh no, the logs are getting backed up!
I have no special gift, I am only passionately curious. --Albert Einstein
I used to consult for a client whose server room was in the women's restroom. It had the largest unallocated space of any room in the building, so they stuffed in two low-boy cabinets full of DEC gear right next to the ladies' crapper.
I had to remember to knock before rebooting.
You kids these days just have it too fscking soft, I tell you...
That is all.
http://www.tfb.net/~nicl/images/?image=EthernetKil ler.jpg
That'll make any machine room the most dangerous.
at my university they were cleaning up a large server room for renovations of the building. while removing the equipment, the crew stumbled on a stray bundle of wires that seemingly led to nowhere. they literally dissappeared into a brick wall. they decided to tear down the wall, and lo and behold there was a unix machine. it had been walled-up while still running. it had an uptime of over 15 years!
Not strictly rats-nest, but it follows from the previous.
This goes back to the sixties, when computers had twitching reels of mag tape and paper tape was king.
The company had a regular overnight run. A control tape was put into the high speed tape reader, all the relevant mag tapes mounted, and the computer got on with its six hour job (about 20 second job by todays standards). Originally there was an operator on duty, but he blatantly had nothing to do, so they decided they could do without him.
But as soon as the operator disappeared, the job started failing at dead of night.
OK, bring back the operator - he can fix the problem and restart the phase which went wrong.
But, as soon as the operator came back, the problem went away. And this was the pattern - if they watched the system, it worked perfectly. But left alone, it invariably failed.
So an engineer decided to sit there and not touch anything. He told the operator to go away, as if he was't there. Which he did, turning the lights out and leaving our hero in the dark - except for the glow of the high-speed tape reader, which shines a strong light through the holes in the punched tape onto photocells. And as he watches, a moth appears and flies through the pool of light, confusing the tape reader and aborting the job.
At the old company that I used to work for. They had no server room yet so the servers just sat in the corner. One day I was working late and the cleaning staff came in. The first thing that they did was to plug the vaccuum cleaner into the UPS that the main server was hooked up to.
It seems they had always been doing that!
134340: I am not a number. I am a free planet!
A similar story was told to me by an "old timer" who claimed to have been told this by a technician at a Digital training course.
There was a site in London, in a multi-story building facing the Thames. Apparently the operators room had a nice view of the river and some shipping wharves.
There VAX/VMS system was crashing about once a month. not every month, but it happened enough that it was obvious that if the system would crash, it would be sometime in a particular 24 hour period. The system had been checked thoroughly, and has so many parts replaced it was almost a new system, but still it crashed.
In the end, a Digital technetium was dispatched to spend the night in the server room and be they're when the system was due to crash. He was standing at the window, looking at the UK Navy Battleship that was tied up at the wharf, by the light of a full moon. As he was watching the ship was slowly rising on the tide, until the radar antenna reached the level of the window, and behind him, the system crashed.
Once a month, the tide was especially high, and the Battleship's radar lined up with the server room window. Shielding the window fixed the problem.
Democracy isn't about no one telling you what to do. It's about everyone telling you what to do.
When our old NEC mainframe came out we had a few tiles with 18"x10" holes formerly used for cabling in our raised floor. We still don't have exactly enough tiles, but furniture is arranged better now, a year later, so that these holes are strategically covered by desks, shelves or other equipment.
Better than accidently wheeling your chair over that duct tape patch! I kid you not.
Actually my college still has them, and they work.
My boss was telling me about how a guy working in the AC system kicked up some dust and it triggered the Halon system. A voice came on to announce they had 15 seconds to get out of the room before it would be deployed. My boss of course hits the button and stops the countdown. But he lets ago, apperantly you have to hold the button until someone can come by and turn off the system. So he and the network admin go diving out of the room just as the halon is released.
There are other labs on campus that have Halon warning labels on them also, and I wouldn't dare try to check if its true.
...
"I'm sorry, but comfortable is the last thing I want in my server room. I want it unbearably cold, and noisy. I want items scattered dangerously around the floor. I want random floor tiles to be missing. I want a very old sandwich of undetermined origin sitting half-eaten in the corner. I want the first thought of any person that enters my server room to be "Dear $DEITY, I must get out of this place IMMEDIATELY!"
Time flies like an arrow, fruit flies like a banana.
Heard a great one while working at a steel plant.
A few years before I started, they were having some issues with a server apparently rebooting every evening, around 11pm. They checked everything they could think of, and found no problems with they system. After a few weeks, they just couldn't nail down the problem, so one of the guys stayed late to see what happened.
Around 11pm, in walks one of the night cleaners. She reaches down, unplugs the server from the UPS, and plugs in the vacuum sweeper. She cleans the carpet, unplugs the sweeper, plugs back in the server, and leaves.
We suggested that she use a wall outlet, and our server problems were fixed.
Back in '86 I had a top-of-the-line Corona IBM-PC clone (cost me nearly $5,000 then). It had those big full-height floppy drives (two!) and was a very well-built, sturdy unit.
I was working as a computer hardware technician at the time, and I had recently bought a bunch of 256k memory chips. I brought my computer to work to show it off to the guys, and also install the memory where I had a nice anti-static station.
So there I was with all my buddies, showing off my toy. I open the case of my computer, ready to wow them, and at least a pound of dog kibbles spills out of the case. Dog kibbles are strewn all over the computer motherboard. We all kind of stood there for a moment, dumbfounded.
Eventually, I discovered the cause. My house was infested with mice, this I had known. But what I didn't know was that, in the middle of the night, mice will steal dog kibbles from the dog dish, and hide them in little places they can get to later. Apparently, they had been climbing in through the full-height floppy drives and storing the kibbles.
Interestingly, it never seemed to affect the computer!
If I see ANYONE near the server room with a camera this week, I'll personally remove your jimmies and make them part of megapod 3.
You have been warned.
(we're in the middle of a rebuild, so it's major chaos before restoration to order)
-- There is no sig line, only Zuul.
It is OK to smoke cigarettes in the server room at Philip Morris. They keep ashtrays there for the sysadmins.
FreeSpeech.org
I did a contract for Teal Datacenter, the datacenter for the state of California. Arcus (now known as Iron Mountain) ahd come in to take our disator-recovery tape offsite.
However, this guy was not familiar with our datacenter.
He pushed his dolly full of tapes (about 75 canisters per day) up to the ramp that leads out of the raised floor area, but stopped when the double doors didn't open. So he stopped at the base of the ramp, walked around the hand rail, and pushed the button that opened the doors.
I might mention that this is before people thought of putting those little flip-up plastic panels on the EPO buttons. Yes, he pushed the Emergency Power Off button.
I never heard what happened to him, but I never saw him again...
If all you have is a hammer, everything looks like a nail.
Halon Escape looks like fun.
Friends don't help friends install M$ junk.
Worked at a large financial institution that used Tandem servers for some system processing.
For those of you that don't know, Tandems run diagnostics on themselves and will phone in service orders to replacement parts as they get iffy or fail.
But they never stop running...
Anyway there was some remodeling going on in the building and the carpenters doing the remodeling didn't want to get dust or dirt on the lovely Tandem computers so they covered them fairly completely with plastic sheeting.
We start getting an assload of fault notices on the Tandem cluster, our Tandem tech calls us to tell us that he's gotten 20 service requests in the last five minutes.
We go down to where the machines are and they have very neatly and in an air tight manner shrink wrapped themselves...
We tear / cut (always good to have a knife on you!) the plastic off and luckily all was well.
We then cordoned off the servers with some drop cloths and asked the nice union carpenters to not do that again.
More...
Fleas, Crabs - pubic lice, scabies. And that is just from the Dell servers. The Compaq servers are the really dirty sluts.
Saskboy's blog is good. 9 out of 10 dentists agree.
IT nerd in bar: Sometime I post on slashdot without previewing. Hot Chick: Oh wow! Let me call my twin sister so we can get together for a weekend in a cabin.
MMO Quests are like orgasms:
You may solo them, I prefer them in a group.
A number of years ago a local bank and a local newspaper upgraded from the stoneage and discarded their Burroughs systems.. Being that I was the one that installed the PC's and token ring (yuk) I just HAD to snag the beasts.
:-)
I got all my friends and a parade of pickup trucks and we got all the iron into my home, with much bner to help us.
They took up ALL of the free space in my house. Good thing I was single then! A 2,200sq ft. house jammed with terminals, CPU's and disk drives where you couldn't walk through it all.
The discs were freaking HUGE, as in the size of trash can lids and held a whopping 5 megabytes.
It booted up from a punch tape and that was the code to begin the disc IO routines so it could boot up a little more and do try to do something useful. NO CRT's on this system! Teletype's only! A REAL MAN'S COMPUTER! YEAH!!
Well, we had to be dumb and power it up. The power cord for the CPU alone was 1.5" in diameter!
I powered up the CPU and disc drive (sounded like an airport in there) and the lights dimmed down pretty low. That should have been a warning sign. But Nooooooooooooooo! Crank it up baby, MORE POWER!
We loaded the paper tape and the discs spun up and all my buds flipped on the terminals. POW!!!
Total blackness. It blew my fuses (yes, old screw in type) and set my fuse box on fire. We had to call the FD to come put it out because I didn't have a fire extinguisher at home and you can spray water on an electrical fire.
The FD wrote me up a nice little fine for violating numerous safety codes and the city inspector cited me for operating a business in my home after seeing all the iron.
So much for my own private mainframe. We gutted the units and now I have hundreds of HUGE capacitors for a future project that will involve Tesla coils, water, and flux capacitors..
I don't even dare mention my current computer room. I really do not have any real idea how many computers I have now. I can see 7 running right now on the hub. I have stacks of stuff that would give any normal tech nightmares. After 25+ years at being a tech you tend to accumulate a few spare parts... Arrrrgh!!
tacos equipment
Nero-burning ROM for Linux!
My one-time employer had a room full of servers and test machines.
To keep them running 24/7, the machines were on an internal generator, not the standard power lines.
The air conditioners were on the standard power lines.
One Friday night, the local power company had a blackout.
What was found Monday morning is left as an exercise for the reader.
Leave the fire by the door, and then enter the room. Much safer.
Shop as usual. And avoid panic buying.