What Bizarre IT Setups Have You Seen?

MicklePickle wonders: "I was talking to a co-worker the other day about the history of our company, (which shall remain nameless), and he started reminiscing about some of the IT hacks that our company did. Like running 10BaseT down a storm water drain to connect two buildings, using a dripping tap to keep the sewerage U-bend full of water in a computer room, (huh?). And some not so strange ones like running SCSI out to 100m, and running a major financial system on a long forgotten computer in a cupboard. I know that there must be a plethora of IT hacks around. What are some you've seen?"

Comments

the U-Bend

[Helix150]

Just to clarify- the U-Bend is what prevents bathrooms and drains from smelling horrible. Inside the drain, shower water, sink water and toilet waste all mix together. As you can imagine this smells horrible. So, where every toilet, sink, shower, etc connects to the drain system there is a 'u-bend'- a downward dip in the pipe which stays full of water. This prevents air from flowing out of the empty drain.
Most sinks have their u-bend visible under the sink and look like this:
http://twenteenthcentury.com/uologos/ubend_shaded. png
Water flows in the top, and out the back. Because the back is higher than the bottom of the bend, the bottom stays full of water at all times, preventing air from passing.

Problem is, if you leave a drain long enough without water passing through it, the water in the u-bend can evaporate, leaving an empty pipe and allowign the nasty sewer smell to escape. Thus, leave a faucet dripping to keep the U-Bend full!

Re:the U-Bend

[Ninjaesque One]

I would think that running the water for 5 minutes while using it(exempli grata:to wash greasy face after long day at work) would cost less than leaving it dripping.

But I guess you guys aren't responsible for utility bills and stuff.

Re:the U-Bend

[TFoo]

The U-Bend isn't just for smell, it is also a safety issue: sewer gases can be poisonous or even explosive if allowed to collect.

http://en.wikipedia.org/wiki/Sewer_gas

Re:the U-Bend

[Helix150]

running the water at any sort of regular interval will keep the u-bend full. For the U-bend to evaporate would take weeks or months probably. Even the slightest drip should more than counter the evaporation. And it probably seemed like a better solution than a server room which stank up every month for no apparent reason :)

Re:the U-Bend

[Animats]

Yeah, and it's a huge hassle in vacant commercial buildings. Somebody needs to run every water tap and flush every toilet about once a month, or the whole place will stink up. Then the smell gets into the carpeting, which makes it hard to rent the building.

For special situations, there are calibrated drip valves. These are often found as part of fire sprinkler systems, which usually have a drain valve for when you need to drain the system for maintenance. The water from the drain valve has to go somewhere, which usually means a sewer connection. But you can't hook a water line to a sewer line; there are situations when you'd suck sewerage into the water system. So there has to be a vacuum break open to air. After the vacuum break, there's a U-trap with water to keep sewer gas inside. But since such drains are seldom used, the water will evaporate. So a tiny bit of water has to be dripped into the drain to keep up with evaporation. There are special "drip valves" for this.

One of the things you need to know about if you run large data centers.

Re:the U-Bend

[Pig Hogger]

So a tiny bit of water has to be dripped into the drain to keep up with evaporation. There are special "drip valves" for this.

I was surprised to see, in a large warehouse store, some automatic urinal flush reservoirs (they flush the toilets every so often) whose output was hooked to about 30 small half-inch pipes going into the floor. The reservoirs were installed about 20 feet high on columns.

Some amount of cogitation was needed to realize that each of those small pipes was headed to the traps of the floor drains installed throughout the store...

Now that's a (plumbing) hack in the true meaning of the term!

Re:the U-Bend

[senaattori]

I pour vegetable oil into the u-bend of our server room's sewerage. Vegetable oil itself doesn't evaporate very quickly. As it floats on top of water, it forms a layer which prevent's the water from evaporating.

Re:the U-Bend

[ozbird]

Fill the trap with cooking oil - it will stop the smell and will not evaporate as quickly as water would.

Please don't. It's a hassle to remove in the sewage treatment works, and can solidify into a oil/water goo that clogs the pipes.

Instead, simply fit the plug or cover the drain - it keeps the smell out, and reduces evaporation. (If fitting the plug might cause the sink to overflow due to a dripping tap, you probably don't have an evaporation problem.)

Re:the U-Bend

[GerryHattrick]

When I started, pre-computer, the accounts department had rows of incredibly noisy mechanical 'Marchant' multiplying machines, each on a resonant all-steel desk. The production department kindly sent up inch-thick felt pads, and the racket subsided. Then came re-equipment with the precious 'electronic' machines (Anita*) with a line of hot number-valves ('tubes') along the top. But accountants are traditionalists, so the felt pads loyally stayed on, as a kind of sympathetic magic for quiet calculation. Ventilation was supposed to come from below, but of course it never came through thick felt. Thus the Anitas were extra popular - because if you put your meat pies on the generous top during the morning tea-break, they'd be hot for everyone by lunchtime. * http://www.xnumber.com/xnumber/photo_anita_C_VIII. htm

Re:the U-Bend

[QuickFox]

Fill the trap with cooking oil

No need to fill it, a few drops are enough. Oil floats on water, it spreads and forms a thin film on the surface. You get a lid that efficiently prevents water evaporation.

Re:the U-Bend

[qwijibo]

I'm glad you don't design server rooms I use. You've solved one problem while introducing various others based on a single superstitious belief.

Air conditioners are a common feature in server rooms. The ones used in this context maintain a certain humidity level. If the humidity is too high, there's risk of damage to the hardware, if it's too low it's easier for static to build up, which is bad for the hardware. When the humidity is too high, the air conditioners remove water from the air. This water needs to go somewhere, so a drain is a good choice. If the humidity is too low, they need a supply of water, for which tap water is sufficient.

Drains are also a common feature in server rooms because if water does get in there, they do not want to have the room flooded.

Re:the U-Bend

[winnabago]

No need to fill it, a few drops are enough. Oil floats on water, it spreads and forms a thin film on the surface. You get a lid that efficiently prevents water evaporation.

This is starting to sound like the introduction for the most boring Mythbusters ever.

"And then we waited for several weeks, comparing the rate of evaporation to our control toilet...."

Re:the U-Bend

[Hoi Polloi]

Slashdot, where you can not only learn about the gory details of OS kernels and hardware but also the nitty-gritty on plumbing!

Re:the U-Bend

[Secrity]

Almost all of the data centers that I have been in have some number of Liebert air handlers in the server room. All of these Liebert air handlers have an evaporator which requires a drain and most of them have a humidifier that requires a water source.

Re:the U-Bend

[crvtec]

News for plumbers. S#it that matters.

Nothing THAT bad...

[thepropain]

Two worst I've seen: 1) While I've chopped patch patch cables in half and turned 'em into crossovers, this one place I toured got a good deal on pre-made crossovers and chopped & spliced them into patch cables for over 50 PCs; 2) Where I work now, a former employee jacked a cable modem straight into a Win9x peer-to-peer network, despite my protests (scary part of that was that he said, and I quote, "Oh, I do this all the time and it's never been a problem before." I spent the next week reinstalling Win98 and software...)

300 wires with a conduit sawed off

[plover]

Back in the '80s we had a brand new computer room that had 300 shielded twisted pairs heading to 300 distant stations. The entire place was shiny, painted white, beautifully installed, all running through three large plastic conduits, one to each floor, hung professionally from the ceiling. A textbook illustration of beautiful wiring.

The fire marshall came in and said "you can't have those low-voltage wires run through that conduit, that conduit is designed for high voltage wiring." So the electricians came in and sawed off their beautiful conduits, leaving the wires draped between the four-foot-spaced supports. They tie-wrapped the bundles every foot or two, but it still looked like a dead python hanging between branches.

To this day I still can't fathom what the hell that inspector was thinking.

Re:300 wires with a conduit sawed off

[ars]

You have it backwards.

The problem is not the low voltage cable - it's that since it's meant for high voltage cable someone could install some later not realizing that there is some low voltage stuff there.

Yes, there really is a code about that - not mixing high and low voltage in the same conduit.

You can, I guess, claim that the conduit is "low voltage". But if it looks like it's for high voltage you might not get away with that.

Re:300 wires with a conduit sawed off

[AJWM]

The fire marshall was (giving the benefit of the doubt) probably thinking that if there was a high voltage conduit, sooner or later somebody would run a high voltage cable through it. Can't have high and low voltage wiring in the same conduit.

(Of course the reasons for all this are probably lost in the mists of time going back to fabric-insulated wires hung on insulators nailed to the studs. You'd think with modern wiring with obvious differences between 12 ga high voltage cable and cat-5e wires it wouldn't matter ... but then I've seen some pretty bizarre wiring setups that were "just temporary" or quick hacks, I can just see somebody provide a whole new meaning to "power over ethernet".)

Re:300 wires with a conduit sawed off

[Dibblah]

No, there is a good reason behind this.

A transformer is basically two lengths of wire running alongside each other for a long distance. If you run a low voltage circuit next to mains cable, you can potentially (in the case of a loop) create dangerous inductive heating in the low voltage cable - Not to mention the problems of the induced voltage in whatever equipment is connected to it.

Re:300 wires with a conduit sawed off

[igb]

Low voltage is more dangerous than high voltage, for a given power consumption. If the concern is heating in the duct, as it often is, 10KW@48V is a completely different animal to 10KW@415V three-phase. I'm always amazed at how flimsy cables are in 110VAV territories, compared to standard practice in 240VAC areas, and I can only think that back in the day, someone convinced themselves (wrongly) that you need cross-sectional area in the conductor for the volts, not the amps. Which in turn is why Americans complain about how weedy electric cookers are, and don't generally boil water with electric kettles: I can have 7.5KW into an electric cooker (30A, 240V) or 3KW into a kettle (13A, 240V), while Americans can't get 70A into a cooker.

Heating effect in a given cable rises by the inverse square of the voltage for constant power, which is why power transmission is done at 25KV or more. I don't think there's much OLE on railways in the USA, but it would be interesting to see what voltages they selected for that.

ian

Dungeon radio

[Centurix]

I worked at one place where our room was a couple of floors underground (very depressing place) and we wanted to listen to the cricket on the radio (pre internet days). Armed with a crappy radio we found we could get perfect reception by connecting to the air conditioning vents with a set of crocodile clips purchased from Tandy's.

Another one I remember is something low-tec invented by some admin staff, we had a policy set in place that locked workstations after 5 minutes of activity, the PC's were severely locked down so you couldn't change this. Turned out the admin section of the company despised this as they would do something on their accounts package, talk to someone on the phone and by the time the phone call had ended the PC had locked itself requiring their password to unlock it. One lady actually took a small clock, took the plastic front off and attached a piece of paper to the second hand, when she wasn't doing anything, she placed the mouse in front of the clock so that when the second hand went past, it moved the mouse slightly stopping it from locking. When the guys in tech support found it, she was visited by practically every IT person just to see it in action.

Re:Dungeon radio

[ShaunC]

we had a policy set in place that locked workstations after 5 minutes of activity

And the PHB's wondered why productivity was in the toilet... :)

Re:Dungeon radio

[lightyear4]

Speaking of radio:

A few months back during the summer, I was enjoying the lovely AC of a rather large server room at my university. On top of a rack in the midst of the farm sat a much abused radio, likely discarded even by the janitors, splattered with paint, and employing a rather frightening extension to its antenna. Its tuner was taped into position, and its headphone jack was connected to one of the machines.

This, of course, supplies the world with a live stream of the campus radio station.

Re:Dungeon radio

[RESPAWN]

At one of my previous places of employment, one of the higher ups decided they wanted to play music over the overhead speakers. Instead of installing a MUZAK system or even connecting the existing MUZAK box (currently serving up the hold music) her solution was to place a radio in the janitorial closet, lift the handset on the phone, dial the extension for the overhead intercom, and set the handset on the floor next to the radio. It worked, but every time somebody went in to the closet to retrieve some supplies, you could hear everything they said and did in the room.

Along a similar vein, in my last apartment, I decided that I wanted to be able to listen to music in the bathroom while I took a shower. Specifically, I want to listen to the MP3s stored on my computer. Digging through my junk boxes, I found an old battery operated FM transmitter from ~1999 and an old FM Walkman from ~1985. I also had several sets of extra computer speakers that I'd just managed to acquire over the years. The last component in this project was my old Marantz stereo receiver that I haven't used in a while because the volume potentiometer needs to be replaced -- it will only output sound when the volume is cranked to less than sensible levels and then it would only work for about 30 minutes, after which I think it got too hot to operate any further. 30 minutes should be fine for music in the shower, though.

Anyway, I had to run a stereo mini-jack to RCA cable from the rear speaker output on my PC to a pair of RCA female to female adapters to about 20 more feet of RCA cable which then plugged in to the input of my Marantz receiver. From there, I used a 1/4" to 1/8" headphone adapter to output the sound to a 1/8" to 1/8" male to male stereo cable feeding the FM transmitter. I did consider plugging the transmitter directly in to the computer, but the thought was that I would have a better chance of receiving the signal if I used the Marantz as an amp to boost the gain in to the transmitter. In the bathroom I then hooked the Walkman up to a set of 2.1 computer speakers, using a nail in the wall to mount the walkman as high up the wall as the cable would allow, since the cable also acted as the antenna. In theory, it worked, but in practice the signal from the transmitter was just too weak to reach the Walkman. The transmitter ran off AA batteries, so I decided to see what would happen if I connected a 9V battery to the leads. (Thinking that maybe if I upped the voltage, I could get more transmitting power.) The result was the release of the magic smoke and the end of my silly project.

Coat Hangers

[tymbow]

I've seen untwisted coat hangers covered in electrical tape and twisted together used to supply AC between two buildings in tropical weather in PNG. The wiring to the main building was bad enough but using coat hangers to supply power to the small hut that housed the computer equipment was priceless. I should also point out that they did not have power outlets for the computers either. They just cut the plugs off, stripped the wires, twisted them together and covered it in electrical tape.

Re:Coat Hangers

[BigBlockMopar]

The wiring to the main building was bad enough but using coat hangers to supply power to the small hut that housed the computer equipment was priceless.

Far East, right? For some reason, they have the same cultural aversion to proper wiring as Middle Easterners have to proper plumbing. [shaking head in belief but oh-my-God-what-can-you-do. At Home Depot, they refuse to sell stuff which the "associate" thinks will be used improperly where customer safety might be compromised. "No sir, 18-gauge lamp cord is not suitable to feed your dryer... Yes, it will generate heat, but I can guarantee it won't be in your dryer... 8/3 by code in Ontario.... Yes, I know it's more expensive, copper is a precious metal.... No, I will not cut you 18/2.... Sir, the building codes in Taiwan are highly suspect already, we've all seen in the news how many "modern" buildings collapsed last time you had a 6.1 quake; this is what ONTARIO requires.... well sir, you're more than welcome to take it up with the store manager, here's my phone, I'll dial 831 for you right now.... Oh, he met with you, laughed at you, and told you we wouldn't sell it to you? Yes, sir, that's why I like working at Home Depot, I know I tried to protect the little children in that house from burning up because you're an uncircumcised philistine. Have an adequate day."]

Moderators: If you don't believe me, Home Depot is hiring. After a week there, you WILL believe. Two years of home-built bidets using kitchen sink side-sprayers (note 1: kitchen sink side sprayers are controlled by the faucet, and it's assumed you're always there when they're on. note 2: kitchen sink side sprayers are rated only for the 5-10 PSI or so they see from aerator/diverter back pressure, not the full 50-75PSI of municipal water pressure) attached to municipal water pressure ("Why it burst? You sold me defective sprayer! What you mean I connect direct to city water? I cannot eh-do that, is connect straight to toilet. I come back from tree week in Yemen and find flood and notice taped to door. Now out $142,000 in flood damage to condo units beneath me! Insurance said I not use right part, they not pay! You pay! You pay!"). Two years of stoves burning through lamp cords ("why do I have to change my stove cord every time my wife uses more than one burner at once?"). Two years of actually having to work to convince people who *tell me they're plumbers* (ie. a guy who scraped together enough beer money to buy a pickup truck and a hammer and who now thinks he's a plumber - a trade which requires at least 5 years of schooling, people! Only 3 more to become a doctor!) that they can't use vinyl tubing to connect natural gas on a water heater for a little old lady who is dumb enough to let him into her house!

Find "Holmes on Homes". Shareaza, Torrents, etc. Download and watch a few episodes.

Jesus fucking wept.

Coathangers? I haven't seen them, but I'm a fervent believer.

Re:Server room heating & worker Safety

[darkonc]

At UBC we had a tiny (10'x10' computer room with a number of (un)pleasantly heat-generating computers (a couple of SUNs and a stack of SGI's). We managed to get the extra wiring put in place to handle the machines (a number of which required a 20AMP plug), but we never managed to get extra AC installed. This didn't bother us until summer came. ... and the build-up of heat would occasionaly trip the thermal breakers in some of the machines.

After begging facilities since the previous year to upgrade the AC (and having one last big machine installed), we 'solved' the problem by buying a small, window-type AC, and poking it out the door. With this setup, we could generally get the room to stablize at around 30C (about 86F).

This worked until facilities showed up and complained that we needed to go through them to get any sort of AC installed, and demanding that we stop using the offending unit. (but required us to continue with the un-responsive process of getting the room AC upgraded).

Peter resolved the impass by calling the health and safety group, and keeping the door closed until they arrived the next morning to inspect a worksite with a temperature of over 100F.
The AC was upgraded in well under a week.

Router at the end of a pier

[jgaynor]

My previous job was in Network Operations at a University. Our Marine Science department had a large grant-funded sensor network running in a river somewhere in South Jersey that needed to talk to their machines on campus. They did this by getting a 56k leased line dropped out to the end of a long pier, to which they connected a cisco 2500 series router (state of the art at the time). It was housed in a box with just enough ventilation to keep it soaked in condensation, but not enough to allow for adequate cooling. Because of the heat it was on a permanent shutdown/reboot loop for most of spring, summer and early fall. They were lucky if they got more than a few hours of readings per day.

Re:Router at the end of a pier

[afidel]

Wonder if that was before Cisco offered extended range products. These days you can get gear from Cisco that will survive in the Sahara in a NEMA3(sealed) outdoor enclosure.

Re:Router at the end of a pier

[The-Bus]

Sure, it might survive the Sahara or the Himalayas, but this is southern New Jersey we're talking about!

...it really is the answer

[techno-vampire]

About ten years ago, I was working for what was then a small, startup ISP doing tech support. For about the first two years I was there, we often had to talk new customers through locking down their modems to 2400 baud in the registration/installation program, because that server often worked best at low speeds. (We also showed them how to reset it to the proper speed afterwards because our POPs were just fine.) I later found out that this was because whoever set up our one and only (at that time) registration server had multiplexed 42 modems through one COM port.

Re:...it really is the answer

[AbRASiON]

This reminds me of working for an ISP in Melbourne about 5 years back.

My tech knowledge is a bit rusty but if I recall we had a fairly bad firmware on our dial in modems / boxes which caused the winmodems to disconnect a lot (I know they sucked 7 -> 10 years ago but most ISP's seemed ok with winmodems 5 years back)
Anyhow I got tired of dealing with angry customers trying to get a reliable connection with their winmodems so I gave them a string which forced the modems to connect at 33.6 baud instead of 56k, I then set the string to report the PORT speed and not the modem handshake speed and bobs your uncle! Customers loved me "He got me a 57600 connection!" all the time.

Be damned if I recall the string but I think it started with AT....

Seal it up

[crossmr]

I had an instructor who used to work in industry. He'd told me about a company he was consulting for. They had a Novell box that they administered remotely. During some remodeling, the small closet/room it was in was sealed with drywall. It was 4 years before the box required maintenance and someone went about trying to find it and realized what had happened.

Re:Seal it up

[Scarletdown]

During some remodeling, the small closet/room it was in was sealed with drywall. It was 4 years before the box required maintenance and someone went about trying to find it and realized what had happened.

http://www.techweb.com/wire/story/TWB20010409S0012

I hack, therefore I am

[tverbeek]

There was the time back in the late 1980s when the multiplexed underground cable that my college was using to provide terminal connectivity for their new-fangled computerized class registration stopped working, and they ran dozens of hastily-created RS-232 cables from the data center to the hall where the action was, half a block away... secured to the sidewalk by duct tape, of course. Which at least didn't remain in place as long as the 10base2 cable that connected two dorms, strung between their 2nd floors (until it was taken out by a lightning strike).

More recent ugly hacks that I can claim personal credit/blame for are mostly of the sort that involve pulling a rabbit out of my ass because a solution needs to be found By Tomorrow Morning... like for deploying 200 installations of Windows 95 in a week (in the days before Ghost, or even backup software that preserved Long FileNames) using DOS boot diskettes, Netware, a utility called lfnbk, and ncopy... or building an e-mail server out of RedHat Linux 6 and spare parts (no, I didn't even have a complete working computer at my disposal) when the company's glorified BBS mail software found itself unable to exchange mail with the standards-compliant system used by a major new business partner.

Re:Bizarre IT setup seen around the country...

[OECD]

How about the 10 MB email limit? That seemed to show up in the last 5 years or so. Before that I've had success with almost every size attachment I've been sent (and I do printing, so I see some pretty fat files.) When was the meeting held where they decided that?

Analog e-mail.

[Anonymous Coward]

When I got hired as an Information Specialist for one of the government sponsored agencies in Hellinois, the people there would write their e-mails on a piece of paper and give those to their previous IT guy. He would then type them up and send them out via a yahoo e-mail. No kidding.

long ping next door

[darkonc]

We shared our internet with the small ISP who sublet a portion of the building from us. They were upgrading their connection to the backbone from a T1 to a microwave link (gives you an idea as to how long ago this was).

At one point, they had changed their routing so that they were using the new link but we hadn't, so we decided to see how a ping went.

A packet between the two machines would go through our router, over the ethernet that the two companies shared, out the (old) external router, and down the coast through Seattle, to California, then back up the coast to Vancouver, and then finally over the same shared ethernet cable that the packet had originally gone out before finally connecting to their router.

A cross-border round trip of a few thousand miles for a net distance of about 60 feet.

Oh, and did I mention that our server room was a converted bank vault?

Honorable Mention

[ShaunC]

I know the poster was looking for funny/interesting anecdotes directly from our community, but for those of you who haven't stumbled across The Daily WTF, hop on over to that site and make it a part of your daily reading. While the focus used to be mostly on programming, it's abstracted itself to the generic IT level in recent months, and you'll see all sorts of bizarre stories there.

The Daily WTF is to IT workers what Jerry Springer is to everyone else. Just when you think you're having a bad day and your life is in the crapper, you can take a few minutes to soak in a situation where somebody else has it much, much worse... :)

Phone cables from modems

[lexarius]

I wanted to try out the option to have the server page me in case of problems. Only problem was that the only phone jack in the server room was on the other side of the room, and I didn't have a phone cable nearly that long. But I did have a box of old ISA modems and short phone cables. My intuition told me that the "Line In" ports were wired directly to the "Phone" ports and didn't require power or actual computers to drive them. So I daisy chained modem cards and short cables together across the ceiling, wedging the actual cards behind cable housing and drop ceiling tiles, until finally I got dialtone. My supervisor commended me for my creativity but made me take it down, since the policy was that the modems were not to be connected to phone lines for fear of people being able to dial in to them or something. Never mind the dedicated internet connection.

Re:Phone cables from modems

[Anml4ixoye]

You reminded me of the time our network admin wanted to setup a failover for our main (high-traffic) website. He figured that he could just add the IP address of our off-site emergency server as a third entry in DNS, since (at least to him) DNS worked by always hitting the first IP, and only moving down the list if it couldn't hit the previous one.

Only it doesn't. It round-robined the requests, so 1/3rd of our traffic was immediately and swiftly rerouted to our emergency site, which some enterprising webmaster had setup to email the webmaster box if anyone hit it (to make sure, I guess, that no one was going to it).

We noticed it because 5 emails came in at once, and then 10 more, and then it didn't stop until Groupwise crashed. We lost all the email in the box, and emails were coming in at some insane rate. We figured it out maybe 3 minutes in, but by the time we logged in and made the change, it was way too late.

Re:Phone cables from modems

[sydb]

Sorry, can't tell whether your story is about the incompetent admin and his DNS "hack" or the Groupwise server which loses all your mail when it gets full. Either way it's good, though.

My direct experience...

[rah1420]

I was the "computer guy" at a fabric processor in a town in Eastern PA that Shall Remain Nameless. Being "The computer guy" meant that they blamed me for the outages, but of course gave me no spending authority to do anything to fix the problems...

About 1 month into the gig, I was in the front office which was connected to the computer room by fiber optic cable (probably the smartest thing this company did.) However, once the fiber terminated at the switch in the office, the horizontal wiring to the workstations was, God help me, silver satin cable. Telephone wire. The shit was everywhere. There were about 100 workstations salted through the plant (which ran high voltage AC and heaters and whatnot) and everyone complained about the server performance. I wasn't even allowed (!) to put a network analyzer on the wire and was too naive/stupid at the time to realize what the problem was. The guy who had the spend authority, the "chief engineer," told me the problem was lack of RAM in the server and was always harping on me to upgrade the memory.

Another time I opened a closet to find a splice of this satin cable (they must have bought it surplus, they had hundreds of reels of the stuff) and the splice was made with, I kid you not, wire nuts.

I lasted 18 months there. I heard they brought an ex-Accenture conslutant in soon after to fix the "computer problems" and she ran the company into the ground.

Unsynchronized air conditioners

[Animats]

Many computer rooms have packaged units which both heat and cool, and some also both humidify and dehumidify. That's fine if you only have one. If you have more than one, they need to be interlocked so you don't get one cooling while another is heating, or one humidifying while another is dehumidifying. If you get into that situation, everything will seem to be just fine, but your energy bills will be maybe 5x what they should be.

Saw that situation in a server room at Stanford a few years ago.

Stupid IT maneuvers

[ximenes]

I have a bunch of stupid cobbled together setups to talk about. It all comes from a combination of poor IT staff at university wages, infintessimal budgets and the overbearing institutional and faculty pressures.

1. A "server room" that was essentially the most worthless room in the entire building, a long skinny room with four windows (perfect for keeping an uneven temperature!). Rather than buy 19" racks or even wire racks, they found a bunch of tables and put one server on each all the way around the edge of the room.
1.a. All of the servers were in fact desktop systems; an Ultra 1 was the mail server, a SPARCstation 5 the print server, a Gateway Pentium Pro 200 desktop the web server, etc.

2. A lab had to be moved one room over, because its current location was deemed too valuable. The original room was designed for a lab, it had 20+ fiber optic networking ports, twist-lock power connections in the ceiling, that sort of thing. The new room had two electrical outlets, no dropped ceiling, and one fiber optic networking port. It had previously been used as a copy room/storage closet. The cost to move the fiber optic wiring (just one room over mind you!) was over $25,000.

So instead, I had the great idea to cut a hole in the common wall (above the drop ceiling line), purchase additional ceiling tiles and cut up 2x4's into wooden supports. The original ceiling boxes containing the networking were put on top of the blocks above the new tiles, and extension cables run through the wall into the new room. In the original room, which was turned into a lounge, you couldn't tell that there was anything funny going on.

The best part is that the lab manager, who insisted they needed every single network port, never used a single one of them in the new room. All of those cables now reside in a box marked "Giant waste of money".

3. The main Windows file server was purchased in 2002 and has an internal RAID (bad idea in my opinion). What was huge then is worthless now; 5 disks that total 135GB. To get more space, the administration begged for a single external 250GB USB drive to host all user data. Nevermind that there is no redundancy, that an external drive is more suspectible to theft or failure, and that USB is unnecessarily slowing things down.

4. A system administrator got it into his head that rackmounting was the way to go (I agree). So he begged for a 19" rack to be ordered, and placed all of his servers into it. Except he doesn't have a single rack mountable server, and he didn't get the rails for any of the cases either. So now he has one $500 rack, and 8 $100 shelves to go in it. Same guy also switched the KVM monitor to a 15" LCD that doesn't support the resolutions of 9 out of 10 systems connected to it.

5. A consultant was brought in to tell us what needed to be done with the computing infrastructure (what DOESN'T need to be done is more the question). His main suggestion was to set up a central backup service just for this college, so as to avoid paying the central university IT group fees to use their central service. OK, thats an idea I guess... except that he wanted us to buy this: http://www.sun.com/storagetek/tape_storage/tape_li braries/sl8500/ (its $200,000). Luckily this one didn't actually come to pass.

Basically every day is a new adventure in ridiculous IT methodology.

2 SUNs == remote control

[darkonc]

With our company was based in Vancouver, we determined that we could get much better bandwidth charges in Seattle, so most of our live servers were there. Two of our larger machines were SUN 450 boxes (bought because, back then, Oracle didn't have full support for Linux). After I set them up, we pulled out the graphics cards that they came with and shipped the cards and monitors back the Vancouver (they were part of the bundle). Then I connected the two machines with null modem cables, Port A - Port B. and Port B to Port A.

Once the graphics cards were removed, the machines defaulted to booting with Serial consoles. This meant that if anything went seriously wrong, just about anything other than hardware maintenance could be done by SSHing to machine X and using a terminal program to connect to the console port of machine Y (or vice versa).
This included the ability to do a complete wipe and install, needing only to instruct the CoLo staff to insert the install CD (which were left on top of the machines) into the appropriate box.

One of the monitors ended up on my desk. I can't remember who got the other one.

TV in disguise

[smurfsurf]

I and a few guys were doing customer phone support in a remote building (ten years ago or such some). Soccer euro cup was up, and a collegue was desperate to find a way to watch the games, as the company (ISP) has just started operation, and callers were few and knowledable (so it was actually fun). Opening the cable funnel, he saw a TV cable. He spliced it up and connected it to a RJ45 jack. He then installed a TV tuner card into his PC, build a network cable look-alike to connect the TV card to the fake network jack, and voila - you could not see he was tapping the TV signal (the cable funnel was very visible, the computer was under the desk).

As we left the building about a year later, the fake jack was left there. I wonder what kind of head scratching this caused for the future tenants :-)

collision detection?

[trb]

In the early 80's, I was working for a company that did lots of its own kernel hacking on UNIX and VMS systems. They had a habit of implementing lots of their own software systems, rather than using standard ones. Some were not very clever. For instance, they had a communication "protocol" that ran over ethernet cable, but it didn't handle collisions. Yes, we had thick ethernet running to every office, and when anyone wanted to use it, they'd run out in the hall and yell to make sure it wasn't in use. If there was contention, data would be corrupted. Eventually, we punted on this stupidity and used TCP/IP.

Re:collision detection?

[LittleBigLui]

so your interpretation of CSMA/CD = Constant Shouting Might Allow Copying Data

Redneck Network Alarms

[mkcmkc]

A Very Large Telecom Corp(TM) had let a contract for a hardware subsystem that was to be connected to their very expensive network monitoring system (probably HP Openview). Anyway, the vendor couldn't quit make this work. So, to satisfy the contract, they had a tape monkey with a laptop in the NOC. Whenever an event happened on the subsystem, he'd manually copy the message into a dialog box on the master monitoring system, at which point it'd pop up on the regular NOC alarm system...

Sprinklers from hell

[plopez]

This happened just this past year.

We had moved into larger building with a server room in the basemnent (cue ominous music).

We rapidly began to run out of space so decided to place the chief sysape in the basement near the servers, which made sense. We cleaned up some items in the basement, moved them into storage, carpeted, dry walled etc. Since it was in the basement it needed an egress window with a steel casing and ladder. This actually turned the office into a nice garden level. You could look out the window and watch the sprinklers, see trees and grass etc.

On day, the chief sysape comes in and notices water on the floor. He looks over at the egress window and there is about 2 feet of water collected in the base of the exit well.

Well, they shut down the water to the entire building. Luckily the server room actually had about an 18 inch raised floor, so no damage.

To make a long story short, upon investigation it turned out that when the sprinkler system was installed, instead of capping off the ends of the plastic piping, they folded it over and crimped it. They relied on the mass of the dirt to keep the ends crimped, and for years it worked. Until the egress well was installed and the dirt was disturbed. Once it was disturbed, the crimps began to fail under water pressure. Leading to a near IT disaster.

Network closet A/C suction

[b0s0z0ku]

Network closet shared with an A/C unit. First time I went in there, I couldn't open the door - I had to force it and it opened with a hiss. Turns out that the A/C system was installed without return ductwork and was sucking all of its intake air through a window that was open approximately two inches.

-b.

Newb Haxx

[NotoriousHood]

When I started working at the school where I still work we were in two separate office buildings separated by more than 100m. I eventually ran coax through the wall via a light fixture, along a fence for about 100 feet into a tree to the roof (where it was held down by a sack of river rocks attached to some plywood) over the roof down a rain gutter and under the door. The building landlord was actually ok with this setup. It was mostly hidden except for the hop from the building to the fence and the whole tree to roof span.

Space Station Video Hack

[cyclone96]

If you've ever seen TV coverage of a Progress or Soyuz docking to the International Space Station, you've probably seen the ubiquitous black and white docking camera video with data overlayed on it as the vehicle approached the docking target.

Unfortunately, this television signal was only within the Russian Segment, and could only be downlinked through Russian communication assets over Russian ground sites. That limited the video to around 10 minutes each orbit, and required the docking to physically occur over Russia.

The US segment downlinks television via the Tracking and Data Relay Satellites (TDRS), which have more or less worldwide coverage. But the US segment and Russian Segment systems used incompatible video standards and weren't physically connected.

Yup, two video systems that cost tens of millions to develop, and they can't talk to each other. Classic "square peg, round hole" problem.

So we devised a setup where the crew ran a cable from the Russian Segment TV system into an IBM A31p laptop which converts the Russian SECAM signal to US NTSC video. The output from the laptop is connected to another cable strung down the stack into the US video system and downlinked via TDRS. Voila, greatly increased video coverage thanks to a lowly Thinkpad.

Details of this being tested can be found here: http://www.spaceref.com/news/viewsr.html?pid=18791

Peace and quiet and a good cup of coffee...

[ScrappyLaptop]

The server room was a fairly large closet with an a/c outlet and a combination of wire racks and IKEA shelves. Nothing too bad, there; it all worked and everything was strapped down in case of a quake. However, to get to the server room you had to go through the breakroom and pass by the kitchen. Which had maybe two outlets and a hardwired coffee maker. Which shared a breaker panel with the server room in the hallway behind both rooms. Can't tell you how many times the admin assistant killed the server room trying to shut off the coffee maker. On a Friday, at 5 p.m... Funny thing is, the Head Cheese only worried about his coffee, not the servers that housed our precious COBOL and account information and wouldn't authorize either a separate breaker panel or a good UPS. Then again, that same admin assistant had to print out the boss' email and Excel spreadsheets and then re-type in his modifications...

Re:The most messed up LAN, evar!

[Tore S B]

The backbone was a five port AUI concentrator... it was too primitive to be called a hub. (AUI was Sun's insane proprietary ethernet connector.)

It doesn't get more primitive than a hub. It was known as a "fanout unit" back then, though, or some other names. AUI was not Sun proprietary, it was an open standard, and for near a decade was the standard interface between a machine and the physical layer.
a BNC co-ax hub used just to hook up workstations in a star topology... for whatever reason, they decided that ring topology wasn't good enough to string five lightly used workstations together.

Presuming that by "ring" you mean "bus", a hubbed star-wired network is still a bus topology. Possibly they did this for reliability reasons (So that one could not just unplug ones T-joint and bring down the whole BNC loop) but that's just a guess.

Re:http://bash.org/?5273

[Gordonjcp]

Been there, done that while running a small hosting company out of a friend's flat. All the servers were in the living room. We'd borrowed a (for the time) fairly meaty PII-350 to act as DNS and database server. Then we bought another machine, went to give the borrowed PII back and - we'd *lost* it! Where TF was it? We could ping it, it was working and serving up requests, but we couldn't find it. So we ran

find(1)

until we heard the disk rattling. It was under a pile of jackets behind the sofa.

MS Exchange in place of a mail server

[SgtChaireBourne]

I've seen people try to use MS Exchange in place of a mail server.

Hey, you did ask.

Re:MS Exchange in place of a mail server

[big dumb dog]

...I seen some try to use IIS as a web server, too.

Booting a PDP11 with no boot ROMs

[Gordonjcp]

I have a PDP-11/73 that I run occasionally (http://pdp11.kicks-ass.net). It's got about 70M of disk space, two 10M removable drives, and two 8" floppies. All of these are (or were) booted by typing the name of the boot device you want at the boot prompt. Now, for those of you who have only ever seen modern PCs, these machines don't necessarily have any kind of BIOS or anything like that. In this case, the boot loaders were held in a pair of EPROMs on an add-in board along with the console serial port and LTC (line time clock that uses a small transformer to provide a mains-derived 50Hz clocked interrupt). One day, one of these EPROMs failed. No boot image. "Oh dear", I said, or something very similar.

Well now, remember I said it had no BIOS? What it *does* have is an octal debugger, similar to DEBUG in MS-DOS, called ODT. This is actually built into the microcode of the CPU; the CPU requires a console serial port to be present to even POST. If it's not there, a little LED lights on the edge of the CPU board and the machine will never come out of halt. So, at worst, all you need to do is hit <BREAK> type in the boot loader code on the terminal, and the machine will boot. Right?

Right. But that's a pain in the gluteus maximus, because it means typing in a load of stuff like

@001000/012700^J
  001002/174400^J
  001004/012760^J
...

and so on for a few dozen lines. There must be an easier way. What, like burn them into an EPROM? Well yes, but I don't have an EPROM burner. What I *do* have, though, is a VT-510 terminal, which allows you to program key sequences into the function keys. So, what I do now is power up the terminal and the PDP11, press HALT and then RESET on the front panel, hit a key sequence on the terminal, drop back into RUN once the disk seeks (controller is ready) and it's booted.

Yes, I'm buying an EPROM blower off eBay...

Continuous wiring

[SgtChaireBourne]

I recall installing some additional power outlets in one building in the tropics and worrying about violating code by doing it myself. The building was wired with 8-gauge multiple strand copper wire (for 110VAC 60Hz) with one hot, one neutral and (ostensibly) one ground. Once I got a look under the faceplate, however, I realized that code wouldn't be a problem.

The wiring method was bizarre: at each terminal (screw) the wire was stripped of insulation by tearing for about 2 inches (~5 cm), wrapped around the screw and then continued on its way to the next. It went like that for all the outlets we examined.

Re:Sodomized service

[MichaelSmith]

At my former job we used fibre cable under major roads to link CCTV cameras to our control room environment. One day a road worker drilled down into the road with some sort of hole digger and wrapped 100 metres of fibre around his machine, exactly like rolling pasta around a fork.

VHS backups

[jdfox]

I used to be the internal LAN support at a large multinational hardware vendor. Most of the company was on Mac desktops and Unix servers, but the accounts department felt they were mavericks who could run their own IT, so they opted for DOS, Lotus 1-2-3 and a Netware server. OK guys, if you think you can do it better, then maybe you can. Go for it.

They also figured that server backups were probably a good idea, since they routinely handled millions of pounds of transactions per day in that one office alone.

And since they were accountants, they naturally picked the cheapest backup solution they could dig up, which was a 40-dollar backup box that used VHS video cassettes, underneath a beancounter's desk, right by his foot. I shit you not: every few weeks, it would occur to him that a backup hadn't been done in a while, so he'd shove the VHS cassette into the backup box with his foot, then nudge the start button with his foot, and return to counting beans. The cassette would pop out when it was finished, and that was proof positive for them of a job done properly. They never even bought a second VHS cassette. Amazingly, the thing never stretched to snapping point, but it was undoubtedly unusable for restores (it never occurred to them to do test restores), making it genuinely much, much worse than useless.

At the office on the other side of town was the accounts department for another division. They also used VHS backups, but felt that doing backups was a bit beneath them really, so instead they had the office cleaner shove the VHS cassette into the 40-dollar backup box next to the office door every night on her way out. One night she was home with the flu, and hadn't left instructions for her replacement to do the "backup". Sure enough, the server crashed that night, and the stale backup wouldn't restore. The poor cleaner was immediately fired, but not the asshats who delegated mission-critical IT chores to a cleaner, on dimestore reject equipment.

I felt duty-bound to tell these fucking morons that they were really making a false savings on backup equipment, and needed to buy real backup gear, with someone trained to monitor the state of the scheduled nightly backups and do scheduled test-restores. This company was pulling in 13 billion US dollars in revenue a year, so 1500 dollars for an internal tape drive and a copy of Cheyenne to protect hundreds of millions of dollars worth of data sounded like a pretty unbeatable deal to me.

Not to them though. "You IT people", quoth a senior beancounter, shaking his head, when I took the purchase requisition to his desk for signature. "It's always more money for the latest damn thing, isn't it."

Cheapest of all would have been for them to simply use the central Unix servers, which were run properly with tested and reliable disaster recovery by experienced sysadmins. I tried explaining that there'd be no change to their DOS PCs, and they'd still have the same F: G: and H: drives, with no visible change to their working environment. I even offered to pay for the new client software. They'd save money, and get vastly better care of precious data.

The reply: "Heh heh heh! And then next year there'll be some reason why we all have to get rid of 1-2-3. And after that there'll be some reason why we have to get rid of DOS. No thanks! Heh heh heh! You guys never quit, do you!"

Re:Bedlam DL3

[z0idberg]

That link just screams NSFW.

Re:http://bash.org/?5273

[kylegordon]

A certain somebody who may remain nameless, but may or may not also be the parent poster, used a similar method to rattle the noisy old disks on a server that lived in my bedroom at ungodly hours in the morning....

The Curious Case of the Magic SCSI Clock

[jolyonr]

A while ago I ran an Amiga software development company. Our designer (Mark) had an Amiga 4000 with various external SCSI devices running off a notoriously unreliable Commodore A4091 SCSI card.

I went to his desk once trying to access a file on his external drives, and I kept getting disk errors. I called him over, and he said "Oh! That disk won't work unless you open up the system clock and resize it to this kind of size, and put it on the screen here". He opened the old analogue-face clock program that came with the amiga, resized it to about 200 pixels square, and stuck it in the top right of his screen.

I stood there smiling. He was, after all, a designer.

The file opened fine though after he did that.

I did some messing around on his machine afterwards. I was convinced there was some kind of obscure problem that we were missing - incorrect termination or bad cables maybe. I put the clock incident down to coincidence.

I could find nothing else wrong - but I still couldn't access the disk. So, I opened the clock application. I tried it on one side of the screen. File would not open. Moved it to the top right corner. The file opened. I did this about ten times as I couldn't believe the results myself. Every time I had the clock in the top right corner, the external SCSI disk behaved itself. I tried different applications, none of them worked in the same way - it had to be the clock.

I was completely spooked by the whole thing, and decided this was something sent by the Gods of SCSI to taunt me. The logical side of my mind believes that it is probably some obscure DMA issue, the rest of my mind believes the machine was possessed.

The thing I was never able to figure out was how Mark discovered the SCSI-healing properties of the Magic Amiga Clock and why he felt it was perfectly normal behaviour for his machine!

Jolyon

Re:The Curious Case of the Magic SCSI Clock

[DoofusOfDeath]

My old-ish CRT monitor makes different whining noises depending on the content of the screen.

Your problem may have been a matter of EM emissions from the monitor cable: when the clock was positioned in that way, the EM from the monitor and/or its cable were such that they didn't interfere with the SCSI cable's signal.

Where do I start...

[Linker3000]

1) The computer room floor built with a 4 foot void rather than 4 inches because the builder read the plans wrong. Mid you, there was room for a lot of kit in this 'split level' computer room.

2) The Netware 3.x file server which was a Toshiba T3200 plasma screen laptop locked inside a filing cabinet (a very secure solution on a military base). While I was working on it, a telephone began to ring in the next drawer up. I mentioned this to someone as nobody seemed to have heard it and the reply was "Oh, we don't answer that one"

3) The Olivetti M24 (AT&T 6300) that lived in a milking shed in the middle of a dusty field that eventually died and had to have a 2-3 inch layer of 'field' vacuumed out.

4) The computer room built with the existing radiators walled in but not turned off - took ages for the aircon guys to figure out why the room never cooled to the calculated temp.

5) The installation test of a new halon system (with a cylinder of CO2) where the engineers had not properly screwed the nozzle onto the 'j' pipe in the centre of the room. When the system was fired, the nozzle shot through the false ceiling, the gas followed it and the pressure blew down all the ceiling ties - the computer room looked like a scene from Die Hard.

6) The school network that comprised 5+ 'backbones' of 10Base2, each with around 20-30 D-Link *hubs* wired directly to cat5 outlets. Netware servers strategically placed round the building acted as repeaters with 2-3 NICs in each. We also found some Cat4 cable buried directly into the walls (no trunking).

7) 140m of Ethernet coax buried below a school field to link two buildings.

8) The over-length Token Ring network that included specially designed and developed repeaters that had to be 'tuned' using a screwdriver to adjust variable resistors to get the timing 'just right' so that the whole thing worked.

I have to add that I was *always* the support person brought in to sort things out - not the one creating the mess.

Re:Where do I start...

[TheMCP]

Well, I had one place hire me to sort out their IT... they had a weird proprietary wiring system that worked only with weird proprietary network cards and talked only to a weird proprietary server. I've never seen any of this garbage before or since. All the wires were about 1/2" thick and were run along the hallways, because they'd never heard of the idea that you could have wiring *installed*. And the server was down most of the time, they'd actually poke at it once a day until it went up for an hour or so so they could exchange files, before it crashed again.

So, I spent about $200,000 having actual ethernet installed and replacing all the computers in the (relatively small) company since everything they had was so ancient it couldn't even be connected to a contemporary network, set up a nice reliable server and backups, and after several months of intense work had everything running.

Then just as it was all stable, the boss called me into his office and explained calmly that our lease on the space we were in would be running out and he'd decided that we were in fact going to move, so I should plan the move of our network and equipment, bring in my wiring contractor to handle the new space, and ensure that we'd be back up and running in the new space in minimal time. Okay, no problem boss, when will we be moving? "In about half an hour." That's right folks, he didn't bother telling anybody that we'd be moving until half an hour before we did it, and I had just spent large amounts of money wiring a space we were about to move out of. And then for the new space of course you can't get a good wiring contractor on half an hour's notice, so all I could do was get a pile of long 10-base-T cables delivered and distribute hubs throughout the space and tape wires to the floor. I wanted to cry.

A few weeks later a psychotic middle manager who hated me because she couldn't understand what I did managed to push me out of the company and replace me with some kid who didn't even know what half the stuff I'd installed was, but he was willing to kowtow to her. I was terminated for "insubordination", for the unforgiveable offense of telling the kid that he couldn't plug the high volume laser printer into the UPS for the main server because it would overload the UPS and result in a shutdown. While the middle manager was gleefully screaming at me about what a nasty horrible person I am and that I was fired, the UPS was screaming from overload. I hear the UPS took the server down about 5 minutes after I walked out the door, and I knew offhand that that particular UPS, once it overloaded, would refuse to come back up until it'd had a (timed) 4 hour cooldown period. So, after the server I'd installed had been stable for a year, it died 5 minutes after I walked out the door and the new guy just couldn't make it go.

They'd also forgotten to ask me to tell them anything, like the admin passwords for any of the workstations, the BIOS passwords for anything, etc, which of course as a professional I would have been happy to tell them right up until they escorted me out the door. A week later they realized that they had hundreds of thousands of dollars of equipment that they couldn't reconfigure. They wheedled someone at the company I'd been friendly with to ask me for the passwords. I asked her "Did they offer to give you anything, like maybe a bonus, if you get the passwords out of me?" She said no. I told her that come to think of it I'd forgotten all the passwords since I didn't need them any more.

Re:the U-Bend AKA trap

[un1xl0ser]

Can we please put the 'itsatrap' tag to good use folks?

Using the heat

[Anomalous Cowbird]

I have been told that, back in the late 70s or early 80s, when a new courthouse/office building was built in a nearby county, someone got the idea to use the heat generated in the computer room to augment the building's heating system.

As I heard it, during the first winter, the gas company sent inspectors to check the pipelines, test the meters, etc., because they couldn't imagine that a building of that size could use so little gas in the wintertime.

Re:Using the heat

[badspyro]

Does that mean that by us staying in our parents basements, we are being enviromentaly freindly?

Re:Server room heating & worker Safety

[Anonymous Coward]

I worked in an office on a Sperry Univac BC7 mini computer. It had a LED panel that displayed certain error messages. I learned how to send messages to the panel. They had air conditioning, but were too cheap to set it to a comfortable level. One day I had a brainstorm while sitting there sweating. I sent "overheat warning" to the panel. I pointed it out to the office manager. He immediately turned the air on.

It's DNS in France

[Punk Walrus]

I worked for a HUGE multinational ISP once. We had just gotten France hooked up and they had been running fairly well for about six months after two years of testing. About 100k customers used the service.

One day, DNS went down. This had happened in the UK a lot, so we barked up the wrong tree for hours thinking it was a Keyring issue over the Transatlantic connection. Nope. Hours later, we found the DNS for France was on a different subnet. This led to discovering that their DNS service was on a set of IPs that pointed to one MAC. Finally, the people in charge of the data center said, "That's not our subnet. I don't know where you are getting DNS from.

We traced back and back through routers, entering territory that got scarier and scarier. It went to an older building that were were in the process of closing down and selling. It also had a data center, but that room had been dark for months, and DNS had been working up until now. Back and back we went.

Finally we found that the trace went through a disused subnet through a former office LAN in that building. This traced it back to an office, which traced it back to... ... a 386 LCD laptop. The machine had died because the logs had filled up the 1.2 gb hard drive. We couldn't believe it until someone rebooted the damn thing, and DNS came back up. We had been running production DNS on this thing for over 2 years.

Turns out that when the French network architecture was being set up, they had to transfer DNS somewhere temporarily as part of a testbed, so some guy had an old laptop in his office he just hooked up. Then he was laid off before we went live. Nobody ever switched it back, and since the office space was being abandoned, no one every went into the office to turn anything off, figuring it was somebody else's problem.

A week later, French DNS was running on a production server.

I am impressed it lasted that long on such a platform.

We also used to run the flight schedules for Lufthansa. It was a Windows NT 3.5.1 system that was running on a 486, and was running some proprietary terminal service and scheduler. It crashed once every 31 days (there was some bug where it would crash after xxxx hours which was between 30-31 days). The only way to fix it was to hard reboot the box, and the directions were scary: "Go down to the older server room, and find an unlabeled shelf next to the first door near the panic switch. On the bottom of that shelf is a box which is behind a stack of old 10base hubs. Hold down the power button until the green light goes off. You may have to lie on the floor on your stomach to reach the button. Count to ten, power back on. Make sure the amber light labeled 'turbo' is lit on bootup. If not, repeat, but wait 60 seconds before powering back up."

I sure hope they got that fixed, it was last like that in 2000.

remote monitoring

[VAXcat]

Back in the day, I used to have to go in late on Saturday nights and copy & compress the system disk on our PDP-11/70 system. It was located in the typical loud, cold computer room. THe copy process could take anywhere from 1 to 5 hours, depending on the amount of fragmentation on the system disk. I didn't want to wait in the loud room, and I didn't want to get up from my comfy chair in my office at the other end of the building to continually check on the progress. One night, tuning around on my FM radio in my office, I heard a funny sort of noise at 98.5 MHz. Its rythmic structure reminded me of the sound the disks made while they were seeking during this copy process. Sure enough, thise old school disk drives, with their Emitter Coupled Logic (which uses about a pound of electrons to do anything) were generating lots of EM noise, which was, I'm guessing, getting coupled to the power line and thence to my radio. After that, I could kick back and have a few beers, and listen to the radio to know when the copy was over, without going back and forth to check.

Re:Bizarre IT setup seen around the country...

[Anonymous Coward]

Even worse, I've seen morons trying to use email as a file server.

"Secure" data center

[The Mayor]

I once worked for a dot-bomb e-commerce company. We had a product that tied into several major credit card issuers (i.e. >40% worldwide market share for issued credit cards). As part of the installation and maintenance of the product, I got to spend many weeks in MAE East (perhaps the biggest data center in the world). From what I've read about the Baby Bells' special networking rooms when the NSA scandal broke last year, I wouldn't be surprised if these servers shared one of those special rooms with the NSA routers.

The data center was about 5 floors below ground level. No form of wireless communications worked whatsoever--cell phones, pagers, etc. Once I parked my car, I had to go to an unlabeled metal door with a tiny camera on the top. Security guards would buzz me in and require me to sign in at their station. Then I would get buzzed in to the main data center room that contained another room inside of it. From there, I had to enter a password into another security system and place my palm on a palm scanner. Inside this room was another security guard--I would have to sign in with them, too. Then I would enter a different password into another security system, and place my head in front of this retinal scanner. This would buzz me into another room with the cages for each of the clients. There was a padlock on the cage, behind which were our servers. The servers required two separate smart IDs to be placed into an external card reader so that there had to be at least 2 people there to perform any maintenance. The servers themselves were locked down pretty tightly, too. It all seemed pretty insane as far as security goes, but I understood--these computers contained every credit card for the credit card issuer.

Well, after about 3 days of going to this data center, everyone got to know me. They would sign in for me to speed up the process. The security guard behind the door with the palm scanner used to get very hot, so she would often block the door open, thus defeating the palm scanner. The retinal scanner also had problems, often requiring about 3 tries before it would read correctly, so that door was often blocked open, too. Then, one day one of us had forgotten our smart card. We started cursing, as the round trip to pick up the card was about 45 minutes, so we tried it with only one smart card. Bingo. It worked. So then we tried it with no card. Seems the card readers weren't functioning properly. So, overall, we were able to defeat all of the security measures except for the padlock, and all because the security staff (getting paid 2 bucks above minimum wage, no doubt) all "knew" us. In my humble opinion, it would have been far smarter to *not* have the security guard in the foyer behind the palm scanner. After all, social engineering is probably the most common form of circumventing security.

Another funny thing about this was that we had a rather difficult security audit for all code releases. We had a bunch of ex-NSA employees working for us that were rather good about it, too. We would also hire outside auditors to do reviews of major code releases. It was all fantastastic, except for one thing: code patches didn't get the same scrutiny as code releases. In fact, they got none. Well, in order to expedite the release of one particular feature (that required emailing confirmation to customers), we packaged it as a "patch". No security audits. And for something that required the installation of a mail server! Furthermore, the code base had access to the record-level encryption used to store the credit cards. So, basically, if I had wanted, I could have installed a bit of code that would have decrypted all of the credit cards of users of our software and emailed them to a third party. I could not believe it. It's a good thing I have what I consider to be high moral and ethical standards.

I realized through this ordeal that security measures are not put in place to ensure security. They are put in place to give people the perception of security. And, furthermore, automation and removing the human element are good things for security. People should be used to monitor and oversee automated security, not to be actively involved in that automated security.

One of these things is not like the others ...

[SgtChaireBourne]

...There is esmtpverbs control parameter in AD which by default does not play very nice with some other smtp servers(its arguable whose fault is that - every party claims its other party to blame) .

Easy enough to deduce: On the one side we have all the other MTAs which, unsurprisingly, are able to communicate well with each other. That leaves, on the other side, one system the odd man out which does not play very nice with the others.

What's wrong with admitting that it's still broken or don't normal rules of any kind apply to that one vendor ?

Security incident

[larien]

Well, there was the time I managed to cause a security incident...

At an old workplace, there was a server (ok, a Sun Ultra 1, but it was running Oracle) which no-one seemed to know where it was, but it was on the network, running OK. I resolved to track it down...

First plan was to have it write something on the screen asking whoever saw it to call me. No joy; guess no-one went there.

Then I figured that it had a sound card & speaker - I also knew it would play .au files natively so went a searching and found a line from Monty Python's Holy Grail: specifically, "Help, Help! I'm being repressed". I then set up a cron job to cat this file to /dev/audio every 15 minutes. Unfortunately, all someone could hear was "help, help" from outside the comms room it was in and assumed someone was trapped inside. Security guard looks around and eventually finds the server with my name on the monitor.

At least we found out where the damn thing was, which was useful when some numpty builder cut the ethernet cable while working in the room.

Long Runs

[nuintari]

Had a friend who had a box colo'd for free at a fairly run down ISP. It was due to some contractual obligation they had to live up to, so they were not happy about it. Was supposed to have a 10 mbit connection to the switch, and pretty much unmetered out over the backbone, but it never saw more than 2 mbit. Turns out, the ISP put an extra long ethernet line place, 500 feet of signal killing goodness. Just to make his service suck, in a way not easy to see unless you pulled up the raised floor, and noticed that all that wire was in fact, one single wire, going back, and forth, and back and forth.

Another friend has the dilbert boss, decided that the router _needs_ to be at one end of the building, far away from the office. A distance of 350 yards, add even more when you take into account the fact that the wiring goes from the ground floor at one end, up above the second floor, spans the building, and comes back down to the ground. Then, he decides that 10 base T is too slow, and he demands gigE. My friend tries explaining to him why all of this is nuts of course, but the man never listens, and says, "just buy a bunch of repeaters." So, this gigE run of nearly 400 yards, is daisy chained together, every 100 yards or so, by a 4 port gigE switch. Most of them live up in the drop ceiling above the second floor, and have some ludicrous power lines run to them. Store and forward nightmare. Boss is currently pissed because it is slow, and because the gigE switches didn't make his 3 mbit cable modem go any faster.

The "Clean" Room

[MWoody]

Well, this one might not be entirely in the spirit of the original question since it's not a cool "hack" so much as it is just an amusing error in planning, but here we go anyway:

Back in '95, my father was a VP of research for a large manufacturer of transmissive and reflective coatings for various glass applications (think insulated windows for the simplest example of said product) in Palo Alto, California. I was 15 and in highschool at the time, and having spent many a year trying like hell to keep a series of shitty no-name x86 computers up and running well enough to play the latest games, I had a sufficient skillset (and my dad had sufficient clout) to get me a job in their IT department. I did pretty well, and quickly found that users generally only got mean-spirited when made to look stupid, so a small dose of humility coupled with an interest in details on their primary task - "While I fix these printer drives you accidentally deleted, I was curious, what does a spectral photometer do?" - kept me out of trouble. Long story short, next year when I switched to full-time for the summer break, my boss actually brought me for a one-day business trip to our plant in Tempe, Arizona.

Now, you've got to realize, a business trip for a 16 year old (this was '96 now) is freakin' AWESOME. I was nervous as hell, had been up since the crack of dawn to take a red-eye with my boss out to the plant, and was deathly afraid I'd do something to embarrass not just me, but my father for having recommended me. So it was pretty unnerving to learn that my first job involved going into a large clean room production area, kept free from particles that could settle on the film during that specific type of sputtering process. We're talking the full disposable "bunny suit" that covered everything but the eyes, even with little slippers, and an airlock-type blower to clean you of all particles before entering.

The problem was a simple fix, really. The brand of 486 motherboard we were using at the time had a tendency, in about 1 out of every 3 units, to burn out the CMOS battery much earlier than you'd expect. And for a manufacturing-floor computer, not having a correct internal clock was a bad thing, not to mention that the lab techs had to go through some errors at startup with BIOS setings no longer being saved. So I suited up, cleaned off the replacement part and my tools as ordered, and went to find the bad machine.

That took some doing, oddly enough, since these computers were rarely shut down due to a 24/7 production schedule, so I had to go through back records on hand to find the lab techs' notes during the last power cycle on which computer had the boot errors. But, once located, the terminal was taken offline and I was able - after being told I had 20 minutes for the repair, tops, before the company would start to lose money as they needed that terminal again - to drag it off to a quiet, out of the way corner for the swap.

But see, there was a problem in the planning stages when this plant was set up. The PCs they used to control the machines were pretty complicated to configure, and the machines run in the clean room were just slightly modified versions of those used in the full-on manufacturing area in the main plant in Palo Alto. It was actually only a pretty small fraction of these production machines that had to operate in a clean environment. So when it came time to set these terminals up, they carefully washed off the outside of the older computers - computers, mind you, that have been sitting on a 24/7 PRODUCTION FLOOR with 10+ lab techs nearby at all times and various debris kicked up from the manufacturing process - and shuffled them off into the clean room.

So picture the scene: our hero, an extremely nervous 16-year-old on his first business trip in full head-to-toe bunny suit gear in the corner of a white, immaculately clean production floor opens his target computer to find a system so full of dust that he can't even SEE the goddamn cards inside. We're talking full-