Slashdot Mirror
What Bizarre IT Setups Have You Seen?
MicklePickle wonders: "I was talking to a co-worker the other day about the history of our company, (which shall remain nameless), and he started reminiscing about some of the IT hacks that our company did. Like running 10BaseT down a storm water drain to connect two buildings, using a dripping tap to keep the sewerage U-bend full of water in a computer room, (huh?). And some not so strange ones like running SCSI out to 100m, and running a major financial system on a long forgotten computer
in a cupboard. I know that there must be a plethora of IT hacks around. What are some you've seen?"
Two worst I've seen: 1) While I've chopped patch patch cables in half and turned 'em into crossovers, this one place I toured got a good deal on pre-made crossovers and chopped & spliced them into patch cables for over 50 PCs; 2) Where I work now, a former employee jacked a cable modem straight into a Win9x peer-to-peer network, despite my protests (scary part of that was that he said, and I quote, "Oh, I do this all the time and it's never been a problem before." I spent the next week reinstalling Win98 and software...)
"You know you're narcissistic when you quote yourself in your sigs." -- PRoPAiN!
My previous job was in Network Operations at a University. Our Marine Science department had a large grant-funded sensor network running in a river somewhere in South Jersey that needed to talk to their machines on campus. They did this by getting a 56k leased line dropped out to the end of a long pier, to which they connected a cisco 2500 series router (state of the art at the time). It was housed in a box with just enough ventilation to keep it soaked in condensation, but not enough to allow for adequate cooling. Because of the heat it was on a permanent shutdown/reboot loop for most of spring, summer and early fall. They were lucky if they got more than a few hours of readings per day.
About ten years ago, I was working for what was then a small, startup ISP doing tech support. For about the first two years I was there, we often had to talk new customers through locking down their modems to 2400 baud in the registration/installation program, because that server often worked best at low speeds. (We also showed them how to reset it to the proper speed afterwards because our POPs were just fine.) I later found out that this was because whoever set up our one and only (at that time) registration server had multiplexed 42 modems through one COM port.
Good, inexpensive web hosting
I had an instructor who used to work in industry. He'd told me about a company he was consulting for. They had a Novell box that they administered remotely. During some remodeling, the small closet/room it was in was sealed with drywall. It was 4 years before the box required maintenance and someone went about trying to find it and realized what had happened.
Thanks to the War on Drugs, it's easier to buy meth than it is to buy cold medicine!
When I got hired as an Information Specialist for one of the government sponsored agencies in Hellinois, the people there would write their e-mails on a piece of paper and give those to their previous IT guy. He would then type them up and send them out via a yahoo e-mail. No kidding.
At one point, they had changed their routing so that they were using the new link but we hadn't, so we decided to see how a ping went.
A packet between the two machines would go through our router, over the ethernet that the two companies shared, out the (old) external router, and down the coast through Seattle, to California, then back up the coast to Vancouver, and then finally over the same shared ethernet cable that the packet had originally gone out before finally connecting to their router.
A cross-border round trip of a few thousand miles for a net distance of about 60 feet.
Oh, and did I mention that our server room was a converted bank vault?
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
I wanted to try out the option to have the server page me in case of problems. Only problem was that the only phone jack in the server room was on the other side of the room, and I didn't have a phone cable nearly that long. But I did have a box of old ISA modems and short phone cables. My intuition told me that the "Line In" ports were wired directly to the "Phone" ports and didn't require power or actual computers to drive them. So I daisy chained modem cards and short cables together across the ceiling, wedging the actual cards behind cable housing and drop ceiling tiles, until finally I got dialtone. My supervisor commended me for my creativity but made me take it down, since the policy was that the modems were not to be connected to phone lines for fear of people being able to dial in to them or something. Never mind the dedicated internet connection.
I was the "computer guy" at a fabric processor in a town in Eastern PA that Shall Remain Nameless. Being "The computer guy" meant that they blamed me for the outages, but of course gave me no spending authority to do anything to fix the problems...
About 1 month into the gig, I was in the front office which was connected to the computer room by fiber optic cable (probably the smartest thing this company did.) However, once the fiber terminated at the switch in the office, the horizontal wiring to the workstations was, God help me, silver satin cable. Telephone wire. The shit was everywhere. There were about 100 workstations salted through the plant (which ran high voltage AC and heaters and whatnot) and everyone complained about the server performance. I wasn't even allowed (!) to put a network analyzer on the wire and was too naive/stupid at the time to realize what the problem was. The guy who had the spend authority, the "chief engineer," told me the problem was lack of RAM in the server and was always harping on me to upgrade the memory.
Another time I opened a closet to find a splice of this satin cable (they must have bought it surplus, they had hundreds of reels of the stuff) and the splice was made with, I kid you not, wire nuts.
I lasted 18 months there. I heard they brought an ex-Accenture conslutant in soon after to fix the "computer problems" and she ran the company into the ground.
Mit der Dummheit kämpfen Götter selbst vergebens.
In the early 80's, I was working for a company that did lots of its own kernel hacking on UNIX and VMS systems. They had a habit of implementing lots of their own software systems, rather than using standard ones. Some were not very clever. For instance, they had a communication "protocol" that ran over ethernet cable, but it didn't handle collisions. Yes, we had thick ethernet running to every office, and when anyone wanted to use it, they'd run out in the hall and yell to make sure it wasn't in use. If there was contention, data would be corrupted. Eventually, we punted on this stupidity and used TCP/IP.
A Very Large Telecom Corp(TM) had let a contract for a hardware subsystem that was to be connected to their very expensive network monitoring system (probably HP Openview). Anyway, the vendor couldn't quit make this work. So, to satisfy the contract, they had a tape monkey with a laptop in the NOC. Whenever an event happened on the subsystem, he'd manually copy the message into a dialog box on the master monitoring system, at which point it'd pop up on the regular NOC alarm system...
"Not an actor, but he plays one on TV."
This happened just this past year.
We had moved into larger building with a server room in the basemnent (cue ominous music).
We rapidly began to run out of space so decided to place the chief sysape in the basement near the servers, which made sense. We cleaned up some items in the basement, moved them into storage, carpeted, dry walled etc. Since it was in the basement it needed an egress window with a steel casing and ladder. This actually turned the office into a nice garden level. You could look out the window and watch the sprinklers, see trees and grass etc.
On day, the chief sysape comes in and notices water on the floor. He looks over at the egress window and there is about 2 feet of water collected in the base of the exit well.
Well, they shut down the water to the entire building. Luckily the server room actually had about an 18 inch raised floor, so no damage.
To make a long story short, upon investigation it turned out that when the sprinkler system was installed, instead of capping off the ends of the plastic piping, they folded it over and crimped it. They relied on the mass of the dirt to keep the ends crimped, and for years it worked. Until the egress well was installed and the dirt was disturbed. Once it was disturbed, the crimps began to fail under water pressure. Leading to a near IT disaster.
putting the 'B' in LGBTQ+
When I started working at the school where I still work we were in two separate office buildings separated by more than 100m. I eventually ran coax through the wall via a light fixture, along a fence for about 100 feet into a tree to the roof (where it was held down by a sack of river rocks attached to some plywood) over the roof down a rain gutter and under the door. The building landlord was actually ok with this setup. It was mostly hidden except for the hop from the building to the fence and the whole tree to roof span.
The fire marshall was (giving the benefit of the doubt) probably thinking that if there was a high voltage conduit, sooner or later somebody would run a high voltage cable through it. Can't have high and low voltage wiring in the same conduit.
... but then I've seen some pretty bizarre wiring setups that were "just temporary" or quick hacks, I can just see somebody provide a whole new meaning to "power over ethernet".)
(Of course the reasons for all this are probably lost in the mists of time going back to fabric-insulated wires hung on insulators nailed to the studs. You'd think with modern wiring with obvious differences between 12 ga high voltage cable and cat-5e wires it wouldn't matter
-- Alastair
The server room was a fairly large closet with an a/c outlet and a combination of wire racks and IKEA shelves. Nothing too bad, there; it all worked and everything was strapped down in case of a quake. However, to get to the server room you had to go through the breakroom and pass by the kitchen. Which had maybe two outlets and a hardwired coffee maker. Which shared a breaker panel with the server room in the hallway behind both rooms. Can't tell you how many times the admin assistant killed the server room trying to shut off the coffee maker. On a Friday, at 5 p.m... Funny thing is, the Head Cheese only worried about his coffee, not the servers that housed our precious COBOL and account information and wouldn't authorize either a separate breaker panel or a good UPS. Then again, that same admin assistant had to print out the boss' email and Excel spreadsheets and then re-type in his modifications...
I've seen people try to use MS Exchange in place of a mail server.
Hey, you did ask.
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.
Well now, remember I said it had no BIOS? What it *does* have is an octal debugger, similar to DEBUG in MS-DOS, called ODT. This is actually built into the microcode of the CPU; the CPU requires a console serial port to be present to even POST. If it's not there, a little LED lights on the edge of the CPU board and the machine will never come out of halt. So, at worst, all you need to do is hit <BREAK> type in the boot loader code on the terminal, and the machine will boot. Right?
Right. But that's a pain in the gluteus maximus, because it means typing in a load of stuff like and so on for a few dozen lines. There must be an easier way. What, like burn them into an EPROM? Well yes, but I don't have an EPROM burner. What I *do* have, though, is a VT-510 terminal, which allows you to program key sequences into the function keys. So, what I do now is power up the terminal and the PDP11, press HALT and then RESET on the front panel, hit a key sequence on the terminal, drop back into RUN once the disk seeks (controller is ready) and it's booted.
Yes, I'm buying an EPROM blower off eBay...
At my former job we used fibre cable under major roads to link CCTV cameras to our control room environment. One day a road worker drilled down into the road with some sort of hole digger and wrapped 100 metres of fibre around his machine, exactly like rolling pasta around a fork.
http://michaelsmith.id.au
That link just screams NSFW.
A certain somebody who may remain nameless, but may or may not also be the parent poster, used a similar method to rattle the noisy old disks on a server that lived in my bedroom at ungodly hours in the morning....
A while ago I ran an Amiga software development company. Our designer (Mark) had an Amiga 4000 with various external SCSI devices running off a notoriously unreliable Commodore A4091 SCSI card.
I went to his desk once trying to access a file on his external drives, and I kept getting disk errors. I called him over, and he said "Oh! That disk won't work unless you open up the system clock and resize it to this kind of size, and put it on the screen here". He opened the old analogue-face clock program that came with the amiga, resized it to about 200 pixels square, and stuck it in the top right of his screen.
I stood there smiling. He was, after all, a designer.
The file opened fine though after he did that.
I did some messing around on his machine afterwards. I was convinced there was some kind of obscure problem that we were missing - incorrect termination or bad cables maybe. I put the clock incident down to coincidence.
I could find nothing else wrong - but I still couldn't access the disk. So, I opened the clock application. I tried it on one side of the screen. File would not open. Moved it to the top right corner. The file opened. I did this about ten times as I couldn't believe the results myself. Every time I had the clock in the top right corner, the external SCSI disk behaved itself. I tried different applications, none of them worked in the same way - it had to be the clock.
I was completely spooked by the whole thing, and decided this was something sent by the Gods of SCSI to taunt me. The logical side of my mind believes that it is probably some obscure DMA issue, the rest of my mind believes the machine was possessed.
The thing I was never able to figure out was how Mark discovered the SCSI-healing properties of the Magic Amiga Clock and why he felt it was perfectly normal behaviour for his machine!
Jolyon
Please read my Canon EOS tech blog at http://www.everyothershot.com
1) The computer room floor built with a 4 foot void rather than 4 inches because the builder read the plans wrong. Mid you, there was room for a lot of kit in this 'split level' computer room.
2) The Netware 3.x file server which was a Toshiba T3200 plasma screen laptop locked inside a filing cabinet (a very secure solution on a military base). While I was working on it, a telephone began to ring in the next drawer up. I mentioned this to someone as nobody seemed to have heard it and the reply was "Oh, we don't answer that one"
3) The Olivetti M24 (AT&T 6300) that lived in a milking shed in the middle of a dusty field that eventually died and had to have a 2-3 inch layer of 'field' vacuumed out.
4) The computer room built with the existing radiators walled in but not turned off - took ages for the aircon guys to figure out why the room never cooled to the calculated temp.
5) The installation test of a new halon system (with a cylinder of CO2) where the engineers had not properly screwed the nozzle onto the 'j' pipe in the centre of the room. When the system was fired, the nozzle shot through the false ceiling, the gas followed it and the pressure blew down all the ceiling ties - the computer room looked like a scene from Die Hard.
6) The school network that comprised 5+ 'backbones' of 10Base2, each with around 20-30 D-Link *hubs* wired directly to cat5 outlets. Netware servers strategically placed round the building acted as repeaters with 2-3 NICs in each. We also found some Cat4 cable buried directly into the walls (no trunking).
7) 140m of Ethernet coax buried below a school field to link two buildings.
8) The over-length Token Ring network that included specially designed and developed repeaters that had to be 'tuned' using a screwdriver to adjust variable resistors to get the timing 'just right' so that the whole thing worked.
I have to add that I was *always* the support person brought in to sort things out - not the one creating the mess.
AT&ROFLMAO
Can we please put the 'itsatrap' tag to good use folks?
v4sw6PU$hw6ln6pr4F$ck 4/6$ma3+6u7LNS$w2m4l7U$i2e4+7en6a2X h
I worked for a HUGE multinational ISP once. We had just gotten France hooked up and they had been running fairly well for about six months after two years of testing. About 100k customers used the service.
... a 386 LCD laptop. The machine had died because the logs had filled up the 1.2 gb hard drive. We couldn't believe it until someone rebooted the damn thing, and DNS came back up. We had been running production DNS on this thing for over 2 years.
One day, DNS went down. This had happened in the UK a lot, so we barked up the wrong tree for hours thinking it was a Keyring issue over the Transatlantic connection. Nope. Hours later, we found the DNS for France was on a different subnet. This led to discovering that their DNS service was on a set of IPs that pointed to one MAC. Finally, the people in charge of the data center said, "That's not our subnet. I don't know where you are getting DNS from.
We traced back and back through routers, entering territory that got scarier and scarier. It went to an older building that were were in the process of closing down and selling. It also had a data center, but that room had been dark for months, and DNS had been working up until now. Back and back we went.
Finally we found that the trace went through a disused subnet through a former office LAN in that building. This traced it back to an office, which traced it back to...
Turns out that when the French network architecture was being set up, they had to transfer DNS somewhere temporarily as part of a testbed, so some guy had an old laptop in his office he just hooked up. Then he was laid off before we went live. Nobody ever switched it back, and since the office space was being abandoned, no one every went into the office to turn anything off, figuring it was somebody else's problem.
A week later, French DNS was running on a production server.
I am impressed it lasted that long on such a platform.
We also used to run the flight schedules for Lufthansa. It was a Windows NT 3.5.1 system that was running on a 486, and was running some proprietary terminal service and scheduler. It crashed once every 31 days (there was some bug where it would crash after xxxx hours which was between 30-31 days). The only way to fix it was to hard reboot the box, and the directions were scary: "Go down to the older server room, and find an unlabeled shelf next to the first door near the panic switch. On the bottom of that shelf is a box which is behind a stack of old 10base hubs. Hold down the power button until the green light goes off. You may have to lie on the floor on your stomach to reach the button. Count to ten, power back on. Make sure the amber light labeled 'turbo' is lit on bootup. If not, repeat, but wait 60 seconds before powering back up."
I sure hope they got that fixed, it was last like that in 2000.
No need to fill it, a few drops are enough. Oil floats on water, it spreads and forms a thin film on the surface. You get a lid that efficiently prevents water evaporation.
This is starting to sound like the introduction for the most boring Mythbusters ever.
"And then we waited for several weeks, comparing the rate of evaporation to our control toilet...."
Dammit Otto, you have lupus.
Slashdot, where you can not only learn about the gory details of OS kernels and hardware but also the nitty-gritty on plumbing!
It is by the juice of the coffee bean that thoughts acquire speed, the teeth acquire stains. The stains become a warning
Does that mean that by us staying in our parents basements, we are being enviromentaly freindly?
Back in the day, I used to have to go in late on Saturday nights and copy & compress the system disk on our PDP-11/70 system. It was located in the typical loud, cold computer room. THe copy process could take anywhere from 1 to 5 hours, depending on the amount of fragmentation on the system disk. I didn't want to wait in the loud room, and I didn't want to get up from my comfy chair in my office at the other end of the building to continually check on the progress. One night, tuning around on my FM radio in my office, I heard a funny sort of noise at 98.5 MHz. Its rythmic structure reminded me of the sound the disks made while they were seeking during this copy process. Sure enough, thise old school disk drives, with their Emitter Coupled Logic (which uses about a pound of electrons to do anything) were generating lots of EM noise, which was, I'm guessing, getting coupled to the power line and thence to my radio. After that, I could kick back and have a few beers, and listen to the radio to know when the copy was over, without going back and forth to check.
There is no God, and Dirac is his prophet.
News for plumbers. S#it that matters.
At an old workplace, there was a server (ok, a Sun Ultra 1, but it was running Oracle) which no-one seemed to know where it was, but it was on the network, running OK. I resolved to track it down...
First plan was to have it write something on the screen asking whoever saw it to call me. No joy; guess no-one went there.
Then I figured that it had a sound card & speaker - I also knew it would play .au files natively so went a searching and found a line from Monty Python's Holy Grail: specifically, "Help, Help! I'm being repressed". I then set up a cron job to cat this file to /dev/audio every 15 minutes. Unfortunately, all someone could hear was "help, help" from outside the comms room it was in and assumed someone was trapped inside. Security guard looks around and eventually finds the server with my name on the monitor.
At least we found out where the damn thing was, which was useful when some numpty builder cut the ethernet cable while working in the room.
Had a friend who had a box colo'd for free at a fairly run down ISP. It was due to some contractual obligation they had to live up to, so they were not happy about it. Was supposed to have a 10 mbit connection to the switch, and pretty much unmetered out over the backbone, but it never saw more than 2 mbit. Turns out, the ISP put an extra long ethernet line place, 500 feet of signal killing goodness. Just to make his service suck, in a way not easy to see unless you pulled up the raised floor, and noticed that all that wire was in fact, one single wire, going back, and forth, and back and forth.
Another friend has the dilbert boss, decided that the router _needs_ to be at one end of the building, far away from the office. A distance of 350 yards, add even more when you take into account the fact that the wiring goes from the ground floor at one end, up above the second floor, spans the building, and comes back down to the ground. Then, he decides that 10 base T is too slow, and he demands gigE. My friend tries explaining to him why all of this is nuts of course, but the man never listens, and says, "just buy a bunch of repeaters." So, this gigE run of nearly 400 yards, is daisy chained together, every 100 yards or so, by a 4 port gigE switch. Most of them live up in the drop ceiling above the second floor, and have some ludicrous power lines run to them. Store and forward nightmare. Boss is currently pissed because it is slow, and because the gigE switches didn't make his 3 mbit cable modem go any faster.
--Nuintari
slashdot : where an opinion can be wrong.