Slashdot Mirror
What Bizarre IT Setups Have You Seen?
MicklePickle wonders: "I was talking to a co-worker the other day about the history of our company, (which shall remain nameless), and he started reminiscing about some of the IT hacks that our company did. Like running 10BaseT down a storm water drain to connect two buildings, using a dripping tap to keep the sewerage U-bend full of water in a computer room, (huh?). And some not so strange ones like running SCSI out to 100m, and running a major financial system on a long forgotten computer
in a cupboard. I know that there must be a plethora of IT hacks around. What are some you've seen?"
Just to clarify- the U-Bend is what prevents bathrooms and drains from smelling horrible. Inside the drain, shower water, sink water and toilet waste all mix together. As you can imagine this smells horrible. So, where every toilet, sink, shower, etc connects to the drain system there is a 'u-bend'- a downward dip in the pipe which stays full of water. This prevents air from flowing out of the empty drain.. png
Most sinks have their u-bend visible under the sink and look like this:
http://twenteenthcentury.com/uologos/ubend_shaded
Water flows in the top, and out the back. Because the back is higher than the bottom of the bend, the bottom stays full of water at all times, preventing air from passing.
Problem is, if you leave a drain long enough without water passing through it, the water in the u-bend can evaporate, leaving an empty pipe and allowign the nasty sewer smell to escape. Thus, leave a faucet dripping to keep the U-Bend full!
--IronHelix
I worked at one place where our room was a couple of floors underground (very depressing place) and we wanted to listen to the cricket on the radio (pre internet days). Armed with a crappy radio we found we could get perfect reception by connecting to the air conditioning vents with a set of crocodile clips purchased from Tandy's.
Another one I remember is something low-tec invented by some admin staff, we had a policy set in place that locked workstations after 5 minutes of activity, the PC's were severely locked down so you couldn't change this. Turned out the admin section of the company despised this as they would do something on their accounts package, talk to someone on the phone and by the time the phone call had ended the PC had locked itself requiring their password to unlock it. One lady actually took a small clock, took the plastic front off and attached a piece of paper to the second hand, when she wasn't doing anything, she placed the mouse in front of the clock so that when the second hand went past, it moved the mouse slightly stopping it from locking. When the guys in tech support found it, she was visited by practically every IT person just to see it in action.
Task Mangler
I've seen untwisted coat hangers covered in electrical tape and twisted together used to supply AC between two buildings in tropical weather in PNG. The wiring to the main building was bad enough but using coat hangers to supply power to the small hut that housed the computer equipment was priceless. I should also point out that they did not have power outlets for the computers either. They just cut the plugs off, stripped the wires, twisted them together and covered it in electrical tape.
After begging facilities since the previous year to upgrade the AC (and having one last big machine installed), we 'solved' the problem by buying a small, window-type AC, and poking it out the door. With this setup, we could generally get the room to stablize at around 30C (about 86F).
This worked until facilities showed up and complained that we needed to go through them to get any sort of AC installed, and demanding that we stop using the offending unit. (but required us to continue with the un-responsive process of getting the room AC upgraded).
Peter resolved the impass by calling the health and safety group, and keeping the door closed until they arrived the next morning to inspect a worksite with a temperature of over 100F.
The AC was upgraded in well under a week.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
How about the 10 MB email limit? That seemed to show up in the last 5 years or so. Before that I've had success with almost every size attachment I've been sent (and I do printing, so I see some pretty fat files.) When was the meeting held where they decided that?
One man's -1 Flamebait is another man's +5 Funny.
When I got hired as an Information Specialist for one of the government sponsored agencies in Hellinois, the people there would write their e-mails on a piece of paper and give those to their previous IT guy. He would then type them up and send them out via a yahoo e-mail. No kidding.
I know the poster was looking for funny/interesting anecdotes directly from our community, but for those of you who haven't stumbled across The Daily WTF, hop on over to that site and make it a part of your daily reading. While the focus used to be mostly on programming, it's abstracted itself to the generic IT level in recent months, and you'll see all sorts of bizarre stories there.
:)
The Daily WTF is to IT workers what Jerry Springer is to everyone else. Just when you think you're having a bad day and your life is in the crapper, you can take a few minutes to soak in a situation where somebody else has it much, much worse...
Thanks to the War on Drugs, it's easier to buy meth than it is to buy cold medicine!
I wanted to try out the option to have the server page me in case of problems. Only problem was that the only phone jack in the server room was on the other side of the room, and I didn't have a phone cable nearly that long. But I did have a box of old ISA modems and short phone cables. My intuition told me that the "Line In" ports were wired directly to the "Phone" ports and didn't require power or actual computers to drive them. So I daisy chained modem cards and short cables together across the ceiling, wedging the actual cards behind cable housing and drop ceiling tiles, until finally I got dialtone. My supervisor commended me for my creativity but made me take it down, since the policy was that the modems were not to be connected to phone lines for fear of people being able to dial in to them or something. Never mind the dedicated internet connection.
I and a few guys were doing customer phone support in a remote building (ten years ago or such some). Soccer euro cup was up, and a collegue was desperate to find a way to watch the games, as the company (ISP) has just started operation, and callers were few and knowledable (so it was actually fun). Opening the cable funnel, he saw a TV cable. He spliced it up and connected it to a RJ45 jack. He then installed a TV tuner card into his PC, build a network cable look-alike to connect the TV card to the fake network jack, and voila - you could not see he was tapping the TV signal (the cable funnel was very visible, the computer was under the desk).
:-)
As we left the building about a year later, the fake jack was left there. I wonder what kind of head scratching this caused for the future tenants
You have it backwards.
The problem is not the low voltage cable - it's that since it's meant for high voltage cable someone could install some later not realizing that there is some low voltage stuff there.
Yes, there really is a code about that - not mixing high and low voltage in the same conduit.
You can, I guess, claim that the conduit is "low voltage". But if it looks like it's for high voltage you might not get away with that.
-Ariel
If you've ever seen TV coverage of a Progress or Soyuz docking to the International Space Station, you've probably seen the ubiquitous black and white docking camera video with data overlayed on it as the vehicle approached the docking target.
1
Unfortunately, this television signal was only within the Russian Segment, and could only be downlinked through Russian communication assets over Russian ground sites. That limited the video to around 10 minutes each orbit, and required the docking to physically occur over Russia.
The US segment downlinks television via the Tracking and Data Relay Satellites (TDRS), which have more or less worldwide coverage. But the US segment and Russian Segment systems used incompatible video standards and weren't physically connected.
Yup, two video systems that cost tens of millions to develop, and they can't talk to each other. Classic "square peg, round hole" problem.
So we devised a setup where the crew ran a cable from the Russian Segment TV system into an IBM A31p laptop which converts the Russian SECAM signal to US NTSC video. The output from the laptop is connected to another cable strung down the stack into the US video system and downlinked via TDRS. Voila, greatly increased video coverage thanks to a lowly Thinkpad.
Details of this being tested can be found here: http://www.spaceref.com/news/viewsr.html?pid=1879
Worst...sig...ever!
I've seen people try to use MS Exchange in place of a mail server.
Hey, you did ask.
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.
At my former job we used fibre cable under major roads to link CCTV cameras to our control room environment. One day a road worker drilled down into the road with some sort of hole digger and wrapped 100 metres of fibre around his machine, exactly like rolling pasta around a fork.
http://michaelsmith.id.au
No, there is a good reason behind this.
A transformer is basically two lengths of wire running alongside each other for a long distance. If you run a low voltage circuit next to mains cable, you can potentially (in the case of a loop) create dangerous inductive heating in the low voltage cable - Not to mention the problems of the induced voltage in whatever equipment is connected to it.
They also figured that server backups were probably a good idea, since they routinely handled millions of pounds of transactions per day in that one office alone.
And since they were accountants, they naturally picked the cheapest backup solution they could dig up, which was a 40-dollar backup box that used VHS video cassettes, underneath a beancounter's desk, right by his foot. I shit you not: every few weeks, it would occur to him that a backup hadn't been done in a while, so he'd shove the VHS cassette into the backup box with his foot, then nudge the start button with his foot, and return to counting beans. The cassette would pop out when it was finished, and that was proof positive for them of a job done properly. They never even bought a second VHS cassette. Amazingly, the thing never stretched to snapping point, but it was undoubtedly unusable for restores (it never occurred to them to do test restores), making it genuinely much, much worse than useless.
At the office on the other side of town was the accounts department for another division. They also used VHS backups, but felt that doing backups was a bit beneath them really, so instead they had the office cleaner shove the VHS cassette into the 40-dollar backup box next to the office door every night on her way out. One night she was home with the flu, and hadn't left instructions for her replacement to do the "backup". Sure enough, the server crashed that night, and the stale backup wouldn't restore. The poor cleaner was immediately fired, but not the asshats who delegated mission-critical IT chores to a cleaner, on dimestore reject equipment.
I felt duty-bound to tell these fucking morons that they were really making a false savings on backup equipment, and needed to buy real backup gear, with someone trained to monitor the state of the scheduled nightly backups and do scheduled test-restores. This company was pulling in 13 billion US dollars in revenue a year, so 1500 dollars for an internal tape drive and a copy of Cheyenne to protect hundreds of millions of dollars worth of data sounded like a pretty unbeatable deal to me.
Not to them though. "You IT people", quoth a senior beancounter, shaking his head, when I took the purchase requisition to his desk for signature. "It's always more money for the latest damn thing, isn't it."
Cheapest of all would have been for them to simply use the central Unix servers, which were run properly with tested and reliable disaster recovery by experienced sysadmins. I tried explaining that there'd be no change to their DOS PCs, and they'd still have the same F: G: and H: drives, with no visible change to their working environment. I even offered to pay for the new client software. They'd save money, and get vastly better care of precious data.
The reply: "Heh heh heh! And then next year there'll be some reason why we all have to get rid of 1-2-3. And after that there'll be some reason why we have to get rid of DOS. No thanks! Heh heh heh! You guys never quit, do you!"
A while ago I ran an Amiga software development company. Our designer (Mark) had an Amiga 4000 with various external SCSI devices running off a notoriously unreliable Commodore A4091 SCSI card.
I went to his desk once trying to access a file on his external drives, and I kept getting disk errors. I called him over, and he said "Oh! That disk won't work unless you open up the system clock and resize it to this kind of size, and put it on the screen here". He opened the old analogue-face clock program that came with the amiga, resized it to about 200 pixels square, and stuck it in the top right of his screen.
I stood there smiling. He was, after all, a designer.
The file opened fine though after he did that.
I did some messing around on his machine afterwards. I was convinced there was some kind of obscure problem that we were missing - incorrect termination or bad cables maybe. I put the clock incident down to coincidence.
I could find nothing else wrong - but I still couldn't access the disk. So, I opened the clock application. I tried it on one side of the screen. File would not open. Moved it to the top right corner. The file opened. I did this about ten times as I couldn't believe the results myself. Every time I had the clock in the top right corner, the external SCSI disk behaved itself. I tried different applications, none of them worked in the same way - it had to be the clock.
I was completely spooked by the whole thing, and decided this was something sent by the Gods of SCSI to taunt me. The logical side of my mind believes that it is probably some obscure DMA issue, the rest of my mind believes the machine was possessed.
The thing I was never able to figure out was how Mark discovered the SCSI-healing properties of the Magic Amiga Clock and why he felt it was perfectly normal behaviour for his machine!
Jolyon
Please read my Canon EOS tech blog at http://www.everyothershot.com
Can we please put the 'itsatrap' tag to good use folks?
v4sw6PU$hw6ln6pr4F$ck 4/6$ma3+6u7LNS$w2m4l7U$i2e4+7en6a2X h
I have been told that, back in the late 70s or early 80s, when a new courthouse/office building was built in a nearby county, someone got the idea to use the heat generated in the computer room to augment the building's heating system.
As I heard it, during the first winter, the gas company sent inspectors to check the pipelines, test the meters, etc., because they couldn't imagine that a building of that size could use so little gas in the wintertime.
I worked in an office on a Sperry Univac BC7 mini computer. It had a LED panel that displayed certain error messages. I learned how to send messages to the panel. They had air conditioning, but were too cheap to set it to a comfortable level. One day I had a brainstorm while sitting there sweating. I sent "overheat warning" to the panel. I pointed it out to the office manager. He immediately turned the air on.
I worked for a HUGE multinational ISP once. We had just gotten France hooked up and they had been running fairly well for about six months after two years of testing. About 100k customers used the service.
... a 386 LCD laptop. The machine had died because the logs had filled up the 1.2 gb hard drive. We couldn't believe it until someone rebooted the damn thing, and DNS came back up. We had been running production DNS on this thing for over 2 years.
One day, DNS went down. This had happened in the UK a lot, so we barked up the wrong tree for hours thinking it was a Keyring issue over the Transatlantic connection. Nope. Hours later, we found the DNS for France was on a different subnet. This led to discovering that their DNS service was on a set of IPs that pointed to one MAC. Finally, the people in charge of the data center said, "That's not our subnet. I don't know where you are getting DNS from.
We traced back and back through routers, entering territory that got scarier and scarier. It went to an older building that were were in the process of closing down and selling. It also had a data center, but that room had been dark for months, and DNS had been working up until now. Back and back we went.
Finally we found that the trace went through a disused subnet through a former office LAN in that building. This traced it back to an office, which traced it back to...
Turns out that when the French network architecture was being set up, they had to transfer DNS somewhere temporarily as part of a testbed, so some guy had an old laptop in his office he just hooked up. Then he was laid off before we went live. Nobody ever switched it back, and since the office space was being abandoned, no one every went into the office to turn anything off, figuring it was somebody else's problem.
A week later, French DNS was running on a production server.
I am impressed it lasted that long on such a platform.
We also used to run the flight schedules for Lufthansa. It was a Windows NT 3.5.1 system that was running on a 486, and was running some proprietary terminal service and scheduler. It crashed once every 31 days (there was some bug where it would crash after xxxx hours which was between 30-31 days). The only way to fix it was to hard reboot the box, and the directions were scary: "Go down to the older server room, and find an unlabeled shelf next to the first door near the panic switch. On the bottom of that shelf is a box which is behind a stack of old 10base hubs. Hold down the power button until the green light goes off. You may have to lie on the floor on your stomach to reach the button. Count to ten, power back on. Make sure the amber light labeled 'turbo' is lit on bootup. If not, repeat, but wait 60 seconds before powering back up."
I sure hope they got that fixed, it was last like that in 2000.
I once worked for a dot-bomb e-commerce company. We had a product that tied into several major credit card issuers (i.e. >40% worldwide market share for issued credit cards). As part of the installation and maintenance of the product, I got to spend many weeks in MAE East (perhaps the biggest data center in the world). From what I've read about the Baby Bells' special networking rooms when the NSA scandal broke last year, I wouldn't be surprised if these servers shared one of those special rooms with the NSA routers.
The data center was about 5 floors below ground level. No form of wireless communications worked whatsoever--cell phones, pagers, etc. Once I parked my car, I had to go to an unlabeled metal door with a tiny camera on the top. Security guards would buzz me in and require me to sign in at their station. Then I would get buzzed in to the main data center room that contained another room inside of it. From there, I had to enter a password into another security system and place my palm on a palm scanner. Inside this room was another security guard--I would have to sign in with them, too. Then I would enter a different password into another security system, and place my head in front of this retinal scanner. This would buzz me into another room with the cages for each of the clients. There was a padlock on the cage, behind which were our servers. The servers required two separate smart IDs to be placed into an external card reader so that there had to be at least 2 people there to perform any maintenance. The servers themselves were locked down pretty tightly, too. It all seemed pretty insane as far as security goes, but I understood--these computers contained every credit card for the credit card issuer.
Well, after about 3 days of going to this data center, everyone got to know me. They would sign in for me to speed up the process. The security guard behind the door with the palm scanner used to get very hot, so she would often block the door open, thus defeating the palm scanner. The retinal scanner also had problems, often requiring about 3 tries before it would read correctly, so that door was often blocked open, too. Then, one day one of us had forgotten our smart card. We started cursing, as the round trip to pick up the card was about 45 minutes, so we tried it with only one smart card. Bingo. It worked. So then we tried it with no card. Seems the card readers weren't functioning properly. So, overall, we were able to defeat all of the security measures except for the padlock, and all because the security staff (getting paid 2 bucks above minimum wage, no doubt) all "knew" us. In my humble opinion, it would have been far smarter to *not* have the security guard in the foyer behind the palm scanner. After all, social engineering is probably the most common form of circumventing security.
Another funny thing about this was that we had a rather difficult security audit for all code releases. We had a bunch of ex-NSA employees working for us that were rather good about it, too. We would also hire outside auditors to do reviews of major code releases. It was all fantastastic, except for one thing: code patches didn't get the same scrutiny as code releases. In fact, they got none. Well, in order to expedite the release of one particular feature (that required emailing confirmation to customers), we packaged it as a "patch". No security audits. And for something that required the installation of a mail server! Furthermore, the code base had access to the record-level encryption used to store the credit cards. So, basically, if I had wanted, I could have installed a bit of code that would have decrypted all of the credit cards of users of our software and emailed them to a third party. I could not believe it. It's a good thing I have what I consider to be high moral and ethical standards.
I realized through this ordeal that security measures are not put in place to ensure security. They are put in place to give people the perception of security. And, furthermore, automation and removing the human element are good things for security. People should be used to monitor and oversee automated security, not to be actively involved in that automated security.
--Be human.