Slashdot Mirror
What Bizarre IT Setups Have You Seen?
MicklePickle wonders: "I was talking to a co-worker the other day about the history of our company, (which shall remain nameless), and he started reminiscing about some of the IT hacks that our company did. Like running 10BaseT down a storm water drain to connect two buildings, using a dripping tap to keep the sewerage U-bend full of water in a computer room, (huh?). And some not so strange ones like running SCSI out to 100m, and running a major financial system on a long forgotten computer
in a cupboard. I know that there must be a plethora of IT hacks around. What are some you've seen?"
I worked at one place where our room was a couple of floors underground (very depressing place) and we wanted to listen to the cricket on the radio (pre internet days). Armed with a crappy radio we found we could get perfect reception by connecting to the air conditioning vents with a set of crocodile clips purchased from Tandy's.
Another one I remember is something low-tec invented by some admin staff, we had a policy set in place that locked workstations after 5 minutes of activity, the PC's were severely locked down so you couldn't change this. Turned out the admin section of the company despised this as they would do something on their accounts package, talk to someone on the phone and by the time the phone call had ended the PC had locked itself requiring their password to unlock it. One lady actually took a small clock, took the plastic front off and attached a piece of paper to the second hand, when she wasn't doing anything, she placed the mouse in front of the clock so that when the second hand went past, it moved the mouse slightly stopping it from locking. When the guys in tech support found it, she was visited by practically every IT person just to see it in action.
Task Mangler
I've seen untwisted coat hangers covered in electrical tape and twisted together used to supply AC between two buildings in tropical weather in PNG. The wiring to the main building was bad enough but using coat hangers to supply power to the small hut that housed the computer equipment was priceless. I should also point out that they did not have power outlets for the computers either. They just cut the plugs off, stripped the wires, twisted them together and covered it in electrical tape.
After begging facilities since the previous year to upgrade the AC (and having one last big machine installed), we 'solved' the problem by buying a small, window-type AC, and poking it out the door. With this setup, we could generally get the room to stablize at around 30C (about 86F).
This worked until facilities showed up and complained that we needed to go through them to get any sort of AC installed, and demanding that we stop using the offending unit. (but required us to continue with the un-responsive process of getting the room AC upgraded).
Peter resolved the impass by calling the health and safety group, and keeping the door closed until they arrived the next morning to inspect a worksite with a temperature of over 100F.
The AC was upgraded in well under a week.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
Yeah, and it's a huge hassle in vacant commercial buildings. Somebody needs to run every water tap and flush every toilet about once a month, or the whole place will stink up. Then the smell gets into the carpeting, which makes it hard to rent the building.
For special situations, there are calibrated drip valves. These are often found as part of fire sprinkler systems, which usually have a drain valve for when you need to drain the system for maintenance. The water from the drain valve has to go somewhere, which usually means a sewer connection. But you can't hook a water line to a sewer line; there are situations when you'd suck sewerage into the water system. So there has to be a vacuum break open to air. After the vacuum break, there's a U-trap with water to keep sewer gas inside. But since such drains are seldom used, the water will evaporate. So a tiny bit of water has to be dripped into the drain to keep up with evaporation. There are special "drip valves" for this.
One of the things you need to know about if you run large data centers.
I and a few guys were doing customer phone support in a remote building (ten years ago or such some). Soccer euro cup was up, and a collegue was desperate to find a way to watch the games, as the company (ISP) has just started operation, and callers were few and knowledable (so it was actually fun). Opening the cable funnel, he saw a TV cable. He spliced it up and connected it to a RJ45 jack. He then installed a TV tuner card into his PC, build a network cable look-alike to connect the TV card to the fake network jack, and voila - you could not see he was tapping the TV signal (the cable funnel was very visible, the computer was under the desk).
:-)
As we left the building about a year later, the fake jack was left there. I wonder what kind of head scratching this caused for the future tenants
If you've ever seen TV coverage of a Progress or Soyuz docking to the International Space Station, you've probably seen the ubiquitous black and white docking camera video with data overlayed on it as the vehicle approached the docking target.
1
Unfortunately, this television signal was only within the Russian Segment, and could only be downlinked through Russian communication assets over Russian ground sites. That limited the video to around 10 minutes each orbit, and required the docking to physically occur over Russia.
The US segment downlinks television via the Tracking and Data Relay Satellites (TDRS), which have more or less worldwide coverage. But the US segment and Russian Segment systems used incompatible video standards and weren't physically connected.
Yup, two video systems that cost tens of millions to develop, and they can't talk to each other. Classic "square peg, round hole" problem.
So we devised a setup where the crew ran a cable from the Russian Segment TV system into an IBM A31p laptop which converts the Russian SECAM signal to US NTSC video. The output from the laptop is connected to another cable strung down the stack into the US video system and downlinked via TDRS. Voila, greatly increased video coverage thanks to a lowly Thinkpad.
Details of this being tested can be found here: http://www.spaceref.com/news/viewsr.html?pid=1879
Worst...sig...ever!
No, there is a good reason behind this.
A transformer is basically two lengths of wire running alongside each other for a long distance. If you run a low voltage circuit next to mains cable, you can potentially (in the case of a loop) create dangerous inductive heating in the low voltage cable - Not to mention the problems of the induced voltage in whatever equipment is connected to it.
They also figured that server backups were probably a good idea, since they routinely handled millions of pounds of transactions per day in that one office alone.
And since they were accountants, they naturally picked the cheapest backup solution they could dig up, which was a 40-dollar backup box that used VHS video cassettes, underneath a beancounter's desk, right by his foot. I shit you not: every few weeks, it would occur to him that a backup hadn't been done in a while, so he'd shove the VHS cassette into the backup box with his foot, then nudge the start button with his foot, and return to counting beans. The cassette would pop out when it was finished, and that was proof positive for them of a job done properly. They never even bought a second VHS cassette. Amazingly, the thing never stretched to snapping point, but it was undoubtedly unusable for restores (it never occurred to them to do test restores), making it genuinely much, much worse than useless.
At the office on the other side of town was the accounts department for another division. They also used VHS backups, but felt that doing backups was a bit beneath them really, so instead they had the office cleaner shove the VHS cassette into the 40-dollar backup box next to the office door every night on her way out. One night she was home with the flu, and hadn't left instructions for her replacement to do the "backup". Sure enough, the server crashed that night, and the stale backup wouldn't restore. The poor cleaner was immediately fired, but not the asshats who delegated mission-critical IT chores to a cleaner, on dimestore reject equipment.
I felt duty-bound to tell these fucking morons that they were really making a false savings on backup equipment, and needed to buy real backup gear, with someone trained to monitor the state of the scheduled nightly backups and do scheduled test-restores. This company was pulling in 13 billion US dollars in revenue a year, so 1500 dollars for an internal tape drive and a copy of Cheyenne to protect hundreds of millions of dollars worth of data sounded like a pretty unbeatable deal to me.
Not to them though. "You IT people", quoth a senior beancounter, shaking his head, when I took the purchase requisition to his desk for signature. "It's always more money for the latest damn thing, isn't it."
Cheapest of all would have been for them to simply use the central Unix servers, which were run properly with tested and reliable disaster recovery by experienced sysadmins. I tried explaining that there'd be no change to their DOS PCs, and they'd still have the same F: G: and H: drives, with no visible change to their working environment. I even offered to pay for the new client software. They'd save money, and get vastly better care of precious data.
The reply: "Heh heh heh! And then next year there'll be some reason why we all have to get rid of 1-2-3. And after that there'll be some reason why we have to get rid of DOS. No thanks! Heh heh heh! You guys never quit, do you!"
I have been told that, back in the late 70s or early 80s, when a new courthouse/office building was built in a nearby county, someone got the idea to use the heat generated in the computer room to augment the building's heating system.
As I heard it, during the first winter, the gas company sent inspectors to check the pipelines, test the meters, etc., because they couldn't imagine that a building of that size could use so little gas in the wintertime.
I worked in an office on a Sperry Univac BC7 mini computer. It had a LED panel that displayed certain error messages. I learned how to send messages to the panel. They had air conditioning, but were too cheap to set it to a comfortable level. One day I had a brainstorm while sitting there sweating. I sent "overheat warning" to the panel. I pointed it out to the office manager. He immediately turned the air on.
I once worked for a dot-bomb e-commerce company. We had a product that tied into several major credit card issuers (i.e. >40% worldwide market share for issued credit cards). As part of the installation and maintenance of the product, I got to spend many weeks in MAE East (perhaps the biggest data center in the world). From what I've read about the Baby Bells' special networking rooms when the NSA scandal broke last year, I wouldn't be surprised if these servers shared one of those special rooms with the NSA routers.
The data center was about 5 floors below ground level. No form of wireless communications worked whatsoever--cell phones, pagers, etc. Once I parked my car, I had to go to an unlabeled metal door with a tiny camera on the top. Security guards would buzz me in and require me to sign in at their station. Then I would get buzzed in to the main data center room that contained another room inside of it. From there, I had to enter a password into another security system and place my palm on a palm scanner. Inside this room was another security guard--I would have to sign in with them, too. Then I would enter a different password into another security system, and place my head in front of this retinal scanner. This would buzz me into another room with the cages for each of the clients. There was a padlock on the cage, behind which were our servers. The servers required two separate smart IDs to be placed into an external card reader so that there had to be at least 2 people there to perform any maintenance. The servers themselves were locked down pretty tightly, too. It all seemed pretty insane as far as security goes, but I understood--these computers contained every credit card for the credit card issuer.
Well, after about 3 days of going to this data center, everyone got to know me. They would sign in for me to speed up the process. The security guard behind the door with the palm scanner used to get very hot, so she would often block the door open, thus defeating the palm scanner. The retinal scanner also had problems, often requiring about 3 tries before it would read correctly, so that door was often blocked open, too. Then, one day one of us had forgotten our smart card. We started cursing, as the round trip to pick up the card was about 45 minutes, so we tried it with only one smart card. Bingo. It worked. So then we tried it with no card. Seems the card readers weren't functioning properly. So, overall, we were able to defeat all of the security measures except for the padlock, and all because the security staff (getting paid 2 bucks above minimum wage, no doubt) all "knew" us. In my humble opinion, it would have been far smarter to *not* have the security guard in the foyer behind the palm scanner. After all, social engineering is probably the most common form of circumventing security.
Another funny thing about this was that we had a rather difficult security audit for all code releases. We had a bunch of ex-NSA employees working for us that were rather good about it, too. We would also hire outside auditors to do reviews of major code releases. It was all fantastastic, except for one thing: code patches didn't get the same scrutiny as code releases. In fact, they got none. Well, in order to expedite the release of one particular feature (that required emailing confirmation to customers), we packaged it as a "patch". No security audits. And for something that required the installation of a mail server! Furthermore, the code base had access to the record-level encryption used to store the credit cards. So, basically, if I had wanted, I could have installed a bit of code that would have decrypted all of the credit cards of users of our software and emailed them to a third party. I could not believe it. It's a good thing I have what I consider to be high moral and ethical standards.
I realized through this ordeal that security measures are not put in place to ensure security. They are put in place to give people the perception of security. And, furthermore, automation and removing the human element are good things for security. People should be used to monitor and oversee automated security, not to be actively involved in that automated security.
--Be human.