Slashdot Mirror
What Bizarre IT Setups Have You Seen?
MicklePickle wonders: "I was talking to a co-worker the other day about the history of our company, (which shall remain nameless), and he started reminiscing about some of the IT hacks that our company did. Like running 10BaseT down a storm water drain to connect two buildings, using a dripping tap to keep the sewerage U-bend full of water in a computer room, (huh?). And some not so strange ones like running SCSI out to 100m, and running a major financial system on a long forgotten computer
in a cupboard. I know that there must be a plethora of IT hacks around. What are some you've seen?"
The fire marshall came in and said "you can't have those low-voltage wires run through that conduit, that conduit is designed for high voltage wiring." So the electricians came in and sawed off their beautiful conduits, leaving the wires draped between the four-foot-spaced supports. They tie-wrapped the bundles every foot or two, but it still looked like a dead python hanging between branches.
To this day I still can't fathom what the hell that inspector was thinking.
John
I worked at one place where our room was a couple of floors underground (very depressing place) and we wanted to listen to the cricket on the radio (pre internet days). Armed with a crappy radio we found we could get perfect reception by connecting to the air conditioning vents with a set of crocodile clips purchased from Tandy's.
Another one I remember is something low-tec invented by some admin staff, we had a policy set in place that locked workstations after 5 minutes of activity, the PC's were severely locked down so you couldn't change this. Turned out the admin section of the company despised this as they would do something on their accounts package, talk to someone on the phone and by the time the phone call had ended the PC had locked itself requiring their password to unlock it. One lady actually took a small clock, took the plastic front off and attached a piece of paper to the second hand, when she wasn't doing anything, she placed the mouse in front of the clock so that when the second hand went past, it moved the mouse slightly stopping it from locking. When the guys in tech support found it, she was visited by practically every IT person just to see it in action.
Task Mangler
I've seen untwisted coat hangers covered in electrical tape and twisted together used to supply AC between two buildings in tropical weather in PNG. The wiring to the main building was bad enough but using coat hangers to supply power to the small hut that housed the computer equipment was priceless. I should also point out that they did not have power outlets for the computers either. They just cut the plugs off, stripped the wires, twisted them together and covered it in electrical tape.
After begging facilities since the previous year to upgrade the AC (and having one last big machine installed), we 'solved' the problem by buying a small, window-type AC, and poking it out the door. With this setup, we could generally get the room to stablize at around 30C (about 86F).
This worked until facilities showed up and complained that we needed to go through them to get any sort of AC installed, and demanding that we stop using the offending unit. (but required us to continue with the un-responsive process of getting the room AC upgraded).
Peter resolved the impass by calling the health and safety group, and keeping the door closed until they arrived the next morning to inspect a worksite with a temperature of over 100F.
The AC was upgraded in well under a week.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
Yeah, and it's a huge hassle in vacant commercial buildings. Somebody needs to run every water tap and flush every toilet about once a month, or the whole place will stink up. Then the smell gets into the carpeting, which makes it hard to rent the building.
For special situations, there are calibrated drip valves. These are often found as part of fire sprinkler systems, which usually have a drain valve for when you need to drain the system for maintenance. The water from the drain valve has to go somewhere, which usually means a sewer connection. But you can't hook a water line to a sewer line; there are situations when you'd suck sewerage into the water system. So there has to be a vacuum break open to air. After the vacuum break, there's a U-trap with water to keep sewer gas inside. But since such drains are seldom used, the water will evaporate. So a tiny bit of water has to be dripped into the drain to keep up with evaporation. There are special "drip valves" for this.
One of the things you need to know about if you run large data centers.
I have a bunch of stupid cobbled together setups to talk about. It all comes from a combination of poor IT staff at university wages, infintessimal budgets and the overbearing institutional and faculty pressures.
i braries/sl8500/ (its $200,000). Luckily this one didn't actually come to pass.
1. A "server room" that was essentially the most worthless room in the entire building, a long skinny room with four windows (perfect for keeping an uneven temperature!). Rather than buy 19" racks or even wire racks, they found a bunch of tables and put one server on each all the way around the edge of the room.
1.a. All of the servers were in fact desktop systems; an Ultra 1 was the mail server, a SPARCstation 5 the print server, a Gateway Pentium Pro 200 desktop the web server, etc.
2. A lab had to be moved one room over, because its current location was deemed too valuable. The original room was designed for a lab, it had 20+ fiber optic networking ports, twist-lock power connections in the ceiling, that sort of thing. The new room had two electrical outlets, no dropped ceiling, and one fiber optic networking port. It had previously been used as a copy room/storage closet. The cost to move the fiber optic wiring (just one room over mind you!) was over $25,000.
So instead, I had the great idea to cut a hole in the common wall (above the drop ceiling line), purchase additional ceiling tiles and cut up 2x4's into wooden supports. The original ceiling boxes containing the networking were put on top of the blocks above the new tiles, and extension cables run through the wall into the new room. In the original room, which was turned into a lounge, you couldn't tell that there was anything funny going on.
The best part is that the lab manager, who insisted they needed every single network port, never used a single one of them in the new room. All of those cables now reside in a box marked "Giant waste of money".
3. The main Windows file server was purchased in 2002 and has an internal RAID (bad idea in my opinion). What was huge then is worthless now; 5 disks that total 135GB. To get more space, the administration begged for a single external 250GB USB drive to host all user data. Nevermind that there is no redundancy, that an external drive is more suspectible to theft or failure, and that USB is unnecessarily slowing things down.
4. A system administrator got it into his head that rackmounting was the way to go (I agree). So he begged for a 19" rack to be ordered, and placed all of his servers into it. Except he doesn't have a single rack mountable server, and he didn't get the rails for any of the cases either. So now he has one $500 rack, and 8 $100 shelves to go in it. Same guy also switched the KVM monitor to a 15" LCD that doesn't support the resolutions of 9 out of 10 systems connected to it.
5. A consultant was brought in to tell us what needed to be done with the computing infrastructure (what DOESN'T need to be done is more the question). His main suggestion was to set up a central backup service just for this college, so as to avoid paying the central university IT group fees to use their central service. OK, thats an idea I guess... except that he wanted us to buy this: http://www.sun.com/storagetek/tape_storage/tape_l
Basically every day is a new adventure in ridiculous IT methodology.
Some amount of cogitation was needed to realize that each of those small pipes was headed to the traps of the floor drains installed throughout the store...
Now that's a (plumbing) hack in the true meaning of the term!
Once the graphics cards were removed, the machines defaulted to booting with Serial consoles. This meant that if anything went seriously wrong, just about anything other than hardware maintenance could be done by SSHing to machine X and using a terminal program to connect to the console port of machine Y (or vice versa).
This included the ability to do a complete wipe and install, needing only to instruct the CoLo staff to insert the install CD (which were left on top of the machines) into the appropriate box.
One of the monitors ended up on my desk. I can't remember who got the other one.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
I and a few guys were doing customer phone support in a remote building (ten years ago or such some). Soccer euro cup was up, and a collegue was desperate to find a way to watch the games, as the company (ISP) has just started operation, and callers were few and knowledable (so it was actually fun). Opening the cable funnel, he saw a TV cable. He spliced it up and connected it to a RJ45 jack. He then installed a TV tuner card into his PC, build a network cable look-alike to connect the TV card to the fake network jack, and voila - you could not see he was tapping the TV signal (the cable funnel was very visible, the computer was under the desk).
:-)
As we left the building about a year later, the fake jack was left there. I wonder what kind of head scratching this caused for the future tenants
You reminded me of the time our network admin wanted to setup a failover for our main (high-traffic) website. He figured that he could just add the IP address of our off-site emergency server as a third entry in DNS, since (at least to him) DNS worked by always hitting the first IP, and only moving down the list if it couldn't hit the previous one.
Only it doesn't. It round-robined the requests, so 1/3rd of our traffic was immediately and swiftly rerouted to our emergency site, which some enterprising webmaster had setup to email the webmaster box if anyone hit it (to make sure, I guess, that no one was going to it).
We noticed it because 5 emails came in at once, and then 10 more, and then it didn't stop until Groupwise crashed. We lost all the email in the box, and emails were coming in at some insane rate. We figured it out maybe 3 minutes in, but by the time we logged in and made the change, it was way too late.
Random Musings
If you've ever seen TV coverage of a Progress or Soyuz docking to the International Space Station, you've probably seen the ubiquitous black and white docking camera video with data overlayed on it as the vehicle approached the docking target.
1
Unfortunately, this television signal was only within the Russian Segment, and could only be downlinked through Russian communication assets over Russian ground sites. That limited the video to around 10 minutes each orbit, and required the docking to physically occur over Russia.
The US segment downlinks television via the Tracking and Data Relay Satellites (TDRS), which have more or less worldwide coverage. But the US segment and Russian Segment systems used incompatible video standards and weren't physically connected.
Yup, two video systems that cost tens of millions to develop, and they can't talk to each other. Classic "square peg, round hole" problem.
So we devised a setup where the crew ran a cable from the Russian Segment TV system into an IBM A31p laptop which converts the Russian SECAM signal to US NTSC video. The output from the laptop is connected to another cable strung down the stack into the US video system and downlinked via TDRS. Voila, greatly increased video coverage thanks to a lowly Thinkpad.
Details of this being tested can be found here: http://www.spaceref.com/news/viewsr.html?pid=1879
Worst...sig...ever!
When I started, pre-computer, the accounts department had rows of incredibly noisy mechanical 'Marchant' multiplying machines, each on a resonant all-steel desk. The production department kindly sent up inch-thick felt pads, and the racket subsided. Then came re-equipment with the precious 'electronic' machines (Anita*) with a line of hot number-valves ('tubes') along the top. But accountants are traditionalists, so the felt pads loyally stayed on, as a kind of sympathetic magic for quiet calculation. Ventilation was supposed to come from below, but of course it never came through thick felt. Thus the Anitas were extra popular - because if you put your meat pies on the generous top during the morning tea-break, they'd be hot for everyone by lunchtime. * http://www.xnumber.com/xnumber/photo_anita_C_VIII. htm
They also figured that server backups were probably a good idea, since they routinely handled millions of pounds of transactions per day in that one office alone.
And since they were accountants, they naturally picked the cheapest backup solution they could dig up, which was a 40-dollar backup box that used VHS video cassettes, underneath a beancounter's desk, right by his foot. I shit you not: every few weeks, it would occur to him that a backup hadn't been done in a while, so he'd shove the VHS cassette into the backup box with his foot, then nudge the start button with his foot, and return to counting beans. The cassette would pop out when it was finished, and that was proof positive for them of a job done properly. They never even bought a second VHS cassette. Amazingly, the thing never stretched to snapping point, but it was undoubtedly unusable for restores (it never occurred to them to do test restores), making it genuinely much, much worse than useless.
At the office on the other side of town was the accounts department for another division. They also used VHS backups, but felt that doing backups was a bit beneath them really, so instead they had the office cleaner shove the VHS cassette into the 40-dollar backup box next to the office door every night on her way out. One night she was home with the flu, and hadn't left instructions for her replacement to do the "backup". Sure enough, the server crashed that night, and the stale backup wouldn't restore. The poor cleaner was immediately fired, but not the asshats who delegated mission-critical IT chores to a cleaner, on dimestore reject equipment.
I felt duty-bound to tell these fucking morons that they were really making a false savings on backup equipment, and needed to buy real backup gear, with someone trained to monitor the state of the scheduled nightly backups and do scheduled test-restores. This company was pulling in 13 billion US dollars in revenue a year, so 1500 dollars for an internal tape drive and a copy of Cheyenne to protect hundreds of millions of dollars worth of data sounded like a pretty unbeatable deal to me.
Not to them though. "You IT people", quoth a senior beancounter, shaking his head, when I took the purchase requisition to his desk for signature. "It's always more money for the latest damn thing, isn't it."
Cheapest of all would have been for them to simply use the central Unix servers, which were run properly with tested and reliable disaster recovery by experienced sysadmins. I tried explaining that there'd be no change to their DOS PCs, and they'd still have the same F: G: and H: drives, with no visible change to their working environment. I even offered to pay for the new client software. They'd save money, and get vastly better care of precious data.
The reply: "Heh heh heh! And then next year there'll be some reason why we all have to get rid of 1-2-3. And after that there'll be some reason why we have to get rid of DOS. No thanks! Heh heh heh! You guys never quit, do you!"
I have been told that, back in the late 70s or early 80s, when a new courthouse/office building was built in a nearby county, someone got the idea to use the heat generated in the computer room to augment the building's heating system.
As I heard it, during the first winter, the gas company sent inspectors to check the pipelines, test the meters, etc., because they couldn't imagine that a building of that size could use so little gas in the wintertime.
I worked in an office on a Sperry Univac BC7 mini computer. It had a LED panel that displayed certain error messages. I learned how to send messages to the panel. They had air conditioning, but were too cheap to set it to a comfortable level. One day I had a brainstorm while sitting there sweating. I sent "overheat warning" to the panel. I pointed it out to the office manager. He immediately turned the air on.
I once worked for a dot-bomb e-commerce company. We had a product that tied into several major credit card issuers (i.e. >40% worldwide market share for issued credit cards). As part of the installation and maintenance of the product, I got to spend many weeks in MAE East (perhaps the biggest data center in the world). From what I've read about the Baby Bells' special networking rooms when the NSA scandal broke last year, I wouldn't be surprised if these servers shared one of those special rooms with the NSA routers.
The data center was about 5 floors below ground level. No form of wireless communications worked whatsoever--cell phones, pagers, etc. Once I parked my car, I had to go to an unlabeled metal door with a tiny camera on the top. Security guards would buzz me in and require me to sign in at their station. Then I would get buzzed in to the main data center room that contained another room inside of it. From there, I had to enter a password into another security system and place my palm on a palm scanner. Inside this room was another security guard--I would have to sign in with them, too. Then I would enter a different password into another security system, and place my head in front of this retinal scanner. This would buzz me into another room with the cages for each of the clients. There was a padlock on the cage, behind which were our servers. The servers required two separate smart IDs to be placed into an external card reader so that there had to be at least 2 people there to perform any maintenance. The servers themselves were locked down pretty tightly, too. It all seemed pretty insane as far as security goes, but I understood--these computers contained every credit card for the credit card issuer.
Well, after about 3 days of going to this data center, everyone got to know me. They would sign in for me to speed up the process. The security guard behind the door with the palm scanner used to get very hot, so she would often block the door open, thus defeating the palm scanner. The retinal scanner also had problems, often requiring about 3 tries before it would read correctly, so that door was often blocked open, too. Then, one day one of us had forgotten our smart card. We started cursing, as the round trip to pick up the card was about 45 minutes, so we tried it with only one smart card. Bingo. It worked. So then we tried it with no card. Seems the card readers weren't functioning properly. So, overall, we were able to defeat all of the security measures except for the padlock, and all because the security staff (getting paid 2 bucks above minimum wage, no doubt) all "knew" us. In my humble opinion, it would have been far smarter to *not* have the security guard in the foyer behind the palm scanner. After all, social engineering is probably the most common form of circumventing security.
Another funny thing about this was that we had a rather difficult security audit for all code releases. We had a bunch of ex-NSA employees working for us that were rather good about it, too. We would also hire outside auditors to do reviews of major code releases. It was all fantastastic, except for one thing: code patches didn't get the same scrutiny as code releases. In fact, they got none. Well, in order to expedite the release of one particular feature (that required emailing confirmation to customers), we packaged it as a "patch". No security audits. And for something that required the installation of a mail server! Furthermore, the code base had access to the record-level encryption used to store the credit cards. So, basically, if I had wanted, I could have installed a bit of code that would have decrypted all of the credit cards of users of our software and emailed them to a third party. I could not believe it. It's a good thing I have what I consider to be high moral and ethical standards.
I realized through this ordeal that security measures are not put in place to ensure security. They are put in place to give people the perception of security. And, furthermore, automation and removing the human element are good things for security. People should be used to monitor and oversee automated security, not to be actively involved in that automated security.
--Be human.
Well, this one might not be entirely in the spirit of the original question since it's not a cool "hack" so much as it is just an amusing error in planning, but here we go anyway:
Back in '95, my father was a VP of research for a large manufacturer of transmissive and reflective coatings for various glass applications (think insulated windows for the simplest example of said product) in Palo Alto, California. I was 15 and in highschool at the time, and having spent many a year trying like hell to keep a series of shitty no-name x86 computers up and running well enough to play the latest games, I had a sufficient skillset (and my dad had sufficient clout) to get me a job in their IT department. I did pretty well, and quickly found that users generally only got mean-spirited when made to look stupid, so a small dose of humility coupled with an interest in details on their primary task - "While I fix these printer drives you accidentally deleted, I was curious, what does a spectral photometer do?" - kept me out of trouble. Long story short, next year when I switched to full-time for the summer break, my boss actually brought me for a one-day business trip to our plant in Tempe, Arizona.
Now, you've got to realize, a business trip for a 16 year old (this was '96 now) is freakin' AWESOME. I was nervous as hell, had been up since the crack of dawn to take a red-eye with my boss out to the plant, and was deathly afraid I'd do something to embarrass not just me, but my father for having recommended me. So it was pretty unnerving to learn that my first job involved going into a large clean room production area, kept free from particles that could settle on the film during that specific type of sputtering process. We're talking the full disposable "bunny suit" that covered everything but the eyes, even with little slippers, and an airlock-type blower to clean you of all particles before entering.
The problem was a simple fix, really. The brand of 486 motherboard we were using at the time had a tendency, in about 1 out of every 3 units, to burn out the CMOS battery much earlier than you'd expect. And for a manufacturing-floor computer, not having a correct internal clock was a bad thing, not to mention that the lab techs had to go through some errors at startup with BIOS setings no longer being saved. So I suited up, cleaned off the replacement part and my tools as ordered, and went to find the bad machine.
That took some doing, oddly enough, since these computers were rarely shut down due to a 24/7 production schedule, so I had to go through back records on hand to find the lab techs' notes during the last power cycle on which computer had the boot errors. But, once located, the terminal was taken offline and I was able - after being told I had 20 minutes for the repair, tops, before the company would start to lose money as they needed that terminal again - to drag it off to a quiet, out of the way corner for the swap.
But see, there was a problem in the planning stages when this plant was set up. The PCs they used to control the machines were pretty complicated to configure, and the machines run in the clean room were just slightly modified versions of those used in the full-on manufacturing area in the main plant in Palo Alto. It was actually only a pretty small fraction of these production machines that had to operate in a clean environment. So when it came time to set these terminals up, they carefully washed off the outside of the older computers - computers, mind you, that have been sitting on a 24/7 PRODUCTION FLOOR with 10+ lab techs nearby at all times and various debris kicked up from the manufacturing process - and shuffled them off into the clean room.
So picture the scene: our hero, an extremely nervous 16-year-old on his first business trip in full head-to-toe bunny suit gear in the corner of a white, immaculately clean production floor opens his target computer to find a system so full of dust that he can't even SEE the goddamn cards inside. We're talking full-