Slashdot Mirror
What Bizarre IT Setups Have You Seen?
MicklePickle wonders: "I was talking to a co-worker the other day about the history of our company, (which shall remain nameless), and he started reminiscing about some of the IT hacks that our company did. Like running 10BaseT down a storm water drain to connect two buildings, using a dripping tap to keep the sewerage U-bend full of water in a computer room, (huh?). And some not so strange ones like running SCSI out to 100m, and running a major financial system on a long forgotten computer
in a cupboard. I know that there must be a plethora of IT hacks around. What are some you've seen?"
Just to clarify- the U-Bend is what prevents bathrooms and drains from smelling horrible. Inside the drain, shower water, sink water and toilet waste all mix together. As you can imagine this smells horrible. So, where every toilet, sink, shower, etc connects to the drain system there is a 'u-bend'- a downward dip in the pipe which stays full of water. This prevents air from flowing out of the empty drain.. png
Most sinks have their u-bend visible under the sink and look like this:
http://twenteenthcentury.com/uologos/ubend_shaded
Water flows in the top, and out the back. Because the back is higher than the bottom of the bend, the bottom stays full of water at all times, preventing air from passing.
Problem is, if you leave a drain long enough without water passing through it, the water in the u-bend can evaporate, leaving an empty pipe and allowign the nasty sewer smell to escape. Thus, leave a faucet dripping to keep the U-Bend full!
--IronHelix
Company moved into a new, larger building. The server room had a heating vent leading into it, and no A/C. They solved it by clogging the vent with a bag full of shredded paper and cutting a hole in the wall to install a small consumer single-room air conditioner.
Two worst I've seen: 1) While I've chopped patch patch cables in half and turned 'em into crossovers, this one place I toured got a good deal on pre-made crossovers and chopped & spliced them into patch cables for over 50 PCs; 2) Where I work now, a former employee jacked a cable modem straight into a Win9x peer-to-peer network, despite my protests (scary part of that was that he said, and I quote, "Oh, I do this all the time and it's never been a problem before." I spent the next week reinstalling Win98 and software...)
"You know you're narcissistic when you quote yourself in your sigs." -- PRoPAiN!
The fire marshall came in and said "you can't have those low-voltage wires run through that conduit, that conduit is designed for high voltage wiring." So the electricians came in and sawed off their beautiful conduits, leaving the wires draped between the four-foot-spaced supports. They tie-wrapped the bundles every foot or two, but it still looked like a dead python hanging between branches.
To this day I still can't fathom what the hell that inspector was thinking.
John
I worked at one place where our room was a couple of floors underground (very depressing place) and we wanted to listen to the cricket on the radio (pre internet days). Armed with a crappy radio we found we could get perfect reception by connecting to the air conditioning vents with a set of crocodile clips purchased from Tandy's.
Another one I remember is something low-tec invented by some admin staff, we had a policy set in place that locked workstations after 5 minutes of activity, the PC's were severely locked down so you couldn't change this. Turned out the admin section of the company despised this as they would do something on their accounts package, talk to someone on the phone and by the time the phone call had ended the PC had locked itself requiring their password to unlock it. One lady actually took a small clock, took the plastic front off and attached a piece of paper to the second hand, when she wasn't doing anything, she placed the mouse in front of the clock so that when the second hand went past, it moved the mouse slightly stopping it from locking. When the guys in tech support found it, she was visited by practically every IT person just to see it in action.
Task Mangler
I've seen untwisted coat hangers covered in electrical tape and twisted together used to supply AC between two buildings in tropical weather in PNG. The wiring to the main building was bad enough but using coat hangers to supply power to the small hut that housed the computer equipment was priceless. I should also point out that they did not have power outlets for the computers either. They just cut the plugs off, stripped the wires, twisted them together and covered it in electrical tape.
After begging facilities since the previous year to upgrade the AC (and having one last big machine installed), we 'solved' the problem by buying a small, window-type AC, and poking it out the door. With this setup, we could generally get the room to stablize at around 30C (about 86F).
This worked until facilities showed up and complained that we needed to go through them to get any sort of AC installed, and demanding that we stop using the offending unit. (but required us to continue with the un-responsive process of getting the room AC upgraded).
Peter resolved the impass by calling the health and safety group, and keeping the door closed until they arrived the next morning to inspect a worksite with a temperature of over 100F.
The AC was upgraded in well under a week.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
My previous job was in Network Operations at a University. Our Marine Science department had a large grant-funded sensor network running in a river somewhere in South Jersey that needed to talk to their machines on campus. They did this by getting a 56k leased line dropped out to the end of a long pier, to which they connected a cisco 2500 series router (state of the art at the time). It was housed in a box with just enough ventilation to keep it soaked in condensation, but not enough to allow for adequate cooling. Because of the heat it was on a permanent shutdown/reboot loop for most of spring, summer and early fall. They were lucky if they got more than a few hours of readings per day.
About ten years ago, I was working for what was then a small, startup ISP doing tech support. For about the first two years I was there, we often had to talk new customers through locking down their modems to 2400 baud in the registration/installation program, because that server often worked best at low speeds. (We also showed them how to reset it to the proper speed afterwards because our POPs were just fine.) I later found out that this was because whoever set up our one and only (at that time) registration server had multiplexed 42 modems through one COM port.
Good, inexpensive web hosting
I had an instructor who used to work in industry. He'd told me about a company he was consulting for. They had a Novell box that they administered remotely. During some remodeling, the small closet/room it was in was sealed with drywall. It was 4 years before the box required maintenance and someone went about trying to find it and realized what had happened.
There was the time back in the late 1980s when the multiplexed underground cable that my college was using to provide terminal connectivity for their new-fangled computerized class registration stopped working, and they ran dozens of hastily-created RS-232 cables from the data center to the hall where the action was, half a block away... secured to the sidewalk by duct tape, of course. Which at least didn't remain in place as long as the 10base2 cable that connected two dorms, strung between their 2nd floors (until it was taken out by a lightning strike).
More recent ugly hacks that I can claim personal credit/blame for are mostly of the sort that involve pulling a rabbit out of my ass because a solution needs to be found By Tomorrow Morning... like for deploying 200 installations of Windows 95 in a week (in the days before Ghost, or even backup software that preserved Long FileNames) using DOS boot diskettes, Netware, a utility called lfnbk, and ncopy... or building an e-mail server out of RedHat Linux 6 and spare parts (no, I didn't even have a complete working computer at my disposal) when the company's glorified BBS mail software found itself unable to exchange mail with the standards-compliant system used by a major new business partner.
http://alternatives.rzero.com/
How about the 10 MB email limit? That seemed to show up in the last 5 years or so. Before that I've had success with almost every size attachment I've been sent (and I do printing, so I see some pretty fat files.) When was the meeting held where they decided that?
One man's -1 Flamebait is another man's +5 Funny.
When I got hired as an Information Specialist for one of the government sponsored agencies in Hellinois, the people there would write their e-mails on a piece of paper and give those to their previous IT guy. He would then type them up and send them out via a yahoo e-mail. No kidding.
At one point, they had changed their routing so that they were using the new link but we hadn't, so we decided to see how a ping went.
A packet between the two machines would go through our router, over the ethernet that the two companies shared, out the (old) external router, and down the coast through Seattle, to California, then back up the coast to Vancouver, and then finally over the same shared ethernet cable that the packet had originally gone out before finally connecting to their router.
A cross-border round trip of a few thousand miles for a net distance of about 60 feet.
Oh, and did I mention that our server room was a converted bank vault?
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
I know the poster was looking for funny/interesting anecdotes directly from our community, but for those of you who haven't stumbled across The Daily WTF, hop on over to that site and make it a part of your daily reading. While the focus used to be mostly on programming, it's abstracted itself to the generic IT level in recent months, and you'll see all sorts of bizarre stories there.
:)
The Daily WTF is to IT workers what Jerry Springer is to everyone else. Just when you think you're having a bad day and your life is in the crapper, you can take a few minutes to soak in a situation where somebody else has it much, much worse...
Thanks to the War on Drugs, it's easier to buy meth than it is to buy cold medicine!
I wanted to try out the option to have the server page me in case of problems. Only problem was that the only phone jack in the server room was on the other side of the room, and I didn't have a phone cable nearly that long. But I did have a box of old ISA modems and short phone cables. My intuition told me that the "Line In" ports were wired directly to the "Phone" ports and didn't require power or actual computers to drive them. So I daisy chained modem cards and short cables together across the ceiling, wedging the actual cards behind cable housing and drop ceiling tiles, until finally I got dialtone. My supervisor commended me for my creativity but made me take it down, since the policy was that the modems were not to be connected to phone lines for fear of people being able to dial in to them or something. Never mind the dedicated internet connection.
I was the "computer guy" at a fabric processor in a town in Eastern PA that Shall Remain Nameless. Being "The computer guy" meant that they blamed me for the outages, but of course gave me no spending authority to do anything to fix the problems...
About 1 month into the gig, I was in the front office which was connected to the computer room by fiber optic cable (probably the smartest thing this company did.) However, once the fiber terminated at the switch in the office, the horizontal wiring to the workstations was, God help me, silver satin cable. Telephone wire. The shit was everywhere. There were about 100 workstations salted through the plant (which ran high voltage AC and heaters and whatnot) and everyone complained about the server performance. I wasn't even allowed (!) to put a network analyzer on the wire and was too naive/stupid at the time to realize what the problem was. The guy who had the spend authority, the "chief engineer," told me the problem was lack of RAM in the server and was always harping on me to upgrade the memory.
Another time I opened a closet to find a splice of this satin cable (they must have bought it surplus, they had hundreds of reels of the stuff) and the splice was made with, I kid you not, wire nuts.
I lasted 18 months there. I heard they brought an ex-Accenture conslutant in soon after to fix the "computer problems" and she ran the company into the ground.
Mit der Dummheit kämpfen Götter selbst vergebens.
Many computer rooms have packaged units which both heat and cool, and some also both humidify and dehumidify. That's fine if you only have one. If you have more than one, they need to be interlocked so you don't get one cooling while another is heating, or one humidifying while another is dehumidifying. If you get into that situation, everything will seem to be just fine, but your energy bills will be maybe 5x what they should be.
Saw that situation in a server room at Stanford a few years ago.
I have a bunch of stupid cobbled together setups to talk about. It all comes from a combination of poor IT staff at university wages, infintessimal budgets and the overbearing institutional and faculty pressures.
i braries/sl8500/ (its $200,000). Luckily this one didn't actually come to pass.
1. A "server room" that was essentially the most worthless room in the entire building, a long skinny room with four windows (perfect for keeping an uneven temperature!). Rather than buy 19" racks or even wire racks, they found a bunch of tables and put one server on each all the way around the edge of the room.
1.a. All of the servers were in fact desktop systems; an Ultra 1 was the mail server, a SPARCstation 5 the print server, a Gateway Pentium Pro 200 desktop the web server, etc.
2. A lab had to be moved one room over, because its current location was deemed too valuable. The original room was designed for a lab, it had 20+ fiber optic networking ports, twist-lock power connections in the ceiling, that sort of thing. The new room had two electrical outlets, no dropped ceiling, and one fiber optic networking port. It had previously been used as a copy room/storage closet. The cost to move the fiber optic wiring (just one room over mind you!) was over $25,000.
So instead, I had the great idea to cut a hole in the common wall (above the drop ceiling line), purchase additional ceiling tiles and cut up 2x4's into wooden supports. The original ceiling boxes containing the networking were put on top of the blocks above the new tiles, and extension cables run through the wall into the new room. In the original room, which was turned into a lounge, you couldn't tell that there was anything funny going on.
The best part is that the lab manager, who insisted they needed every single network port, never used a single one of them in the new room. All of those cables now reside in a box marked "Giant waste of money".
3. The main Windows file server was purchased in 2002 and has an internal RAID (bad idea in my opinion). What was huge then is worthless now; 5 disks that total 135GB. To get more space, the administration begged for a single external 250GB USB drive to host all user data. Nevermind that there is no redundancy, that an external drive is more suspectible to theft or failure, and that USB is unnecessarily slowing things down.
4. A system administrator got it into his head that rackmounting was the way to go (I agree). So he begged for a 19" rack to be ordered, and placed all of his servers into it. Except he doesn't have a single rack mountable server, and he didn't get the rails for any of the cases either. So now he has one $500 rack, and 8 $100 shelves to go in it. Same guy also switched the KVM monitor to a 15" LCD that doesn't support the resolutions of 9 out of 10 systems connected to it.
5. A consultant was brought in to tell us what needed to be done with the computing infrastructure (what DOESN'T need to be done is more the question). His main suggestion was to set up a central backup service just for this college, so as to avoid paying the central university IT group fees to use their central service. OK, thats an idea I guess... except that he wanted us to buy this: http://www.sun.com/storagetek/tape_storage/tape_l
Basically every day is a new adventure in ridiculous IT methodology.
Heck, feel glad that the "U Bend" (a.k.a. trap) had a water faucet. when maintenance boxed in our sink they took off the spigots off the mop sink, but left the drain functional. Then they boxed it in plywood. After the A/C was installed (and had dehumidified the room) - and the building's humidified air was shut off to the room .. no more humidity for us, and the drain dried out.
.. ugly hacks?
... that someone stuck a 10base-T hub on at 200 feet (give or take) because the hub actually strengthened the signal? Then they filled the hub with 8 10baseT computers..
.. they put in BNC .. then ran ARCNet over it (averaged about 300kbit/sec). Then complained it was too slow (well no kidding!!)
... after they'd sliced through it moving office furniture ... (yeah, it kinda worked)..
Hmmm
How about a Netware 3.x server stuck in a closet between two 10base2 (coax) runs, connecting one segment with another (glorified IPX router).
How about a 395 foot run of BNC
How about a 450 foot run of BNC with old, frayed screw on ends. That ohmed out at about 76 ohms. And they wondered why the network was slow.. (I re-crimped all the ends and cut the wire approximately in half, and used a second NIC in the seerver). In that same place, one of the workers figured out that if they took one of the BNC connectors off the T in the back of the PC, the network would go down and they could just sit there and do nothing...
The company that had a 1000 foot run, so instead of buying ARCNet wire
The BNC wire I saw that someone had repaired with a paper clip and electrical tape
And we won't even talk about how many networks I ran into that looked like
Server ----- hub ----- hub ----- hub ----- hub
and they wondered why the people on the 4th hub would lose server connections randomly..
= Grow a brain...
Once the graphics cards were removed, the machines defaulted to booting with Serial consoles. This meant that if anything went seriously wrong, just about anything other than hardware maintenance could be done by SSHing to machine X and using a terminal program to connect to the console port of machine Y (or vice versa).
This included the ability to do a complete wipe and install, needing only to instruct the CoLo staff to insert the install CD (which were left on top of the machines) into the appropriate box.
One of the monitors ended up on my desk. I can't remember who got the other one.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
I and a few guys were doing customer phone support in a remote building (ten years ago or such some). Soccer euro cup was up, and a collegue was desperate to find a way to watch the games, as the company (ISP) has just started operation, and callers were few and knowledable (so it was actually fun). Opening the cable funnel, he saw a TV cable. He spliced it up and connected it to a RJ45 jack. He then installed a TV tuner card into his PC, build a network cable look-alike to connect the TV card to the fake network jack, and voila - you could not see he was tapping the TV signal (the cable funnel was very visible, the computer was under the desk).
:-)
As we left the building about a year later, the fake jack was left there. I wonder what kind of head scratching this caused for the future tenants
In the early 80's, I was working for a company that did lots of its own kernel hacking on UNIX and VMS systems. They had a habit of implementing lots of their own software systems, rather than using standard ones. Some were not very clever. For instance, they had a communication "protocol" that ran over ethernet cable, but it didn't handle collisions. Yes, we had thick ethernet running to every office, and when anyone wanted to use it, they'd run out in the hall and yell to make sure it wasn't in use. If there was contention, data would be corrupted. Eventually, we punted on this stupidity and used TCP/IP.
A small business asked me to help them with their reliability problems. The company had been there for 20 years and the cabling for about 6 workstations and 4 or 5 big production printers consisted of premade cables tacked on the walls and ceiling, and one or more ancient efforts by electricians. There were serveral 10/100 hubs to extend the distance or 'multiplex' a single cable back to the internet connection. I went in several times to plug power into hubs, or pull cable out of a doorframe where it was being pinched. It was apparent that the main issue was this messy, poorly designed cable infrastructure.
So I got a data cable guy in to quote a clean installation-- home runs back to a single switch, cable in conduit, all clean and reliable. It was a few thousand dollars.
The owner decided it was too much $$, so he got an electrician to do it and now he has 3 more switches and even fewer home runs than he had before. But it's now Cat6, so he somehow thinks it's better.
I've already been back to troubleshoot... bad cabling.
A Very Large Telecom Corp(TM) had let a contract for a hardware subsystem that was to be connected to their very expensive network monitoring system (probably HP Openview). Anyway, the vendor couldn't quit make this work. So, to satisfy the contract, they had a tape monkey with a laptop in the NOC. Whenever an event happened on the subsystem, he'd manually copy the message into a dialog box on the master monitoring system, at which point it'd pop up on the regular NOC alarm system...
"Not an actor, but he plays one on TV."
This happened just this past year.
We had moved into larger building with a server room in the basemnent (cue ominous music).
We rapidly began to run out of space so decided to place the chief sysape in the basement near the servers, which made sense. We cleaned up some items in the basement, moved them into storage, carpeted, dry walled etc. Since it was in the basement it needed an egress window with a steel casing and ladder. This actually turned the office into a nice garden level. You could look out the window and watch the sprinklers, see trees and grass etc.
On day, the chief sysape comes in and notices water on the floor. He looks over at the egress window and there is about 2 feet of water collected in the base of the exit well.
Well, they shut down the water to the entire building. Luckily the server room actually had about an 18 inch raised floor, so no damage.
To make a long story short, upon investigation it turned out that when the sprinkler system was installed, instead of capping off the ends of the plastic piping, they folded it over and crimped it. They relied on the mass of the dirt to keep the ends crimped, and for years it worked. Until the egress well was installed and the dirt was disturbed. Once it was disturbed, the crimps began to fail under water pressure. Leading to a near IT disaster.
putting the 'B' in LGBTQ+
The machine with the user accounts on it had a few more hot, high-speed disks than the case was really designed to keep cool. It got hot and beeped. My boss wouldn't consider replacing it or even getting a new case. So I was forced to improvise: I cut a hole in the front panel and fitted a spare case fan into it. Then I realized that the motherboard didn't have another power connector for the case fan ... but I had a spare 5V wall-wart. A little wire-cutting and electrical-taping later, I had an externally cooled disk bay.
That "machine room" sucked. It was in the corner of the basement of a college office building. In the winter, the (crappy, household-type) AC unit iced over and the servers overheated. One summer, the facilities staff decided to power-wash the wood siding of the building. High-pressure water ran up through the wall and rained down right onto the server shelf. The only thing that blew up was the fancy new monitor that had come with the expensive and utterly overpowered RS/6000 just purchased by the library.
A couple of years ago when I visited the campus, they were still using that wall-wart-powered fan to cool the disks ....
-b.
When I started working at the school where I still work we were in two separate office buildings separated by more than 100m. I eventually ran coax through the wall via a light fixture, along a fence for about 100 feet into a tree to the roof (where it was held down by a sack of river rocks attached to some plywood) over the roof down a rain gutter and under the door. The building landlord was actually ok with this setup. It was mostly hidden except for the hop from the building to the fence and the whole tree to roof span.
OK. This was in the late nineties for a small computer hardware firm that had been in business since the '70s... but still, there was no excuse for this. It was a rambling wreck, a crazy collection of every ethernet standard implementation, and a few that were decidedly non-standard, just sort of tossed into place as time went on.
The backbone was a five port AUI concentrator... it was too primitive to be called a hub. (AUI was Sun's insane proprietary ethernet connector.) Hanging off that was a Sun server that was shiny and new when the Soviet Union was still in the news, which was the router to the DMZ, and a media adapter for thicknet. That thumb-thick yellow cable snaked over to the engineering cubes and hardware labs, with "vampire taps" hanging off it everywhere - vampire taps have a screw that drills into the cable, which is how you hook stuff up to thicknet. No lie. These were connected to 10Base-5 thin-net adapters, which hooked up to co-ax concentrators, which hooked up to AUI media adapters which hooked up to the various Sun workstations. I had never seen before, nor have I seen since, a BNC co-ax hub used just to hook up workstations in a star topology... for whatever reason, they decided that ring topology wasn't good enough to string five lightly used workstations together. I have no idea why any of this worked. It usually didn't, and needed various pieces of arcane equipment power-cycled and jiggled and cursed at to get any data to make it from the file servers to the workstations.
It gets worse. Another port on the AUI concentrator went to the Cabletron TPT-2 setup, which took care of accounting, sales, support and executive row. This was like 10Base-T ethernet, with a patch-panel that was wired to RJ-45 jacks in the offices and the cubes, except it was completely incompatible with 10Base-T equipment. Media adapters for all! And when one of the adapters goes down, the whole TPT-2 system locks up, a hundred or so systems. Let's play the hunt-the-locked-adapter! So much fun when the CIO is screaming at you.
I went on vacation, and the engineers were left to figure out how to bring the network back on when one of these adapters froze. You'd think they would unplug the patch cords one at a time in the computer room until the network came back up, but no. They just remembered that I told them power cycling an adapter would usually bring it back online. So they powered down the building. Serious. They needed to reboot the building... by this time all the critical systems were on UPS, so nothing was fuxxored, but still.
I eventually got the penny-pinchers in charge of the business to invest in nice 100B-T and 10B-T switches and AUI adapters and a few nice new Sun servers. Worked much better thereafter.
I was once working in a datacenter on a machine close to the company email servers. There's a message on the console for the email server saying that the email database was corrupted and that some utility had to be run to fix it. An IT guy walks up to the console and looks at the message. He then proceeds to reboot the server. After the reboot, the same message appears. Reboot again. Same message. Reboot again. Same message. The IT guy repeated this 10 or 15 times. Then he walked away perplexed at why the email server wasn't working.
"Fill the trap with cooking oil - it will stop the smell and will not evaporate as quickly as water would."
:)
A good example of lateral thinking. Good thing they're cloning you.
If you've ever seen TV coverage of a Progress or Soyuz docking to the International Space Station, you've probably seen the ubiquitous black and white docking camera video with data overlayed on it as the vehicle approached the docking target.
1
Unfortunately, this television signal was only within the Russian Segment, and could only be downlinked through Russian communication assets over Russian ground sites. That limited the video to around 10 minutes each orbit, and required the docking to physically occur over Russia.
The US segment downlinks television via the Tracking and Data Relay Satellites (TDRS), which have more or less worldwide coverage. But the US segment and Russian Segment systems used incompatible video standards and weren't physically connected.
Yup, two video systems that cost tens of millions to develop, and they can't talk to each other. Classic "square peg, round hole" problem.
So we devised a setup where the crew ran a cable from the Russian Segment TV system into an IBM A31p laptop which converts the Russian SECAM signal to US NTSC video. The output from the laptop is connected to another cable strung down the stack into the US video system and downlinked via TDRS. Voila, greatly increased video coverage thanks to a lowly Thinkpad.
Details of this being tested can be found here: http://www.spaceref.com/news/viewsr.html?pid=1879
Worst...sig...ever!
The server room was a fairly large closet with an a/c outlet and a combination of wire racks and IKEA shelves. Nothing too bad, there; it all worked and everything was strapped down in case of a quake. However, to get to the server room you had to go through the breakroom and pass by the kitchen. Which had maybe two outlets and a hardwired coffee maker. Which shared a breaker panel with the server room in the hallway behind both rooms. Can't tell you how many times the admin assistant killed the server room trying to shut off the coffee maker. On a Friday, at 5 p.m... Funny thing is, the Head Cheese only worried about his coffee, not the servers that housed our precious COBOL and account information and wouldn't authorize either a separate breaker panel or a good UPS. Then again, that same admin assistant had to print out the boss' email and Excel spreadsheets and then re-type in his modifications...
A few years ago I used to get together with some mates every weekend for a little LAN party. The small apartment we met in grew too small for the number of participants, however fortunately one of the newcomers lived in the apartment below, so some CAT 5 was run out on to the balcony, and down to the next floor. Instant room for expansion on the LAN parties.
The cable remained there for many months - rain, hail and shine until this guy moved to a house directly across the street. There was talk of running the CAT 5 across the street alongside the power lines, however it never happened.
On the work front, once I visited a client complaining of slow network performance. Following the cables, I popped my head up into the false ceiling to find a roll of 100m of 10B2 cabling.
RedFive jedi_knight111@hotmail.com
I've seen people try to use MS Exchange in place of a mail server.
Hey, you did ask.
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.
Well now, remember I said it had no BIOS? What it *does* have is an octal debugger, similar to DEBUG in MS-DOS, called ODT. This is actually built into the microcode of the CPU; the CPU requires a console serial port to be present to even POST. If it's not there, a little LED lights on the edge of the CPU board and the machine will never come out of halt. So, at worst, all you need to do is hit <BREAK> type in the boot loader code on the terminal, and the machine will boot. Right?
Right. But that's a pain in the gluteus maximus, because it means typing in a load of stuff like and so on for a few dozen lines. There must be an easier way. What, like burn them into an EPROM? Well yes, but I don't have an EPROM burner. What I *do* have, though, is a VT-510 terminal, which allows you to program key sequences into the function keys. So, what I do now is power up the terminal and the PDP11, press HALT and then RESET on the front panel, hit a key sequence on the terminal, drop back into RUN once the disk seeks (controller is ready) and it's booted.
Yes, I'm buying an EPROM blower off eBay...
I recall installing some additional power outlets in one building in the tropics and worrying about violating code by doing it myself. The building was wired with 8-gauge multiple strand copper wire (for 110VAC 60Hz) with one hot, one neutral and (ostensibly) one ground. Once I got a look under the faceplate, however, I realized that code wouldn't be a problem.
The wiring method was bizarre: at each terminal (screw) the wire was stripped of insulation by tearing for about 2 inches (~5 cm), wrapped around the screw and then continued on its way to the next. It went like that for all the outlets we examined.
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.
At my former job we used fibre cable under major roads to link CCTV cameras to our control room environment. One day a road worker drilled down into the road with some sort of hole digger and wrapped 100 metres of fibre around his machine, exactly like rolling pasta around a fork.
http://michaelsmith.id.au
A little over a year ago I had the first ever chance to do something right with regard to IT infrastructure. Prior to that we were based in an office where our 'server room' was a closet. The room in which the closet was located had power issues, bad air conditioning, etc. and we'd regularly have issues with heat or power. When they looked at moving us I jumped at the chance. Got a 600 square foot room with fully independent 480V power service. That power service includes an APC Symmetra with a nominal 15 minute run time. That's backed up by a 125kW natural gas fired generator. We also extend a tendril out to the MDF in the building to power our ISP's routing gear there. The power system is regularly exercised and tested. Air conditioning is provided by two independent systems and the room is kept at 65F. There are forty servers in the room. One day we decided to see what would happen if there was air conditioning failure. In the space of a half hour the temp in the room went from 65F to 85F. So we know we got the cooling system specs right. Now I'll explain the flaw in our whole system. We depend on someone else for DNS resolution. When they go down, we go down. They're finally seeing the light and putting DNS on somewhat more robust boxes and power systems. Just goes to show, you can plan for everything but you'll never find it all.
Working in a fairly large software company (the technology will probably give it away but I still dont plan to name names) our department had our own private kitchen and espresso machine (because the site canteen was heavily over priced)
We had an honour system for payment - an old desktop PC with a card reader. You swiped your ID badge through a card reader. All this did was extract the card ID string and send it through a shell script to a mysql database which then deducted from your balance the cost of a coffee - hand cash to the secretaries to top up the balance (I'll admit on average most people were in negative balance though every now and then the worst offenders had their balance details mailed to the whole department to shame them into paying up)
The actual purpose of the card reader PC? It was the DHCP server for the (still in use at the time - 2002) token ring network.
$_="Slashdotter";$syn="OTT";s;..;;;sub _{print shift||$_};s!ash!Perl !;s=$syn=ack=i;tr+LLEd+BLAH+;_"Just Another ";_
(I usually use while : ; do echo ^G >/dev/tty1 ; sleep 1 ; done)
If it was the old CFC-based stuff, at least it was neither electrically conductive nor inflammable .....
Je fume. Tu fumes. Nous fûmes!
Recently I did a short stint for a household name UK company in their reporting department. They needed to report accurately on the performance of their call centres which were run on a number of Aspect ACDs, the call data from these ACDs were dumped into some kind of data warehouse.
From their the department needed to create a reliable and accurate reporting infrastructure to deliver key business data throughout the company.
What they had was a "primary layer" using Crystal Enterprise to create a number of huge spreadsheets containing the days data in large Excel files. The "secondary layer" were a Excel spreadsheets filled with VBA code to read the files created by Crystal process them a bit and output them as new files elsewhere. The "tertiary layer" was another lot of Excel spreadsheets filled with more VBA code to read the secondary layer and add drop down boxes and graphs etc to format the data. In some cases there was a "quatenary layer" which read the tertiary layer and did some more processing.
Needless to say all the VBA in the Excel files was mainly recorded macros with no attempt to check whether any of the various files it relied on were present or showing the correct data. All the files were scattered willy nilly around the network and all of them had to be ran manually in the correct order every day.
Frighteningly they had a huge backlog of further Excel files to write to do yet more processing and they were rebooting Crystal on a daily basis. I left after a couple of weeks because I didn't want to be around when the whole thing collapsed in a giant mess around their ears.
I believe you are referring to "Bedlam DL3".
Until quite recently I worked at an Aussie telco, not one of the massive ones but in the top 5. I noticed the guy at the desk next to me had a mobile phone plugged into a charger, stashed on the floor under his desk, 24/7. I asked him about it once, and he darkly warned me never to unplug it, answer it if it rang, or otherwise touch it.
I later discovered its purpose; basically it all came to do with "legacy" MMS messages, which are MMS to phones which don't have MMS capabilities (SMS only). The network instead sent the message to the customer as an email (SMTP, which is what MMS uses anyway), with the sender address of a certain "special" service. Apparently, due to the way the providers' network had been designed, if the GSM service it believed these messages were "from" was not "alive" on the HLR, it would refuse to send these thousands of legacy messages to customers.
I always wondered what would happen if one of the cleaners came through afterhours and decided to help himself to a free phone...
Many years ago I was called on a Friday lunchtime by a very scared IT manager.
The computer that was running their organisation had been struck by lightning.
They offered me a lot of money to get the system up and running by monday 09.00.
Naturally I accepted the challenge and made my way to the site, it was unbelievable.
The lightning hadn't actual struck the computer, it had hit a power line 100 yards away, but the damage was incredible.
I thought he was joking when he said it had melted.
Well over the course of the weekend I managed to either build, repair, or source replacements for everything except the memory. It was now about 03.00 monday and I hadn't slept, eaten, or had a beer since about 12.00 Friday.
Fed up by now I hacked up a little serial interface and connected it to one of the tape drives that had survived the blast, man those things were tough, then created a little board on vero that took a memory address and serialised it. This was hacked into the system and I punched in the boot code.
10 minutes later, a monitor sprang to life.
Well when I say sprang, it more like oozed to life.
One character per second, you had to hold a key down until it appeared on screen then release it, but it was working.
Amazingly all their records had survived as well! ( found out later that all the tape drives were run off a seperate mains feed with a surge supressor, at least one engineer must have known what he was doing.)
They paid me off and ran like that for two days before they could get a replacement ram board.
They also figured that server backups were probably a good idea, since they routinely handled millions of pounds of transactions per day in that one office alone.
And since they were accountants, they naturally picked the cheapest backup solution they could dig up, which was a 40-dollar backup box that used VHS video cassettes, underneath a beancounter's desk, right by his foot. I shit you not: every few weeks, it would occur to him that a backup hadn't been done in a while, so he'd shove the VHS cassette into the backup box with his foot, then nudge the start button with his foot, and return to counting beans. The cassette would pop out when it was finished, and that was proof positive for them of a job done properly. They never even bought a second VHS cassette. Amazingly, the thing never stretched to snapping point, but it was undoubtedly unusable for restores (it never occurred to them to do test restores), making it genuinely much, much worse than useless.
At the office on the other side of town was the accounts department for another division. They also used VHS backups, but felt that doing backups was a bit beneath them really, so instead they had the office cleaner shove the VHS cassette into the 40-dollar backup box next to the office door every night on her way out. One night she was home with the flu, and hadn't left instructions for her replacement to do the "backup". Sure enough, the server crashed that night, and the stale backup wouldn't restore. The poor cleaner was immediately fired, but not the asshats who delegated mission-critical IT chores to a cleaner, on dimestore reject equipment.
I felt duty-bound to tell these fucking morons that they were really making a false savings on backup equipment, and needed to buy real backup gear, with someone trained to monitor the state of the scheduled nightly backups and do scheduled test-restores. This company was pulling in 13 billion US dollars in revenue a year, so 1500 dollars for an internal tape drive and a copy of Cheyenne to protect hundreds of millions of dollars worth of data sounded like a pretty unbeatable deal to me.
Not to them though. "You IT people", quoth a senior beancounter, shaking his head, when I took the purchase requisition to his desk for signature. "It's always more money for the latest damn thing, isn't it."
Cheapest of all would have been for them to simply use the central Unix servers, which were run properly with tested and reliable disaster recovery by experienced sysadmins. I tried explaining that there'd be no change to their DOS PCs, and they'd still have the same F: G: and H: drives, with no visible change to their working environment. I even offered to pay for the new client software. They'd save money, and get vastly better care of precious data.
The reply: "Heh heh heh! And then next year there'll be some reason why we all have to get rid of 1-2-3. And after that there'll be some reason why we have to get rid of DOS. No thanks! Heh heh heh! You guys never quit, do you!"
A certain somebody who may remain nameless, but may or may not also be the parent poster, used a similar method to rattle the noisy old disks on a server that lived in my bedroom at ungodly hours in the morning....
There was that elderlish guy at a large german university where I used to work whose network was no longer working. We did the standard procedure (with 3000 machines spread over a city the size of Munich, you don't jump to the desaster area right away): Check if routers worked, pinged some machines in the building he was in, looked at the logs... to no avail: I had to go over to him (3km, in deepest winter, at -15C, with rush hour traffic jams that rendered cars basically unusable)... ... When I arrived, I ran some diagnostics on the machine, but it seemed there really was no network connected. I checked the cable box in the wall and the one on his machine. We used to deploy 10m-cables, because some genius bought them in bulk, and the PHB figured the were "cheaper" because of this bulk-buying - even if the distance between wall and machine was less than a meter.
Well, the cable was connected both to the wall, and to the computer. Unfortunately, it was clearly CUT right in the middle. When I questioned that elder superhero, he stated that he found out years ago that you could use a copper cable as an TV antenna, and he received a memo the day before that WiFi was now available in all university buildings - so he decided to cut the Cat5 to serve as a WiFi antenna....
Screw the FSM - Real geeks believe in the Invisible Pink Unicorn
A while ago I ran an Amiga software development company. Our designer (Mark) had an Amiga 4000 with various external SCSI devices running off a notoriously unreliable Commodore A4091 SCSI card.
I went to his desk once trying to access a file on his external drives, and I kept getting disk errors. I called him over, and he said "Oh! That disk won't work unless you open up the system clock and resize it to this kind of size, and put it on the screen here". He opened the old analogue-face clock program that came with the amiga, resized it to about 200 pixels square, and stuck it in the top right of his screen.
I stood there smiling. He was, after all, a designer.
The file opened fine though after he did that.
I did some messing around on his machine afterwards. I was convinced there was some kind of obscure problem that we were missing - incorrect termination or bad cables maybe. I put the clock incident down to coincidence.
I could find nothing else wrong - but I still couldn't access the disk. So, I opened the clock application. I tried it on one side of the screen. File would not open. Moved it to the top right corner. The file opened. I did this about ten times as I couldn't believe the results myself. Every time I had the clock in the top right corner, the external SCSI disk behaved itself. I tried different applications, none of them worked in the same way - it had to be the clock.
I was completely spooked by the whole thing, and decided this was something sent by the Gods of SCSI to taunt me. The logical side of my mind believes that it is probably some obscure DMA issue, the rest of my mind believes the machine was possessed.
The thing I was never able to figure out was how Mark discovered the SCSI-healing properties of the Magic Amiga Clock and why he felt it was perfectly normal behaviour for his machine!
Jolyon
Please read my Canon EOS tech blog at http://www.everyothershot.com
1) The computer room floor built with a 4 foot void rather than 4 inches because the builder read the plans wrong. Mid you, there was room for a lot of kit in this 'split level' computer room.
2) The Netware 3.x file server which was a Toshiba T3200 plasma screen laptop locked inside a filing cabinet (a very secure solution on a military base). While I was working on it, a telephone began to ring in the next drawer up. I mentioned this to someone as nobody seemed to have heard it and the reply was "Oh, we don't answer that one"
3) The Olivetti M24 (AT&T 6300) that lived in a milking shed in the middle of a dusty field that eventually died and had to have a 2-3 inch layer of 'field' vacuumed out.
4) The computer room built with the existing radiators walled in but not turned off - took ages for the aircon guys to figure out why the room never cooled to the calculated temp.
5) The installation test of a new halon system (with a cylinder of CO2) where the engineers had not properly screwed the nozzle onto the 'j' pipe in the centre of the room. When the system was fired, the nozzle shot through the false ceiling, the gas followed it and the pressure blew down all the ceiling ties - the computer room looked like a scene from Die Hard.
6) The school network that comprised 5+ 'backbones' of 10Base2, each with around 20-30 D-Link *hubs* wired directly to cat5 outlets. Netware servers strategically placed round the building acted as repeaters with 2-3 NICs in each. We also found some Cat4 cable buried directly into the walls (no trunking).
7) 140m of Ethernet coax buried below a school field to link two buildings.
8) The over-length Token Ring network that included specially designed and developed repeaters that had to be 'tuned' using a screwdriver to adjust variable resistors to get the timing 'just right' so that the whole thing worked.
I have to add that I was *always* the support person brought in to sort things out - not the one creating the mess.
AT&ROFLMAO
Once upon a time (+-1989), we had a set of some 50 Apollo workstations linked up via a Token Ring network. Not only did that ring have a habit of being physically broken every so often, the worst part was that there was no file server. Everybody stored his own files on his own machine. Project accounts were housed on the machine of the project owner. Nice and orderly, huh?
Well... except for the fact that there were people who didn't have a personal machine. Their data was initially housed on the machine of someone they cooperated most with. When disks filled up, new people without a machine would end up on whatever disk happened to have spare capacity. Then we (or rather "they", as I was there but not part of the IT gang) found that the amount of data people store over time outgrows the size of their disks, especially if you have shared project accounts. So ever so often, accounts had to be moved around. And sure enough, before you knew it the owner of the machine to which some high profile project had just been moved would complain that his box was overloaded doing other people's I/O. And just when that had been sorted out, there typically would be a reorganisation involving people switching offices or desks. Sometimes the machine followed its owner (not all were equally fast and some of them had black&white displays that nobody wanted to inherit), but most often IT would object to moving the boxes. By now the physical link between the data and its owner is totally gone. In the end, most people didn't have a clue what machines their files were stored on.
And now the fun really starts. We relied a lot on students and interns. In those days, if a student had seen a computer before, either it was a Commodore 68 or an early standalone PC. They didn't have a clue what the network was used for, so whenever they were done and went home, they'd physically switch off the machine they had been using. To make matters worse, even just keeping employee data storage away from the student machines was not an option, because there were not nearly enough student machines around. Typically, students would use the machine of an employee who happened to be out of the office on that day or during that night. Oh, what sweet memories... Not!
Not to mention the backup problems... Nor the fact that we also had a parallel experimental ethernet network with non-Apollo machines, of couse also without a proper file server. After a while, some data was being stored there instead. Now where the ... did I save that f... file last week???
Linux user since early January 1992.
Can we please put the 'itsatrap' tag to good use folks?
v4sw6PU$hw6ln6pr4F$ck 4/6$ma3+6u7LNS$w2m4l7U$i2e4+7en6a2X h
I work for an outsourcing support company. The ISP/Cableco who was our client shall remain nameless, but here's what happened: They transitioned to a completely new billing system and whoever sold it to them had assured them the record would transition seamlessly from the old system, but they didn't. A percentage (forgot how many) of customer records had been goofed up in the transition. To fix this the home office needed copies of certain screens (about five different ones for each account) from the old billing system, so they knew what values to type in by hand on the new system.
They didn't have the time at the office to look up all these records themselves. Their solution was for us, the company providing customer support, to (I'm not making this up) take screen captures of the needed screens on the old billing system, print them out, and fax them to the home office halfway across the country. This faxing had to be done after normal business hours as our machine had to be free for other use during the day. Anyway, to give you an idea how much paper this was wasting, we were measuring usage by the ream, not the sheet. All of this paper was being fed through an autofeed on the fax/copier that would occasionally jam up, too. I volunteered for this assignment for the overtime.
Anyway, things were moving slowly and bugging from the fax/copier hiccups (this was a 30,000 page/month duty cycle copier that was used at closer to 50,000 pages according to HR) at about 2:00am I came up with a better idea, why are we printing these out and wasting all this paper when we can send them the screen shots themselves? I was noticing the readability was quite poor from some samples we sent back and forth with the other side. So we pasted the screen shots into Word documents, and then we tried to send the enormous Word documents over email, but the company's email server was complaining about the attachment size.
Well a co-worker had a GMail account he had been playing around with (this is when GMail was brand new, so he had gotten an invite for it) so we decided to try it because of the 10MB attachment limit. We could just upload the files and give the other side the log-in info and they could download the attachments at their leisure. Well, that didn't work out either, I think some of the files may have been too big or there was an issue getting the files uploaded, I don't remember anymore. But it didn't work out.
My next idea was to go back to faxing, but paperless. There was one workstation that had a dial-up modem instead of a NIC card, it was normally used to test access numbers for dial-up ISPs clients we had, but we hadn't used it in so long we didn't remember the login name to get into it, the password we were pretty sure of. After trying to guess it for a couple minutes I got the idea of booting the machine using a Knoppix CD and looking in C:\Documents and Settings\ to see what user folders were there (as I'd spot the correct login name amongst them). After we got logged into the machine we used a flash drive to transfer the files from the other machines we had been using to compile the Word screen shot documents. Then we'd open a document and fax it using Windows XP's built-in fax capabilities to the fax machine at the home office. So soon we had an over 50 page Word file printing over fax to the east coast with several more just like it queued up. It was moving slow but seemed to be working. The idea was we could now leave and let the machine fax the rest of the night.
I was excused to go home about 4am. I came in the next day around 1:30pm and found the fax calls had been interrupted around nine that morning. Apparently the home office had called because the same page was repeating over and over again on their side (which they naturally claimed must be an issue from our end). I didn't hear how they finished getting the records transmitted, but I think they went back to paper faxing again.
Now that I think about this, It would have probably been a better idea (if we'd had more than one day to do this) to just to take the huge Word documents and burn them to a CD and then Overnight the CD to the home office.
Twelve years ago, I worked in the I.T. department of a small private college in upstate New York (around 4000 students). Our main server room was fairly meticulous and well set-up (we had a perfectionist geek as the main sysadmin at the time). One summer, that building was scheduled for new electrical service to some new science labs in the upper floors of the building. A few hours after the electricians showed up, they came running to our offices and insisted we follow them down to the basement. There, they showed us the wiring to our server room, installed just a couple of years earlier: it was not actually physically connected. The wires had a small gap, and the electricity was simply arcing over. One serious bump of the box would probably move them enough to cut our power.
So their first task was to fix this. They would turn off the power for 30 minutes while we ran all the servers on UPSs, then temporarily reconnect power for awhile to recharge the UPSs, then turn off the power again and work... took all day at this rate.
$nice = $webHosting + $domainNames + $sslCerts
I have been told that, back in the late 70s or early 80s, when a new courthouse/office building was built in a nearby county, someone got the idea to use the heat generated in the computer room to augment the building's heating system.
As I heard it, during the first winter, the gas company sent inspectors to check the pipelines, test the meters, etc., because they couldn't imagine that a building of that size could use so little gas in the wintertime.
I worked in an office on a Sperry Univac BC7 mini computer. It had a LED panel that displayed certain error messages. I learned how to send messages to the panel. They had air conditioning, but were too cheap to set it to a comfortable level. One day I had a brainstorm while sitting there sweating. I sent "overheat warning" to the panel. I pointed it out to the office manager. He immediately turned the air on.
When I was in the USAF and working in Saudi Arabia in the 1980s, we used data and voice hookups provided by the local telco to connect 3 sites scattered out there in the Kingdom...
Trying to trace some problems with one of the voice circuits, in one wiring closet we found the voice circuit had 3 twisted wire splices in one 6 foot section of cable...
Later, as a contractor running the email system for a not very large, but well known US government agency...
A few months after the agency had moved a lot of people into the new Ronald Reagan office building in Washington, I noticed their GroupWise server didn't seem to be functioning.
I called the local GroupWise administrator and told her they needed to take a look at it.
She called me back an hour later to say they hadn't been able to locate the server and were still looking, but she's sure she had seen it recently...
Two days later they found it in a closet in the old building they had moved out of months before...Still running, but someone had finally shut down the network in the building...
Goofy, Geeky Gifts and More!
I worked for a HUGE multinational ISP once. We had just gotten France hooked up and they had been running fairly well for about six months after two years of testing. About 100k customers used the service.
... a 386 LCD laptop. The machine had died because the logs had filled up the 1.2 gb hard drive. We couldn't believe it until someone rebooted the damn thing, and DNS came back up. We had been running production DNS on this thing for over 2 years.
One day, DNS went down. This had happened in the UK a lot, so we barked up the wrong tree for hours thinking it was a Keyring issue over the Transatlantic connection. Nope. Hours later, we found the DNS for France was on a different subnet. This led to discovering that their DNS service was on a set of IPs that pointed to one MAC. Finally, the people in charge of the data center said, "That's not our subnet. I don't know where you are getting DNS from.
We traced back and back through routers, entering territory that got scarier and scarier. It went to an older building that were were in the process of closing down and selling. It also had a data center, but that room had been dark for months, and DNS had been working up until now. Back and back we went.
Finally we found that the trace went through a disused subnet through a former office LAN in that building. This traced it back to an office, which traced it back to...
Turns out that when the French network architecture was being set up, they had to transfer DNS somewhere temporarily as part of a testbed, so some guy had an old laptop in his office he just hooked up. Then he was laid off before we went live. Nobody ever switched it back, and since the office space was being abandoned, no one every went into the office to turn anything off, figuring it was somebody else's problem.
A week later, French DNS was running on a production server.
I am impressed it lasted that long on such a platform.
We also used to run the flight schedules for Lufthansa. It was a Windows NT 3.5.1 system that was running on a 486, and was running some proprietary terminal service and scheduler. It crashed once every 31 days (there was some bug where it would crash after xxxx hours which was between 30-31 days). The only way to fix it was to hard reboot the box, and the directions were scary: "Go down to the older server room, and find an unlabeled shelf next to the first door near the panic switch. On the bottom of that shelf is a box which is behind a stack of old 10base hubs. Hold down the power button until the green light goes off. You may have to lie on the floor on your stomach to reach the button. Count to ten, power back on. Make sure the amber light labeled 'turbo' is lit on bootup. If not, repeat, but wait 60 seconds before powering back up."
I sure hope they got that fixed, it was last like that in 2000.
Some years ago I was the computer guy for a fairly large network installation in a student dormatory. Because our 19in racks are (still) in the bicycle cellar, our equipment used to die of too much condensing wetness quite regularily.
So I ordered replacement parts and took the network switch out of the 19in rack into my apartment to replace all the components (thanks HP for their modular design).
On the way I encountered a panicking student, who knew me as the guy responsible for the internet, and asked me _WHILE_ I was carrying an insanely large box with a giant bunch of network connectors, "Do you know that Internet stopped working, like, 5 minutes ago?"
The phone lines in our office use RJ45 connectors. I expect they are intended to be used with the type of fancy phone system one would expect in a building with four whole rooms (2 of which actually seem intended to be used as offices)
After the person who hooked up our phone lines informed us that there was no way to use an old phone with these connectors, no converter, no nothing, we would need to buy new phones (which Bell South had a fine selection of, of course), and then each of the offices would need to be wired professionally (which Bell South could do!)...
I found that we still had: 1) A crimping tool 2) Leftover CAT5 cable 3) more splitters than could possibly be justified by a single DSL line.
I know nothing about phones, or electrical anything beyond my kindergarden science fair project on Conductivity (pennies DO conduct, oatmeal lids DON'T!). This is a piss-poor job.
Every phone in our office plugs into a splitter. An RJ11 crimped onto a CAT5 cable (two dangling wires) plugs into the same [read:wrong] end of the same splitter, which then plugs into the wall's RJ45 connector.
At the other end is a nest of wires in which many small wires pulled out of a CAT5 cable connect every line of the terminal to the same connection on the terminal as the only "real" phone line in the building (which is located in the 4'^2 "lobby")
I'm pretty certain the terminal is not intended to have four wires stacked onto the same connection. But as I said, I'm not an electrical anything. For all I know, this is how it was meant to be done. All I know is: It works.
-- 'The' Lord and Master Bitman On High, Master Of All
I used to work part time as a volunteer engineer at a community radio station. Aside from the digital audio playout system playing music off a Novell server, we had no money for IT. We occasionally experienced problems with stuttering music, caused by people transferring large files across the network which were causing packet collisions and interrupting the critical stream of audio data to the studio playout machine. This was prior to MP3 and the like, so the best audio file compression we got was 4:1 and the required bitrate was pretty hefty.
I was given a new 100Mb hub, the theory being that 100Mb is faster than the 10Mb we already had, so it would solve the problem. Not so! Those large files would still collide with the audio streaming because we had no intellient routing, traffic prioritising or cash to pay for a decent solution.
I discovered that the new hub would auto-sense the 10/100Mb speed from the NIC at the other end, but had no internal 10Mb100Mb switch capability. In other words it was effectively two hubs in one package with separate 10Mb and 100Mb buses inside. That turned out to be advantageous in the end. I set up all the audio workstations to run at 100Mb, and all the administrative workstations to run at 10Mb, so effectively we had two separate networks, one for audio and the other for admin. So the secretaries could continue sending their large files around, printing and so on, and it didn't affect the audio operations. There was only one PC that needed access to both audio and admin, and I solved that by simply giving it two NICs (making sure they weren't bridging).
It ran that way for several years. I believe more recently they employed the services of an IT contractor, who promptly saw fit to replace my old 10/100 hub with a fancy new switch. Almost immediately the stuttering problems returned. I don't know what they did to remedy that, but it seems to be better now. Presumably they have a bit more of an IT budget these days.
Je fume. Tu fumes. Nous fûmes!
Comment removed based on user account deletion
Back in the day, I used to have to go in late on Saturday nights and copy & compress the system disk on our PDP-11/70 system. It was located in the typical loud, cold computer room. THe copy process could take anywhere from 1 to 5 hours, depending on the amount of fragmentation on the system disk. I didn't want to wait in the loud room, and I didn't want to get up from my comfy chair in my office at the other end of the building to continually check on the progress. One night, tuning around on my FM radio in my office, I heard a funny sort of noise at 98.5 MHz. Its rythmic structure reminded me of the sound the disks made while they were seeking during this copy process. Sure enough, thise old school disk drives, with their Emitter Coupled Logic (which uses about a pound of electrons to do anything) were generating lots of EM noise, which was, I'm guessing, getting coupled to the power line and thence to my radio. After that, I could kick back and have a few beers, and listen to the radio to know when the copy was over, without going back and forth to check.
There is no God, and Dirac is his prophet.
So 10 years ago I worked at a small ISP where the owner lived in a small apartment. In his apartment, he had a couple bonded ISDN lines that he then resold bandwidth off to a smaller business. Ever time the power went out (and the UPS ran out of juice), the ISDN equipment had to be manually reset for his and the customers lines to come back up. He went on vacation and of course the power went out. When I tried to get access to the apartment, the leasing company told me no, not without a signed letter allowing me access. I thought a bit, then rummaged around in his desk at work, found a contract he had signed, cut his signature out, taped it to a hastily typed letter saying I was allowed access to the apartment, and faxed it over to the complex. I then waited 30 seconds and called them up, saying he had just faxed it over to them and did they see it. They said "Yep, come over and get the keys" Made a couple copies before I gave them back. It was too easy.
Even worse, I've seen morons trying to use email as a file server.
Between the Plain, &, and % codes you really could get away with things like:
ATZ0%F0&UC&K2O1F&F which by the way is (IIRC) a perfectly acceptable init string to reset the modem to chipset default and then turn off some reporting that nobody uses anyway. It might also disable one of the 56K compression algorythms.
I once worked for a dot-bomb e-commerce company. We had a product that tied into several major credit card issuers (i.e. >40% worldwide market share for issued credit cards). As part of the installation and maintenance of the product, I got to spend many weeks in MAE East (perhaps the biggest data center in the world). From what I've read about the Baby Bells' special networking rooms when the NSA scandal broke last year, I wouldn't be surprised if these servers shared one of those special rooms with the NSA routers.
The data center was about 5 floors below ground level. No form of wireless communications worked whatsoever--cell phones, pagers, etc. Once I parked my car, I had to go to an unlabeled metal door with a tiny camera on the top. Security guards would buzz me in and require me to sign in at their station. Then I would get buzzed in to the main data center room that contained another room inside of it. From there, I had to enter a password into another security system and place my palm on a palm scanner. Inside this room was another security guard--I would have to sign in with them, too. Then I would enter a different password into another security system, and place my head in front of this retinal scanner. This would buzz me into another room with the cages for each of the clients. There was a padlock on the cage, behind which were our servers. The servers required two separate smart IDs to be placed into an external card reader so that there had to be at least 2 people there to perform any maintenance. The servers themselves were locked down pretty tightly, too. It all seemed pretty insane as far as security goes, but I understood--these computers contained every credit card for the credit card issuer.
Well, after about 3 days of going to this data center, everyone got to know me. They would sign in for me to speed up the process. The security guard behind the door with the palm scanner used to get very hot, so she would often block the door open, thus defeating the palm scanner. The retinal scanner also had problems, often requiring about 3 tries before it would read correctly, so that door was often blocked open, too. Then, one day one of us had forgotten our smart card. We started cursing, as the round trip to pick up the card was about 45 minutes, so we tried it with only one smart card. Bingo. It worked. So then we tried it with no card. Seems the card readers weren't functioning properly. So, overall, we were able to defeat all of the security measures except for the padlock, and all because the security staff (getting paid 2 bucks above minimum wage, no doubt) all "knew" us. In my humble opinion, it would have been far smarter to *not* have the security guard in the foyer behind the palm scanner. After all, social engineering is probably the most common form of circumventing security.
Another funny thing about this was that we had a rather difficult security audit for all code releases. We had a bunch of ex-NSA employees working for us that were rather good about it, too. We would also hire outside auditors to do reviews of major code releases. It was all fantastastic, except for one thing: code patches didn't get the same scrutiny as code releases. In fact, they got none. Well, in order to expedite the release of one particular feature (that required emailing confirmation to customers), we packaged it as a "patch". No security audits. And for something that required the installation of a mail server! Furthermore, the code base had access to the record-level encryption used to store the credit cards. So, basically, if I had wanted, I could have installed a bit of code that would have decrypted all of the credit cards of users of our software and emailed them to a third party. I could not believe it. It's a good thing I have what I consider to be high moral and ethical standards.
I realized through this ordeal that security measures are not put in place to ensure security. They are put in place to give people the perception of security. And, furthermore, automation and removing the human element are good things for security. People should be used to monitor and oversee automated security, not to be actively involved in that automated security.
--Be human.
And if such a time period had passed, all the sewer gases in your home would have long since been evacuated from the piping via your stack anyway.
Not if other toilets in the building are in use.
rj
I've got several bad IT scenarios that I could share, but this one is the best I've personally seen.
The building my company is in was never designed to house IT infrastructure - even new newer additions are not build to house IT infrastructure. But we've made do and put infrastructure where we can, even if it means sharing a closet with the cleaning crew. One such closet had and old catalyst 2900 24 port switch in it, with the ports facing upwards. The connection are nice and hardwired into the switch, and generally looks like a rats nest (it pre-dates my employment). Recently in the renovation of the floor above it, the construction crew needed to poor a new floor. It didn't occur to them to plug the holes that had been drilled through the old floor over time, or cap the stubbed in conduit that had been put in. As a result of this the switch with it's ports facing upwards was filled with concrete. Completely and utterly filled. The cleaning crew reported that the fans made an awful noise for days. The switch never failed and is still in operation, although we can't unplug any cables or plug any new wiring into it.
At a different company, the server administrator was deeply convinced that rack mount server cases were evil and caused problem. He didn't want to use tower or desktop cases either, his reasoning? They were ugly (his words). The solution? He ordered servers, removed the motherboards from them, zip tied them into the rack via their screw-down holes, and placed the hard drives directly on top with cardboard "protecting" them from the motherboard. 5 racks of servers mounted just like this. I quite after a week of working for them, as I didn't want to be anywhere near that place when a zip-tie failed.
http://www.accelerateglobalwarming.com
10 MB email limit I wouldn't allow that on any mail server I run.. 1 MB tops. Email is not for large file transfers.
Back in 1999-2000 time frame I was working at an office that had been acquired by a larger company.
Said larger company used Exchange for all their email needs. Our office used an old desktop running linux. It was shoved in a corner of the server room but mainly forgotten.
I had set a limit of 1MB on email attachments on the linux server after some secretaries decided to start swapping 40MB word docs back and forth via email instead of the shared drives. They pretty much killed the mail server simply because it didn't have the disk space or IO speed to handle 40 MB attachments.
One day the HR department in the larger company decided to send out a 20MB PR fluff make you feel good about working here newletter.. via email.. to all 500+ employees.. Completely crashed the exchange server. Our little email server happily rejected the garbage and kept on trucking.
About 9 months after I left that company I got a call to rebuild the linux server. The hard drive had failed and no one knew anything about it.
At another company there was an old Novell Netware 3.x server in a closet.. covered in about an inch of dust with an uptime in the 4 year range.
-Jerry
10+ years ago, I used to work for a Networking Software company called Artisoft in their Tech Support office. There were two notable hacks that come to mind. The first one I only heard about (file this one under Urban Legend if you please), but someone needed to network their farm house to their barn. Their solution: Barbed wire that was twisted just enough to simulate CAT 3 wiring. Worked ok, except for the dropout when the cows would lean against the fence.... The other one was a case I worked on. Gent in Florida or Louisiana had lost conectivity between two machines on a coaxial network. Now, as 80% of existing network failure can be attributed to cabling, I had him check the span.... He had to look under the house (the structure turned out to be on stilts in the swamps) and lo and behold the cable was broken. By an alligator/crocodile living under the house.... I wished him luck and closed the case....
Welcome to my world. I work for a CLEC/ISP with coverage throughout Oregon. In the last year no less then three times have construction crews drilled through massive cables in three difference cities. The best was when they pull the spegetti fork type pull, under a major highway. It took them three days to get it all fixed. The problem with that mess is they just can't splice the wire together but need to replace meters of cable under asphalt.
If you ever want to find some buried fiber in Oregon just put a backhoe in a field. When you check it in the morning it will be pointed to the neared fiber.
...There is esmtpverbs control parameter in AD which by default does not play very nice with some other smtp servers(its arguable whose fault is that - every party claims its other party to blame) .Easy enough to deduce: On the one side we have all the other MTAs which, unsurprisingly, are able to communicate well with each other. That leaves, on the other side, one system the odd man out which does not play very nice with the others.
What's wrong with admitting that it's still broken or don't normal rules of any kind apply to that one vendor ?
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.
-One computer room in a pork plant had its air intake on the roof... right next to the exhaust from the rendering exhaust. Now, while the smell itself was awful, the worst part was that the sulphur and other noxious chemicals would eat the computers alive. The IT group had to install a special device that monitored the decay levels of samples of metals such as copper, silver, and gold to see how long the machines would last.
-One computer room was made out of a new office on the third floor of one plant. But there were no elevators to get the hardware up into the room. So they cut a hole in the floor, used a gigantic crane, and lifted two guys with each fully-populated rack up to the room. And the crane was still three inches short, so the two guys would have to do a wheelie with 1000+ lbs. of equipment to get it in the room.
-The same plant with the hole in the floor was also keen on bringing in electricians who were severely brain damaged. How bad? Imagine a 110V cord strung across the air with no support. And the other end of the cord with the three prongs was "hot". And someone once plugged it into the wall socket. I hear it doubles as a cattle prod now.
-Two computer rooms needed air conditioning, so they simply carved holes in the walls and hung out Wal-Mart brand home air conditioners. With no other insulation. In the winter sometimes frost appeared on the machines.
-Many of the computer functions were relegated to plant employees with their own unique vocabulary. I had the privilege of speaking with one woman in Texas who sounded exactly like Boomhauer on "King of The Hill." Honest. Another one always tried one-upping the "smart" IT guys back at corporate. On one occasion, she said that her monitor was "tricating". This was the word she used to mean that the emulation was off. I informed her that there is no such word as "tricating," at which point she told me that since I was young I likely hadn't heard of that word before. Funny, neither had Merriam-Webster.
--Chag
I know a similar limit showed up at my university (to remain nameless) shortly after a friend of mine backed up his hard drive as a uuencoded email message to himself. It was only on the server for a couple of hours but still...
In the quest for Internet service and TV in our barracks area in Iraq (a tent, mind you), we did silly, silly things.
- Cat 5e running out in all directions from my tent - some partially buried in gravel/sand, some over roads, etc. At 1000ft per box, I must have strung nearly a mile of cat5e over there. (FYI, cat5e will survive a run over a moderately trafficked gravel road for about a month and a half.)
- Three different Wi-Fi access points of different models / configurations. Having no external antennas for the "primary" access point, improved signal and range was accomplished by shimmying up to the top of a tent pole and mounting the access point to a platform hung from the pole. The whole mess was then covered up with a cardboard box with vent holes cut in the side to try and keep dust from clogging the unit.
- Tons of splitters and hundreds of feet of network cable to share the few TV antennas available. Starting at the cable in my friend's trailer, I once attempted to determine which antenna he was actually hooked up to. A complete loop and a half around the entire housing complex later, I still hadn't found the actual antenna.
The best I have seen was with the Arizona Department of Transportation (ADOT).
ADOT inherited a building in the easment next to the I-10/I-17 intersection near the AZ Fairgrounds that formerly contained some heavy cranes. These cranes were meant to lift heavy equipment onto trucks and were suspended on rails som 30 feet from the warehouse floor.
When the state inherited the building they decided to lease the downstairs to the Arizona Magazine for printing and assembling that fine pictoral magazine. The area in the rafters where the aforementioned crane resided was useless to them... so along comes the genius.
They used the heavy beams meant to support the crane as the basis for hanging a plywood floor. On this hanging / suspended plywood floor they would put in "Office Space" and lease that to ADOT for their IT development group.
It gets better.
Yes, all the electrisity and data wiring came into the building and was drapped across the gap between the wall and the suspended floor.
Yes the floor moved noticably, although it was too big for a single person to shift it on their own. After all there were 10 cubicles, 2 offices and a conference room on the floor.
Yes they put up pseudo walls separating areas of the room.
Yes, it was a warehouse so the ceiling and walls were just corregated steel... Yes it was in Phoenix Arizona... a dessert.
They did provide air conditioning so it was warm, but not unreasonably.
Another nice feature of the buiding was that it was partially beneath the I-17 N to I-10 E ramp that was about 60 feet off the ground. Every once in a while you would hear the clunk of someone Super Big Gulp hitting the metal roof, or the lite tap of a cigarette butt. At one point an ADOT truck in the parking lot was crushed by a truck tire that came off and went over the railing. Another truck was damaged by a water tank that came off another vehicle.
The best part about the IT solution was that they also put the servers up there. We had a separate room where all the servers were. It had extra air conditioning blowing on it, but it was not a contained room, it was firly open with walls that went part way to the cieling and a gap between the floor and the print shop below.
So the server room was in a metal building with no insulation in the dessert on a plywood floor suspended above a print shop under a highway were large things occassionally rain down. I shudder to think what would happen during a power outage.
Slowly waving my hand - "This is not the sig you are looking for."
At an old workplace, there was a server (ok, a Sun Ultra 1, but it was running Oracle) which no-one seemed to know where it was, but it was on the network, running OK. I resolved to track it down...
First plan was to have it write something on the screen asking whoever saw it to call me. No joy; guess no-one went there.
Then I figured that it had a sound card & speaker - I also knew it would play .au files natively so went a searching and found a line from Monty Python's Holy Grail: specifically, "Help, Help! I'm being repressed". I then set up a cron job to cat this file to /dev/audio every 15 minutes. Unfortunately, all someone could hear was "help, help" from outside the comms room it was in and assumed someone was trapped inside. Security guard looks around and eventually finds the server with my name on the monitor.
At least we found out where the damn thing was, which was useful when some numpty builder cut the ethernet cable while working in the room.
Sounds like 10Base5 or "Thicknet", which was the original Ethernet cabling spec.
I heard some stories of the early days of Voice over IP being deployed at Sprint. Here in KC Sprint is a major employer. I was at lunch at a restaurant when several Sprint guys (apparently from IT) came in and started telling war stories. One told of the time an entire building phone system failed at the flip of a single switch. The techs had just deployed a trial of VoIP for the entire building. They had, of course, placed the new VoIP server int eh data center witht he rest of the servers. Being new, they hadn't completed all the "official" labeling, and it was a trial run anyway. After a while one of the PHBs of the center was walking by and noticed an unlabeled machine in the rack. Policy said that nothing unofficial ran there, and if it wasn't labeled it was official. He powered it down and went to the phone to call some techs to get rid of the offending machine. Of course, the phone was dead. Next room, next phone. Dead. Continue ad nauseaum across the whole building with people trying to find a functional phone. Cell phones work, but nobody can call in to the building. Eventually the techs realize the VoIP server is messing up, go the the center and discover the dead phone server. While they mill around waiting for the server to come back up the PHB comes back by to tell them about the strange machine he found. He notices it back on and promptly shuts it back off. Needless the say a heated discussion of PHBs, power switches, and corporate policy and common sense ensued.
Had a friend who had a box colo'd for free at a fairly run down ISP. It was due to some contractual obligation they had to live up to, so they were not happy about it. Was supposed to have a 10 mbit connection to the switch, and pretty much unmetered out over the backbone, but it never saw more than 2 mbit. Turns out, the ISP put an extra long ethernet line place, 500 feet of signal killing goodness. Just to make his service suck, in a way not easy to see unless you pulled up the raised floor, and noticed that all that wire was in fact, one single wire, going back, and forth, and back and forth.
Another friend has the dilbert boss, decided that the router _needs_ to be at one end of the building, far away from the office. A distance of 350 yards, add even more when you take into account the fact that the wiring goes from the ground floor at one end, up above the second floor, spans the building, and comes back down to the ground. Then, he decides that 10 base T is too slow, and he demands gigE. My friend tries explaining to him why all of this is nuts of course, but the man never listens, and says, "just buy a bunch of repeaters." So, this gigE run of nearly 400 yards, is daisy chained together, every 100 yards or so, by a 4 port gigE switch. Most of them live up in the drop ceiling above the second floor, and have some ludicrous power lines run to them. Store and forward nightmare. Boss is currently pissed because it is slow, and because the gigE switches didn't make his 3 mbit cable modem go any faster.
--Nuintari
slashdot : where an opinion can be wrong.
The one where ambient humidity serves as an insulator and thereby prevents the aggregation of static charge, which is a tremendous risk in a room with that many highly charged moving parts.
I realize this is nitpicking (and the rest of your post was right on) but I think you meant to say "where ambient humidity serves as a conductor and thereby prevents the aggregation of static charge..."
Higher humidity makes the air a poorer dielectric, meaning that static charges dissipate before they can build up to significant voltages. With dry air, the air is a better insulator, hence higher-voltage static charges. (This is why the kids' trick where you scuff your feet on the carpet, particularly while wearing rubber-footed one-piece jammies, and then shock the beejesus out of someone, only works in the winter.) Naturally, anything that produces sparks -- particularly my favorite, Van de Graaf generators -- work far better in dry air than wet.
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
Hehe.. I remember pointing this out to a colleague. I said it was insane, he said that I was a MS hater.
10 minutes later as we were on our way to lunch, my colleague had to withdraw money from one of the brand spanking new wintendo ATM's that had just been installed. The ATM bluescreened while he was making a withdrawal (after it said - wait for your funds). So basically, he got his card back but the money never came. He checked the account and noticed that the money had left his bank account (without any money actually being paid out to him). He spent his entire lunch break arguing with bank staff about getting his money back while I went to eat. I never heard him say anything pro MS after that.
I find it incredible that - in addition to running Windows on the ATM, it had no concept of transactions.
A friend of mine does networks for a major oil company. He brought back pictures of an installation in (I think) Angola.
The first notable point was that the outside of the building seemed to be polka-dotted. Those were the bullet holes. Nobody was firing at the building per se, it was just downrange from one of the local hot spots.
The telecom room for the building was on the second floor. My friend's company rented the fifth floor, and some other company rented the floor above that. There were no data risers in the building. Instead, there was a hole about a foot across that had been knocked through the wall on the second floor, and a couple of six-inch holes on the fifth and sixth floors. The data cables ran out through the wall on the second floor, up the outside of the building (among all the previously mentioned bullet holes), and back in through the walls on the fifth and sixth floors. My friend's company had gone to the extra time and expense of running conduit, but the company above just had a 40' swag of cable hanging there in the breeze.
The telecom room itself looked like the aftermath of a "will it blend" episode. Take 20 drums of assorted wire product and unspool it all, wad it all up into a 10'x12'x8' snarl, then start grabbing random segments and pulling until you can nail that particular chunk of wire to some point on the wall. That's what it looked like. When they needed to fix something, they put one guy upstairs with a handset on the line and a radio, then sent another guy with a radio down to the telecom room to wave a toner around until the guy upstairs started to hear noise.
My friend also told a story about renting a phone line from the Chadian national government. That's all it was.. the same kind of POTS line you have to the phone on your wall, for a cost of something like $10k per month. One of the things he had to do was install a modem on the line (which saw about 95% use since it served a whole office), but the line itself was so unbalanced that he couldn't get a decent signal. He mentioned this to the local telecoms expat, who said, "oh yeah.. come on. You'll enjoy this."
They got their security people and drove over to the phone substation (which was run by the Chadian military and had armed guards outside), and before the truck even stopped, the expat was out and stomping his way past the guards, through the door, and into the wiring frame. As my friend came in behind him (pretty much thinking, "okay, we're dead"), he realized that he could hear voices speaking English througout the building. The military had tapped their line, but since it was used so much, they were running the sound through a loudspeaker rather than just listening to it on a handset.
Halfway down the row of wires, the expat stopped, pointed at the wire that dropped down from the celing and tapped into the frame, and shouted, "IS THIS A WIRETAP? IS IT?" By that time, *everyone* in the facility was there watching, and the colonel who ran the place was saying, "no no! Is not a tap," despite the fact that everyone could hear the voices of the people using the line over the loudspeaker.
The expat yanked the wire loose and said, "damn right it isn't a wiretap.. it's rubbish. We spotted it two seconds after you put it in. Here.. let me show you what to do," then proceeded to do a quick lecture-demo on professional wiretapping techniques.
Well, this one might not be entirely in the spirit of the original question since it's not a cool "hack" so much as it is just an amusing error in planning, but here we go anyway:
Back in '95, my father was a VP of research for a large manufacturer of transmissive and reflective coatings for various glass applications (think insulated windows for the simplest example of said product) in Palo Alto, California. I was 15 and in highschool at the time, and having spent many a year trying like hell to keep a series of shitty no-name x86 computers up and running well enough to play the latest games, I had a sufficient skillset (and my dad had sufficient clout) to get me a job in their IT department. I did pretty well, and quickly found that users generally only got mean-spirited when made to look stupid, so a small dose of humility coupled with an interest in details on their primary task - "While I fix these printer drives you accidentally deleted, I was curious, what does a spectral photometer do?" - kept me out of trouble. Long story short, next year when I switched to full-time for the summer break, my boss actually brought me for a one-day business trip to our plant in Tempe, Arizona.
Now, you've got to realize, a business trip for a 16 year old (this was '96 now) is freakin' AWESOME. I was nervous as hell, had been up since the crack of dawn to take a red-eye with my boss out to the plant, and was deathly afraid I'd do something to embarrass not just me, but my father for having recommended me. So it was pretty unnerving to learn that my first job involved going into a large clean room production area, kept free from particles that could settle on the film during that specific type of sputtering process. We're talking the full disposable "bunny suit" that covered everything but the eyes, even with little slippers, and an airlock-type blower to clean you of all particles before entering.
The problem was a simple fix, really. The brand of 486 motherboard we were using at the time had a tendency, in about 1 out of every 3 units, to burn out the CMOS battery much earlier than you'd expect. And for a manufacturing-floor computer, not having a correct internal clock was a bad thing, not to mention that the lab techs had to go through some errors at startup with BIOS setings no longer being saved. So I suited up, cleaned off the replacement part and my tools as ordered, and went to find the bad machine.
That took some doing, oddly enough, since these computers were rarely shut down due to a 24/7 production schedule, so I had to go through back records on hand to find the lab techs' notes during the last power cycle on which computer had the boot errors. But, once located, the terminal was taken offline and I was able - after being told I had 20 minutes for the repair, tops, before the company would start to lose money as they needed that terminal again - to drag it off to a quiet, out of the way corner for the swap.
But see, there was a problem in the planning stages when this plant was set up. The PCs they used to control the machines were pretty complicated to configure, and the machines run in the clean room were just slightly modified versions of those used in the full-on manufacturing area in the main plant in Palo Alto. It was actually only a pretty small fraction of these production machines that had to operate in a clean environment. So when it came time to set these terminals up, they carefully washed off the outside of the older computers - computers, mind you, that have been sitting on a 24/7 PRODUCTION FLOOR with 10+ lab techs nearby at all times and various debris kicked up from the manufacturing process - and shuffled them off into the clean room.
So picture the scene: our hero, an extremely nervous 16-year-old on his first business trip in full head-to-toe bunny suit gear in the corner of a white, immaculately clean production floor opens his target computer to find a system so full of dust that he can't even SEE the goddamn cards inside. We're talking full-