Slashdot Mirror
Stunts, Idiocy, and Hero Hacks
snydeq writes "InfoWorld's Paul Venezia serves up six real-world tales of IT stunts and solutions that required a touch of inspired insanity to pull off, proving once again that knowing when to throw out the manual and do something borderline irresponsible is essential to day-to-day IT work. 'It could be server on the brink of shutting down all operations, a hard drive that won't power up vital data, or a disgruntled ex-employee who's hidden vital system passwords on the network. Just when all seems lost, it's time to get creative and don your IT daredevil cap, then fire up the oven, shove the end of a pencil into the motherboard, or route the whole city network through your laptop to get the job done,' Venezia writes."
I once fixed an issue that was holding up the operations of a $50 million dollar a year company with one well placed rubber band.
I got "lucky" to solve a problem for someone back in college: she had written her thesis on a 3.5 floppy, had no backup (this is when you had to go to the "computing center" to work, as practically no one had a machine of their own, so you had to take all your stuff with you), and had run the disk through the washing machine.
She came in, crying hysterically (it actually took a few tries just to figure out what was wrong), and realized what had happened. I had one of the few "eureka!" moments of my life, and grabbed another floppy, carefully cut it open, did the same with her disk, then air-dried it. I put the platter in the "new" disk, with its dry fabric covering (whatever that stuff was...), taped it shut, and put it in the Mac (SE...no hd) and yep, the disk was readable and I was able to get her thesis off and onto a network drive, then we copied it back onto a new disk and assured her I'd hold onto the thesis on the network drive until the end of the semester.
Funny thing, she kept the disk I had used, taped around the edges, and the next year I saw her again and asked how things were, and she was still using it. Go figure.
I don't have a story on the the scale of any of these but I remember fixing my Commodore 1541 drive by adding an extra screw. The drive belt had gotten stretched somehow and I was getting all kinds of read errors, and being poor, decided to attempt a repair. Turns out there was an empty screw hole near the drive belt, I put one in, stretched the belt around the screw to take up the extra slack and that drive was still working when I finally got rid of my C64 a couple years later.
I know, cool story bro.
I worked with Paul a long time ago at a mom&pop in NH. And I know that he personally did the drive trick and it worked. It was a 9 gig scsi drive with an smtp mqueue on it. He was extremely elated that it had worked, and his portrayal of the story to a wide-eyed netadmin noob (me) was one of those late-night, sipping coffee at the Red Arrow while the raid rebuilds sorta memories that you'll take to your grave.
slashdot: where everyone yells sarcastic metaphors to themselves to understand the issue
Back when I was a computer tech for one of the big retailers, I had a customer bring in a machine that wouldn't boot. After interrogating the customer a little more, it turned out he had tried 'upgrading' his CPU, and in the process had broken off one of the Athlon XP's (shows age) pins by inserting the CPU in the wrong orientation.
The dude couldn't afford anything new, so I offered my most MacGyver-ish attempt. I went over to the car-audio shop, grabbed some speaker wire, spliced out some copper about the same size as a pin, and voila!
After bending some of the pins back with a mechanical-pencil tip, and inserting the new 'pin' into the socket below the missing pin on the CPU (cut to semi-correct length), it booted right up! He took it home and all was well. I don't work for said company any more, but how long that 'fix' lasted is questionable.
Never told the boss about that one.
'We are trying to prove ourselves wrong as quickly as possible, because only in that way can we find progress.' RPF
The first "stunt" depends on your point of view. If you have nicely brainwashed and duped by marketing material that "Vendor gear good, PC bad" that may sound as a stunt. If you actually know what you are doing you can run networks for years on this.
Nearly any laptop today has the forwarding grunt of an upper end of a 3800, there are plenty of servers that are on par with a 7200 or low end 7600 and most supervisor modules. You can run a network on this on a daily basis and do a _LOT_ of things a Cisco cannot do or cannot do at sufficient performance.
To put the so called "stunt" into a perspective, I used to run a production installation with 20+ 802.1q trunks via 800MHz Via EPIAs with 600+ entry ACL lists including content filtering with VRRP failover, load balancing to multiple upstream uplinks, OSPF, hardware accel-ed openvpn and ipsec, 16+ class hierarchy CBQ QoS and a few more bells and whistles. For years. Not for 48 hours.
Nothing wrong with it if you can do it. If you cannot - well, not everything in life is learned on CCXX and RedRat certification courses. C'est la vie.
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
I had non-booting server board back in the late 90s and managed to track the problem to a scratch through one of the traces on the bottom of the board. Something had fallen between the board and the offsets and had worn through the circuit.
Having nothing to lose, I fired up the soldering gun and pulled out the only wire I had from a pair of speakers and sure enough, once the circuit was made the board booted and remained stable long enough for us to order and install a replacement.
Trust me, they will come in handy in the lab at some point. Even for sudden headcrab infestations.
He who knows best knows how little he knows. - Thomas Jefferson
I once took my laptop and used it to set up an Apache + DNS server while replacing a webserver that died. All I did was to post a "Emergency Maintenance" page while we swopped out the server.
Every IT guy who has been in the trenches for 10+ years has "I once" stories. Oftentimes they salvaged hundreds of thousands of rands of damages for the company, or helped mitigate a bad management decision.
The thing is, one of several scenarios invariably happen:
1 - You get no recognition because no one understands what you did. ("Oh, you had another web server running on your laptop, that's dandy!")
2 - You get an accusing look. ("How was it possible that this happened? Sure you fixed it but this should not have happened, make sure it doesn't happen again.") - I saw something like this happen to a senior network admin once, something totally out of IT's control that occurred due to a bad management decision not to buy a spare router. We used an old PC with IPtables to route traffic on a network over a weekend while our suppliers tried to source one.
3 - The dark suit analogy: Doing a good job is like spilling coffee on a dark suit, you feel warm all over, but nobody notices.
Being in IT is a bitch, and management doesn't help - IT is honouring the impossible promises of management to unthankful clients.
Seven Days with Ubuntu Unity
I once repaired a critical UPS that was attached to a critical database server actively recording data in the middle of a test shot with jumper cables and the battery from my truck. All that just to replace a fan that kept sending the UPS in to panic mode for an overheating battery and trying to start a shutdown sequence on the database server.It was a 12v power source for the UPS (old, old equipment) coming out of the AC to DC power supply. The UPS was part of a suite of equipment that included the database server, the array, a backup device, a network switch and the UPS hardwired to each of them in it's own rack. Don't ask me who made it. All I know is it was an Informix based DB and the maker was some esoteric, specific solution company I never heard of and before my time anyway. All I knew was the replacement parts had a 2 week lead time and I have no idea why this company chose to hold up such critical data with such arcane and unsupportable equipment. But, I had to shutdown the UPS to do the work but the battery didn't have enough juice to support the 30 minutes it was going to take to do the work. The battery power would have been killed once the unit was off anyway.
So I attached my jumper cables and the 600 amp battery from my truck to the output rails on the UPS, after the control switches. From there it was just juice to the rails and then to the server and it's data array. The car battery had about 45-55 minutes of juice for the suite to run on full-tilt. So I shut the UPS down and the servers, thankfully, stayed up! Had a box fan blowing on the battery and jumper cables. I disassembled the UPS case, cut the bad fan out and spliced the old connector on to the new fan I got at a local surplus store for $3. Plugged it all in, reassembled and turned the UPS on. It went through diagnostics and everything went green. Then the overload light started blinking and the warning chime came on. I pulled the jumper cables off and the overload warning went away and things stayed stable. The fan stayed on and nothing went down.
I probably should have gotten an award for it because it was a test shot for a multi-billion dollar contract but I was more afraid of disciplinary action over the risk than getting any praise for it. As far as I know, to this day, only two other people at that company know what happened
Look for scratches on the bottom side, brush with toothpaste (the plain one, no additional abrasive ingredients), rinse, read.
In the Dim Times, my company had a couple of hard drives (those newfangled 3.5" Scuzzy drives) that wouldn't spin up and had critical data on them. My solution:
2. Hold the drive in my fingertips (so the platters were parallel with my palm)
3. Power on the computer, then "snap" the drive with a twist parallel to the platters, relying upon inertia to break the stiction.
4. Recover data from now-spun-up drives.
5. Power down, then physically destroy the interface pins on the drive to ensure nobody tried to use it again.
Since then, I've used that trick several times on dead/dying hard drives. As long as the heads are trying to move (indicating electrical life), it's worked every time.
Everybody gets what the majority deserves.
Looks like somebody doesn't want email access today, huh? (End BOFH mode)
I once was a sysop for a small company's Data General system, where large datasets were stored as TAR archives on nine-track tapes; some poor soul had copied TO the tape instead of FROM the tape, and desperately needed to recover a file that was still there on the part of the tape beyond the end of the inadvertent write. You could read up to the added end-of-tape marker, but the tape just wouldn't read any further. Screwed, yes? Well, not quite. I set the system to rereading the damaged tape, waited 'till just before it reached the offending end-of-tape marker, and briefly put my thumb on the roller that measured tape travel, causing the drive to jump the tape ahead ('cause the sensor said "the tape is not moving!") and right past the EOT marker. Voila! The system read out the rest of the files on the tape, fortunately including the one they really needed, and I was briefly a hero. Hero never lasts, of course, but it was fun.
"My strength is as the strength of ten men, for I am wired to the eyeballs on espresso."
I'm surprised they didn't mention the technique for unsticking recalcitrant half-height RLL and MFM hard disk drives by slamming them gently, but firmly, down onto a smooth horizontal surface (like your desktop). They would occasionally stick when the heads became goo-ed to the platters due to breakdown (or solidification, I was never sure which) of the lubricating material. When all other hope was abandoned, and you knew the drive was headed for the graveyard, a good, solid (but gentle) whack would often get it spinning again. The idea was to keep the drive as parallel as humanly possible to the horizontal surface. It was one of the few hardware tricks I had to summon male assistance to handle--my hand was not large enough to get the necessary firm one-handed grasp on the drive. Boy, do I feel old. Probably because I am old.
"Here's what's happening. You're starting to drive like your Dad..." - Red Green
Back in my days as an engineer at Boeing, I supported some automated test equipment on the factory floor. One day, one of the ATE failed to download the required s//w update, so I was called out to investigate. It turned out that the network drop adjacent to the equipment had been disconnected in the nearby network closet. (locked, of course). So I, with the factory manager in two, called the IT department to get it plugged back in.
Me: "I'm in the Renton plant, at column XYZ and we need this network drop reconnected. Production has been halted."
IT Operator: "OK. We'll start a ticket on that. But standard turn-around is 24 hours".
Me: "We can't wait 24 hours. We need to get this equipment updated to get the line up and running. Is there any way to escalate this?"
IT Operator: "Sorry. That drop is was identified as being inactive and was unplugged."
Of course it was inactive. The ATE is only powered up when needed. At other times, the little light on the switch in the closet would be off.
At this point, the factory manager asked for the phone. Very calmly, he spoke to the IT operator.
Manager: "You can cancel that ticket. My engineer assures me that he can reconnect the drop once he gains access to the network closet. The plant fire department is just downstairs and we'll have them bring up a fire axe to open the door."
The IT department dispatched a tech who arrived within 15 minutes.
Have gnu, will travel.
Back in the early 60's I was on a three man combat targeting team and we had two minuteman missiles to startup and target one day. So we went to the first site and the maintenance team had just finished installing a new guidance and computer package and the nuclear warhead. They closed the 80 ton door that protects the missile and so it was out turn to perform.
We started up the on-board computer and ran some checks and then began loading in the targeting data such as whether it was a air burst or ground burst and all of the war-plans associated with it as well as the launch codes and targets.
After this is accomplished then the guidance package goes through some testing and self calibration and finally becomes "ready"
Ready is actually called "Strategic Alert" and lights a green light on our console.
The missile system sat in strategic alert for a few minutes and so we figured we had completed our job and would button up the site and head to our second site.
Suddenly the "Launch Commanded" light lit on the console and a fraction of a second later the "Launch in Progress" light also lit.
I quickly popped out a bunch of the circuit breakers on adjoining panels causing the support equipment to stop functioning.
At this stage we did not know if we had a bad console (portable between sites) or a computer failure on-board. Anyway the missile did not blow the umbilical nor launch so we believe we stopped it just in time. If we tried to check our technical data then we would have been dead most likely.
We contacted job control and they agreed not to attempt a restart and rather have maintenance replace the guidance/computer package yet again and return it to Autonetics for repair.
The next site we went to for startup went perfect and the console worked flawlessly...
That has been nearly 50 years ago now and i still occasionally wonder if the missile had actually entered "launch" or if the on-board computer was giving erroneous launch status.
And in the end, the love you take is equal to the love you make
"bubblegum and duct-tape fix"
Yeah.. as if a Watch-guard, NOKIA, or CISCO firewall are any better than a Laptop running Debian.
Sounds like the writer knows nothing about firewalls at all.
Do not look at laser with remaining good eye.
It was the 3rd year of undergrad and my roomie and come to me with a problem: his brother's computer goes black on booting up to windows. Safe Mode wouldn't work. Finals was the following week and his brother had 2 papers on the computer, unfinished, which had to be turned in on Monday. I told him to bring the computer down and I'd do what I could.
I hooked his computer up to my KVM (best low-space hobby troubleshooter investment I ever made). I booted up and the diagnosis was definitely correct. The second Windows tried booting, the screen would go black and Safe Mode crashed on each attempt. He didn't have a recovery disc for his factory installation so I just had to wing it... without seeing my actions on screen.
I booted up to XP Pro on my computer (which he was using on his) and wrote down all the keystrokes, tabs, enters, etc. in order to get down a method of setting the display settings to minimum settings (Windows key, up x times, right once, etc.). That didn't change anything. I then set out to uninstall the drivers, again writing down the operation as performed on my computer and then repeating the process blind on his. That still didn't fix it. "Oh!," I thought, "Maybe I'll uninstall the device and reboot... duh!" I did that blind, rebooted, and the desktop was viewable. I spent the next 3 hours removing viruses and malware.
Lesson to the brother: Don't install ATI drivers for their built-in software overclocking when you have an NVidia card.
Lesson to me: Fix computer, get beer.
Dude, if you have to neg your sack surprises, you have some pretty big issues to take care of with your wife.
I think he's married to a prostitute and she's paying in kind for IT services rendered.
To have a right to do a thing is not at all the same as to be right in doing it