Slashdot Mirror
Stunts, Idiocy, and Hero Hacks
snydeq writes "InfoWorld's Paul Venezia serves up six real-world tales of IT stunts and solutions that required a touch of inspired insanity to pull off, proving once again that knowing when to throw out the manual and do something borderline irresponsible is essential to day-to-day IT work. 'It could be server on the brink of shutting down all operations, a hard drive that won't power up vital data, or a disgruntled ex-employee who's hidden vital system passwords on the network. Just when all seems lost, it's time to get creative and don your IT daredevil cap, then fire up the oven, shove the end of a pencil into the motherboard, or route the whole city network through your laptop to get the job done,' Venezia writes."
I once fixed an issue that was holding up the operations of a $50 million dollar a year company with one well placed rubber band.
I got "lucky" to solve a problem for someone back in college: she had written her thesis on a 3.5 floppy, had no backup (this is when you had to go to the "computing center" to work, as practically no one had a machine of their own, so you had to take all your stuff with you), and had run the disk through the washing machine.
She came in, crying hysterically (it actually took a few tries just to figure out what was wrong), and realized what had happened. I had one of the few "eureka!" moments of my life, and grabbed another floppy, carefully cut it open, did the same with her disk, then air-dried it. I put the platter in the "new" disk, with its dry fabric covering (whatever that stuff was...), taped it shut, and put it in the Mac (SE...no hd) and yep, the disk was readable and I was able to get her thesis off and onto a network drive, then we copied it back onto a new disk and assured her I'd hold onto the thesis on the network drive until the end of the semester.
Funny thing, she kept the disk I had used, taped around the edges, and the next year I saw her again and asked how things were, and she was still using it. Go figure.
Back when I was a computer tech for one of the big retailers, I had a customer bring in a machine that wouldn't boot. After interrogating the customer a little more, it turned out he had tried 'upgrading' his CPU, and in the process had broken off one of the Athlon XP's (shows age) pins by inserting the CPU in the wrong orientation.
The dude couldn't afford anything new, so I offered my most MacGyver-ish attempt. I went over to the car-audio shop, grabbed some speaker wire, spliced out some copper about the same size as a pin, and voila!
After bending some of the pins back with a mechanical-pencil tip, and inserting the new 'pin' into the socket below the missing pin on the CPU (cut to semi-correct length), it booted right up! He took it home and all was well. I don't work for said company any more, but how long that 'fix' lasted is questionable.
Never told the boss about that one.
'We are trying to prove ourselves wrong as quickly as possible, because only in that way can we find progress.' RPF
The first "stunt" depends on your point of view. If you have nicely brainwashed and duped by marketing material that "Vendor gear good, PC bad" that may sound as a stunt. If you actually know what you are doing you can run networks for years on this.
Nearly any laptop today has the forwarding grunt of an upper end of a 3800, there are plenty of servers that are on par with a 7200 or low end 7600 and most supervisor modules. You can run a network on this on a daily basis and do a _LOT_ of things a Cisco cannot do or cannot do at sufficient performance.
To put the so called "stunt" into a perspective, I used to run a production installation with 20+ 802.1q trunks via 800MHz Via EPIAs with 600+ entry ACL lists including content filtering with VRRP failover, load balancing to multiple upstream uplinks, OSPF, hardware accel-ed openvpn and ipsec, 16+ class hierarchy CBQ QoS and a few more bells and whistles. For years. Not for 48 hours.
Nothing wrong with it if you can do it. If you cannot - well, not everything in life is learned on CCXX and RedRat certification courses. C'est la vie.
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
I once took my laptop and used it to set up an Apache + DNS server while replacing a webserver that died. All I did was to post a "Emergency Maintenance" page while we swopped out the server.
Every IT guy who has been in the trenches for 10+ years has "I once" stories. Oftentimes they salvaged hundreds of thousands of rands of damages for the company, or helped mitigate a bad management decision.
The thing is, one of several scenarios invariably happen:
1 - You get no recognition because no one understands what you did. ("Oh, you had another web server running on your laptop, that's dandy!")
2 - You get an accusing look. ("How was it possible that this happened? Sure you fixed it but this should not have happened, make sure it doesn't happen again.") - I saw something like this happen to a senior network admin once, something totally out of IT's control that occurred due to a bad management decision not to buy a spare router. We used an old PC with IPtables to route traffic on a network over a weekend while our suppliers tried to source one.
3 - The dark suit analogy: Doing a good job is like spilling coffee on a dark suit, you feel warm all over, but nobody notices.
Being in IT is a bitch, and management doesn't help - IT is honouring the impossible promises of management to unthankful clients.
Seven Days with Ubuntu Unity
I once repaired a critical UPS that was attached to a critical database server actively recording data in the middle of a test shot with jumper cables and the battery from my truck. All that just to replace a fan that kept sending the UPS in to panic mode for an overheating battery and trying to start a shutdown sequence on the database server.It was a 12v power source for the UPS (old, old equipment) coming out of the AC to DC power supply. The UPS was part of a suite of equipment that included the database server, the array, a backup device, a network switch and the UPS hardwired to each of them in it's own rack. Don't ask me who made it. All I know is it was an Informix based DB and the maker was some esoteric, specific solution company I never heard of and before my time anyway. All I knew was the replacement parts had a 2 week lead time and I have no idea why this company chose to hold up such critical data with such arcane and unsupportable equipment. But, I had to shutdown the UPS to do the work but the battery didn't have enough juice to support the 30 minutes it was going to take to do the work. The battery power would have been killed once the unit was off anyway.
So I attached my jumper cables and the 600 amp battery from my truck to the output rails on the UPS, after the control switches. From there it was just juice to the rails and then to the server and it's data array. The car battery had about 45-55 minutes of juice for the suite to run on full-tilt. So I shut the UPS down and the servers, thankfully, stayed up! Had a box fan blowing on the battery and jumper cables. I disassembled the UPS case, cut the bad fan out and spliced the old connector on to the new fan I got at a local surplus store for $3. Plugged it all in, reassembled and turned the UPS on. It went through diagnostics and everything went green. Then the overload light started blinking and the warning chime came on. I pulled the jumper cables off and the overload warning went away and things stayed stable. The fan stayed on and nothing went down.
I probably should have gotten an award for it because it was a test shot for a multi-billion dollar contract but I was more afraid of disciplinary action over the risk than getting any praise for it. As far as I know, to this day, only two other people at that company know what happened
Look for scratches on the bottom side, brush with toothpaste (the plain one, no additional abrasive ingredients), rinse, read.
In the Dim Times, my company had a couple of hard drives (those newfangled 3.5" Scuzzy drives) that wouldn't spin up and had critical data on them. My solution:
2. Hold the drive in my fingertips (so the platters were parallel with my palm)
3. Power on the computer, then "snap" the drive with a twist parallel to the platters, relying upon inertia to break the stiction.
4. Recover data from now-spun-up drives.
5. Power down, then physically destroy the interface pins on the drive to ensure nobody tried to use it again.
Since then, I've used that trick several times on dead/dying hard drives. As long as the heads are trying to move (indicating electrical life), it's worked every time.
Everybody gets what the majority deserves.
Back in my days as an engineer at Boeing, I supported some automated test equipment on the factory floor. One day, one of the ATE failed to download the required s//w update, so I was called out to investigate. It turned out that the network drop adjacent to the equipment had been disconnected in the nearby network closet. (locked, of course). So I, with the factory manager in two, called the IT department to get it plugged back in.
Me: "I'm in the Renton plant, at column XYZ and we need this network drop reconnected. Production has been halted."
IT Operator: "OK. We'll start a ticket on that. But standard turn-around is 24 hours".
Me: "We can't wait 24 hours. We need to get this equipment updated to get the line up and running. Is there any way to escalate this?"
IT Operator: "Sorry. That drop is was identified as being inactive and was unplugged."
Of course it was inactive. The ATE is only powered up when needed. At other times, the little light on the switch in the closet would be off.
At this point, the factory manager asked for the phone. Very calmly, he spoke to the IT operator.
Manager: "You can cancel that ticket. My engineer assures me that he can reconnect the drop once he gains access to the network closet. The plant fire department is just downstairs and we'll have them bring up a fire axe to open the door."
The IT department dispatched a tech who arrived within 15 minutes.
Have gnu, will travel.
Back in the early 60's I was on a three man combat targeting team and we had two minuteman missiles to startup and target one day. So we went to the first site and the maintenance team had just finished installing a new guidance and computer package and the nuclear warhead. They closed the 80 ton door that protects the missile and so it was out turn to perform.
We started up the on-board computer and ran some checks and then began loading in the targeting data such as whether it was a air burst or ground burst and all of the war-plans associated with it as well as the launch codes and targets.
After this is accomplished then the guidance package goes through some testing and self calibration and finally becomes "ready"
Ready is actually called "Strategic Alert" and lights a green light on our console.
The missile system sat in strategic alert for a few minutes and so we figured we had completed our job and would button up the site and head to our second site.
Suddenly the "Launch Commanded" light lit on the console and a fraction of a second later the "Launch in Progress" light also lit.
I quickly popped out a bunch of the circuit breakers on adjoining panels causing the support equipment to stop functioning.
At this stage we did not know if we had a bad console (portable between sites) or a computer failure on-board. Anyway the missile did not blow the umbilical nor launch so we believe we stopped it just in time. If we tried to check our technical data then we would have been dead most likely.
We contacted job control and they agreed not to attempt a restart and rather have maintenance replace the guidance/computer package yet again and return it to Autonetics for repair.
The next site we went to for startup went perfect and the console worked flawlessly...
That has been nearly 50 years ago now and i still occasionally wonder if the missile had actually entered "launch" or if the on-board computer was giving erroneous launch status.
And in the end, the love you take is equal to the love you make
Like they say: "Hardware is the part of the computer that you can kick..."
Non-Linux Penguins ?
When you are spending so long doing something awkward it's normally worth sitting back for a few minutes and reconsidering the goal and approach.
Goal: Recover documents off computer.
Solution 1: Spend hours writing down key strokes and working blind.
Solution 2: Plug harddrive into another computer and retrieve files.
Solution 3: Use VGA mode or any Windows install disk to recover drivers.
Most of the time when you are working hard it's because you are doing it wrong.