Stunts, Idiocy, and Hero Hacks

snydeq writes "InfoWorld's Paul Venezia serves up six real-world tales of IT stunts and solutions that required a touch of inspired insanity to pull off, proving once again that knowing when to throw out the manual and do something borderline irresponsible is essential to day-to-day IT work. 'It could be server on the brink of shutting down all operations, a hard drive that won't power up vital data, or a disgruntled ex-employee who's hidden vital system passwords on the network. Just when all seems lost, it's time to get creative and don your IT daredevil cap, then fire up the oven, shove the end of a pencil into the motherboard, or route the whole city network through your laptop to get the job done,' Venezia writes."

Comments

Re:Floppy disk in the wash

[LMacG]

I don't think you understand the term "got lucky." Oh right, I'm reading Slashdot ...

Depends on the point of view

[arivanov]

The first "stunt" depends on your point of view. If you have nicely brainwashed and duped by marketing material that "Vendor gear good, PC bad" that may sound as a stunt. If you actually know what you are doing you can run networks for years on this.

Nearly any laptop today has the forwarding grunt of an upper end of a 3800, there are plenty of servers that are on par with a 7200 or low end 7600 and most supervisor modules. You can run a network on this on a daily basis and do a _LOT_ of things a Cisco cannot do or cannot do at sufficient performance.

To put the so called "stunt" into a perspective, I used to run a production installation with 20+ 802.1q trunks via 800MHz Via EPIAs with 600+ entry ACL lists including content filtering with VRRP failover, load balancing to multiple upstream uplinks, OSPF, hardware accel-ed openvpn and ipsec, 16+ class hierarchy CBQ QoS and a few more bells and whistles. For years. Not for 48 hours.

Nothing wrong with it if you can do it. If you cannot - well, not everything in life is learned on CCXX and RedRat certification courses. C'est la vie.

Car Battery

[Ynsats]

I once repaired a critical UPS that was attached to a critical database server actively recording data in the middle of a test shot with jumper cables and the battery from my truck. All that just to replace a fan that kept sending the UPS in to panic mode for an overheating battery and trying to start a shutdown sequence on the database server.It was a 12v power source for the UPS (old, old equipment) coming out of the AC to DC power supply. The UPS was part of a suite of equipment that included the database server, the array, a backup device, a network switch and the UPS hardwired to each of them in it's own rack. Don't ask me who made it. All I know is it was an Informix based DB and the maker was some esoteric, specific solution company I never heard of and before my time anyway. All I knew was the replacement parts had a 2 week lead time and I have no idea why this company chose to hold up such critical data with such arcane and unsupportable equipment. But, I had to shutdown the UPS to do the work but the battery didn't have enough juice to support the 30 minutes it was going to take to do the work. The battery power would have been killed once the unit was off anyway.

So I attached my jumper cables and the 600 amp battery from my truck to the output rails on the UPS, after the control switches. From there it was just juice to the rails and then to the server and it's data array. The car battery had about 45-55 minutes of juice for the suite to run on full-tilt. So I shut the UPS down and the servers, thankfully, stayed up! Had a box fan blowing on the battery and jumper cables. I disassembled the UPS case, cut the bad fan out and spliced the old connector on to the new fan I got at a local surplus store for $3. Plugged it all in, reassembled and turned the UPS on. It went through diagnostics and everything went green. Then the overload light started blinking and the warning chime came on. I pulled the jumper cables off and the overload warning went away and things stayed stable. The fan stayed on and nothing went down.

I probably should have gotten an award for it because it was a test shot for a multi-billion dollar contract but I was more afraid of disciplinary action over the risk than getting any praise for it. As far as I know, to this day, only two other people at that company know what happened

Fire Axe

[PPH]

Back in my days as an engineer at Boeing, I supported some automated test equipment on the factory floor. One day, one of the ATE failed to download the required s//w update, so I was called out to investigate. It turned out that the network drop adjacent to the equipment had been disconnected in the nearby network closet. (locked, of course). So I, with the factory manager in two, called the IT department to get it plugged back in.

Me: "I'm in the Renton plant, at column XYZ and we need this network drop reconnected. Production has been halted."

IT Operator: "OK. We'll start a ticket on that. But standard turn-around is 24 hours".

Me: "We can't wait 24 hours. We need to get this equipment updated to get the line up and running. Is there any way to escalate this?"

IT Operator: "Sorry. That drop is was identified as being inactive and was unplugged."

Of course it was inactive. The ATE is only powered up when needed. At other times, the little light on the switch in the closet would be off.

At this point, the factory manager asked for the phone. Very calmly, he spoke to the IT operator.

Manager: "You can cancel that ticket. My engineer assures me that he can reconnect the drop once he gains access to the network closet. The plant fire department is just downstairs and we'll have them bring up a fire axe to open the door."

The IT department dispatched a tech who arrived within 15 minutes.