Slashdot Mirror
Tales From the Tech Trenches
GMGruman writes "Anyone in IT has a story or two involving stupid users, crazy co-workers, kludgy technology, and airhead managers. Lisa Blackwelder has collected top tales of the tech trenches, covering user antics, office politics, and unusual technical challenges that IT pros faced (usually) with aplomb, insight, and savvy."
http://www.infoworld.com/print/146869
Because there's really no reason to post that shit on two pages to cram in more ads.
You're better off reading The Daily WTF for these types of stories. It's better written than InfoWorld could ever hope to be.
There's no -1 for "I don't get it."
This month's horror story concerns a user reporting a nasty security issue.
The user comes up to the helldesk and reports that they have a, quote, "mysterious cable" coming out the front of their computer. Given that at $company we pay a little more attention to security than, say, Gawker, one of my fellow Ops techs was dispatched to the user's desk to determine what this cable could be and why it was so mysterious.
A few minutes later, he returned, having successfully traced the mysterious cable out the front USB port all the way to the keyboard.
Upon reporting this finding, another tech asked who the user was--and then noted that she had given said keyboard to said user, who had plugged the keyboard into the USB port herself.
In Xanadu did Kubla Khan
A stately pleasure dome decree
Several years ago, I was the Network Administrator at a medium sized manufacturing company. We had a problem with some of our employees viewing inappropriate content while at work and we didn't have a robust content filtering system at the time. So, I set up a sniffer port at our Internet ingress point and connected a laptop with Driftnet to get a real time picture, literally, of what was being accessed. One of my fellow IT workers saw what I was up to and decided to DoS me. He created an image of a Nazi Flag with my picture superimposed and the words "Network Nazi" underneath. He then uploaded the image to his personal web server with an auto-refresh script and for the next 20 minutes, the only thing I saw was the image he created / posted.
Seriously, my gripe here isn't about being spread over two pages, my gripe is that there is literally shit everywhere on that site. I have never seen such a "busy" lay out, with the facebook shit on the side and ads on the other, topped off with text ads in the middle of the article.. fuck that.
Eat sleep die
At one point I did tech support for a company that had inherited a block of 1 way cable modems from a company they bought. One way cable modems are the very definition of asynchronous - 56Kb upstream & 3MB downstream at the time (55:1 ratio) - as they bond a dialup connection upstream with a cable connection downstream. Not only does this compound the number of problems - all the problems of a dialup modem and the problems of a cable modem with the added joy of bonding issues - but customers were completely unable to grasp the asynchronous nature of the process.
One customer in particular was quite upset that it took so long to upload his files. I can only blame myself as I asked "What kind of files are you working with?" I then endured a 20 minute rant on how it didn't matter what kind of file it was because he had downloaded the files quickly from the newsgroups earlier and it shouldn't take hours to repost his new donkey porn videos to a different newsgroup.
Obviously, the constant barrage of stories linking back to InfoWorld here, posted always by the same employees at InfoWorld, is the result of money changing hands between InfoWorld and Slashdot. This is a slow week for news, so the story is even more tedious than normal. Still, they've already pad for the linkage, so they gotta fill it with something...
Copiers here. I put a note over the power point (covering the plugs) saying "Don't unplug this without our permission." They fllipped the note up and did it again. We then taped the thing down properly. Our next planned step was international picture language: disconnected cable and man with bat.
Indeed, everyone in IT has stories about how everyone except themselves are idiots.
Read Dilbert instead
The unfriendly solution is to tell the company that suppliers the cleaners that if it happens again you cahnge companies. If it's important equipment then it's the only realistic thing to do.
So a company I worked for had about 80 people in manufacturing and design engineering in one building. Some electrical contractors were doing upgrades to the manufacturing area -- pretty normal. But two guys on the crew wanted to make sure the line they were working on was dead, so they went to the main breaker panel and started systematically flipping breakers to identify their circuit. They had worked their way through all the circuits powering the engineering workstations, crashing Unix machines right and left, and had started on the circuits powering the PC board stuffing robots, etc., in the manufacturing area. The breaker panel was visible in that area, so one of the manufacturing managers figured out what was happening and put a stop to it. Still, it cost us in engineering most of a day to recover.
The manufacturing VP was a cool guy... he immediately walked the whole crew out the front door and called their boss to report their firing from the job... and said the tools they left behind would be sent to them. A lucky friend of mine got to pour all their tools randomly into a moderate-sized crate and wheel it to the loading dock.
Ohh... as to your naming contest... my contribution: Rita Reboot
.... doesn't mean you should.
Best story I've got this year is attempting to help a customer repair a corrupted Exchange information store.
What we found while helping this customer run the repair tools was that they had put their Exchange databases on a software RAID1 between an internal SATA drive and an external USB drive...
All the while, the server in question already had an internal hardware RAID controller with a RAID5 with roughly 3 times available space as the size of their Exchange databases.
On top of that, their backup was a backup to disk folder, on the same software RAID1.
It appeared that the internal SATA drive, which was not properly mounted in the first place due to the fact that the server chassis was designed for hot-plug SAS on a backplane, had failed some time prior, and now the USB drive was starting to experience excessive bad blocks.
We were able to repair their database after moving it to the hardware RAID.
Reply inanely here if you too have had a cleaning lady pop a core something (switch|router|something better) to plug in a vacuum cleaner.
/me raises hand.
Company servers in the middle of the nightly backup. It was an MS-DOS based system. The batch files would start the backup normally and were programmed to reboot the machine at the end of the cycle to clear everything. We couldn't figure out why a backup that should have taken about three hours was done in one. Eventually the boss spent the night in the server room so he could watch the screen to see what happened. The cleaners would come in about an hour after midnight, unplug the server, vacuum, re-plug it and it would come up as if the backup had run.
When our name is on the back of your car, we're behind you all the way!
I used to work at a shipping company. They used to provide higher volume customers with a PC that ran their shipping software. One of my responsibilities was to maintain this equipment at the customer sites (upgrade software, fix PCs, etc.). So the company assigned me to the Florida West Coast (Naples, Ft. Myers, etc.). I arrived at the facility with a list of about a dozen customers to see that week. My first task was to get the spare equipment to replace/install at customer sites. There was supposed to be a small room with all the equipment. I went to the room but it was empty. I called the home office... They said something to the effect that "It's a large room with a whole bunch of PCs, monitors, keyboards. You can't miss it." So I looked again. Even though it was a good sized warehouse, there were only four or so rooms. Nope, couldn't find all that equipment. Finally got in touch with the manager there.... Yup, my predecessor had loaded all the equipment into a truck and taken it away.
I called my office. And yeah, I figured that something suspicious had happened but I had to play it dumb (can't go around accusing someone of theft if I wasn't certain). Call went something like:
"Hey, the former admin took all the equipment away. Where did he take it?"
"What do you mean he took the equipment?"
"I understand that he loaded everything into a truck last week and drove off. Let me know where and I will see about moving it back."
"What do you mean he took the equipment?"
"The facility manager said he took the whole day loading everything up into a Ryder truck. Then he drove off."
"Where did he take them?"
"I don't know. I just got here today. "
The beauty of his move was that he maintained all the inventory... So when it came time to see how much equipment was supposed to be there, everything showed as empty or at customer sites or disposed off... A roomful of brand new equipment was marked as "Disposed" or "Sent back"...
I "graduated" from a desktop technician to a full fledged web developer quite some time back -- but yes, I do indeed have some interesting stories from back-in-the-day. One of my favorites is the exceptionally overweight guy who called me to fix a problem with his MacBook Pro after having sat on it. The odd part to me was that he somehow thought that the issue was related to software, and he didn't even admit to what he'd done until I turned over the Mac, and asked him point blank about the huge freaking dent in the bottom. Needless to say, the computer had a cracked motherboard and had to take a trip with me to the nearest Apple Retail Store.
Ok, well I have to take some blame because I was involved in this, but while working for a major retailer I was one of two engineers fixing the power going to a pair of 6509's. They had redundant power supplies, and both the backups were bad. I had sent them both back, and received the RMA units the same day. After scheduling the change, and getting all the paperwork filled out we were ready to begin. Because we anticipated issues with at least one of the units, anything in this Datacenter seemed to be cursed, we called in a proactive ticket with Cisco. As we lined up the 30 amp plug and had it seated in the plug housing (attached to a local UPS) the engineer I was working with began inserting the 20 pound power supply into the chassis.
Just as he was sliding it I noticed THE CABLE HOUSING WAS SLIDING OUT OF THE POWER SUPPLY!!! I was starting to shout for him to stop and the two exposed solder points contacted the outside of the power supply. Needless to say, milliseconds later, Sparky (who hadn't checked the screw that held the housing in place on the power supply) was cowering in the corner, the operator on duty ran in the DC and had to yell over our now popped ears what the fuck just happened. Occording to her it was a very large bang, to me it was like a lightning bolt in front of my eyes.
I was already reaching for the leather strap to yank him off it, when I saw he was on the ground and the UPS had locally blown it's fuse. Thankfully he wasn't hurt, and it only took me about 36 hours of explaining to TAC what happened to get the unit back up to 100%. Before that night I never thought I'd call and say, "The unit arc'ed out and I watched it ground through the chassis... we're gonna need some parts." From now on I write the instructions such that it's painfully fucking obvious "DON'T FLIP THE POWER TO THE ON POSITION ON THE FEED UNTIL THE UNIT IS SECURE!!!"
Sparky doesn't do IT anymore.
Another soloution is to plug the important equipment in using something so obviously different from a normal plug and socket that the cleaner doesn't even think to unplug it. Something like an IEC 60309 connector or so.
Or just have power hardwired into the rack and a lock on the rack door.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
Or have cleaners that are not beyond stupid?
The server rooms where I do system work have special outlets set aside for "unclean" use like vacuums or power-tools.
Then again the cleaning crew knows enough to not break shit ;)
Or have cleaners that are not beyond stupid?
That would mean hiring cleaners smart enough to know what minimum wage is and to threaten to report an employer that tries to pay less!
If you pay peanuts, you get monkeys!
I watched fascinated as a desktop tech attempted to get more mileage out of a laser toner by shaking it -= then when that failed, held it over his head, stood underneath and prised it open with a screwdriver. When I finished laughing (not that the inhalation of toner is healthy) I suggested using the hottest possible water to clean the toner out of his clothes. The following week the same person rolled out a number of new PCs, and connected them to the network with 5 metre cat5 cables - when I went up to fix the alleged routing problem I found he'd used his "nouse" (and muscles) to stretch the cables. The job sheet clearly said 10m cables, guess he was just lazy. Oh, another time he "discovered" that an entire batch of new replacement hdds was dodgy - by hooking them up one-by-one, with the the drives sitting on the metal case (shorting the circuitry) and booting the test machine. Finally, in response to all the complaints his father stepped in a moved him out of Desktop support - to Project Management. No surprise that company went into receivership the following year. (I'm looking at you Adam!).
Then there was the new starter in the server room - not 15 minutes after being read the rules on *not* touching the mainframes or the tape robots ("just stick to labeling blank tapes") he climbed into a tape robot "to see how it worked". It was only luck and a lot of micro-switches that prevented the robot from tearing his head off. (there's a special place in Hell for people who "look" with their fingers). I should have seen the signs when he first tried to install Outlook on his PC - I asked wtf he was doing - he replied that the "don't fuck with the equipment" rules were only for people who weren't computer experts (he held a MSCE), and that surely everyone knew Lotus was an "end-of-life" product.... One guess who the employer was (manufacturer of said mainframes and tape robots).
Both these people are still employed (elsewhere) in the industry, the latter is considered an IT industry authority in the local PC user's group.
This lawn supervisor was out on a sprinkler maintenance job and he started working on a Findlay sprinkler head with a Langstrom 7 gangly wrench. Just then, this little apprentice leaned over and said, “You can’t work on a Findlay sprinkler head with a Langstrom 7 wrench.” Well this infuriated the supervisor, so he went and got Volume 14 of the Kinsley manual, and he reads to him and says, “The Langstrom 7 wrench can be used with the Findlay sprocket.” Just then, the little apprentice leaned over and said, “It says sprocket not socket!
Display some adaptability.
A client of a previous consultant job owned a wild life rescue. One day she brought in her laptop that a Raccoon had used as a litter box (It was a Gateway, so not far off.) Rubber gloves, masks and moving to the companies attached garage with a card table was required to get the data off the HD and into the new one she purchased. The garage reeked of Raccoon musk for quite a while.