Slashdot Mirror
What is the Worst Tech Mistake You Ever Made?
"In the interest of full disclosure, this is mine:
I was working at a Fortune 50 bank as a consultant. I was due to go on vacation for a week and the company did not have webmail. I decided that I would try forwarding emails to my corporate account. (I know this was a bad idea, and probably against several corporate policies.) I set it up so that any email that came in would forward to my consulting company's account. My mistake was I also left Delivery Receipt on. This was not Microsoft, it was Lotus Notes. The system began forwarding the incoming mail to my account. But then it would get a Delivery Receipt, which in turn would be forwarded to my account, which would generate another delivery receipt, ad infinitum. When I got back from vacation they claimed I had brought down the email system for 4 hours. This incident caused the bank to stop allowing consultants to set up email rules. What's your story?"
Thankfully, that's the worst I've done so far.
Prevent email address forgery. Publish SPF records for y
None that I've done come to mind - I tend to make lots of little stupid mistakes rather than occasional huge cock-ups. But I had a client that had a CIO who was actively hostile to the idea of any kind of computer security what-so-ever. Waste of time and money for a made up threat he said.
They were running 13 servers at remote locations (and I mean remote, as in out in the boonies 4 hours from nowhere on back roads) and these servers were unpatched, had out of date or innactive anti-virus and were connected to the net via a combination of satellite and dedicated (always on) dialup. Their communications were secured with nothing more than Windows 2000's built in VPN.
Needless to say, my audit report told them that they had big beefy powerful angels on their side since they hadn't yet had a noticable intrusion. (They had no way of detecting one, but at least the servers weren't hosting porn sites.) I warned them that a virus or worm would come along though and knock the whole thing out. The CIO scoffed at my report, called me an alarmist and said that my opinions were right up there with the Y2K doomsayers.
When Slammer hit, I had described the vulnerabilities and outcome so accurately that this guy actually accused me of writing it myself. Took the whole corporate network down and they couldn't bring it back up until their techs visited each site. It took two teams seven days to get to all the sites. The company lost 6 business days, three customers and a months worth of transaction records.
Needless to say the CIO was demoted (they didn't fire him, which I consider itself a major tech mistake) and had me re-issue my audit report which they then followed to the letter taking every precaution I suggested.
I was a young pup in the Army, during a training exercise. My Commander told me to kill the network, to "simulate" it's loss. We were operating a frequency hopping radio network, which of course is based on time. As the master node, I controlled the time. I pumped my transmitter to full power, and slowly pulled the stations that could recieve my signal out of time. Lowered power, pulled a smaller number of stations even farther out of time. Wash, rinse, repeat.
Commander thought I was brilliant, and so did I. I had fractured our network into at least 10 different domains. No one could talk to anyone, effectively "simulating" an enemy jamming attempt. It would take hours to restore the network, with many mad commo guys having to drive about with Pluggers, early GPS devices, to restore each radio to propper time.
Then a tank flipped. Someone died. No one could call for help. I am so damn smart.
No moon black, At 2 in the morning, in an upside down tank, the gunner figured out how to put his radio in plain text to call for help. It took him almost half an hour.
If voting were effective, it would be illegal by now.
My best singke mistake was after a long night of re-installing an updated version of solaris on a SparcServer 2, I needed to clear out the /tmp dir sor some stupid reason. So, I did the old: "mkdir newdir ; mv * newdir"
/tmp. I was in /.
/usr/lib/libc.so.0
/usr/sbin/static there are 5 statically compiled binaries: cp, ln, mv, rcp, and tar. /newdir/usr/sbin/static/mv /newdir/* / would have fixed it.
I wasn't in
My next command was 'ls'. It returned: unable to find
AAAAARGH!
I now know how to solve that under solaris. Under
Ever since then, my prompt has had my current directory in it. That experience certainly made me more careful.
Better (or worse) was when a stupid service rep came in to replace a bad CPU on a sun e10000. The idiot shut down the sub-system, and powered off the board correctly. He then managed to pull out the wrong board, despite the blinken lights. Of course it was the peoplesoft domain. Running year end reporting.
AAAAARGH!
Zapman
Similar to the if (0 == c) trick in C, I've been trying to train my fingers to type DELETE WHERE whenever I mean to type "DELETE". Then, I fill out the WHERE clause and only then go back to say what table to delete from.
This also gives you time to ponder the wisdom of first running a SELECT statement with the same WHERE clause and comtemplate whether you want to do this.
Aliasing rm to rm -i will do nothing if you use the -f flag, as you did. -f overrides -i.
.jpg
However, accidentally separating a wildcard from text is an infrequent mistake that can cause much pain. For example, typing rm -rf *
Zsh, by default, will complain at you and ask you if you *really* mean it if you use a bare wildcard with an rm command. Invaluable, and has saved my ass a few times.
May we never see th
This wasn't my mistake, but I guy I work with left a where clause condition of an update SQL statement and released the work to production. In development he was working on a system with a reduced data volumes and it updated a few stray records and he didn't notice. In test, the test cases didn't cause the SQL to run. In production it wiped out about 20000 records and had to full scan a very large table to boot. We actually found it because of the performance problem first. It took us a few weeks to reconstruct all the erased data.
My biggest mistake was in my first programming job years ago. I intentionally wrote an infinite loop into a program that was running on a very powerful (for the time) reasearch unix box used at the Naval research lab where I had an internship. It was a sonar imaging optimization routine and I would let it run for short periods (10-30 seconds typically) and then CTRL-C it to force it to stop and inspect the log file to find the results. I was new to unix and so I would use "ps" as opposed to "ps -aux" to see what processes I had running. I had multiple sessions up and managed to leave one of my programs running, switched sessions, ran ps which showed no processes running and went to lunch. The sysadmin was also a meeting and then lunch. When I returned I had a bunch of nastygrams telling me to kill my job immediately, not to run processes that hog the CPU because other projects couldn't use the system and to get approval before running long running jobs because the CPU time was billed (this was around 1985). I actually sat down, ran ps again, saw no job, and wrote back saying I didn't know what they were talking about. The sysadmin (who had returned from lunch) came over to visit me and educated me on a whole bunch of things.
rm -rf from the command line? Lucky bastards!
I once added the following to a cronjob
rm -rf $foo/*
My intention was to wipe contents of a directory that I was reusing. Unfortunately "foo" was unset. The cronjob ran overnight with rm -rf traversing every NFS mounted drive in the company. I remember coming in at 10 the next morning and thinking "christ what kind of idiot deleted all of my files?", and then "shit! that idiot deleted everyone's files" and then "shit that idiot is me!".
Ever since then I usually do something like
rm -rf ${foo:?}
mkdir $foo
Later as I recovered my composure I started thinking "Now why can't those idiots set their umask correctly?".
The only positive aspect of what happened was that it revealed a weakness in the backup procedures being following by the IS department.
Personally I count my self lucky to have had the benefit of such a humbling experience w/out loosing my job.
I've adapted that idea to a lot of other situations; my SQL queries always start out as "-- delete ..." until I'm sure about what I'm typing.
Dewey, what part of this looks like authorities should be involved?