Lessons From Your Toughest Software Bugs

Nerval's Lobster writes: Most programmers experience some tough bugs in their careers, but only occasionally do they encounter something truly memorable. In developer David Bolton's new posting, he discusses the bugs that he still remembers years later. One messed up the figures for a day's worth of oil trading by $800 million. ('The code was correct, but the exception happened because a new financial instrument being traded had a zero value for "number of days," and nobody had told us,' he writes.) Another program kept shutting down because a professor working on the project decided to sneak in and do a little DIY coding. While care and testing can sometimes allow you to snuff out serious bugs before they occur, some truly spectacular ones occasionally end up in the release... despite your best efforts.

Comments

Hardly devastating, but a waste of several hours

[Rei]

Program crashing at startup? Okay, let's add debugging statements.

Can't get the debugging statements to execute? Okay, let's try removing code.

Doesn't fix the problem? Okay, let's keep removing more... and more...

A couple hours later, so much code was removed that the entire program had become nothing more than an empty main function that still crashed. This led to the following rule which I try to follow to this day: Make sure that you're actually compiling and executing the same copy of the code that you're modifying. ;)

You are not qualified to debug your own code

[myowntrueself]

I recall a proverb, something like

"It takes twice as much intelligence to debug code as it took to write it.
So if you code to the best of your ability you are, by definition,
not qualified to debug it."

Re:You are not qualified to debug your own code

[bloodhawk]

The full quote is

“Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.”
- Brian Kernighan

I used to use this as my signature a few years back to try and make devs think about what they are writing. It is nearly always better to make the code simple and readable than to try and produce the best possible code. No it isn't as fun, but it is a damn side better for those that have to try and decipher your clever coding tricks later.

Re:Compiler bugs are the worst

[sectokia]

The absolute worst I've had was a soft cpu in a altera fpga. It shipped with a C compiler. A programmer came to me to explain how his program would crash if he changed the order in which subroutines were defined. After carefully checking the logic it, there was nothing wrong with his code. So i then trawled through the assembly. Again i could find nothing wrong And thought i was losing my mind. I had to painstakingly check the cpu state after each instruction until i eventually found one instruction that did not set a flag as per the manual, and the assembler matched the manual. It was a fault that would only trigger it you did a certain conditional jump after a certain fetch increment then store sequence. It was a bug in the cpu pipeline logic. I learnt a valuable lesson never to trust anything. We wasted allot of time because we were convinced we must have been the source of the fault.

Re: Compiler optimizer bugs

[Anonymous Coward]

A compiler guy here, who used to work for one of the RISC companies. Most compiler bugs are not that difficult to debug. But I worked on instruction scheduling and register allocation, hence always got assigned all the weird bugs. The most memorable one for me was actually a hardware bug - most people don't realize but most of the commercial microprocessors have a lot of bug in them. See published erratas and you will find many bugs. A few years after the particular generation of this processor was on the market, I got assigned a bug from this commercial DBMS vendor (I.e. very important customer) on this weird crash bug. It took me forever to figure out but it turns out to be a bug in the processor that corrupts a particular register (due to the register renaming logic screwing up in a rare combination of instructions) that is dependent on the timing and the instruction combination. It became anothet errata item, and I ended up implementing a workaround - if you notice some benign but odd code sequence a compiler generates, there might be a good reason behind :)

Re:Compiler optimizer bugs

[Jeremi]

I was working at my first job writing my first program ever that was not a homework assignment, I decided to write it as a multi-threaded program

^^^ 2015 nominee for most terrifying sentence on Slashdot :)

Re:Passing Parameters with Side Effects

[Anonymous Coward]

Actually, what you're describing is formally defined as undefined behavior in the C and C++ standards.

Undefined behavior:

doSomething(pixel[i++],pixel[i++],pixel[i++]); /* function call commas are NOT sequence points, so the result is undefined */

Refer to the Sequence point article. The [3] citation says

"Clause 6.5#2 of the C99 specification: "Between the previous and next sequence point an object shall have its stored value modified at most once by the evaluation of an expression. Furthermore, the prior value shall be accessed only to determine the value to be stored."

Pay spectial attention to see point #4 under "Sequence points in C and C++", because that talks about your exact problem. But beware that you'd still have a bug even if you hid the increment inside of a function, because order of argument evaluation is not specified (as oppposed to undefined behavior, which can cause nasal demons or format your hard drive).

Fixed with least diff:

int r=pixel[i++], g=pixel[i++], b=pixel[i++]; /* commas between declarators ARE sequence points */
doSomething(r,g,b);

See also: S.O. questions related to undefined behavior and sequence points in C and C++.