Slashdot Mirror
Lessons From Your Toughest Software Bugs
Nerval's Lobster writes: Most programmers experience some tough bugs in their careers, but only occasionally do they encounter something truly memorable. In developer David Bolton's new posting, he discusses the bugs that he still remembers years later. One messed up the figures for a day's worth of oil trading by $800 million. ('The code was correct, but the exception happened because a new financial instrument being traded had a zero value for "number of days," and nobody had told us,' he writes.) Another program kept shutting down because a professor working on the project decided to sneak in and do a little DIY coding. While care and testing can sometimes allow you to snuff out serious bugs before they occur, some truly spectacular ones occasionally end up in the release... despite your best efforts.
Some of the bugs I've beat my head against the wall over the most are compiler bugs. It's easy to have the mindset that the compiler is infallible, and so programmers don't usually debug in a way that tests whether fundamentals like operators are really working right. This was particularly bad developing for Windows CE back around 2000 when you had to build for 3 different processors (Arm, MIPS and SH3). I ran into a number of optimizer bugs usually related to binary operators. The usual solution was precompiler directives to disable the optimizer around a specific block of code.
Better known as 318230.
Program crashing at startup? Okay, let's add debugging statements.
Can't get the debugging statements to execute? Okay, let's try removing code.
Doesn't fix the problem? Okay, let's keep removing more... and more...
A couple hours later, so much code was removed that the entire program had become nothing more than an empty main function that still crashed. This led to the following rule which I try to follow to this day: Make sure that you're actually compiling and executing the same copy of the code that you're modifying. ;)
I'll never forget the last thing grandma said to me before she died: "What are you doing in here with that knife?!?"
I had a bug once where red and blue values were swapping places across thousands of pixels that took quite a while to hunt down once. It turns out there was a function doSomething called with parameters (pixel[i++],pixel[i++],pixel[i++]) while doing transformations. The compiled code pushed the third parameter onto the stack first, so it was using the red value from the array in the blue spot and vise-versa across the entire image.
When ARM first came out on some philips CPUs it had bugs in the C compiler. The IT department called us hardware engineers in after being stuck on a bug for months. The problem with programmers is to many of them work at a high level, and they hit a wall at some abstraction layer, usually at assembly code. The other problem with these compiler bugs was as you removed unrelated code, they went away, as the compiler had pointer corruption issues. So to get the vendor to fix it, you often had to submit an entire copy of your code project. Sometimes we had to submit images of entire machines because the compiler would interact with an IDE and with Windows. These days we use only open source compilers to ensure we arnt held up and can identify and fix problems quickly.
Back in the 80's, I was working on a project with three other programmers. Nobody had heard of version control back then; we were using VAX/VMS and it would keep a few versions of a file around after you changed it, which seemed good enough (after all, we all trusted each other, right?)
Well, I don't remember the exact bug(s), but one day I fixed something, and tested it. Fine. A few days later the bug came back. So I went back, fixed it again (wait, didn't I already make this change?). A few days later it came back again.
It turned out that one of the other guys had fixed a different bug, which I had introduced with my fix. So, his fix was to change the code back the way it was. We went back and forth a few times un-doing each others' changes before we realized what was going on. Seeing a revision log with comments on the changes might have helped...
Have you read my blog lately?
I recall a proverb, something like
"It takes twice as much intelligence to debug code as it took to write it.
So if you code to the best of your ability you are, by definition,
not qualified to debug it."
In the free world the media isn't government run; the government is media run.
Bug 1 (my fault) : Took over working on a financial application that took an identifier and enriched them with all sorts of useful data. The original programmer had left, and nobody at the company knew anything about how it worked. Soon after, we were troubleshooting an issue reported by a client that the output data wasn't consistent between runs. I grabbed a list of all the unique security IDs I could find (about 100k) and pushed them through a couple of times just to try and replicate the issue. HOWEVER... it turns out the application was actually using the Bloomberg "By Security" interface under the hood. That was a service where you drop a list of IDs onto Bloomberg's FTP server, and they would respond with data... for a fee of $1 per security. The client got an unexpected bill of nearly $200k that month, and I had the most awkward talk ever with my boss. Fortunately, Bloomberg forgave the charges, and it turns out they were actually responsible for the inconsistent data - which was fixed on their end shortly thereafter.
Bug 2 (not my fault) : A client/server application is returning odd responses to a particular query. Developer (we'll call him "Jason") inserts a switch into the code that dumps this query out to a hardcoded folder on the server. The code then gets checked into production WITH THE SWITCH TURNED ON. It went undetected for nearly a year because the query wasn't terribly high volume. But slowly and steadily, the query files built up over time. Our IT had lots of money to play with, so server space was not an issue. Unfortunately, the number of files was. Server performance went steadily downward every so often, until finally this query would make it crash every time. When we eventually tracked down the cause, there were millions of files sitting in the same folder of every single server in the group. It took nearly three days just to get the OSs to delete the files without falling over.
A stray ; 30 years ago in some C took me a week to find, replacing the intended body of a loop with an empty block IIRC. I have ever since tried always to { } statement blocks so that it is easy to tell what was intended...
Also I strongly echo the "make sure that you're editing what you're running/debugging" comment elsewhere. Still horribly easy to get that one wrong in lots of different ways...
Rgds
Damon
http://m.earth.org.uk/
First job out of college doing tech support for a big corp. One day thousands of Win2000 computers start taking multiple hours to boot up. Nobody can figure out what the problem is, got like 20 people working on it for almost two weeks.
After digging through logs and error messages I discover than some idiot who had denied doing anything had sent out an update via our client management software to add a new local user for support purposes. He didn't do this via a script, rather "recorded" him adding it to a machine and then sent out a copy of the files and registry entries that had changed. Unbeknownst to this genius, the local security database is an binary (pretty sure encrypted) file that you can't just go copying between machines.
I put together a script that repaired the local database and fixed the problem in a couple minutes. But literally had thousands of workers sitting around doing nothing waiting for computers to boot for like 2 weeks.
//TODO: Insert catchy phrase
I once had a hiesenbug, which was a simple dereferrenced pointer. The problem is that I had a couple thousand lines of code, and the bug wasn't where I was recently coding. Every coder knows to check for bugs in their most recent code, but a derefferenced pointer can be anywhere in the code. Anyway, I decided to break down and pray for help. Then within moments I read through a random line of code in some random file and debugged the problem. Since then, I often pray I do well in general, then I don't get stuck on a brick wall of tech, that God helps me while I code, and a host of other cool stuff. I find things flow more smoothly since then and I don't fight with code. I know God is real, and I've come to discover prayer does help too. In addition to that, I've been more careful with pointer math, biasing array memory structures more.
God spoke to me
My favourite head scratcher - back using Motorola's version of Unix, we had a voice response (IVR) application that would poll for activity, and otherwise sit idle using the sleep() command. The code had interrupt handlers SIGUSR (iirc) that would perform "real-time" activities as necessary (handling call hang ups, touch tone digit receipt, etc). When running under a load test scenario during a quality cycle, we kept running into scenarios where 1 in a 1000 or so instances of our event handlers were NOT handling the activities such as call hangups, missing digits, etc.
After MUCH digging, having witnessed our interrupt handling code, half way through a trace, simply stop executing, we did a reverse disassemble of the sleep command, and found this jewel: a SETJMP on invocation, and a LONGJMP back to the stack location when the SIGALRM timer that it set ran out. Assumption being that while in the sleep() call, no other code would be executing. In reality, if our event handlers where running when the the SIGARLM timer ran out, the sleep call did a LONGJMP, restoring the stack back to its original state, wiping our interrupt handler off the stack.
When Motorola was confronted, the first reaction was "no, we didn't do that. We're looking at the code." Only when we showed them the disassembled output did they admit there was an issue with the release of software we were using.
That one took 4 days for me to track down as a junior programmer at the time, some 25 years ago.
Two bugs come to mind, one that I wrote and fixed, one that I fixed but did not create. The one that I created was an assembler bug, code written in UKY-502 assembler (military computer). I screwed up one op code, specifying LK (load constant) instead of L (load from memory address). The difference in the code was one bit, but I had to single-step through the code to find the bug - took me hours for one stinking bit.
The other bug, also on the UYK-502 computer, was a bug in the micro-code. The guy who wrote the micro-code for one particular instruction had ignored the user guide for the bit-slice processor and had implemented a read-modify-write operation in a single micro-code instruction. It worked for him because the timing hardware was slow enough. Unfortunately, a couple of years later, the manufacturer of one of the chips in the timing hardware improved the internal workings of the chip so that one of the line dropped sooner than it did on older versions of the chip (NB: the chip still met the same specs - it was just faster). Debugging was a pain. The computer used a back-plane, and the timing hardward and the bit-slice processor were on difference cards. When we put either card on a extender so we could connect a logic analyser, the delay added by the traces on the extender caused the problem to go away. It took two of a week to find the problem. The fix was to update the microcode ROMs for every computer that received the new timer card.
linquendum tondere
For about 10 years I was a troubleshooter, they'd assign me something to work on and then interrupt me for a big ass bug.
First big bug? Linux system would crash after about a week. Diagnosis? When it crashed it was out of FDs. Turns out a kernel resource was opening a file, exiting, and never closing the fd. Time to find? About a week. Time to diagnose? About a minute. Time to fix? About 10 minutes.
How did I find it? Waiting until it died, did some built in command to see WTF happened, looked at the source code, fixed.
Second big bug. System would reboot randomly within an hour to a week due to a watchdog timer firing. Even had a "magic" laptop that made it crash more often. Diagnosis? When you read from a register the chip would sometimes hang. Time to diagnose? About a month, most of that waiting for the damned system to crash. Didn't help I only had 1 JTAG, I couldn't do anything else while waiting for the sytem to crash. I spent a lot of time looking for interesting websites during that month. Time to fix? For me, about 30 seconds. It was a system status register, nobody cared except the hardware folks, I quit reading it. For the hardware folks? Don't know, don't care.
How did I find it? It was a cellphone. When it restarted JTAG was initialized at the reboot point. I found the point in software that initialized the memory controller. As the system never lost power memory was intact. Found the process crashing. Then I created an in-memory array. As the code progressed I updated this in-memory array, stuff like "code does something, I put 0x10 into my array. Code does something else, 0x20 into my array". After a couple days of "it's just reading a register, I messed up somewhere" I finally concluded "reading this register causes it to crash about 1 time in 10,000"
Third big bug? Cellphone base station. Card handled 3 T1 lines, did the analog/digital and digital/analog muxing for each call. Cells would randomly drop out after a day or so, they didn't come back until you rebooted the system. It's a base station, you never reboot the system. After about 3 months of this I got asked to look into it. I'm like, dafuq? It's a DSP issue, I don't know jack about DSP, I'm screwed. Honestly, I had no idea how to even approach this problem.
The fix? I was telling myself how screwed I was, and I'd never get a raise, and generally killing time reading the docs. Found a library call that said "do not call this during an ISR". It was being called from an ISR. Sent email to the DSP folks asking them to comment out that line, they did and sent me the binary blob to load onto the card. I did, problem went away.
One night while coding half asleep, I wrote the following to increment a variable in C++
x = x++;
The problem with this code is that it is an undefined behavior. It looks okay at first glance, and then when you consider the machine code that would be built from it, a bit of ambiguity arises. The problem comes in with the = sign vs the ++ operator. Both of which are assignment operators for the x variable, but it is not well defined which assignment should happen first/last. The code in use was actively being used in both MSVC and GCC environments, each producing opposite assignment ordering. This was awesome to debug, since the code "worked" on one platform but not the other!
http://blogs.msdn.com/b/rick_s...
Read this years ago, and thought it was interesting at the time...I've saved the link for years. Really detailed story about finding a really complicated bug in MS Word way back in the day.
Chaos, panic, disorder...my work here is done.
I gave up on the concept that I would be able to write and debug programs correctly the first time. Now all the central data structures in any long-lived control system get error-checking code added to them. For example, the sorted-list code is built with a checker to ensure it stays in order. The communications code gets error-checking. The PID controllers get min/max testing, etc.
Every once in a while I come across a bugs that are not in the source code. Often they are compiler errors. Sometimes the bugs involve a rare C/C++ or operating system eccentricity. Sometimes the errors are caused by obscure library changes. Sometimes they are hardware errors.
Especially with the embedded micro-controllers, I leave the consistency checking code in, because you just can't assume the everything always works. The nature of software bugs change with time, and it is not always in the way a programmer would expect. I am frequently surprised by how obscure some of the bugs are.
Back in my student days I had a runaway pointer. On one of mid-1980s Motorola 68000 Macs, it would trigger the power-off function if it wasn't running under a debugger. Talk about frustrating.
At least it was consistent.
Remember, this was back in the days before protected memory. Also, if memory serves, the MacOS and applications always ran in "supervisor mode" (analogous to "ring 0" on Intel chips), so your program 0wned the machine while it was running.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Oh man, that's happened to me twice, with several hours lost in each instance. I've sworn to never allow it to happen a third time.
The novel is The Bug by Ellen Ullman.
Here's quote from one of the reviewshttps://www.kirkusreviews.com/book-reviews/ellen-ullman/the-bug/:
In theory, theory and practice are the same; in practice they're different. (Yogi Berra & A. Einstein)
Some people, when trying to analyze a buggy program, think "I know, I'll use a debugger". Now they have two buggy programs to analyze.
-- a grumpy old programmer
I don't care if it's 90,000 hectares. That lake was not my doing.
Because it stymied me for weeks years back when I first started in C++. I'd written some code that made assumptions about where variables were initialised and what happened when said variable were returned, using some custom stuff in operator= and the constructor. (irrelevant detail: I wanted to be able to return sub-matrices of a matrix that could be assigned to to overwrite the relevant parts of the full matrix. Think matlab A([1 2 3], [3 4 5]) = B overwrites part (but not all) of matrix A style. And I was fairly new to C++).
Worked great without optimisation.
Broke horribly when optimisation was turned on.
It was a learning curve, but eventually google turned up a little thing called return value optimisation (or something-or-other ellision, it seems to have a few names). Basically, by design, how code executes (literally what it does) can be a direct function of your optimisation flags. Specifically what assignment operators etc get called, and in what order, when you start returning classes from functions.
I know it's not technically a bug - after all, it's right there in page 5 billion point 2 of the spec - but still, it marked the end of my "my god C++ is amazeballs and can do no wrong" phase.
Oh, damn... yeah, done that as well. Frustrating as hell, because it just doesn't make sense until you finally figure out you're not even debugging the code you're working with.
Other variations of "the impossible is happening" include:
* Syncing to new code, recompiling, and crashing. Crashes only go away once you force a full rebuilt to update stale precompiled headers. // --- crashes here. all variables are integers).
* Program crashes mysteriously, and only is fixed after the machine is rebooted (likely some process in RAM has been corrupted).
* When you get automated crash debug reports from hundreds of thousands of customers, you eventually realize that a staggering number of people simply have bad hardware, due to the impossible crashes that occur (e.g. a = b + c;
* Compiler or hardware bugs - thankfully much more rare than they used to be.
Irony: Agile development has too much intertia to be abandoned now.
I had a job with a group managing shared minicomputers. One program I was writing was to log someone off after being inactive for some time, to free up a port for other users. So my loop to check every 5 minutes involved incrementing the time to wake up by 5 minutes on each iteration. Ie, it woke up at a specific time. So it would theoretically wake up at 12:00, 12:05, 12:10, etc.
The problem was that this operating system for some reason blocked when sending the alert message to someone's terminal. There was possibly some non-blocking way to do this with some extra effort, but it didn't seem like any additional effort was needed. However some user type Control-S on his terminal and then went off to lunch, probably typed it by accident. So a warning message went to his terminal, but blocked because of the Control-S. So the program was stuck until he came back from lunch and typed Control-Q. At which point this unblocked my program which then printed out one after the other on everyone's terminal in two buildings:
"your terminal has been idle and you will be logged off in 15 minutes",
"your terminal has been idle and you will be logged off in 10 minutes",
"your terminal has been idle and you will be logged off in 5 minutes",
"logging off due to inactivity."
This was shortly followed by a line of people coming into the office to complain, including my boss.
"recompiling and rerunning said program when one is sure it is now bug-free"
That's a neat trick Please let us know how you do it.
putting the 'B' in LGBTQ+
I was working on an embedded system recently that had a 5 minute timer to shut off the machine. We had received customer complaints that the machine occasionally shut off early. The code was a simple while loop that ran some pid controls and every loop checked "If (run_time > 5 minutes): exit;". I ran the machine in the lab for a while and sure enough, it shut off early once in a while. I looked through, and eventually SCOURED the code, assuming there was a subtle bug, such as clock corruption due to interrupts, or some kind of type conversion mistake, I couldn't find anything. I eventually set up a serial printout from the machine so I could see what was happening. And it would run and then print out "5 minutes elapsed, shutting down". No glitches or resets (which is what I was expected). So now I'm staring at this one line "If (run_time > 5 minutes): exit;", pulling my hair out. Finally in a moment of insane desperation, I added another line to the while loop. "if (4000 > 5000): print("Something is very wrong!"); I carry the machine to the lab and set it up, and IT PRINTS. Every few minutes or so it pops up on the display. So now I'm just like "fuck everything" how can I possibly run code if I can't even trust the basic principal that the computer will do what I tell it too. So the first thing I do is add triple checks to all critical comparisons, that eliminates the symptoms for now but I know it's going to cause weird problems forever if I leave it like that. Ok so the execution is buggy, I get out the scope and check the power line and various other things and it looks ok, but I notice at this point that the problem never occurs when the machine is running empty, only when it's loaded, so I clip ferrites everywhere you can possibly fit one and spend half a day putting metal covers on everything. As I run the machine this time I'm practically holding my breath, 1 run good, 2, 3. I'm getting super excited at this point, then bam "Something is very wrong!" prints and I die a little inside. After walking out to my car and screaming at the sky for a while, I get back to it. At least I know it has something to do with noise. Since the machine can't possibly be more shielded a take a look at the schematic, it looks normal, but there's a bunch of funky stuff on the reset line. I ask around and nobody knows why its there. It's got a regular pull up resistor, but somebody added a diode in series, and a ferrite bead right before the pin. Due to the voltage drop the MCLR is only being pulled up the 3.9v instead of 5v, so that's not good. Then I take a look at the ferrite on the board and it's sticking off the board with a coil of wire through it not 2 inches from a brushed motor the size of my fist. It must be acting like a transformer secondary. I shorted the diode and the ferrite and the problem never happened again!
while (something) {
// do_stuff
} while (something_else);
It compiles, is legal C, and loops endlessly if something_else is true.
It can be done in a careless moment when switching a complex piece of code from a while () loop to a do-while () loop.
We had a program that was doing session matching of RTP streams (via RTCP). We had to be able to handle a potentially very high load.
Things had been going OK - development progressing, QA testing going well. And then one day our scaling tests took a nosedive. Whereas we had been handling tens of thousands of RTP sessions with decent CPU load, suddenly we were running at 100% CPU with an order of magnitude fewer sessions.
I spent over a week inspecting recent commits, profiling, etc. I could see where it was happening in a general sense, but couldn't pin down the precise cause. And then a comment by one of the other developers connected up with everything I'd been looking at.
Turns out that we had been using a single instance of an object to handle all sessions going through a particular server, but that resulted in incorrect matching - it was missing a vital identifier. So an additional field had been added to hold the conversation ID, and an instance was created for each conversation.
Now, that in itself wasn't an issue - but the objects were stored in a hash table. Objects for the same server but different conversations compared non-equal ... but the conversation ID hadn't been included as part of the hashcode calculation. So all conversation objects for a particular server would hash the same (but compare different).
We had 3 servers and tens of thousands of conversations between endpoints. Instead of the respective server objects being approximately evenly spread across the hash map, they were all stuck into a single bucket per server ... so instead of a nice amortised O(1) lookup, we instead effectively had an O(N) lookup for these objects - and they were being looked up a lot.
The effect was completely invisible under low load and in unit tests. The hash codes weren't verified as being different in the unit tests as there was the theoretical possibility that the hashcodes being verified as different could end up the same with a new version of the compiler/library/etc.
On the level of someone changing order of columns in an indexing for no particular reason, possibly because it looked better to have the index column in alphabetical order.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
... [did something] With absolutely no backup. ...
Lesson learnt: Backup
Looking around, it seems that most people take 'tough' to mean 'spectacular'; I disagree with that. I think some of the most difficult bugs are the subtle ones that don't give many symptoms, or which masquerade as something else.
Probably the hardest one to solve - or the one that required most insight - was in an application is worked with on Windows NT. The architecture was messy, to say the least, with anonymous pipes everywhere, but the real trouble came from the toolset, which tempted developers into doing stupid things. I think it was written using a an IDE for C++ from Borland (I forget the name), and they had got this 'brilliant' idea of making a number of objects that you could drag onto your design surface to create a Windowed application with automatically generated code behind. One class of objects were for things like FTP, etc, which was used in a central place. The problem, as it turned out, after I had thought deeply about it, was that network communication is asynchronous by its very nature, whereas the graphical toolset in Windows is non-reentrant, meaning that it is not a good idea to call functions that update the desktop before they have returned from a previous call. See what I mean: When a network packet arrives, you update your progress bar or whatever, which looks cool - but if the next packet arrives too soon, it tends to kill not just the application, but the whole desktop. The solution was to not use the network objects at all and instead rely on POSIX network calls running in a separate thread and communicating to the main loop via a pipe. Not quite synchronous, but much more robust.
Many of the "hard" bugs discussed in the article do not seem very hard. Divide by zero errors and a +Inf in an input file are straightforward issues that should be caught using standard practice techniques (bounds checking and exception handling). Two of these three hard bugs would have been easy to catch with version control and continuous integration. It seems like the article is more about dealing with other people's crappy code and poor software development practice rather than debugging nasty bugs.
The nastiest bugs are almost always race conditions, which are by their nature non-deterministic and may not be reproducible across time or certain hardware.
Fast Federal Court and I.T.C. updates
It is the anonymous CORBAConnection variable that is create in the function call. Programmers create this anonymous variables all the time and never thing that it will bite them is the ass. Well, this one did and nearly took down the company too. Here is the explanation behind it.
CORBA communication is asynchronous, and thus COBRA connect object lives past the function that created it. When the communication thread that was using the connection is finished the original calling function that created it, has passed out of scope so there is no destructor called implicitly. And since there is no explicit variable, we cannot call the destructor explicitly either. With no way to call a destructor, there is no way to reclaim the memory, used thus the memory leak.
The solution was to explicitly declare a variable for the CORBA connection object and then call the destructor when it finished.