Slashdot Mirror
Lessons From Your Toughest Software Bugs
Nerval's Lobster writes: Most programmers experience some tough bugs in their careers, but only occasionally do they encounter something truly memorable. In developer David Bolton's new posting, he discusses the bugs that he still remembers years later. One messed up the figures for a day's worth of oil trading by $800 million. ('The code was correct, but the exception happened because a new financial instrument being traded had a zero value for "number of days," and nobody had told us,' he writes.) Another program kept shutting down because a professor working on the project decided to sneak in and do a little DIY coding. While care and testing can sometimes allow you to snuff out serious bugs before they occur, some truly spectacular ones occasionally end up in the release... despite your best efforts.
Thanks for posting
Here's an archived copy: https://archive.is/VpWQl
Some of the bugs I've beat my head against the wall over the most are compiler bugs. It's easy to have the mindset that the compiler is infallible, and so programmers don't usually debug in a way that tests whether fundamentals like operators are really working right. This was particularly bad developing for Windows CE back around 2000 when you had to build for 3 different processors (Arm, MIPS and SH3). I ran into a number of optimizer bugs usually related to binary operators. The usual solution was precompiler directives to disable the optimizer around a specific block of code.
Better known as 318230.
... can't stop now, I'm in the middle of a nasty debugging session.
OMG....a Meme...
Program crashing at startup? Okay, let's add debugging statements.
Can't get the debugging statements to execute? Okay, let's try removing code.
Doesn't fix the problem? Okay, let's keep removing more... and more...
A couple hours later, so much code was removed that the entire program had become nothing more than an empty main function that still crashed. This led to the following rule which I try to follow to this day: Make sure that you're actually compiling and executing the same copy of the code that you're modifying. ;)
I'll never forget the last thing grandma said to me before she died: "What are you doing in here with that knife?!?"
I had a bug once where red and blue values were swapping places across thousands of pixels that took quite a while to hunt down once. It turns out there was a function doSomething called with parameters (pixel[i++],pixel[i++],pixel[i++]) while doing transformations. The compiled code pushed the third parameter onto the stack first, so it was using the red value from the array in the blue spot and vise-versa across the entire image.
When ARM first came out on some philips CPUs it had bugs in the C compiler. The IT department called us hardware engineers in after being stuck on a bug for months. The problem with programmers is to many of them work at a high level, and they hit a wall at some abstraction layer, usually at assembly code. The other problem with these compiler bugs was as you removed unrelated code, they went away, as the compiler had pointer corruption issues. So to get the vendor to fix it, you often had to submit an entire copy of your code project. Sometimes we had to submit images of entire machines because the compiler would interact with an IDE and with Windows. These days we use only open source compilers to ensure we arnt held up and can identify and fix problems quickly.
Back in the 80's, I was working on a project with three other programmers. Nobody had heard of version control back then; we were using VAX/VMS and it would keep a few versions of a file around after you changed it, which seemed good enough (after all, we all trusted each other, right?)
Well, I don't remember the exact bug(s), but one day I fixed something, and tested it. Fine. A few days later the bug came back. So I went back, fixed it again (wait, didn't I already make this change?). A few days later it came back again.
It turned out that one of the other guys had fixed a different bug, which I had introduced with my fix. So, his fix was to change the code back the way it was. We went back and forth a few times un-doing each others' changes before we realized what was going on. Seeing a revision log with comments on the changes might have helped...
Have you read my blog lately?
I recall a proverb, something like
"It takes twice as much intelligence to debug code as it took to write it.
So if you code to the best of your ability you are, by definition,
not qualified to debug it."
In the free world the media isn't government run; the government is media run.
Hahaha, I see this happening so often at work.
I think the most memorable one for me was when a specific order of load and store instructions caused the data cache to emit incorrect information.
Bug 1 (my fault) : Took over working on a financial application that took an identifier and enriched them with all sorts of useful data. The original programmer had left, and nobody at the company knew anything about how it worked. Soon after, we were troubleshooting an issue reported by a client that the output data wasn't consistent between runs. I grabbed a list of all the unique security IDs I could find (about 100k) and pushed them through a couple of times just to try and replicate the issue. HOWEVER... it turns out the application was actually using the Bloomberg "By Security" interface under the hood. That was a service where you drop a list of IDs onto Bloomberg's FTP server, and they would respond with data... for a fee of $1 per security. The client got an unexpected bill of nearly $200k that month, and I had the most awkward talk ever with my boss. Fortunately, Bloomberg forgave the charges, and it turns out they were actually responsible for the inconsistent data - which was fixed on their end shortly thereafter.
Bug 2 (not my fault) : A client/server application is returning odd responses to a particular query. Developer (we'll call him "Jason") inserts a switch into the code that dumps this query out to a hardcoded folder on the server. The code then gets checked into production WITH THE SWITCH TURNED ON. It went undetected for nearly a year because the query wasn't terribly high volume. But slowly and steadily, the query files built up over time. Our IT had lots of money to play with, so server space was not an issue. Unfortunately, the number of files was. Server performance went steadily downward every so often, until finally this query would make it crash every time. When we eventually tracked down the cause, there were millions of files sitting in the same folder of every single server in the group. It took nearly three days just to get the OSs to delete the files without falling over.
A stray ; 30 years ago in some C took me a week to find, replacing the intended body of a loop with an empty block IIRC. I have ever since tried always to { } statement blocks so that it is easy to tell what was intended...
Also I strongly echo the "make sure that you're editing what you're running/debugging" comment elsewhere. Still horribly easy to get that one wrong in lots of different ways...
Rgds
Damon
http://m.earth.org.uk/
A trigger on a busy table was using a Rule Based Optimizer
We had done a 'rough' system test for upgrading from 9i to 10g, but the system did not have a realistic production load put on it
The DBA group placed the upgrade into production and suddenly the system drags to a crawl
It took us a very short amount of time to figure out the problem, but a few hours to deal with the existing change control process and satisfying a DBA manager, who failed to let us know that there was a major change with the database release, that dropping the hint entirely (he had been on the team that introduced it years earlier) would be the best way to go since the new Cost Based Optimizer would recognize the query and make adjustments for it.
Wherever You Go, There You Are
I have a bug in javascript that I can't fix.
I can't remember what it is now but it's documented in the code that if you remove the Are you sure? prompt (or remove the now-hidden debug statement), the code doesn't work. When you display the variable, or just wait and ask, then the code does work.
Every couple years when someone scans thru the code, they'll spend a day or two trying to figure out what's really happening.
First job out of college doing tech support for a big corp. One day thousands of Win2000 computers start taking multiple hours to boot up. Nobody can figure out what the problem is, got like 20 people working on it for almost two weeks.
After digging through logs and error messages I discover than some idiot who had denied doing anything had sent out an update via our client management software to add a new local user for support purposes. He didn't do this via a script, rather "recorded" him adding it to a machine and then sent out a copy of the files and registry entries that had changed. Unbeknownst to this genius, the local security database is an binary (pretty sure encrypted) file that you can't just go copying between machines.
I put together a script that repaired the local database and fixed the problem in a couple minutes. But literally had thousands of workers sitting around doing nothing waiting for computers to boot for like 2 weeks.
//TODO: Insert catchy phrase
I wrote some Assembly (Saturn processor in case anyone cares) once that ran differently when run on a even memory address, vs run on an odd memory address. That was no fun to figure out, since the debugger always ran it on an even memory address!
One of my toughest bugs didn't exist.
My code was actually working correctly, but the debugger until certain conditions would display wrong values. I wasted a lot of time trying to find the bug in my code.
I once had a hiesenbug, which was a simple dereferrenced pointer. The problem is that I had a couple thousand lines of code, and the bug wasn't where I was recently coding. Every coder knows to check for bugs in their most recent code, but a derefferenced pointer can be anywhere in the code. Anyway, I decided to break down and pray for help. Then within moments I read through a random line of code in some random file and debugged the problem. Since then, I often pray I do well in general, then I don't get stuck on a brick wall of tech, that God helps me while I code, and a host of other cool stuff. I find things flow more smoothly since then and I don't fight with code. I know God is real, and I've come to discover prayer does help too. In addition to that, I've been more careful with pointer math, biasing array memory structures more.
God spoke to me
Write code to test your code. Hit every edge case hard, every boundary condition.
All too often we tend to test our code by just running the overall program, but this is not good enough. Running the overall program does not introduce a wide enough range of input parameters to every function.
Write test code. Write code to log your inputs and outputs to files early in the development cycle. Don't get swamped down in the land of trying to debug code that was never written to be debugged.
I had many many tough bugs back in the day before I learned this lesson. Once I got this behind me, it was a lot easier.
My favourite head scratcher - back using Motorola's version of Unix, we had a voice response (IVR) application that would poll for activity, and otherwise sit idle using the sleep() command. The code had interrupt handlers SIGUSR (iirc) that would perform "real-time" activities as necessary (handling call hang ups, touch tone digit receipt, etc). When running under a load test scenario during a quality cycle, we kept running into scenarios where 1 in a 1000 or so instances of our event handlers were NOT handling the activities such as call hangups, missing digits, etc.
After MUCH digging, having witnessed our interrupt handling code, half way through a trace, simply stop executing, we did a reverse disassemble of the sleep command, and found this jewel: a SETJMP on invocation, and a LONGJMP back to the stack location when the SIGALRM timer that it set ran out. Assumption being that while in the sleep() call, no other code would be executing. In reality, if our event handlers where running when the the SIGARLM timer ran out, the sleep call did a LONGJMP, restoring the stack back to its original state, wiping our interrupt handler off the stack.
When Motorola was confronted, the first reaction was "no, we didn't do that. We're looking at the code." Only when we showed them the disassembled output did they admit there was an issue with the release of software we were using.
That one took 4 days for me to track down as a junior programmer at the time, some 25 years ago.
This is why printf is (usually) self-debugging
There are few things more exhilarating than writing code.
Controlling the machine with just a few commands? Cool.
Debugging said program, with just a few hours of stress? Not so fun.
But, recompiling and rerunning said program when one is sure it is now bug-free? Like hammering the gas pedal of a muscle car!
Oops, missed one. Screetch.
Three of my favorite bugs or gaffs.
(1) endlessly tweaking and commenting my autoexec.bat file. Only to eventually overwrite something with nothing. With absolutely no backup. Lesson learned? Don't waste time tweaking autoexec.bat
(2) putting the computers at all our department's campus-spanning internet-connected locations into a tight "give me the code again" loop, and then going home for the night. Lesson learned? Normally polite campus sys admins spend much of their time counting the tens of gigabytes of data they reluctantly ship.
(3) one can gain full-time summer employment from the pursuit of a single bug. Nerdy organic chem professor has custom chem. sim program made for the previous year's grad. students' thesis. Only it doesn't work. Lesson learned? Tiny variables need 32-bits of precision.
I come here for the love
about unpublished errata. According to the lead engineer (at a major cpu vendor) there are more hardware bugs than software bugs.
Two bugs come to mind, one that I wrote and fixed, one that I fixed but did not create. The one that I created was an assembler bug, code written in UKY-502 assembler (military computer). I screwed up one op code, specifying LK (load constant) instead of L (load from memory address). The difference in the code was one bit, but I had to single-step through the code to find the bug - took me hours for one stinking bit.
The other bug, also on the UYK-502 computer, was a bug in the micro-code. The guy who wrote the micro-code for one particular instruction had ignored the user guide for the bit-slice processor and had implemented a read-modify-write operation in a single micro-code instruction. It worked for him because the timing hardware was slow enough. Unfortunately, a couple of years later, the manufacturer of one of the chips in the timing hardware improved the internal workings of the chip so that one of the line dropped sooner than it did on older versions of the chip (NB: the chip still met the same specs - it was just faster). Debugging was a pain. The computer used a back-plane, and the timing hardward and the bit-slice processor were on difference cards. When we put either card on a extender so we could connect a logic analyser, the delay added by the traces on the extender caused the problem to go away. It took two of a week to find the problem. The fix was to update the microcode ROMs for every computer that received the new timer card.
linquendum tondere
Stop writing so many of them?
That is all.
For about 10 years I was a troubleshooter, they'd assign me something to work on and then interrupt me for a big ass bug.
First big bug? Linux system would crash after about a week. Diagnosis? When it crashed it was out of FDs. Turns out a kernel resource was opening a file, exiting, and never closing the fd. Time to find? About a week. Time to diagnose? About a minute. Time to fix? About 10 minutes.
How did I find it? Waiting until it died, did some built in command to see WTF happened, looked at the source code, fixed.
Second big bug. System would reboot randomly within an hour to a week due to a watchdog timer firing. Even had a "magic" laptop that made it crash more often. Diagnosis? When you read from a register the chip would sometimes hang. Time to diagnose? About a month, most of that waiting for the damned system to crash. Didn't help I only had 1 JTAG, I couldn't do anything else while waiting for the sytem to crash. I spent a lot of time looking for interesting websites during that month. Time to fix? For me, about 30 seconds. It was a system status register, nobody cared except the hardware folks, I quit reading it. For the hardware folks? Don't know, don't care.
How did I find it? It was a cellphone. When it restarted JTAG was initialized at the reboot point. I found the point in software that initialized the memory controller. As the system never lost power memory was intact. Found the process crashing. Then I created an in-memory array. As the code progressed I updated this in-memory array, stuff like "code does something, I put 0x10 into my array. Code does something else, 0x20 into my array". After a couple days of "it's just reading a register, I messed up somewhere" I finally concluded "reading this register causes it to crash about 1 time in 10,000"
Third big bug? Cellphone base station. Card handled 3 T1 lines, did the analog/digital and digital/analog muxing for each call. Cells would randomly drop out after a day or so, they didn't come back until you rebooted the system. It's a base station, you never reboot the system. After about 3 months of this I got asked to look into it. I'm like, dafuq? It's a DSP issue, I don't know jack about DSP, I'm screwed. Honestly, I had no idea how to even approach this problem.
The fix? I was telling myself how screwed I was, and I'd never get a raise, and generally killing time reading the docs. Found a library call that said "do not call this during an ISR". It was being called from an ISR. Sent email to the DSP folks asking them to comment out that line, they did and sent me the binary blob to load onto the card. I did, problem went away.
One night while coding half asleep, I wrote the following to increment a variable in C++
x = x++;
The problem with this code is that it is an undefined behavior. It looks okay at first glance, and then when you consider the machine code that would be built from it, a bit of ambiguity arises. The problem comes in with the = sign vs the ++ operator. Both of which are assignment operators for the x variable, but it is not well defined which assignment should happen first/last. The code in use was actively being used in both MSVC and GCC environments, each producing opposite assignment ordering. This was awesome to debug, since the code "worked" on one platform but not the other!
I once contracted with a shop that had a process that generated garbled output data rows. It appeared to be extra stuff that didn't affect (over-write) the intended rows. The shop had added an extra processing step to filter out the garbage rows and eventually just worked around the glitch.
They had asked me to try to track it down, among other projects, because they were newbie programmers. I couldn't figure it out either because it never appeared in my intermediate trace statements. I put a trace (print) statement before every "write" in the program. None of the prints showed the garbage, yet garbage ended up in the output file. Head-scratcher galore. I was supposed to be "the expert", and thus feeling a bit deflated.
On I think the last day of my contract, I was running a test copy of the code with some changes to perform speed tests. I went to try a certain speed tweak, and I suddenly spotted the error: the file handle variable was re-used for another non-handle purpose, something like this:
The actual handle name was something like "qhand". But a regular variable, "quantity on hand" ended up "qhand" also, the same name as the file handle.
When it dawned on me what happened, I started screaming like a wildman and the others popped out of their cubicles to see what was going down. They took my coffee away :-)
As far as the link on goofy video game bugs, I remember somebody discovered that if you don't put a game cartridge in all the way, certain characters dance and spin randomly and rapidly in the sky.
It created an Internet meme, and spoofs started appearing all over, typically using stop-motion with live actors. I forgot the nickname of the meme, but I found it hilarious. It took my mind off the handle bug.
Table-ized A.I.
Been there done that. Lol yep, lost half a day on it...
http://blogs.msdn.com/b/rick_s...
Read this years ago, and thought it was interesting at the time...I've saved the link for years. Really detailed story about finding a really complicated bug in MS Word way back in the day.
Chaos, panic, disorder...my work here is done.
Yeah.. That's why the Democrats are pro open immigration too.
This has happened to me too. Which is why all of the code now includes a startup banner that displays the time and date on which it was built.
I gave up on the concept that I would be able to write and debug programs correctly the first time. Now all the central data structures in any long-lived control system get error-checking code added to them. For example, the sorted-list code is built with a checker to ensure it stays in order. The communications code gets error-checking. The PID controllers get min/max testing, etc.
Every once in a while I come across a bugs that are not in the source code. Often they are compiler errors. Sometimes the bugs involve a rare C/C++ or operating system eccentricity. Sometimes the errors are caused by obscure library changes. Sometimes they are hardware errors.
Especially with the embedded micro-controllers, I leave the consistency checking code in, because you just can't assume the everything always works. The nature of software bugs change with time, and it is not always in the way a programmer would expect. I am frequently surprised by how obscure some of the bugs are.
Always check your compiler.
Not a Republican, but all us libertarians understand that in order to sell a thing there must exist people who can buy that thing. And, since the vast majority of things cost money, the more people who have money the more of those things you can sell. This isn't rocket science folks.
Time is what keeps everything from happening all at once.
Back in my student days I had a runaway pointer. On one of mid-1980s Motorola 68000 Macs, it would trigger the power-off function if it wasn't running under a debugger. Talk about frustrating.
At least it was consistent.
Remember, this was back in the days before protected memory. Also, if memory serves, the MacOS and applications always ran in "supervisor mode" (analogous to "ring 0" on Intel chips), so your program 0wned the machine while it was running.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Oh man, that's happened to me twice, with several hours lost in each instance. I've sworn to never allow it to happen a third time.
The novel is The Bug by Ellen Ullman.
Here's quote from one of the reviewshttps://www.kirkusreviews.com/book-reviews/ellen-ullman/the-bug/:
In theory, theory and practice are the same; in practice they're different. (Yogi Berra & A. Einstein)
Because it stymied me for weeks years back when I first started in C++. I'd written some code that made assumptions about where variables were initialised and what happened when said variable were returned, using some custom stuff in operator= and the constructor. (irrelevant detail: I wanted to be able to return sub-matrices of a matrix that could be assigned to to overwrite the relevant parts of the full matrix. Think matlab A([1 2 3], [3 4 5]) = B overwrites part (but not all) of matrix A style. And I was fairly new to C++).
Worked great without optimisation.
Broke horribly when optimisation was turned on.
It was a learning curve, but eventually google turned up a little thing called return value optimisation (or something-or-other ellision, it seems to have a few names). Basically, by design, how code executes (literally what it does) can be a direct function of your optimisation flags. Specifically what assignment operators etc get called, and in what order, when you start returning classes from functions.
I know it's not technically a bug - after all, it's right there in page 5 billion point 2 of the spec - but still, it marked the end of my "my god C++ is amazeballs and can do no wrong" phase.
Oh, damn... yeah, done that as well. Frustrating as hell, because it just doesn't make sense until you finally figure out you're not even debugging the code you're working with.
Other variations of "the impossible is happening" include:
* Syncing to new code, recompiling, and crashing. Crashes only go away once you force a full rebuilt to update stale precompiled headers. // --- crashes here. all variables are integers).
* Program crashes mysteriously, and only is fixed after the machine is rebooted (likely some process in RAM has been corrupted).
* When you get automated crash debug reports from hundreds of thousands of customers, you eventually realize that a staggering number of people simply have bad hardware, due to the impossible crashes that occur (e.g. a = b + c;
* Compiler or hardware bugs - thankfully much more rare than they used to be.
Irony: Agile development has too much intertia to be abandoned now.
I had a job with a group managing shared minicomputers. One program I was writing was to log someone off after being inactive for some time, to free up a port for other users. So my loop to check every 5 minutes involved incrementing the time to wake up by 5 minutes on each iteration. Ie, it woke up at a specific time. So it would theoretically wake up at 12:00, 12:05, 12:10, etc.
The problem was that this operating system for some reason blocked when sending the alert message to someone's terminal. There was possibly some non-blocking way to do this with some extra effort, but it didn't seem like any additional effort was needed. However some user type Control-S on his terminal and then went off to lunch, probably typed it by accident. So a warning message went to his terminal, but blocked because of the Control-S. So the program was stuck until he came back from lunch and typed Control-Q. At which point this unblocked my program which then printed out one after the other on everyone's terminal in two buildings:
"your terminal has been idle and you will be logged off in 15 minutes",
"your terminal has been idle and you will be logged off in 10 minutes",
"your terminal has been idle and you will be logged off in 5 minutes",
"logging off due to inactivity."
This was shortly followed by a line of people coming into the office to complain, including my boss.
I was working on an embedded system recently that had a 5 minute timer to shut off the machine. We had received customer complaints that the machine occasionally shut off early. The code was a simple while loop that ran some pid controls and every loop checked "If (run_time > 5 minutes): exit;". I ran the machine in the lab for a while and sure enough, it shut off early once in a while. I looked through, and eventually SCOURED the code, assuming there was a subtle bug, such as clock corruption due to interrupts, or some kind of type conversion mistake, I couldn't find anything. I eventually set up a serial printout from the machine so I could see what was happening. And it would run and then print out "5 minutes elapsed, shutting down". No glitches or resets (which is what I was expected). So now I'm staring at this one line "If (run_time > 5 minutes): exit;", pulling my hair out. Finally in a moment of insane desperation, I added another line to the while loop. "if (4000 > 5000): print("Something is very wrong!"); I carry the machine to the lab and set it up, and IT PRINTS. Every few minutes or so it pops up on the display. So now I'm just like "fuck everything" how can I possibly run code if I can't even trust the basic principal that the computer will do what I tell it too. So the first thing I do is add triple checks to all critical comparisons, that eliminates the symptoms for now but I know it's going to cause weird problems forever if I leave it like that. Ok so the execution is buggy, I get out the scope and check the power line and various other things and it looks ok, but I notice at this point that the problem never occurs when the machine is running empty, only when it's loaded, so I clip ferrites everywhere you can possibly fit one and spend half a day putting metal covers on everything. As I run the machine this time I'm practically holding my breath, 1 run good, 2, 3. I'm getting super excited at this point, then bam "Something is very wrong!" prints and I die a little inside. After walking out to my car and screaming at the sky for a while, I get back to it. At least I know it has something to do with noise. Since the machine can't possibly be more shielded a take a look at the schematic, it looks normal, but there's a bunch of funky stuff on the reset line. I ask around and nobody knows why its there. It's got a regular pull up resistor, but somebody added a diode in series, and a ferrite bead right before the pin. Due to the voltage drop the MCLR is only being pulled up the 3.9v instead of 5v, so that's not good. Then I take a look at the ferrite on the board and it's sticking off the board with a coil of wire through it not 2 inches from a brushed motor the size of my fist. It must be acting like a transformer secondary. I shorted the diode and the ferrite and the problem never happened again!
This happens in Visual Studio all the time. Often you'll find that a unit test session doesn't properly release the SUT, especially when you have "CopyLocal = true" on the reference in the unit test project. There'll be a missing FreeLibrary() call on the .dll to balance out the LoadLibrary() calls and for whatever reason Windows doesn't catch it so Clean Solution/Rebuild Solution doesn't actually overwrite the previous version of the .dll. Usually the only way to fix it is to close Visual Studio and reopen it although occasionally we've had to reboot a developer workstation as well.
Reading TFS, it's clear that the lesson is to blame someone else. Be it the professor or the guys who built the financial instruments. Let's try to roll a few "MBA types" under the bus while we're at it. So long as it doesn't stick to me!
ugh. this one stumped me, and was deeply unsatisfying since it was never resolved. There was this model of NIC used in our school where Iworked hw support. The card worked for years, dated to 1995 or so. We used Debian, I think this was back in the Potato days. We thought we were safe on Debian. The upgrade happened, and the NIC LED simply would not turn on with the upgraded system. We modified the network and driver config files, modified the kernel configurations, passed load-time boot parameters, checked old/new installation settings, hacked the driver, recompiled, re-installed old and new versions to re-verify that the old distribution worked while the new one didn't. My colleagues weren't slouches either. In retrospect, we should have abandoned the efforts, it was a waste of time and effort. Even then, the cards were nearly worthless. But we were young.
while (something) {
// do_stuff
} while (something_else);
It compiles, is legal C, and loops endlessly if something_else is true.
It can be done in a careless moment when switching a complex piece of code from a while () loop to a do-while () loop.
Been there done that!
"GET / HTTP/1.0" 200 51230 "-" "Mozilla/4.0 (compatible; Setec Astronomy)"
In the 90s I prototyped some communications code was to bootstrap a supecomputing job on 6-10 sites with 64-128 nodes running the job at each site. We ran flawless rehearsals where the site operators reserved perhaps 10% of their machines for half a day so we could submit test jobs without much queue latency. Since there were other groups preparing jobs for an annual conference, machine time was scarce. We wouldn't get to run a full-scale test until the week of the conference, when networks and sites were reconfigured for the occasion. It kept failing with assertion failures suggesting dead-locks in the synchronization code.
After hours of staring at code and debug logs, I finally submitted a full-scale test run where every node was configured to run their task under gdb inside an xterm with the DISPLAY set to a laptop I'd borrowed at the conference show floor. This way, I could wait for one node to crash out, leaving the rest hanging waiting for peer messages. Then I went through hundreds of gdb instances looking at stack traces and inspecting application state.
I found that the hierarchical all-to-all message exchange was desynchronizing due to an unfortunate design blunder where I wrote some code as if it was an MPI task with a barrier-synchronization primitive, but no such primitive existed at this early bootstrapping phase and across multiple sites. My brute-force solution was to rescue this broken design by enforcing its naive implementation invariants... I added virtual clocks, counters, and reordering message buffers to make sure that all messages from one phase of the communication where consumed at a receiver before it would process messages from a later phase, even if those later phase messages arrived first due to skew among all the communicating nodes.
That's on par with rebooting the wrong machine.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
We had a program that was doing session matching of RTP streams (via RTCP). We had to be able to handle a potentially very high load.
Things had been going OK - development progressing, QA testing going well. And then one day our scaling tests took a nosedive. Whereas we had been handling tens of thousands of RTP sessions with decent CPU load, suddenly we were running at 100% CPU with an order of magnitude fewer sessions.
I spent over a week inspecting recent commits, profiling, etc. I could see where it was happening in a general sense, but couldn't pin down the precise cause. And then a comment by one of the other developers connected up with everything I'd been looking at.
Turns out that we had been using a single instance of an object to handle all sessions going through a particular server, but that resulted in incorrect matching - it was missing a vital identifier. So an additional field had been added to hold the conversation ID, and an instance was created for each conversation.
Now, that in itself wasn't an issue - but the objects were stored in a hash table. Objects for the same server but different conversations compared non-equal ... but the conversation ID hadn't been included as part of the hashcode calculation. So all conversation objects for a particular server would hash the same (but compare different).
We had 3 servers and tens of thousands of conversations between endpoints. Instead of the respective server objects being approximately evenly spread across the hash map, they were all stuck into a single bucket per server ... so instead of a nice amortised O(1) lookup, we instead effectively had an O(N) lookup for these objects - and they were being looked up a lot.
The effect was completely invisible under low load and in unit tests. The hash codes weren't verified as being different in the unit tests as there was the theoretical possibility that the hashcodes being verified as different could end up the same with a new version of the compiler/library/etc.
On the level of someone changing order of columns in an indexing for no particular reason, possibly because it looked better to have the index column in alphabetical order.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
One I'll always remember was some Actionscript or Javascript which would never happen with the debugging console open, but would always halt the program if the debugging console was closed.
It turned out to be a call to console.log, which is a fatal error in IE if the debug console isn't visible at the moment.
When viewing this I got the footer quote "%DCL-MEM-BAD, bad memory VMS-F-PDGERS, pudding between the ears".
I find it very suitable to this article.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
If you do not put any new code in it you ain't gonna introduce any new bug in it
"tough bugs weren't my fault"
I'm a minority race. Save your vitriol for white people.
Worst bug we ever ran across was a program that absolutely would not work as soon as anyone looked at it to see if it was working or just to observe the GUI. If you did that, it broke. So we spent a LOT of time trying to run it, debug it, rerun it, and no matter what we did it never worked right as long as someone was looking.
But the moment you stopped looking, locked that PC and walked away, the program would run fine on files dropped into the appropriate input hot folder. It would happily do its thing and give you good results in an output folder.
Look at it, however, and it blew up immediately.
We spent a LOT of time trying to fix this, however it's not a product we made. So there's a limit.
We did finally realize the program is licensed only for the PC where it is installed. If you remote into it, the program sees the remote client as if it's the installed machine which has no license and thus it runs in kind of a suicide mode. When you stop looking at it and drop the remote connection, the license assignment returns to normal and the program runs again.
I documented and discovered all of this and solved a major headache for our company, because we had to have this software working. I unlocked the secret and got it working. My FUCKING BOSS stole credit for the discovery and promptly began parading around like she thought of the solution.
Which is funny because guess who they called next time it broke? Me. I know it inside and out, now.
Sig for hire.
Let us now stop and give praise and thanks to gdb's tui, for it doth warn that thy source file's mtime art newer than thy executable's.
I can't count how many times GDB has either saved me from this directly, or short-circuited what might've been a whole day's hair-pulling frustration.
Why the snark?
Yes, I also designed the hardware target of that code, wire-wrapping the first unit, and writing the 'OS' in a mixture of C and asm.
The asm equivalent was accidentally starting my NMI routine with "push hl; push de" and ending it with "pop hl, pop de". That anything worked at all was a minor miracle, and it did for months before I noticed.
Rgds
Damon
http://m.earth.org.uk/
In the 80's, I wrote a pursuit game where you had to run away from something chasing you. To give you a chance, the chaser was dumb and headed for your current position. When objects got to the edge of the screen, they bounced off. I got the signs in the logic of that wrong, which made the chaser head for where you were going instead, so it always got you quicker. Only time I wrote a bug that improved (in some sense) the code...
I was a tester on Desert Bus. Had to check if the score would roll over if it went too high. *sobs*
Lesson learned: Mandating everything must be tested in a live environment is not always the best policy. Sometimes it's better to fiddle with the code.
I was writing an audio framework using ASIO. I had to create the device and get it all setup then start it. On one soundcard everything was fine, worked like a charm. On a different card? boom. reboot. every time. even in the debugger. I spent weeks trying to track it down. Eventually managed to get it in the debugger in release mode to break. as soon as i hit continue, it rebooted. I saw it was crashing as it was calling ADDR+12. I tracked it down (after three weeks of this) to the struct you pass in that give ASIO the callbacks. I had been creating an instance of the struct on the stack and passing it by reference into the ASIO call. One driver copied the contents, one just stored the pointer. Driver calls callback with the now defunct stack address and over it goes. Still gives me shudders.
Most time consuming bug - The AMD cpu stack corruption bug. Errata 721. It took me a year to track it down. Half that period I thought it was a software bug in the kernel, for a month I thought it was memory corruption in gcc. And most of the rest of the time was spent trying to reproduce it reliably and examine the cores from gcc to characterize the bug. Somewhere in there I realized it was a cpu bug. It took a while to reduce the cases enough to be able to reproduce the bug within 60 seconds. And the last week was putting the whole thing together into a bootable USB stick image to send to AMD so they could boot up the test environment and reproduce the bug themselves.
Bug that was the most fun - The 6522 I/O chip was a wonderful multi-feature chip with a lot of capability. There was a hardware timer bug which could jam the timer interrupt if it timed out at just the wrong time.
My general advice: Add assertions for complex pre-conditions instead of assuming that said complex pre-conditions are always properly in place. The more non-stupid assertions you have in your code, the earlier you detect the bug and the easier it is to fix.
-Matt
Lesson 1: Read your own code and understand what it does.
Lesson 2: Reread your own code and understand what it does.
Lesson 3: If you don't understand what it does, that's a bug.
Looking around, it seems that most people take 'tough' to mean 'spectacular'; I disagree with that. I think some of the most difficult bugs are the subtle ones that don't give many symptoms, or which masquerade as something else.
Probably the hardest one to solve - or the one that required most insight - was in an application is worked with on Windows NT. The architecture was messy, to say the least, with anonymous pipes everywhere, but the real trouble came from the toolset, which tempted developers into doing stupid things. I think it was written using a an IDE for C++ from Borland (I forget the name), and they had got this 'brilliant' idea of making a number of objects that you could drag onto your design surface to create a Windowed application with automatically generated code behind. One class of objects were for things like FTP, etc, which was used in a central place. The problem, as it turned out, after I had thought deeply about it, was that network communication is asynchronous by its very nature, whereas the graphical toolset in Windows is non-reentrant, meaning that it is not a good idea to call functions that update the desktop before they have returned from a previous call. See what I mean: When a network packet arrives, you update your progress bar or whatever, which looks cool - but if the next packet arrives too soon, it tends to kill not just the application, but the whole desktop. The solution was to not use the network objects at all and instead rely on POSIX network calls running in a separate thread and communicating to the main loop via a pipe. Not quite synchronous, but much more robust.
I have a very memorable bug:
I got a new work PC and set up my environment. Checked out the repo with the SVN command line, worked on a new feature and tested it.
When it was ready, I got the TortoiseSVN client for windows (ease of use) and submitted. And after that the software didn't work anymore. Had really curious bugs I've never seen before.
After half a day of investigating I found out: TortoiseSVN installs a shell extension DLL which also gets loaded into your process if you use any windows shell functions, for example GetOpenFileName. The shell extension had a bug that caused it to change the C locale of the whole process, which obviously my application didn't expect to get randomly changed.
That was a something marvelous to find out.
Many of the "hard" bugs discussed in the article do not seem very hard. Divide by zero errors and a +Inf in an input file are straightforward issues that should be caught using standard practice techniques (bounds checking and exception handling). Two of these three hard bugs would have been easy to catch with version control and continuous integration. It seems like the article is more about dealing with other people's crappy code and poor software development practice rather than debugging nasty bugs.
The nastiest bugs are almost always race conditions, which are by their nature non-deterministic and may not be reproducible across time or certain hardware.
Fast Federal Court and I.T.C. updates
I do remember struggling with compiler bugs - in particular the C++ compiler we were using on OS/2 way back in the day suffered a few - but I was primarily a machine-code programmer in those days so an inspection of the compiled object code would tell me what was wrong pretty quickly; plus the compiler guys (I think it was Watford) were very responsive and would usually fix things up pretty quickly. So no, the bugs that have most challenged me have always been concurrency-related - deadlocks, race conditions and the like. My earliest experience of this was probably on the Atari ST. I was coding a game in 68000 assembler, one of the early 3D-rendered golf games. Whilst the golfer (a sprite) was taking his swing I needed to pre-render the first 3D frame, so hooked up a hardware interrupt to run the renderer whilst the golfer animation took place (or maybe it was the other way around; doesn't matter). Anyway it all worked nicely except, just occasionally, a rogue red pixel would appear in a random part of the screen. It took me about a month to figure out that there was one variable/memory location being read and written by both bits of code, with no mutex round it. I guess it was good to learn a hard lesson about concurrency early in my programming career.
http://jakepoz.com/soviet_debugging.html
Mine wasn't particularly hard but was particularly funny. I was working on "blocking" for a guided vehicle system. Vehicles followed a guidepath buried in the floor which was broken into segments. It was (mostly) sufficient to make sure that no vehicle was in a segment before another vehicle was allowed to enter it. While developing this code a developer on another project ran into a problem where a small circle in the guidepath could be filled with vehicles which would then deadlock because none had an empty segment in front of them.
I realized my project had a similar configuration, a system with 5 vehicles and a circle with 5 segments. I thought "what is the possibility that all five vehicles will be in the circle at the same time" and did nothing about it. Within 15 minutes of getting all five vehicles working on site they were all sitting deadlocked in the circle. I manually moved one out of the circle to break the deadlock and they soon wound up back in the circle. It was comical, like they were drawn to that area so they could deadlock and take a break.
What I hadn't realized was that the vehicles had to traverse some part of the circle to go between to any two destinations on the guide path. I remind myself of this any time I'm tempted to ignore a problem just because I think it unlikely to happen.
Developing some embedded software we had a common issue when adding new features that the code would crash when outputting strings to the console, until we added some debug code to identify the problem, when the crash would stop happening. We were in a hurry and so the code generally got shipped with the debug code suitably disabled but still present. I had some extra time one day and decided to investigate this, but couldn't find any coding errors. I eventually got around to looking at the output of the linker/locater to discover that the problem was related to trying to print the last declared string to the console. It emerged that the build tools would fail to append the closing null to the last string stored in the initialised memory portion of the image. Stored to EPROM, some of those final strings ended up with a lot of FF characters appended.
Ah, me too.
Red to red, black to black. Switch it on, but stand well back.
"That's ...strange... why did that prod server just alert offline... ooohhh crap..."
Nope, totally haven't done that.
Carpe Canem - Seize the Dog
I have seen... things you can't imagine. I feel like a replicant in the rain, explaining my life and death. ( https://www.youtube.com/watch?v=ZTzA_xesrL8 )
My own bone-headed prize goes to when I was clearing away debris files left by typos from a previous admin, and accidentally deleted "/bin/[" on an old BSD machine. That's a symlink to "/bin/test" and is part of the logic of most shell scripts, There were other typos there, like "/bin/-r" from old mistyped "rm commands" so there was in fact debris to clean. But it took me 3 days to bring that system back up, the bootstrap TK-50 tapes hadn't been tested in years and didn't work anymore, and the mag-tape drive was on the wrong VMEbus to restore from tape with. There were compelling reasons we didn't touch system file on systems running BSD on a VAX lightly.
My boneheaded prize for others goes to the kernel idiots who insisted on building their kernels on their desktop machines and sending them to me to install, rather than publishing their code changes and using the build system I wrote. Unfortunately, they failed to merge the patches that had been in place for 2 years. So when their "new kernel" was installed, it didn't have the hardware support for the disk drives that the manufacturer had upgraded to in new hardware and had never made it into my rack of test hardware. So when we kernel updated the network, an entire continent went mostly offline: only the old servers were up.
Because I'm an absolute paranoid weasel, I'd learned the hard way to make failed kernel upgrades recoverable: Set the boot loader to use the new kernel *once*, and only once, and activate it as the default only if the reboot succeeds running the new kernel. If it's not running the new kernel, keep the old kernel as the default and revert to it on the next reboot. But somebody had to go and powercycle *all* the new servers.
Don't forget Tiberio's Law of Mutually Cancelling Bugs... While doing a code review of otherwise "working code" you fix an obvious flaw, and as it turns out it was preventing another latent bug from manifesting. Now go find it...
Is trading a 'financial instrument' the same as making a bet?
... and testing for the fix on live at the same time. Noticing after an hour of desperation. .... Arrrrrgh!!!!
We suffer more in our imagination than in reality. - Seneca
Neither of these was hard to diagnose. First was back in the 80s, when automated circuit board assembly was new. Got a batch of boards that didn't work. Turns out somebody had loaded capacitors where resisters should have gone, all our RAM lines had capacitors instead of pullups on them. Whoops.
Then about 10 years ago we get an ASIC from the fab. The clock was all over the place, you could hook a scope up to it and watch it vary from, say, 10 MHZ to 500 MHZ. Turns out that, after running a suite of tests on the VHDL before sending the VHDL to the fab, one of the hardware guys forgot to turn his DEBUG switch to OFF. This left a diode in the phase locked loop that prevented the loop from locking. That was a million dollar mistake that also caused a 6 week schedule slip.
Yes, early days of MSVC (v1.52 on win 3.1 IIRC) was one of my most memorable bugs. It appeared in a new release of our app where a counter was incrementing by 2's and severely screwing up a job dispatch system servicing 6000 telco workers. Running the code in the debugger we watched as the counter jumped by two as we stepped thru a single line i++ statement. Sure enough when we opened it up in assembly we found an extra INC op? I rebuilt to binary using the same build tag and environment, the bug disappeared? It wasn't a particularly difficult bug to fix, but the fact that we couldn't reproduce it from source and never found a better explanation than "cosmic ray" or "Microsoft, pfft", is why it has stuck in my mind for 20yrs.
Disclaimer: I currently manage a large and ancient cvs repository, over the last decade or so I have constructed and maintained an automated build system for about a dozen active projects and a couple of dozen legacy versions that services a team of 25-30 devs plus offshore subcontractors. I have had similar head banging moments wrt compiler optimizations. What I have learned from those experiences is that optimisation often has no noticeable impact on the end customer, so unless a developer can convince me that a specific optimization is critical to an application's performance, I always have them turned off and ask our devs to do likewise.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
The original release of Skeleton+ for the Atari 2600 had a bug which caused the "You Win" screen to generate garbage rather than a stable display. The bug turned out to be in Skeleton (hand coded in 6502 ASM) I'd used the label for the code which followed the "You Win" graphics data as the end-of-data marker. When I modified Skeleton to create Skeleton+ that chunk of code got moved, but the I'd forgotten I'd reused the label.
A colleague and I once found a hardware bug that affected ~2000 motorola modems that we were using for a (1990's) mobile app. The problem was the modem became "emotionally attached" to the first tower it found and refused to talk to any other tower even when its original partner was well out of range and other towers were within easy reach. Tough one to crack for a couple of software guys, took a couple of weeks and a trip to Queensland.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
It is not my bug but in a list of epic bugs this one should not be missing.
https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/issues/123
A spurious space caused the install script to delete your root mount point...
Sounds familiar :)
I notice I do it more often than I did 20yrs ago. Some say it's old age but I think it's probably due to regularly working on multiple VM's via a laptop as opposed to the old days of a stand alone dev box sitting under the desk.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
Are very non-obvious like a typo or a whitespace issue. Though I do recall one particularly annoying bug that dealt with encryption/compression and only manifested itself in rare cases. It drove us nuts for months.
'The code was correct, but the exception happened because a new financial instrument being traded had a zero value for "number of days," and nobody had told us,' he writes.
Sounds like it wasn't correct after all...
crash..debug..crash..debug
Mumble, mumble,...@#$!...what moron wrote this code!
Scroll...mumble...scroll..
Oh, I did,...let me read that again.
A friend of mine once described the above phenomena as "source code is like shit, you can't smell your own"
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
This happened when I was still in school. One of my COBOL programs worked fine at first, but then gave results that didn't make any sense at all. I looked at it until my eyes crossed, the code looked fine, but the results didn't. Finally, I was taking a crap, and something occurred to me. After I completed my important business, I took another look and realized that one of the table subscripts was getting set to 0, and writing outside the table.
I learned an important lesson that day. If I get stuck on a problem, a good long crap will fix it.
Slow down, cowboy! It has been 4 hours since you last posted. You must wait another few hours.
I ran into this in college on a Solaris machine. One of my classmates had created a program called "test". Running test always resulted in the program returning immediately with no output. A bunch of us were puzzled as to why his program wasn't working until one bright fellow asked, "What happens if you run ./test?" We all learned a valuable lesson that day about not including the current working directory in $PATH.
Almost all bugs turn out to be my bugs, but the one that still stays with me to this day was when I tried to implement drag-and-drop in a Java application on a Mac back in 2003. I spent a solid month trying to get it to work and it just didn't behave the way the APIs said it should. Finally I mentioned it to another programmer, a friend, and he said oh yeah he'd noticed the same thing.
Apple's impl of the drag-and-drop library had a bug in it. A user *must* support String type DnD in order for other data types to work. Even though I didn't need to support String for my app, even though the String support did nothing, as soon as I added String to the list of types I claimed to support, all the rest of my code immediately worked as expected.
Fuck you, Apple!
Tip: add shell aliases that print a warning. Whoever actually wants to reboot a server needs to use the full path to reboot/shutdown/halt.
Also, when you fix a bug, make sure you're pushing to the right repo. I suppose that's more of a bug in the programmer. Humanity is a bug.
Chewbacon
The Bible is like Wikipedia: written by a bunch of people and verifiable by questionable sources.
...is my top of list: GLARINGLY OBVIOUS yet so buried in code that it's easily overlooked/missed.
Back before I really got to know my good friend Javascript I encountered the ol' truthy vs truth thing. If I remember right it was a single element array with the value 0 that tested to false. Something like var x = [0]; and then if ( x ) equates to false. That one can really unnerve someone not familiar with the pyscho-gf that is Javascript.
I came to the datacenter drunk with a fake ID, don't you want to be just like me?
One bug I remember quite well.
As quite often, "all works without debug, but not if I enable it." (although didn't notice that at that point)
I was receiving data through MIDI, and had built-in (optional) debug functionality, that would print the incoming (raw) data to the screen. It worked so, that it would print values for one command on the first line, next ones on second etc. until screen is full, after which the used part is cleared, and first line printed again.
The core problem was, I was accidentally printing one line too many, which resulted in data being assigned after the used screen mem. It just happened so, that immediately after that was array of least significant bytes for MIDI command callback routines, some of which were modified.
So after that happened, one command whose handler address was changed arrived (a while after the data was corrupted), and the code jumped to wrong position. And not "too wrong" either (actually called keyboard number key handler) but of course didn't call anymore what it was supposed to call.
So how I found & fixed it:
See where the code execution ends up. Number key handler. "But I didn't press any keys?"
At that point I was rather lucky. I just added a breakpoint at the number key handler, and noticed two things:
1)The number key callback was still clearly executed, as before.
2)Breakpoint was never triggered.
The corrupted address was actually a few bytes after the first command of the function, which clearly explained this. At that point I became really suspicious, considering the MIDI command callback wasn't called anymore, I checked the callback array, and noticed quite a few wrong bytes. Set watchpoint there, run code, and it was clearly done by the debug print code. After decrementing the compare value, all was fine again.
And before anyone mentions it, stack contents were rather useless, as there's exactly one jsr call during this all.
And that was, what... Two weeks ago?
I was a young programmer working at my first startup company back in 1999. We had an communication app that talked using CORBA much as today you would used Web Services. We knew we had a memory leak and back then you only had a few megabytes of RAM so a memory leak could chew through all your memory pretty quickly. This bug was causing our servers to crash at least once a day and was in danger of taking down the whole company.
/*code snippet*/
/*Connection parameters*/), &header, &message);
/* continue processing other event */
/* handle error */
I didn't write the code, it was written by one of our senior engineers. He has insisted his code was right but I found the memory leak in his code using a debugger.
returncode = CallCORBA( new CORBAConnect(
if (returncode != ERROR)
{
}
else
{
}
As a software testing intern, I found a crash bug on the test server. I could reproduced it 100%. My boss couldn't reproduced it at all, and subsequently approved the patch for the production server despite my dire warnings. The production server crashed within 24 hours, knocking it offline for three days and costing the company $250K in lost revenues. My internship wasn't renewed and 1/3 of the division got laid off the following month to make up for the lost revenue. As for my boss, he got promoted.
I inherited a 'creaky' legacy system - and the server needed to be rebooted. It hadn't been rebooted in 5+ years.
I did all of the sensible things: I checked that the backups were up to date - and then I manually copied the entire codebase and local data and database to another server before touching anything, just to be sure.
Reboot ... dead.
Restore everything ... still dead.
After a lot of tracking down I discovered that a previous developer had placed critical config in /tmp /tmp was purged on reboot.
I then discovered that the backup system was configured to ignore /tmp - because ... /tmp
That took a lot of effort and guesswork to rebuild.
Now I always copy /tmp before rebooting anything with a long uptime ;)
I am the only one sick of the articles that get submitted by this crustacean?
It is the anonymous CORBAConnection variable that is create in the function call. Programmers create this anonymous variables all the time and never thing that it will bite them is the ass. Well, this one did and nearly took down the company too. Here is the explanation behind it.
CORBA communication is asynchronous, and thus COBRA connect object lives past the function that created it. When the communication thread that was using the connection is finished the original calling function that created it, has passed out of scope so there is no destructor called implicitly. And since there is no explicit variable, we cannot call the destructor explicitly either. With no way to call a destructor, there is no way to reclaim the memory, used thus the memory leak.
The solution was to explicitly declare a variable for the CORBA connection object and then call the destructor when it finished.
Along with prayer, you should take a vow of celibacy and poverty. More for the rest of us.
Stop cheaping out and expecting your end users to report the bugs. You need to actually HIRE QA people and test the software BEFORE delivering it.
I worked with a guy many years ago who coined the term "worthy bugs" that we used when we had a really good one. Two, in particular, I remember decades later. 1. This turned out to be a hardware bug that showed up in our software very intermittently. In the 1980's, National Semiconductor offered the NSC 880, a clone of the 8080: Same instruction set, mostly the same specs. This processor is spec'ed such that on the enable interrupts instruction, interrupts are not actually enabled for one instruction cycle so as to allow for a bit of cleanup (pop or whatever) without interruption. Without this, stacks could become confused, and did. Well, the guys at the fab across the street from where we were doing software development did not implement this one-instruction delay, but kept knowledge of it as a secret errata. When confronted about it after we had traces that proved the error in hardware, their response was "well, we didn't think it would ever come up." Bastards. 2. It turned out that one of the early Intel chipsets implementing PCI would, when doing 64K data transfers that fell exactly on 64K boundaries, deliver the first byte of the the range in place of the last byte. I was working on Ethernet device drivers at the time, so this just looked like data corruption in the driver or the network controller to us. It took a while and many logic analyzer traces to root cause this one to the chip set. Once we knew what was happening, the software work-around was easy, but it did slow down the driver just a bit. At least the chip set guys were unaware of this bug, and it never appeared in the many subsequent chip set implementations of PCI.
Not just a bug; it's a virus, Misterr Anderrson.
Got them moderator blues I blieve I walk out the do', With these mod-points I been gettin', I 'most never post no mo'
This is why I never develop in any language that doesn't have an IDE I can step through the code with a debugger.
That said, I'm a strong believer in defensive programming practices. Not only do they make the software more secure, but they also help catch your own bugs. As the one article in TFS says
Unless out-and-out performance is vital, checking inputs is always a good idea.
Hint: Performance is only vital in very few locations, namely interrupt handlers deep in the Operating System. So it's not likely that performance is vital enough to skip checking the inputs to your functions.
Hint: Checking the inputs to your functions will almost always help you catch logic errors, prevent memory overflows, etc. IOW, they'll save you many man months of debugging by making many things obvious. You just have to be disciplined enough to do them.
Truth is like the sun. You can shut it out for a time, but it ain't goin' away. - Elvis Presley (source: imdb.com)
The default shortcut to submit code for the IDE I work in is F3. This language/IDE has some legacy support features, one of which is assigned to F4 by default. Pressing F4 will take the last piece of code that you submitted to run, and insert it into your current editor window wherever your cursor is.
One day I was working on a 1k odd LOC program. I selected the code from top to bottom, pressed F3 to submit it, and then somehow pressed F4, which copied the 1000 LOC and pasted them into the bottom of the doc (unbeknownst to me).
There were some errors in the log (as the code was still in development), so I would start at the top of the log, find the first error and go and debug it, then resubmit the program, and repeat. Though I fixed the program, the errors kept occurring. Took me about half a day to figure out that I was debugging the top half of a duplicated piece of code. Because I was using CTRL-HOME, CTRL-END, and CTRL-F to navigate around, I never noticed the duplication.
So mad. My first action now is always to disable that damn F4 shortcut.
Late 90s, multi-developer team. One day, my code was crashing, and all my attempts to step thru the code and examine the stacktrace left me scratching my head why it was crashing in my code.
The problem was someone else's code freed the same memory block twice, causing the heap to get corrupted. The program would still run a bit until it crashed in my code.
I would have never figured it out if I decided not to run Purify on the code, and it detected the double free() call.
The various browser DOM implementations are an endless fountain of weird bugs, even today. One of the most bizarre I've tackled was an issue that cropped up in a headless browser test. One input element for currency would intermittently report the reverse of the value entered into it by the automated test. 234.00 became $432.00. Try as I might, I couldn't reproduce the problem in an actual browser, which meant using the slightly less than awesome tools for inspecting state in a headless browser. Suspecting that the jQuery plugin for currency formatting responsible, I debugged its handling of the input end to end, and found nothing. But disabling the plugin suppressed the problem. Identical code on other pages worked without issue. And then, I discovered that commenting out the input element *after* my currency element made the problem go away. So what followed was a few hours of trying to find any way in which the code for managing these two inputs could possibly be interacting, nothing. Finally, I got to bisecting the HTML itself, and found a styling related HTML class on the second input, that would suppress the problem if it was removed. Some combination of unsupported CSS in the headless browser squashing my form elements, the jQuery plugin reformatting input, and the way the test runner entered input into the form resulted in the string getting reversed, sometimes. To this day I still don't know exactly what was happening. The fix ended up being a tiny CSS tweak completely unrelated to my poor currency input.
Hi fellow /.'ers,
The timing is fortuitous; I just started a blog of my own experiences on this very topic, drawn from 35+ years of software development experience: www.geekcrumbs.com
The Glass House and the Drawbridge (most recent post) is on the front page, and will no doubt give some of you a chuckle. Please forgive the advertising down the left side. I'm hoping for my first tropical vacation in fifteen years :-)
Cheers,
ws
Oh, I just have to chime in on this one.
When you fork a copy of someone's git repository? Do not assume that the code you've just inherited matches the binary that you have been using. Make a test compile before you do anything.
Sometimes, what's in the public repository doesn't match what was compiled for the binary. Sheesh.
After 50 years of coding:
1. Beware of doing things the easy way, even if you are in a "crunch". The "easy" way is often the hard way.
2. Beware of clever coding. Clever is a "bug farm" for the next person, even if -you- are the next person.
3. Learn to read at least a little Assembler code (and the Debugger). If the compiler or linker have a bug, it is the only way to figure out what is wrong. And it can be useful for regular bugs, too.
I had a hell of a bug about 8 years ago. Clearly it was a use after free in kernel process running in VxWorks that zero'ed out memory it no longer owned. The way it was observed was when a memory 'free size node' pointer was set to 0, corrupting the memory AVL tree. We couldn't reliably hit it; it had to happen only if the corrupted memory happened to be appended to the free size nodes, which meant it was a discontiguous free'd memory region, and then you wouldn't see the problem until someone allocated memory that had the matching requested size of the corrupted node, which meant we never got the same stack trace twice. To test, we ran a simulation of the environment constantly destroying and re-instantiating the object structure, and would get about 1 hit every 12 hours. This program instantiated tens of thousands of objects from ~250 different classes. The bug was a misunderstood order in a class hierarchy destructor: one class's destructor cleared memory an inheriting class had already freed. Not a big deal to fix, but incredibly difficult to find. We invented this to find it: http://www.google.com/patents/... While I worked on this problem longer than anyone else, I sadly was not included on the patent. :(