Slashdot Mirror
Pet Bugs II - Debugger War Stories
AlphaHelix queries: "A few weeks back there was an article on Pet Bugs, where people were asked about their favorite bugs. I have a different sort of question: what was your greatest debugging challenge? I've been debugging for a long time, from analog circuits all the way up to multi-kLOC multithreaded servers, and I have some pretty grisly war stories, like the time I debugged a problem in a third-party DLL in machine code because the client didn't have the source for it (yay open source.) What was your greatest debugging triumph?" The first time Slashdot did this it was more about bugs that you had encountered (and may not have solved), this one is about bugs in your own projects code and the trials and tribulations you had to go thru to get them fixed.
Once I had to debug a program written in MFC... Wait. Sorry. The memory is too painful to recall.
Not the most difficult bug I ever encountered, but one that didn't pop right out.
Project was porting code from Solaris to AIX, multithreaded app. At one point in the code, two threads were started, and they needed to synchronize with each other.
Anyway, on Solaris, the threads would start and interact properly. On AIX, the system would crash. Turned out that right after a thread was started on Solaris, the scheduler would stop one thread and allow the other one to start up, and from then on both threads existed at the same time as they should.
Under AIX, the scheduler would start a thread, and that thread would run through to completion before the other one even got started. To fix this, we had to add in a rendezvous point at the top of each thread, so that the first thread would stop and wait for the second one to be created.
If tits were wings it'd be flying around.
That one is an oldie :) Back in the Amiga days I had made this game that worked fine on my A500 but that stopped working after a while on most other A500. That was strange as the machines were supposed to be identical and I couldn't make more tests at home.
So I used the Action Replay cartridge. For those who don't know about Action Replay, those were "hardware debuggers" that pluged on the bus and could stop and restore the execution of the running program. They were very powerful debugging tools.
After inspecting the content of the hardware registers thanks to the Action Replay, the result was that on some revision of the A500 motherboard the audio interupts had a slightly different timing that caused an improbable case were the audio samples always stopped playing on offset 0 retrigering an audio interupt as soon as one was handled.
The Amiga was so much fun...
True warriors use the Klingon Google
When we were working on a game title on the Nintendo 64, we were maintaining a parallel version for Windows, as it was substantially easier to develop for. Basically, we created some rendering, sound, controller and I/O code for Windows that duplicated what we'd created on the N64.
At one point, we were trying to find some problems with the camera behavior, so we created a flying camera object that coincided with the real camera. It looked like an old Hollywood camera, though the lens cap, reels, everything was just flat black. Then, we'd set up a fixed camera and watch what the game camera would be trying to do by observing where the flying camera went.
Time passed, and we'd forgotten about the added camera altogether. Then, as we were approaching a critical milestone, we went to bring the N64 build up to date... and the screen was black.
The game seemed to be playing, the menus were there, and the framerate counter was up, but - black. The Z-buffer had a constant value all the way across it, meaning there was some mysterious polygon that was exactly covering the screen all the time.
We were there until something like 2am, trying to figure out what the hell was going on. We were risking blowing this milestone, and with that, taking on a pretty hefty late delivery penalty.
We finally figured it out. Stepping back... well, the N64 engine didn't support back-face culling at that point, whereas the Windows engine did. So what's the upshot of the whole thing?
We'd left the lens cap on.
Says the RIAA: When you EQ, you're stealing bass!
Never trust M$? I'm sorry but it's clearly documented that the asserts are stripped from the release code. The macro to use for code you want to check in debug mode but still execute in release mode is VERIFY()
I'm no fan of Microsoft, but it's a bit easy to blame them for your own mistakes.
True warriors use the Klingon Google
It's really easy to set up a bad DMA chain on most architectures, and when that happens, it can do wonderful damage that's tough to reproduce.
One of the more fun things about it is that DMA generally ignores the MMU completely, so you can consistently trash whatever's at some physical address time and again, ignoring all protection.
Even better, DMA doesn't cause hardware breakpoints, so even if your debugger/system are capable of watching for all writes to a given address or page, it'll still merrily corrupt it.
Even more fun if data has been corrupted, but the correct data is still in the data cache from a previous access, making failures even more unpredictable, often relying on an interrupt or other random bit of interference clearing the cache.
On top of all this, a bad DMA chunk may not manifest itself in an obvious way. The program may crash in random sections well before you realize that the DMA you intended didn't happen, or the program may just keep on running with a single fram graphic glitch or a brief bit of static in the case of DMA meant to go to video or audio hardware. That's easy to miss when you're focussing on the debugger and not the running program.
I've seen products ship weeks late just because of a single hard-to-find DMA glitch.
Says the RIAA: When you EQ, you're stealing bass!
I got stung by that on a Friday before a long weekend in 1984 or 1985. A dirty INT21 hook I was applying to DOS worked on ATs but not on XTs (or was it the other way around?). I had set up a structure on the stack and needed to pass its address to a higher language (prolly K&R C) routine, so PUSH SP seamed like the right thing to do.
Hardly a complex bug, but one where it is non-obvious that a 286 is not a superset of an 86.
Then there was the time I had to download a patch to over a thousand embedded controllers spread over a whole country whose problem was that downloading didn't work.... a truck roll to wach one was not an option. But, that's another story (bootstrapping the fix was horrendously more complex that finding the bug).
You could've hired me.
I worked on code for a satellite ground-control system for a few years. It was all Fortran-5/77 and handled a couple-dozen satellites and ground stations in real-time. The problem was that it was written back in the 60's and the programmers who implemented it had really old slow MV-8000's. There weren't enough spare clock cycles to have decent synchronization between modules so they just depended on different subroutines taking an exact number of cycles to execute so they'd match up with whatever they were talking to. Change a single line in anything and you had to recompile it and time every possible way it could execute. Horrible stuff...
Once I was writing some C code to run on an old Motorolla DSP in an embedded system.
One particular function kept crashing. My debugging tools were very limited in this environment--basically, I had a total of 4 LEDs that I could blink on and off by insert function calls into my code. That and a logic analyzer for when things got really nasty.
Well, things did get really nasty. After reviewing and rewriting that function dozens of times, I finally decided the bug couldn't be in my C code. So I had the compiler spit out the assembly it was generating, brushed up on my DSP assembly, and read through its code on the hunch that there was a bug in the compiler (the compiler was very new and still pretty crappy).
But after spending a couple of days staring at the assembly, I concluded that it was perfectly fine. What else could be going wrong? I started thinking maybe something was going wrong in the link step or in the process of getting the file transferred down onto the embedded controller.
I went and learned more than I wanted to know about XCOFF format and used a little binary file editor to see what the linker output was. Again, everything just as it should be.
I just knew that somehow, what was getting executed was different from what was in the file. So we fired up the logic analyzer, and attached it to the DSP, and set it up to watch the contents of the address bus and data bus at each clock cycle.
This is incredibly painstaking--you have to look at 32 lines of step-functions to read off the address, and 48 lines of step-functions to read off the data (yes, it was a 48-bit data register; go figure) for EACH OPCODE. This will make your eyes bug out in a hurry.
But even then--nothing was wrong! The opcodes being loaded into the processor were exactly what they should be. But on this one particular test-and-branch instruction, the processor would just start to go crazy (address and data lines full of random noise; had to be powered down).
I dug out the processor manual and triple-checked the opcode name and number, addressing mode, and operands. Every bit was correct.
In utter frustration, we decided to call Motorolla to see if we could get some assistance from them. After going through a small maze of transfers, we finally ended up talking to the right person who knew (and quickly told us) that:
That particular addressing mode, when used with that particular
opcode, was known to throw the DSP into a hosed state.
It was a bug in the processor itself. The solution was simply to change my code to use a different addressing mode, and all was well.
Good books, MS tools are weird mainly because I like my commandline a bit too much, but they publish some damn fine books about programming.
This is also a case of found easily by reading the output of gcc -E, you best friend when debugging code that has macro's anywhere near it.
I had never changed a flag that said I wanted it to strip them, so I assumed it wouldn't. Never trust M$
I hate to post a flame, but RTFM. On every compiler or tool you ever use, spend several days reading the manual and all associated docs you can find. Knowing how the compiler works, and how all the tools work is a hallmark of all the finest programmers I know. I used VC++ a handful of times 5 year ago, and I could have told you the asserts were stripped in from release mode. All you have to do is look at the full list of options it puts on the command line. That's relatively easy to find in the menuing system on VC 4.0 (the only version I used). The -DNDEBUG=1 flag turns off asserts.
Kirby
PS: Other then the keyboard and mouse I use, I haven't used a Microsoft product on a daily basis in years. It's about craftsmanship, and knowing your tools.
Cute :).
Laurent
Two situations come to mind.
The first is not so much a specific bug that was difficult to find, as it was the general means that I was forced to use to locate bugs. (Yes, I found quite a few.) Back in the early 80's I was working at IBM in the QA group responsible for testing their VM operating system. We were tasked with taking the existing VM OS and not only were we to improve its performance on multi-processor systems, we were also to improve its reliability by doing extensive reviews and testing. I was responsible for testing the free storage allocator.
Some background: For those who may not be aware, that whole operating system was written in IBM BAL (Basic Assembler Language) using the 370 instruction set. The VM operating system created a virtual machine environment for each user - thus producing the appearance of a machine, identical in [almost] all respects to running on the bare hardware. Those few differences pertained to some optimizations in the the virtual memory management's use of PTLBs (page table look-aside buffers) among others.
So, I needed to test memory allocation on the bare hardware to make sure that it worked okay. Once that was nailed down, I had to test memory allocation when VM was running on VM. But, there was yet another set of optimizations that I needed to test when a VM was running on VM running on VM (i.e. a "3rd level" VM).
It was not possible to just issue VM commands to test the various code paths. So, each test consisted of setting hardware breakpoints at the appropriate hex offset, and single stepping through these allocations. By the time I got to testing the 3rd level VM code, I was tracing and debugging these PTLB calculations and allocations, single stepping through instructions in hexadecimal and verifying multiple levels of indirection to memory pages where those calculations were also in hex. Those were the days! Just a year or so out of college, and I had all to myself a multi-million dollar mainframe computer that could support several hundred people!
The other bug was actually a specific bug that caused much early hair loss. I was working at a place where there was a lot of new employees come on board. Along with that, new departments were being formed, people were being promoted and moving to different groups, and there was a great deal of office moves as a result. So, it soon became a problem finding somebody's office. "Gee, wasn't Mary here just last week?"
Sensing a need, I wrote up a quick REXX program (yes, this was back in the early 80's, too) which did data aquisition through forms and supported the generation of reports sorted by various categories: Name, Department, Room Number, etc. This was pretty straightforward and in a couple days I'd gotten it coded, tested, and all the data populated. As there were only a few hundred people I used a flat file (the other alternative was creating a DB2 database and disk space was very dear back then!)
Rolled it out and received much positive feedback. Except, there was one person who noticed there was an error in the ordering of room numbers. See the format was: "Floor: 1, 2, or 3"; then the "building wing: compass direction: N, S, E, or W", and lastly, "room number: 2-digit number". As this was a rapidly growing organization, managers would be allocated several empty offices in advance for the people they'd hire during the next quarter. Also, the building had just been constructed and some areas and sometimes whole wings were still not yet ready for use so there were many gaps in the data. Here is a selection of the kinds of results I saw for the room numbers, in ascending order:
Why are the rooms in the South wing nicely ordered, but things are really messed up for the East wing? I spent HOURS and HOURS trying to figure this one out. See if you can tell what the problem is before reading ahead.
The problem? There is a single set of comparison operators <, =, and > and they just did the right kind of comparison depending on the data type of the operands. Well, here I was thinking the data was text, but the program was making the comparisons as if these were numbers that contained an exponent part