Slashdot Mirror
How Do You Deal w/ "Heisenbugs"?
horos1 asks: "I was wondering how people out there deal with 'Heisenbugs': bugs that have no logical, programmatic cause (mostly from C and C++ programs and especially threaded C/C++ programs), that may change or disappear if you modify the state of the program.
For example: we have a multi-threaded C++ app which cores in about
5 places, at a different memory address each time, and which disappear if we turn threading off. They seem to be caused because of a memory overrun error, but this too is exceedingly hard to fix with tools like purify, because they tend to give several 'false positives' on memory errors, as well as core when we link with certain libraries. Anyways, this is getting very annoying... Any help with this, as well as pointers on how to deal with bugs like this would be greatly appreciated."
You could use a "Heisenburg Compensator".
Spinlocks
Mutexes
Critical Sections
printf() statements
Breakpoints
Logging
Also keep in mind that many of C and C++ built-ins are not necessarily thread-safe.
Dancin Santa
Several useful suggestions have already been posted. One additional one: Initialization.
Rig macro wrappers for malloc() and calloc() and the like that initialize new memory to 0xdeadbeef if you have a memory debug flag active. This should make your program crash more abruptly when it starts doing something questionable.
This won't help much for finding the race condition or non-thread-safe code causing the problem, but it may give you some idea of what's being stomped, and make the heisenbug more predictable.
Patient: "Doctor, when I write multi-threaded programs in C++ they dump core all over the place and I don't understand why!"
Doctor: "Don't do that."
I know it sounds glib, but it really is the heart of the matter. Writing proper multi-threaded programs is difficult and takes a fair amount of skill. If you or your programming staff don't have the skill to do it, you are better off sticking to something less challenging. This isn't meant as a put down, either: there are plenty of simple solutions to problems that, at first blush, may look like they require threads (or dynamic memory allocation, or any of a number of other complex and error prone tactics). You could replace your multiple threads with a polling loop and a switch statement, or spawn completely separate processes and communicate through pipes (you might be supprised how much performance you can get out of either solution compared to the threaded code).
If you feel that you really must use the cool threaded code (or a complex, dynamically allocted data structure, or whatever) then you may need to dedicate a week or two to a carefull re-examination of the code and the accompanying re-implementation to eliminate race-conditions/memory-leaks/mutual exclusion errors/etc. While there are a lot of cool debugging tools out there, that can help you find some kinds of errors, there is really no substitute for a deep and thorough understanding of your code. You can either spend the time understanding the mess you have, or try replacing it with something less complex but easier to maintain. (but, maybe, harder to extend/scale/etc.)
Here are some of my favorite high-tech problems with low-tech solutions:
- Multi-threading: multiple processes or polling loop with switch statement.
- Search Trees or other tree structures: hash tables.
- Linked lists or other linked structures: dynamically allocated arrays (double when size>=length, halve when size<length-n, like Java vectors)
- Complex decision logic: state machines.
Admittedly, sometimes you really do need to go with the more complex solution, but it is best to avoid it whenever possible. Besides, you can always put the complex stuff in the next version.Debugging tips
0. If you are running linux: apply the patch that causes the thread that had the segfault to dump core. The default Linux semantics (under 2.2.x at least) are for the threads to exit STARTING with the one that had the problem. Then, the LAST thread dumps its program counter / stack info into your core file. The result: you get what look like "random" crashes when really they aren't very random.
There is a patch which fixes this behavior. try this patch .
1. See if you can get your program to crash in a debugger or dump core. I presume that you are getting this by your comments.
Record the places that you get crashes. Each time anyone gets a crash, have them record where, as best as they can.
Try to figure out what's getting overwritten, even if it's not clear when or how.
3. Try to increase the frequency of the crash (e.g. by running on an SMP machine). this usually provides people with more incentive and a better chance to test if any given change really fixed things.
4. The next few hints fall into the category of "reduce the problem code". It sounds to me like you don't have a good feeling as to where the problem is happening. Try to eliminate sections of the code, by any means necessary. examples include one big lock to force serialization, test programs that only excercise certain modules of your code, etc. I know there is often a temptation to just jump in, but some extra scaffolding to reduce the possible problems is almost always valuable on hard debugging problems.
Reduce the amount of data shared between threads. We are using a message passing interface, where each thread more or less has its own data. This has been a big win. We often copy data before passing on to the next thread, just to make sure.
5. Understand what is and isn't thread safe in your libs. For example, did you know that it is almost impossible to make the C++ std string thread safe, without changing the implementation? that's because the implementation is copy on write. So, even when you're not sharing any data between threads, you are....
I hope some of these help. threading problems aren't easy, particularly in c++...
"Heisenbugs", as you call them, are almost always the result of memory management bugs. Arrays boundaries are being overwritten, null pointers are being accessed, freed memory is being used, or some other memory is being misused. It's a very common problem.
Now, one answer to this problem is to use smart pointers and automatic garbage collection. That won't help array boundaries, but you can use an equivalent wrapper for arrays. It's not a bad practice to get into for large-scale C++ development.
Another solution, and one I have found to be *incredibly* useful, is to use a memory profiler. There are loads of memory profilers on the market today. Visual C++ has one included, the GNU Foundation has one, and loads of commercial companies offer profilers. Most of the commercial ones have nice functions, such as pretty (and useful) graphical displays and the ability to profile code that has already been compiled. I remember profiling Solaris (the OS itself) back in the early 90s, finding loads of memory leaks and memory management problems.
Even though I'm not a betting man, I would lay money that if you rid yourself of memory management problems, your code will no longer contain these "Heisenbugs".
--Be human.
Consider using something other than C/C++.
You can never be too paranoid. Check all passed parameters for validity. Do range checks on indices. Verify that a pointer to foo actually points to an object of type foo.
Don't use dynamic memory allocation.
Have a recovery strategy for resource exhaustion.
Check for things that "can't happen".
For every global data structure, you should be able to say which procedures access/modify it, and under what conditions.
Read some good books on real-time and concurrent programming.
Mea navis aericumbens anguillis abundat
I have a lot of can't happen checks. They never, or rarely trigger (anymore). I have to maintain code written by someone less paranoid about buggy hardware (for 100 triggered can't happen except on hardware errors, I've fixed 110 software bugs. This on new untested hardware where I have found hardware bugs by other means.
I've also fixed several crashes because I knew the code well enough to know where it should be an what it should be doing. Once I proved it wasn't in that state I had to figgure out why not, and from there the fix was easy. Unfortunatly figgureing out why I was in the wrong state is hard.
Duplicatable problems are easy to fix. If you can crash in one of 5 cases, then splatter printf's all over those areas. Consider writting your own printf which just writes to memory, not the screen so you don't block. Then when your program crashes you pull that memory from the core file and you know where each function was last. just knowing what function each thread was in last is a big clue.
Finially, code inspections are a must. Get some good programers who have never seen that section of code and have them inspect it. If nothing else it will assure that your comments are meaningful before the programmer quits.
My two favorite examples of past debugging:
www.HearMySoulSpeak.com
Quite seriously. There are lots of debugging things you could do where first thought is "Ug, that would take me forever." True. It might. But if you start now you'll be done that much sooner.
Consistency, too.
The worst thing in the world for debugging is to sit back, go "Hmmmm, I wonder if this is it?", throw in a debug statement and see what happens. That's what you do when you don't really want to find the problem (I used to do that in the old days when I knew that firing up the system took 10 minutes each time). Be thorough and consistent. If you know that your program crashes in one of 12 methods, don't assume it happens in method foo() and only breadcrumb that one -- sit down and do them all. You can always take it out later.
(I have something of a reputation on my team that after a bug has stumped the developers for days, I am given it and I solve it in hours. Normally the first thing I do is turn debugging statements on for every component. Sure, it takes me a little while to do it, and when I'm done I get megs of output, but I almost always find the problem straightaway, too.)
www.HearMySoulSpeak.com
Built in test code that periodically checks structures to make sure the values in them make sense is also very useful and easy to do in C++
Also, having a few "signature" fields in your structures (essentially, unused data items that are initialized to a specific value and never changed) can help locate tough-to-find memory overwrites. Just check the signature field of a structure before using the other data in that structure. If the signature isn't right, the memory has been overwritten
It can also be helpful to have a pointer check routine that looks at the value of a pointer and determines if it is legal for the platform you are on. For example, checking to make sure it falls in a range where the hardware actually has memory, or making sure it doesn't overlap system reserved areas
Milo from Kangaroo Koncepts
when the act of debugging itself changes or masks the bug. The poster is refering to non-deterministic bugs, such as can easily happen when multithreading.