How Would You Improve Today's Debugging Tools?

redelvis asks: "I recently came across an article by MIT's Media Lab on 'The Debugging Scandal and What to Do About It'. It's a few years old now, but it really got me thinking about how little the debugging process has improved over the last 5,10 or even 30 years. I have developed applications using modern IDE debuggers such as Borland's JBuilder, Microsoft's Visual C++, as well as standard tools like gdb and jdb. Despite the slick graphical interfaces, nice thread stack traces and local variable browsers, I still make sure I have on hand plenty of notepads, graph paper, pens and pencils so I can try to build up a picture of what state the program is in and to help me play detective in pinpointing what is going wrong with my (or other peoples) programs. Do other developers have similar problems? Do they find modern IDEs and debuggers have shortcomings in helping track down bugs? What would make a better debugger? Why do you think so much effort been invested in areas such as advanced modelling tools but so little in improving debugging tools?"

Comments

Debugger improvements

[exp(pi*sqrt(163))]

• Reversibility. I'd like to be able to step backwards through code. No, this isn't a joke. Only a small amount of information needs to be stored as each instruction is executed in order to have an emulator go backwards again. You don't necessarily want to run an entire application reversibly. You might want to run a small loop until it crashes and then work backwards to find the cause.
• Easy plugins I want to easily be able to load my own custom code into the debugger. For example I might want a custom routine to conveniently display a custom datastructure, or even give a graphical representation of some data.
• Program control. More cooperation with the code itself. I'd like the debugger to provide a library against which I can link my code so that the code being debugged can control things like which variables are being watched and which breakpoints are active. I could then easily have breakpoints that are enabled only under certain conditions. Especially useful when the best way to detect those conditions is for the code itself to figure it out.

some ideas on why debuggers are in this state

[Shaleh]

a) it is hard to make money on debuggers

b) making a good debugger is definately not easy

c) many developers prefer puzzling out the code on pen and paper then walking it in a debugger. Quite a few I have talked to believe that they make better code if they can solve the problem without resorting to debuggers. This BTW is why Linus does not like kernel debuggers.

d) most of the debuggers are good enough and as we all know from open source, a good enough solution will live forever. The existence of a debugger that mostly works gives most people a reason to spend their time elsewhere.

In my experience I have truly benefited more from programs like Purify or Insure (and now the great valgrind) than I have from a debugger.

My Dream Debugger

[Rary]

Some of these features exist in current debuggers, and some do not. What I want is all of them.

1) The ability to set conditional breakpoints.
2) The ability to see not only a variable's current value, but a stack of all of its previous values.
3) The ability to select a variable's previous value and jump to the line of code that set it to that.
4) The ability to change the value of a variable at any point.
5) The ability to add/change code on the fly.
6) The ability to jump into the debugger at any point in the program, even when I hadn't planned to before running it.
7) Auto-logging of method calls and (optionally) variable values, to be started and stopped as I see fit either while stepping through code or running it.

That's all that comes to mind off the top of my head.

Getting it right the first time

[theonetruekeebler]

Okay, so debugging hasn't gotten any better in the last couple of decades. I gotta say it's because the focus of methodology research has been on the development process itself, and that's an extremely good thing.

The more work goes in before you get to the debugging stage, the less buggy your product will be. A well-designed system does not need as much debugging later. I know I'm idealizing, but I've seen this bear out, especially when version N > 1 comes out and it's been hacked to smithereens because of poor design choices in the previous iteration.

Probably the most amazing bit of self-fulfilling prophecy you'll ever hear on a software project is a manager saying, "We'd better start coding now, because we're going to have a lot of debugging to do."

As for catching things ahead of time, I've always put breadcrumbs in my code to spew to stderr or a Java error console class or wherever. Very easy to #ifdef out later and trivial to turn back on later.

Re:Perhaps....

[Titusdot Groan]

I found that as I started using OO languages and techniques (Objective C and Java + MVC and Design Patterns) I almost stopped using debuggers. I almost always know exactly where the bug is as soon as I see the behaviour and can find it by inspection in a few seconds.

This is also experienced based -- as I get older I use debuggers less and less but my code output has remained constant or improved (even though I don't put in all nighters any more :-)

I'm finding it really hard to get excited by the fact that there is no effort being put into debuggers where as 10 years ago it would have been a big deal.

Perhaps that's also part of the problem -- when I evaluate tools etc. I don't spend a lot of time looking at the debugger these days ...

I don't use debugging tools for much..

[defile]

But I only use debuggers for two purposes.

Purpose 1. Segmentation Fault (core dumped). Uhm, now where did that happen? Whip out gdb, find the line that generated SIGSEGV, and it's usually obvious how it happened. If not, I have it print out a stack backtrace. If I really can't figure it out then, a 5 minute walk around the block and I'll have figured it out as soon as I sit back down.

When writing in high level languages, I'm finding that debuggers are wholly unnecessary. In fact, I can't remember the last time I spent more than 20 minutes trying to track down a bug. *shrug*

It'd be nice if debuggers solved my problems automatically, but I'm really finding that I don't need them. I might even go so far as saying use of debuggers encourages dependency on debuggers, which in turn discourages thinking about the program itself. Not saying that EVERYONE does this, just that some of the best work gets done remarkably well even without debuggers.

The Linux kernel, for example, was largely developed without the aid of a debugger, and the core developers seem to eschew them. Here's a good thread on why the developers don't want to include a debugger.

Purpose 2. On the other hand, debuggers are remarkably good at helping you break program code. With having almost no experience using gdb, I was able to break the license key check on Intel's C Compiler in about an hour. I was amazed at how easy it was to attach a debugger to the compiler and skip the subroutine that performed the license key check. With no debugging symbols to work with. Disassemblers rule. It took another 10 minutes to turn this into a script that could be distributed as a wrapper for icc (called xicc), so all you had to do was set CC=xicc in the Makefile.

Sure I could have used LD_PRELOAD so that time() always returned a date within the trial period, but breaking program code with a debugger is just so gosh darn fun.

Superdebuggers

[jimfrost]

Well, it just so happens that there have been a few debuggers much more functional than the likes of Visual Studio.

I used to work for a company called Saber, who later changed their name to Centerline. We produced a product called Saber-C (and later Saber-C++, and the products were renamed CodeCenter and ObjectCenter).

This product was a C/C++ superdebugger. It started out as a C interpreter developed at Harvard. Now, an interpretive environment allowed a lot of neat features: You could tell whether or not variables had been initialized at reference time, you had complete stack and context information, it was pretty easy to unload and reload code in source modules (dynamic reloading). You also got dynamic type checking, which caught errors in downcasting (for instance). And everyone's favorite, array bounds checking.

With this tool debugging C code often meant: Loading the code, running your test case, and seeing where the debugger complained. The first time you ran your code through it was a humbling experience ... almost all production code has serious (but normally benign) errors.

Unfortunately the tool didn't scale well because it took time to load the interpreted code into the debugger and the heap and runtime requirements of interpreters stressed machines of the time (this was when Sun3s were common, and 3/50s couldn't even have more than 4MB memory).

I think it was 1990 when we released version 3, whose big new feature was object code debugging (it's not really that easy to have object code debugging as well as interpretive code, although we could certainly have implemented it more easily than we did). We of course maintained the ability to dynamically reload code fragments, so you could do a "make", reload the affected modules, and be on your way in a few seconds. This sure beat the several minute wait you normally had with the linker and debugger restart. But what you lost was some of the runtime checks -- including most of what made the product interesting. Still, the ability to have superdebugger capabilities on some code and still scale to large programs was very valuable.

About a year, maybe two years later we got visited by the people who eventually started Purify. They had a neat approach where they'd get most of the benefit of our tool, but in object code, by disassembling the object code, adding extra code to perform checks, and putting it back together. This would give about 80% of the functionality of our debugger but with much lower runtime costs and work even if you didn't have source code.

We ended up re-implementing this functionality such that we had a Purify competitor (and getting sued, and losing, even though Sun later patented the same technique we used).[1] Integrating it into the debugger allowed you to fine-tune the debugging capability you wanted on a module-by-module basis. This was a superb tool.

Now, what are the raw capabilities that made this stuff really neat?

1. Incremental reload of modules. That removes the debugger restart cost (symbol loading is a huge time sink).
2. Supply worst-case scenarios for code. In our tool, we'd initialize heap and stack memory to 0xbfbfbfbf patterns, which tended to trigger bugs in code that had intitialization errors.
3. Track heap allocations. Knowing who allocated something is remarkably valuable, particularly for tracking down memory leaks.
4. Augment the code with error checks that are sensible for the source language. For instance, we knew the type that was assigned to memory so we used that to detect all manner of type mismatches.
5. Augment the code with bounds checking for arrays. The bounds checkers worked with the heap system, which of course tracked the size of all allocations. This caught a lot of off-by-one errors.

The tool also provided two additional interesting features. One was called "action points". Since we had a C interpreter in the product it was no big deal to allow you to type C code into it and run it at arbitrary times. We'd allow you to attach any code you wanted to any line in the program or to just type code at the debug prompt and it would run in the current scope. This was useful for things like adding print statements without recompiling, for instance, and for adding assertions, and many other things.

The other feature was a graphical heap inspector. All this did was display structures as tables of their internal values and allow traversal of references to other structures, which would be displayed (including links between the structures). This made the shape of data a LOT easier to determine.

Several people talked about adding reverse execution. That was actually done for Java by a tools company back in 1996 (maybe 1997). It didn't always work, but it was pretty neat. The problem is that the recording code was pretty expensive and there's a limit to people's patience and what they'll spend on hardware (although I grant that hardware is a lot less of an issue today than it was even in 1997, to say nothing of 1989). Anyway this feature is a lot less interesting if your debugger can find errors at the point they occur.

Anyway, back in 1995 when I started to work with Visual-C++ on a regular basis I was appalled by what Microsoft was calling a "state of the art" debugger. Feh! It was 1985 technology with a pretty face. It really has not improved much since then either, although they do at least have a pessimistic heap allocator now. With their ability to hook in with the compiler they could do a phenomenal job with heap allocation tracking and instrumentation of code with error checks but they don't bother (because, I suppose, they make their money even without it).

Now, using a language with runtime error checking built in - such as Java, C#, or Visual Basic, seriously undermines the amount of effort you have to put into the debugger. Still, I'd like to see a lot better heap debugging features built into compilers and runtime environments.

Hope this gives people some ideas, and that they lead to tools I can use.

[1] One thing that could be done to avoid the Purify patents would be to have the compiler emit the instrumentation. For the OSS world, where you always have the source code, that is eminently practical. Furthermore you can get all the benefits of an interpreter but at higher performance. A big problem with Purify's technology, and the technology we came up with, was that you're doing a lot of heuristics during code inspection. It's easy to miss code or misinterpret code, and of course the code inspector is VERY architecture and compiler dependent. What I would like to see is a gcc/glibc/gdb combination where the compiler, libraries, and debugger all worked together. That wasn't possible back when the vendor controlled the compiler and you didn't have access to library source. Today it is, at least for the OSS world.