Debugging

dwheeler writes "It's not often you find a classic, but I think I've found a new classic for software and computer hardware developers. It's David J. Agan's Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems." Read on for the rest. Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems author David J. Agans pages 192 publisher Amacom rating 9 reviewer David A. Wheeler ISBN 0814471684 summary A classic book on debugging principles

Debugging explains the fundamentals of finding and fixing bugs (once a bug has been detected), rather than any particular technology. It's best for developers who are novices or who are only moderately experienced, but even old pros will find helpful reminders of things they know they should do but forget in the rush of the moment. This book will help you fix those inevitable bugs, particularly if you're not a pro at debugging. It's hard to bottle experience; this book does a good job. This is a book I expect to find useful many, many, years from now.

The entire book revolves around the "nine rules." After the typical introduction and list of the rules, there's one chapter for each rule. Each of these chapters describes the rule, explains why it's a rule, and includes several "sub-rules" that explain how to apply the rule. Most importantly, there are lots of "war stories" that are both fun to read and good illustrations of how to put the rule into practice.

Since the whole book revolves around the nine rules, it might help to understand the book by skimming the rules and their sub-rules:

1. Understand the system: Read the manual, read everything in depth, know the fundamentals, know the road map, understand your tools, and look up the details.
2. Make it fail: Do it again, start at the beginning, stimulate the failure, don't simulate the failure, find the uncontrolled condition that makes it intermittent, record everything and find the signature of intermittent bugs, don't trust statistics too much, know that "that" can happen, and never throw away a debugging tool.
3. Quit thinking and look (get data first, don't just do complicated repairs based on guessing): See the failure, see the details, build instrumentation in, add instrumentation on, don't be afraid to dive in, watch out for Heisenberg, and guess only to focus the search.
4. Divide and conquer: Narrow the search with successive approximation, get the range, determine which side of the bug you're on, use easy-to-spot test patterns, start with the bad, fix the bugs you know about, and fix the noise first.
5. Change one thing at a time: Isolate the key factor, grab the brass bar with both hands (understand what's wrong before fixing), change one test at a time, compare it with a good one, and determine what you changed since the last time it worked.
6. Keep an audit trail: Write down what you did in what order and what happened as a result, understand that any detail could be the important one, correlate events, understand that audit trails for design are also good for testing, and write it down!
7. Check the plug: Question your assumptions, start at the beginning, and test the tool.
8. Get a fresh view: Ask for fresh insights, tap expertise, listen to the voice of experience, know that help is all around you, don't be proud, report symptoms (not theories), and realize that you don't have to be sure.
9. If you didn't fix it, it ain't fixed: Check that it's really fixed, check that it's really your fix that fixed it, know that it never just goes away by itself, fix the cause, and fix the process.

This list by itself looks dry, but the detailed explanations and war stories make the entire book come alive. Many of the war stories jump deeply into technical details; some might find the details overwhelming, but I found that they were excellent in helping the principles come alive in a practical way. Many war stories were about obsolete technology, but since the principle is the point that isn't a problem. Not all the war stories are about computing; there's a funny story involving house wiring, for example. But if you don't know anything about computer hardware and software, you won't be able to follow many of the examples.

After detailed explanations of the rules, the rest of the book has a single story showing all the rules in action, a set of "easy exercises for the reader," tips for help desks, and closing remarks.

There are lots of good points here. One that particularly stands out is "quit thinking and look." Too many try to "fix" things based on a guess instead of gathering and observing data to prove or disprove a hypothesis. Another principle that stands out is "if you didn't fix it, it ain't fixed;" there are several vendors I'd like to give that advice to. The whole "stimulate the failure, don't simulate the failure" discussion is not as clearly explained as most of the book, but it's a valid point worth understanding.

I particularly appreciated Agans' discussions on intermittent problems (particularly in "Make it Fail"). Intermittent problems are usually the hardest to deal with, and the author gives straightforward advice on how to deal with them. One odd thing is that although he mentions Heisenberg, he never mentions the term "Heisenbug," a common jargon term in software development (a Heisenbug is a bug that disappears or alters its behavior when one attempts to probe or isolate it). At least a note would've been appropriate.

The back cover includes a number of endorsements, including one from somebody named Rob Malda. But don't worry, the book's good anyway :-).

It's important to note that this is a book on fundamentals, and different than most other books related to debugging. There are many other books on debugging, such as Richard Stallman et al's Debugging with GDB: The GNU Source-Level Debugger. But these other texts usually concentrate primarily on a specific technology and/or on explaining tool commands. A few (like Norman Matloff's guide to faster, less-frustrating debugging ) have a few more general suggestions on debugging, but are nothing like Agans' book. There are many books on testing, like Boris Beizer's Software Testing Techniques, but they tend to emphasize how to create tests to detect bugs, and less on how to fix a bug once it's been detected. Agans' book concentrates on the big picture on debugging; these other books are complementary to it.

Debugging has an accompanying website at debuggingrules.com, where you can find various little extras and links to related information. In particular, the website has an amusing poster of the nine rules you can download and print.

No book's perfect, so here are my gripes and wishes:

1. The sub-rules are really important for understanding the rules, but there's no "master list" in the book or website that shows all the rules and sub-rules on one page. The end of the chapter about a given rule summarizes the sub-rules for that one rule, but it'd sure be easier to have them all in one place. So, print out the list of sub-rules above after you've read the book.
2. The book left me wishing for more detailed suggestions about specific common technology. This is probably unfair, since the author is trying to give timeless advice rather than a "how to use tool X" tutorial. But it'd be very useful to give good general advice, specific suggestions, and examples of what approaches to take for common types of tools (like symbolic debuggers, digital logic probes, etc.), specific widely-used tools (like ddd on gdb), and common problems. Even after the specific tools are gone, such advice can help you use later ones. A little of this is hinted at in the "know your tools" section, but I'd like to have seen much more of it. Vendors often crow about what their tools can do, but rarely explain their weaknesses or how to apply them in a broader context.
3. There's probably a need for another book that takes the same rules, but broadens them to solving arbitrary problems. Frankly, the rules apply to many situations beyond computing, but the war stories are far too technical for the non-computer person to understand.

But as you can tell, I think this is a great book. In some sense, what it says is "obvious," but it's only obvious as all fundamentals are obvious. Many sports teams know the fundamentals, but fail to consistently apply them - and fail because of it. Novices need to learn the fundamentals, and pros need occasional reminders of them; this book is a good way to learn or be reminded of them. Get this book.

If you like this review, feel free to see Wheeler's home page, including his book on developing secure programs and his paper on quantitative analysis of open source software / Free Software. You can purchase Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems from bn.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.

Comments

i hate debugging

[Anonymous Coward]

cause when i do it, it is often re-bugging

Re:i hate debugging

[Frymaster]

cause when i do it, it is often re-bugging

we have a special process we call "debuggery". debuggery - maxims and arrows

1. be hostile: your application was your friend - your baby. you gave it life. well, no longer. now your application is your enemy. do you admire the intricate house of cards you have built like hiram abif? don't. you have a glue gun now and you are going to do a little explaining about who is boss here! your app is taunting you - it's thinking "what does a chemical/analogue hack like that have that i don't?" well, i'll tell you: an index finger. suitable for hitting the "del" key. make this crystal goddamn clear!
2. kludge everything! the debug stage of the development life cycle is all about kludges. we call it klop - kludge-oriented programming:

   kludge foo = new kludge(specialCase bar);

   you've written that. the debugging phase comes at the end of a project. ie the part closest to the deadline when clueless suits and moneyment confuse line count with product. the pressure is on. the company is on the line. are you going to walk into the glass tower and pitch to the vc's about how yr going to have to go back to the uml's and rebuild x? good luck! can i have your job when you're done? get the tape, get the staples, get the glue.

3. blame others: teamwork is just a code word for being the shepherd to a flock of scapegoats. if you were smart, you'd have been working on cultivating a culture of accepting blame early on in the cycle. this is espescially effective if yr building a client/server thingy. establish early on that most of the failures are on the client(server) side. whichever one you're not writing.

   make yourself documentation czar if possible - then abuse the position to retroactively assign blame to other team members ("the docs explicitly state that we use roman numerals" - "gee, i don't remember that" - "well tough. get coding").if you set it up right you can build an army of debugging minions to do your kluding for you while you, uh, read slashdot...

4. redefine feature sets. the client is a clueless little doughboy who can't tell his ass from his operating system anyway. he's been flaking you on the spec-n-req all year. turn those tables! if a feature is buggy, yank it. if there's a complaint, reference the client to some vaguely-related advisory somewhere (trust me, he won't read all the way down). if he complains say "in light of advisory x we strongly adivse against implementing _______ (feature). a work around may be possible at a future point and we are more than willing to calculate the billing for that additional work now."

all that and echo will solve all yr debuggery problems.

Re:i hate debugging

[aled]

I like your methodology and I would apply it in the case I do any debugging. I don't need it right now, because I didn't found any bugs to fix. It may be related to not doing any testing. Testing is dangerous, something may break. The only maxim I need is "if it compiles, it works". Modern compilers check your program, if I would make some silly error it just catches at sight. And static type checking guarantees the program works right allways. I never look the code again once it compiles. Isn't programming wonderful?
I have to leave now, I have to deliver more quality software.

#9 is wrong

[Anonymous Coward]

What if someone else fixes it?

yuck

[theMerovingian]

Make it fail: Do it again, start at the beginning, stimulate the failure, don't simulate the failure, find the uncontrolled condition that makes it intermittent, record everything and find the signature of intermittent bugs, don't trust statistics too much, know that "that" can happen

Isolate the key factor, grab the brass bar with both hands (understand what's wrong before fixing), change one test at a time, compare it with a good one, and determine what you changed since the last time it worked.

Does anyone else feel dirty after reading this?

The Slashdot 9

[grub]

(this IS slashdot after all)

1) Check your registry.
LINUX ain't got no registry crap!

2) Check your FAT32/CXFS filesystems.
LINUX is JOURNALLED and can do that in the background!

3) Verify your drivers are current.
LINUX is stable with drivers written in COBOL back in the 50's!

4) Defrag your disks.
Defrag?! You must be a WINDOZE LOOSER!!

5) Check your connections on the back of the PC.
HAHAHA! LINUX does that AUTOMATICALLY!!! LOOSERSSSSS!!!

6) Are your cards well seated? Power down and reseat.
HAHAHAAHA! LINUX can HOTSWAP EVARYTHING EVEN CPUs, LOOSERS!

7) Is your OS up to date? Perform a Windows Update.
HAHAAHAHA!!! LINUX can update itself automatically cuz of its LEET HEURISTICS and COOLNESS that MS aint got, LOOSERS!!!

8) Start in "SAFE MODE"
HAHAHA! What's the other? UNSAFE MODE!?!?! LINUX is always safe, LOOSERS!

9) Reinstall Windows.
HAHAHAH! LINUX NEVER NEEDS INSTALLING! Pour the blood from a freshly sacrificed penguin on the disk and it installs AUTOMATICALLY THROUGH AIR!!!!! LOOOOOOOSERS!!!!!

I don't need a book...

[garethwi]

...to learn how to debug. I only need my own sloppy code.

Re:I don't need a book...

[Dukael_Mikakis]

Yeah who needs a book?

System.out.println("1");
ComplexClassInstantiator _cci = new ComplexClassInstantiator((UtilType)ClassGrabber.ge tObjectFromDefaults(_a, _b, _kl1, _z56), new UtilSocket(_p23876, _p5541), new Runnable() { public void run() { runDataSetAnalysis(_p1, _p2, _paramClass); } });
System.out.println("2");

Output: 1
[Error message]
So obviously the error is in the line between the two print statements.

So, I repeat, who needs a book?

Re:I don't need a book...

[Anonymous Coward]

If that's how you code all the time, you do.

The first law of debugging

[ToSeek]

"The most likely source of the current bug is the fix you made to the last one."

Hey...the chicken bones are a valid fix too....

[Dr_Marvin_Monroe]

These "rules" are great, but nothing beats the mystic power of a little goat blood and chicken bones waved over a misbehaving system.

Without these, the average user might be tempted to try and fix it themselves.... Next thing, my job is being "offshored" to a phone bank in India.

No, the chicken bones and a little incantation will keep my job right here, where it belongs.

Re:Hey...the chicken bones are a valid fix too....

[Saeed al-Sahaf]

i love what RMS has done for Free Software. the comment above is a joke, take it as such.

I might consider your request if your comment was... funny?

And the final solution

[aliens]

10) Hammer.

if 10 fails

11) Shotgun.

Congrats problem solved, human destressed.

Re:And the final solution

[Rahga]

Most solutions only go up to ten.... These go to eleven.

Rule 0

[Anonymous Coward]

0. If you're a software guy blame it on hardware, if you're a hardware guy blame it on software.

0.1. Blame it on the user.

0.2. Blame it on your colleague.

0.3. Blame it on your manager.

0.4. Yell at the computer and tell it to work dammit!

0.5. Put head on keyboard and sob.

0.6. Read Slashdot.

0.7. Post on Slashdot.

0.8. Call it a feature not a bug.

Top 10 Rules of Debugging

[ackthpt]

10. Code is _always_ Beta. It's never done until it's no longer in use or support no longer exists.

9. The better the SDK, the more sophisticated the bugs.

8. There's always more bugs in the other guy's (girl's) code.

7. Declaring code bug-free is asking for it to fail at the worst possible time with the greatest visibility.

6. A good design is as likely to have bugs as a bad one. Bugs are equal opportunity.

5. Debugging time is inversely proportional to coding time.

4. If it works the first time, there's a bug, but you won't find it until you roll it out.

3. Debugging is fun. Really! It's when you run out of bugs that you should wonder if you got them all, that's not fun.

2. The most difficult bugs to find are in the most straightforward looking code.

1. That's not a bug, that's a feature.

Re:Top 10 Rules of Debugging

[hburch]

If it works the first time, there's a bug, but you won't find it until you roll it out.

There truly is nothing more scary than a program that works the first time. You know there's a bug, but you cannot find it. It's sitting there, silently laughing at your vane attempts to distrurb it from it's hiding place with your testing.

The bug plots against you, with an evil grin on it's face, biding its time until you finally decide that maybe, somehow, your code actually was correct the first time. Then, just as you stop holding your breath everytime you here about that code piece being executed, BAM! Three critical, blocking bugs are filed against that code.

As you open the code, the bug sits there like a blinking red light of impossibility. You clearly did something wrong. It cannot possible work, much less ever have worked. Somehow, all those tests that your program passed before fail miserably. You must rewrite the entire program from scratch because of your flawed logic in the design.

Fear, programmers of the world! Fear the lurking bug that has learned patience!

Re:Heisenbugs...

[AndroidCat]

When I was working on arcade games, we had a sure-fire method of making bugs go away. However, shipping each coin-op game with an engineer and $40k worth of testing equipment connected to it wasn't really cost-effective.

Effective Technique

[Rick the Red]

I find the best way to uncover bugs is to do a demo for your boss's boss.

Re:Effective Technique

[isomeme]

I knew I'd really become a software manager when I gained the ability to cause code to fail by standing behind the person trying to demo it.

Re:Effective Technique

[MurphyZero]

When preparing a demo, remember never to say the following: "Have a look a this. I (1) finally got all the bugs out or (2) got it to work now." Either comment is a surefire way for it to fail immediately.

Anecdote

[ackthpt]

A dishonest computer repairman dies and finds himself in Hades. The Devil smiles and says, "we've been waiting for you and have your place for eternity all ready." The repairman shudders, but follows the Devil as he is lead down a tunnel. They pass several doors along the way and the repairman peers through portals to see the other condemned up to their necks in feces, languishing in pools of acid and being proded by lesser demons with red hot pokers. The devil finally comes to a door and rubs his hands together. "Here you are, your eternal damnation." The repairman cringes as the door is flung open, but sees only a vast cavern filled with PC's, Mac's, Sun SparcStations, etc. "What? That's it?", he enquires, "I shall spend eternity fixing these then?" "Oh, yes", says the Devil. "Well that's not so bad," the repairman cracks his knuckles and strides into the cavern. "Just one thing", says the Devil as he closes the door, "they've all got intermittent problems."

"AAAHHHHHHHHHHHHH!!!!!!"

The Three R's of Windows Debugging

[iguana]

Retry
Reboot
Reinstall

And that's why I love having source code!

The first rule of debugging

[XiChimos]

No,

The first rule of bugs is, you do not talk about bugs. The second rule of bugs is, YOU DO NOT TALK ABOUT BUGS!

-From a memo to Microsoft's new employees

riiiight

[Anonymous Coward]

"9. If you didn't fix it, it ain't fixed: Check that it's really fixed, check that it's really your fix that fixed it, know that it never just goes away by itself , fix the cause, and fix the process."

Obviously, the author has never used C++Builder.

Re:Rule #10

[Dukael_Mikakis]

... either that or add a comment right before the pertinent code:

/* Code used with permission: Microsoft Corporation */

(Not that your clients would have your source code to look at, but ...)

Common sense strikes again

[ValentineMSmith]

Was it just me or did anyone else get to the bottom of that bullet list and feel let down? Here I was expecting some sort of earth-shattering revelation, and all that the list shows are common sense rules. At the risk of sounding elitist, maybe this was epiphany for someone else. Lord knows it would be a revelation in our QA department, where the list consists of exactly one rule:

1. -Grab a programmer

But still... Someone made money off of that? Heck, look for my new book next week, "Walking to Peoria in 3,976,554 steps", with each step being "Place your rear foot 1.5 feet in front of your front foot."

Unit Test?

[MooseByte]

What is this "unit test" you refer to? If we consider our customer base to be a "unit", does that count?

Yours Truly (All Belongs To Me),

Bill

Re:Windows *Debugging*

[marklark]

Windows Debugging Steps:

1) Re-boot.
2) Re-install.
3) Re-format, Re-boot, Re-install.
4) Re-peat. ;^/

Re: Heisenbugs...

[Alzheimers]

Hardware: The part of the computer that you can kick.
Software: The part of the computer that can kick you.

Re:One Rule For 90% of Bugs

[Anonymous Coward]

Damn, you're pretty lucky if you can tell your customers to run the program with the -log switch, and they have the slightest idea what you're talking about.

I, on the other hand, have to tell my customers "OK, click on the start button... the button in the bottom left, that has start written on it... no, it's not missing, it's right there ('Ohhh, START!' they say). OK, now find out where the hell Microsoft decided to put the command prompt shortcut this week...." (because a helpful sysadmin disabled the run command, but every user can still run CMD...). This process can take up to an hour, and I'm still not even sure if they've typed -log or _log.

Re: Heisenbugs...

[aled]

Computers are deterministic

Oh god, a computer calvinist!

Let me tell you about the group of girls I hear telling the profesor how they were debugging the timing loop to measure an analogic signal, stepping through the loop code in the debugger. Free will kills determinism!

Re:favorite quote: discovery of debugging

[JPZ]

As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs.

-- Maurice Wilkes, 1949

55 years later, programmers are still spending a large part of their lives finding bugs and fixing them...

Even worse, they're probably still spending time finding and fixing bugs this Wilkes guy introduced...