Properly Testing Your Code?

lowlytester asks: "I work for an organization that does testing at various stages from unit testing (not XP style) to various kinds of integration tests. With all this one would expect that defect in code would be close to zero. Yet the number of defects reported is so large that I wonder how much testing is too much? What is the best way to get the biggest bang for your testing buck?" Sometimes it's not the what, it's the how, and in situations like this, I wonder if the testing procedure itself may be part of the problem. When testing code, what procedures work best for you, and do you feel that excessive testing hurts the development process at all?

Comments

Code Review, Code Review, Code Review

[JohnT]

My experience has shown that the number one way to find defects is code reviews performed by other developers who can read the code and also understand the intended functionality. This will catch 90% of all defects before they are even released to QA.
For more information, the developers bible (IMHO) Code Complete (available on Amazon and elsewhere) has some good information on testing strategies and some hard numbers on effectiveness of testing. Good luck.

Re:Code Review, Code Review, Code Review

[ssclift]

Again, violent agreement. Why? Testing is basically just writing the code again, only in a more restricted form. You take a known input, and then program the output expected (rather than derive it another way) and then compare the two implementations.

Inspection, on the other hand, compares the program with it written in another form: human language. Since human language is generally too vague to execute automatically, the only way to test the equivalence of the two is to inspect.

By far the best inspection book is Software Inspection by Tom Gilb. His very generous web site contains a ton of supplementary material.

Remember, proving two statements are the same is the halting problem, and is NP-complete (i.e. you must check all possible solutions). Testing is a measure of code against code, inspection a measure of code against requirements. Together they kill a lot of bugs because they find different discrepancies between three statements of the same problem.

Close the loop

[edhall]

The object of finding bugs isn't to result in fewer bugs by fixing them. It's to result in fewer bugs by not writing them in the first place. The developers need to review found bugs on a regular basis, with the objective of changing development methods to avoid them in the future.

It's all fine and good to say "don't write buggy code in the first place," but this sort of feedback is the only way to get there. What makes this so hard in many organizations -- aside from the usual disrespect many developers have for QA people -- is that developers fear that this process is some sort of performance evaluation. As soon as this happens, the focus shifts from finding better processes to defending existing processes: "It's not really a bug," "There isn't really a better way of doing that," "We just don't have time to do it the 'right' way," and so on.

This is why the feedback needs to be direct from QA to the developers, who are then tasked to categorize bugs and develop recommendations for avoiding them. It's the latter that is the "product" required by management, not a list of bugs with developer's names on them. Management should otherwise get the hell out of the way.

-Ed

Re:the best way to test code...

[Xentax]

I agree -- our own company suffers from giving less effort on code reviews than most of us know we should. People try to save time by under-planning for code reviews, but that saved time is always lost at least twofold in uncaught bugs, extra time for optimization, and so on -- all things that would be identified in a solid code review.

Identify the people in your company that have the best "critical eye" for each language you develop in -- and see to it that they get the time they need to really critique code, either during implementation or at least before starting integration testing (preferably before unit testing, actually). It may be hard to convince management the first time around, but if you account your hours precisely enough, the results WILL speak for themselves, in terms of hours spent overall and on testing/integration.

Xentax