Slashdot Mirror
Properly Testing Your Code?
lowlytester asks: "I work for an organization that does testing at various stages from unit testing (not XP style) to various kinds of integration tests. With all this one would expect that defect in code would be close to zero. Yet the number of defects reported is so large that I wonder how much testing is too much? What is the best way to get the biggest bang for your testing buck?" Sometimes it's not the what, it's the how, and in situations like this, I wonder if the testing procedure itself may be part of the problem. When testing code, what procedures work best for you, and do you feel that excessive testing hurts the development process at all?
This is excellent advise. In my experience, the most stable code comes from pragmatic design followed up by pragmatic coding.
Design your system thoroughly. Identify every component, and the minimum interface required for that component. Carefully document that interface (API) - use Design By Contract (preconditions, postconditions and invariants) if possible.
Moving targets mean that the API will almost certainly have to be extended - documentation on the design and intent of the component/API as a whole will reduce the pain of this process. The responsibility for this documentation is shared between the design and implementation phases. Pay careful attention to documenting assumptions made within the code, e.g. ownership of member/global variables.
When it comes to coding, start with a skeleton. Put in the API function/method as defined, then check/assert every pre/post condition. Think about how any parameter could be out of range, or violate the assumptions you make. Once you are happy you're checking for all illegal use, you can go on to code the internals.
When coding internals, remember that you cannot trust anything (with the possible exception of other code in the same component). Check/assert the return values (and in/out parameters) of all calls you make. Have a well-defined system-level design for error handling, that doesn't allow the real error (or its source, if possible) to get lost.
As for testing, I'm all for the XP method: write your test cases first. This helps you to think about what you API is doing, how you are going to actually use it, and what you can throw at it that may break it (helping you to lock down the pre/postconditions).
You must use regression tests! Testing is useless if its done one, but the code is modified afterwards. Have a library of test cases, and use all of them. Every time a bug is found, add a test case for that bug, and ensure it is regression tested every time.
Code audits can detect and solve a lot of common implementation bugs. Use them to look for unchecked pointer/buffer use, assuming return values or success/failure of functions, and that asserts are correctly and accurately used.
In my experience most bugs do NOT come from implementation errors, but from developer misunderstanding, especially late in a project or in maintenance, or even during bug fixing! A developer must fully understand the code (s)he is working on, and all the assumptions it makes. Never adjust a non-local variable without first checking all other functions that use or modify that variable, and understanding the implications. Never use a function or method without understanding all the side effects (on parameters and scope/global state). This is why all of this information should be documented, and audits performed to ensure that the documentation is accurate.
i-name =twylite [http://public.xdi.org/=twylite], see idcommons.net
I've worked on both ends (dev and test), at M$ and other places, and I've come to one conclusion (I'm sure its not the only correct one).
Developers must test their code.
With a test team backing you up, it becomes too easy to change something, run it once (if at all), and then push it into the next build so the test team can catch your errors. I've found that as a tester, a huge proportion of bugs are simply features implemented where the developer just forgot something stupid. I end up wasting 5 minutes writing a report, my manager assigns the bug back to a developer (hopefully the one who made the mistake but not always), and the developer comes back to the code a week later, spending 20 minutes just trying to figure what s/he wrote a week back.
My point: this wastes 30 minutes of people's time for every little stupid mistake. Pressure your developers to really give a thorough test to the code they write before the check it in, especially if you have a test team, because you just end up wasting more people's time.
Your signatures belong to me.
I agree. The whole concept is flawed. Ultimately the problem with too many bugs is not a "testing" problem, but a "design" and "implementation" problem.
The flaw in the thinking is the assumption that all bugs are inevitable. You accept as given the idea that the bugs have to be found and corrected. It actually is possible to avoid introducing the bugs in the first place.
The sad thing is, it is likely true in every case that *avoiding* the bugs is cheaper than *correcting* the bugs. Yet we keep introducing bugs & assuming they will be found & corrected later.
Study after study has confirmed: Code reviews find more defects, per staff hour, than any other activity, including all kinds of testing.
Aside from that, the benefits of having more than one person aware of each change to the code are significant. If George is sick, or quits, or wants to go on vacation, it's not just George who can make the next change.
Stupid job ads, weird spam, occasional insight at
... is to not make the mistake in the first place. This may sound kind of stupid, but it's true. Don't skip on sleep - so you may stay properly awake, don't run yourself on Coca/Pitr Cola, eat good food, go for walks, and you'll find yourself making far fewer mistakes and producing better quality stuff.
The question is what type of mistake. Is your program crashing a lot? Then see the above poster. Is your program generating the wrong results? Then the problem is that you have not specified rigorously enough. With good engineering specs, the actual code is just data entry.
Programming is the same way. What kinds of bugs are you finding? Are they just stupid bugs, like buffer overflows or off-by-ones (good design, bad implementation), or are they unhandled errors, or are they API mis-matches or faulty algorithms (bad design)?
Have you made any effort to go back and say "Gee, we are getting a lot of off-by-one errors. OK folks, we need to think about our loops."?
And when you find one type of bug, do you go back and identify anyplace else a similar bug may exist?
If you are hitting high and right, and you never adjust your sights, you will NEVER hit the target consistently. If you never feed back the CAUSE of the bugs, you will never eliminate them.
www.eFax.com are spammers
I don't let customers dictate how programs should work. I make them tell me what information they have to enter, and what they want to get back out. I decide on mostly everything in the middle.
I don't let customers dictate how programs should work. I make them tell me what information they have to enter, and what they want to get back out. I decide on mostly everything in the middle.
Then you aren't writing particularly complex software. If your users need software that does sophisticated processing, mathematical or otherwise, then the programmer probably isn't the best person to work out how it should do it. This is true whether you're working on software for pricing derivatives, or for tracking shipments in a supply chain, or for controlling manufacturing machinery. That's why there are notations like UML, so that functional experts can communicate unambiguously to the software developers what a system should be doing. A good programmer knows about programming, a good analyst knows about business processes, some people are both, but only with years of experience, and even then, only within a single industry.
The requirements, specification and alanlysis process is what separates software engineering from "hacking".
How can you be sure you are 'Properly Testing Your Code'?
Actually you can do this by adding more bugs, yes adding them, The technique is called bebugging and the is basicly:
1) Produce code, it contains an unknown number (N) of bugs.
2) Programmer (or bebugger) seeds the code with a number (B) of known new bugs, the number and type of bugs should be determined from bugs found in previous debugging cycles.
3) Code is submitted to testing and some bugs are found (F).
3) The bugs found are examined and categorised as either real bugs (FN) or bebugs (FB).
4) Number of real bugs (N) can be found as the ratio of found bebugs (FB) to unfound bebugs (F).
5) Don't forget to remove all the bebugs.