Slashdot Mirror
Properly Testing Your Code?
lowlytester asks: "I work for an organization that does testing at various stages from unit testing (not XP style) to various kinds of integration tests. With all this one would expect that defect in code would be close to zero. Yet the number of defects reported is so large that I wonder how much testing is too much? What is the best way to get the biggest bang for your testing buck?" Sometimes it's not the what, it's the how, and in situations like this, I wonder if the testing procedure itself may be part of the problem. When testing code, what procedures work best for you, and do you feel that excessive testing hurts the development process at all?
... is to not make the mistake in the first place. This may sound kind of stupid, but it's true. Don't skip on sleep - so you may stay properly awake, don't run yourself on Coca/Pitr Cola, eat good food, go for walks, and you'll find yourself making far fewer mistakes and producing better quality stuff. And _double_check_ everything.
And the answer to that is of course: "No, you should test more, and fix the bugs". And of course, looking over your development model to see why you have so many errors might be a good idea (such as formalizing who can commit code, if you've got lot's of programmer at various skill-levels).
But in real-life, many bugs are not that important, and time-to-market and development cost is more important.
So unless you provide us with more data, such as ...
...I don't think anyone will be able to give any good advice either.
The main thing: testing does absolutely nothing to minimize the number of defects in a particular application.. There are lots of other things that are as important.. ie: are these defect reports being seen by the appropriate developers and are they being acted on, what types of procedures and communication actually exists between the developer and the QA persons (assuming that they are not the same folk)..
The last point isn't as bizarre as it sounds, I've seen lots of places where a QA person enters bugs, but the developers silently reject them ("its not a bug, that's how the program works")
Testing just tries to discover the presence of defects, by itself, it cannot ensure that your product works perfectly (for an application of even moderate complexity, there may be an exponential number of cases and paths to check, most test cases are written for a percentage of those only).. Because of this, if you feel that you're spending too much time testing, perhaps you need to check if your test cases are appropriate to the situation and stage of development..
Another point is that tests can be automated to some degree or the other, perhaps a scriptable tool might assist in lowering some of the drudgery associated with actually assuring the quality of your software...
rant mode = on...Excessive testing ONLY hurts if it takes people away from development at the early or even middle stages of a project and forces them to run tests on incomplete sections of code.. otherwise, there is NO such thing as too many things...
This is excellent advise. In my experience, the most stable code comes from pragmatic design followed up by pragmatic coding.
Design your system thoroughly. Identify every component, and the minimum interface required for that component. Carefully document that interface (API) - use Design By Contract (preconditions, postconditions and invariants) if possible.
Moving targets mean that the API will almost certainly have to be extended - documentation on the design and intent of the component/API as a whole will reduce the pain of this process. The responsibility for this documentation is shared between the design and implementation phases. Pay careful attention to documenting assumptions made within the code, e.g. ownership of member/global variables.
When it comes to coding, start with a skeleton. Put in the API function/method as defined, then check/assert every pre/post condition. Think about how any parameter could be out of range, or violate the assumptions you make. Once you are happy you're checking for all illegal use, you can go on to code the internals.
When coding internals, remember that you cannot trust anything (with the possible exception of other code in the same component). Check/assert the return values (and in/out parameters) of all calls you make. Have a well-defined system-level design for error handling, that doesn't allow the real error (or its source, if possible) to get lost.
As for testing, I'm all for the XP method: write your test cases first. This helps you to think about what you API is doing, how you are going to actually use it, and what you can throw at it that may break it (helping you to lock down the pre/postconditions).
You must use regression tests! Testing is useless if its done one, but the code is modified afterwards. Have a library of test cases, and use all of them. Every time a bug is found, add a test case for that bug, and ensure it is regression tested every time.
Code audits can detect and solve a lot of common implementation bugs. Use them to look for unchecked pointer/buffer use, assuming return values or success/failure of functions, and that asserts are correctly and accurately used.
In my experience most bugs do NOT come from implementation errors, but from developer misunderstanding, especially late in a project or in maintenance, or even during bug fixing! A developer must fully understand the code (s)he is working on, and all the assumptions it makes. Never adjust a non-local variable without first checking all other functions that use or modify that variable, and understanding the implications. Never use a function or method without understanding all the side effects (on parameters and scope/global state). This is why all of this information should be documented, and audits performed to ensure that the documentation is accurate.
i-name =twylite [http://public.xdi.org/=twylite], see idcommons.net
I've worked on both ends (dev and test), at M$ and other places, and I've come to one conclusion (I'm sure its not the only correct one).
Developers must test their code.
With a test team backing you up, it becomes too easy to change something, run it once (if at all), and then push it into the next build so the test team can catch your errors. I've found that as a tester, a huge proportion of bugs are simply features implemented where the developer just forgot something stupid. I end up wasting 5 minutes writing a report, my manager assigns the bug back to a developer (hopefully the one who made the mistake but not always), and the developer comes back to the code a week later, spending 20 minutes just trying to figure what s/he wrote a week back.
My point: this wastes 30 minutes of people's time for every little stupid mistake. Pressure your developers to really give a thorough test to the code they write before the check it in, especially if you have a test team, because you just end up wasting more people's time.
Your signatures belong to me.
Study after study has confirmed: Code reviews find more defects, per staff hour, than any other activity, including all kinds of testing.
Aside from that, the benefits of having more than one person aware of each change to the code are significant. If George is sick, or quits, or wants to go on vacation, it's not just George who can make the next change.
Stupid job ads, weird spam, occasional insight at
Just generating random data and trying to load it caught a lot of bugs, but even more effective was to take a valid image and modify the bytes in it at random, and then try to load it.
Of course, the reason this was so effective, is that the loaders would get mostly what they expect, and then suddenly something illegal. This is the kind of thing you tend to forget about when you write code.
Since it is so easy to attack your program with random data, this kind of testing gives you a lot of bang for the buck, but on the other hand, the bugs it find may not always be those that are likely to occur in practice.
Programming is the same way. What kinds of bugs are you finding? Are they just stupid bugs, like buffer overflows or off-by-ones (good design, bad implementation), or are they unhandled errors, or are they API mis-matches or faulty algorithms (bad design)?
Have you made any effort to go back and say "Gee, we are getting a lot of off-by-one errors. OK folks, we need to think about our loops."?
And when you find one type of bug, do you go back and identify anyplace else a similar bug may exist?
If you are hitting high and right, and you never adjust your sights, you will NEVER hit the target consistently. If you never feed back the CAUSE of the bugs, you will never eliminate them.
www.eFax.com are spammers
I don't let customers dictate how programs should work. I make them tell me what information they have to enter, and what they want to get back out. I decide on mostly everything in the middle.
I don't let customers dictate how programs should work. I make them tell me what information they have to enter, and what they want to get back out. I decide on mostly everything in the middle.
Then you aren't writing particularly complex software. If your users need software that does sophisticated processing, mathematical or otherwise, then the programmer probably isn't the best person to work out how it should do it. This is true whether you're working on software for pricing derivatives, or for tracking shipments in a supply chain, or for controlling manufacturing machinery. That's why there are notations like UML, so that functional experts can communicate unambiguously to the software developers what a system should be doing. A good programmer knows about programming, a good analyst knows about business processes, some people are both, but only with years of experience, and even then, only within a single industry.
The requirements, specification and alanlysis process is what separates software engineering from "hacking".
How can you be sure you are 'Properly Testing Your Code'?
Actually you can do this by adding more bugs, yes adding them, The technique is called bebugging and the is basicly:
1) Produce code, it contains an unknown number (N) of bugs.
2) Programmer (or bebugger) seeds the code with a number (B) of known new bugs, the number and type of bugs should be determined from bugs found in previous debugging cycles.
3) Code is submitted to testing and some bugs are found (F).
3) The bugs found are examined and categorised as either real bugs (FN) or bebugs (FB).
4) Number of real bugs (N) can be found as the ratio of found bebugs (FB) to unfound bebugs (F).
5) Don't forget to remove all the bebugs.
I beg to differ. This is how most developers test their code as well, though manually.
If you're just testing to make sure your code does what it is supposed to do you are likely in BIG, BIG trouble. Users (and black hats) do just the opposite.
Focus just as much on making sure your code doesn't do things it WASN'T designed to do. Or risk a CERT or Security Focus advisory...