Slashdot Mirror
Properly Testing Your Code?
lowlytester asks: "I work for an organization that does testing at various stages from unit testing (not XP style) to various kinds of integration tests. With all this one would expect that defect in code would be close to zero. Yet the number of defects reported is so large that I wonder how much testing is too much? What is the best way to get the biggest bang for your testing buck?" Sometimes it's not the what, it's the how, and in situations like this, I wonder if the testing procedure itself may be part of the problem. When testing code, what procedures work best for you, and do you feel that excessive testing hurts the development process at all?
... is to not make the mistake in the first place. This may sound kind of stupid, but it's true. Don't skip on sleep - so you may stay properly awake, don't run yourself on Coca/Pitr Cola, eat good food, go for walks, and you'll find yourself making far fewer mistakes and producing better quality stuff. And _double_check_ everything.
One wonders how your development has been organized. Everybody here should know the basics of software engineering, including but not limited to:
1) document APIs exactly, including definitions of legal and illegal data sets
2) separate test group from programmers
3) separate quality assurance from both API testing and programmers.
Well, that's the theory. I've never worked in a place where that would have been implemented. Instead, people trying to bring this in have been kicked out. In practice, maybe one should try to get a feeling of each API: how is it supposed to be used? Use each piece of software only in the implicit limits of it's programmer's idea to keep the number of bugs down. Not to mention the obvious coding style mantras.
I think, therefore thoughts exist. Ego is just an impression.
And the answer to that is of course: "No, you should test more, and fix the bugs". And of course, looking over your development model to see why you have so many errors might be a good idea (such as formalizing who can commit code, if you've got lot's of programmer at various skill-levels).
But in real-life, many bugs are not that important, and time-to-market and development cost is more important.
So unless you provide us with more data, such as ...
...I don't think anyone will be able to give any good advice either.
... NOT tested in. If the product is poorly engineered, there should be no surprise at the vast number of bugs, no matter how much testing you do. Crap is crap.
The main thing: testing does absolutely nothing to minimize the number of defects in a particular application.. There are lots of other things that are as important.. ie: are these defect reports being seen by the appropriate developers and are they being acted on, what types of procedures and communication actually exists between the developer and the QA persons (assuming that they are not the same folk)..
The last point isn't as bizarre as it sounds, I've seen lots of places where a QA person enters bugs, but the developers silently reject them ("its not a bug, that's how the program works")
Testing just tries to discover the presence of defects, by itself, it cannot ensure that your product works perfectly (for an application of even moderate complexity, there may be an exponential number of cases and paths to check, most test cases are written for a percentage of those only).. Because of this, if you feel that you're spending too much time testing, perhaps you need to check if your test cases are appropriate to the situation and stage of development..
Another point is that tests can be automated to some degree or the other, perhaps a scriptable tool might assist in lowering some of the drudgery associated with actually assuring the quality of your software...
rant mode = on...Excessive testing ONLY hurts if it takes people away from development at the early or even middle stages of a project and forces them to run tests on incomplete sections of code.. otherwise, there is NO such thing as too many things...
Its all in the same package...
... My bet is that the second half would spend less total time on that program and bug fixes in a real world situation (which was the whole point of the exercise)
If you have a well thought out and well structured design... and if every single function is documented atleast mostly before coding begins... and if all of the theory of interactions fit in... then only a small amount of testing is required
The main problems which require more testing come when corners are cut over the initial design stages, or their not done as fully as they should be doing, or when people dont actually think about what input users can give
For one of my projects I did ages ago for college, half of the class (which was split to give the same coding ability spread in both halfs) were each asked to write this program and the other half were asked to design it, review the design, make a proper testing strategy, document and THEN start writing it
The second half took a little longer to get their program working, but the first half had more bugs, and spent more time testing
Most of them have nothing to do with poor coding, they have to do with poor understanding of the original problem. Which leads to writing a solution for the wrong thing. No amount of debugging will fix a wrong assumption.
If you really want to generate bug free code, you have to keep one rule in mind at ALL times. A bug occuring in the code is a failure in the methodology you are currently using to avoid them. Sounds very basic, but a lot of companies forget that.
;-)
When you have a problem with bugs, you need to figure out where in the process the problem happened. Was the unit spec wrong? Documentation? Implementation? Unit testing procedures? Was it a correctable problem caused by the engineer involved?
If you really want to be bug-free, every time one shows up you have to figure out why it happened, and change things.
Personally, I think the biggest one is to make engineers work 9-5. Not 9-7, or 11-9. Tell them to go home at 5, even if they're in the middle of something. Software engineering is a very complex task that takes a lot of energy and concentration to do right. Just like Doctors who work long hours make mistakes (resulting, often, in people dying), engineers who work too long make mistakes too.
Being in the "zone" is often the death of good code. You get lots of cool code written, but none of it is double-checked, none of it is verified to match spec, and it often ends up afterwards difficult to understand.
Now don't get me wrong, I don't do any of this, and my crap is FULL of bugs, but thats what you need to do if you really want to help it. Writing buggy code is like a public works program for QA people. Who wants a hundred thousand unemployed anal-retentive QA people nitpicking something else like your car's inspection or your tax forms? Better to keep them in the software where they can't do any harm
Mathmatical proofs and unit testing asside make sure that your program dosent crap out when its being used or abused. So get some regular people to bang on it..
... trust the programmers when it comes to testing. You may find some obvious buigs in your own code, but when it comes to runtime testing most programmers tend to emphasize on "correct" user behaviour or maybe the few wrong input sets they've taken care of and completely neglect the pervert fantasy of lusers *g*
;)
Dedicated testers OTOH are perfect in letting a program die in the most obscure ways. Did you know that you can crash a Commodore 64's BASIC interpreter by just typing PRINT 5+"A"+-5 ?
Testing, like documentation, is not improved by sheer quantity. If you are doing 1,000 crap tests then 10,000 crap tests are not going to catch many more bugs.
Look at your bugs and ask yourself: at which stage of testing should this bug have been caught? Then find out why it was not.
If the bug couldn't have been caught by a component of your testing framework then you will have to look at the framework.
Quality audited tests as what you want, you may find that most of your tests are testing for things that never could happen and thus flooding you with worthless reports and giving you a false sense of confidence.
As an aside, I have in the past created useless tests to calm panicy project leaders. "5% of our tests failed" sounds much better then "50% of our tests failed" and it as all achieved by adding unnecessary tests. But at least I knew which ones they were.
I guess you should try to spend a part of your testing budget on improving your design and programming practices.
I've worked on both ends (dev and test), at M$ and other places, and I've come to one conclusion (I'm sure its not the only correct one).
Developers must test their code.
With a test team backing you up, it becomes too easy to change something, run it once (if at all), and then push it into the next build so the test team can catch your errors. I've found that as a tester, a huge proportion of bugs are simply features implemented where the developer just forgot something stupid. I end up wasting 5 minutes writing a report, my manager assigns the bug back to a developer (hopefully the one who made the mistake but not always), and the developer comes back to the code a week later, spending 20 minutes just trying to figure what s/he wrote a week back.
My point: this wastes 30 minutes of people's time for every little stupid mistake. Pressure your developers to really give a thorough test to the code they write before the check it in, especially if you have a test team, because you just end up wasting more people's time.
Your signatures belong to me.
I agree 100%
The only perverted problem with this approach is that in case you are selling your code to some 3rd party company for example, and you are competing with a set of other companies you might have terribly hard time trying to make the customer understand why your development takes 3 times longer than what is promised in other proposals. This is not so much problem when you have a proven track record clearly stating that the complete time wasted is less using your approach. Still, in some cases, the customer might be stuck to some bizarre outsourcing process with inherently excludes all proposals that exceed the shortest estimate by some magic percent.
The point being that many of current customers for software support this bizarre approach of first jumping to lake and then seeing if you hit a stone or not. Ofcourse its not easy job to decide to pay $100 for a candy, if you can get the same thing with just some dog dung added for $33. Especially if the software provider does not say that the dung is included in the offering.
Study after study has confirmed: Code reviews find more defects, per staff hour, than any other activity, including all kinds of testing.
Aside from that, the benefits of having more than one person aware of each change to the code are significant. If George is sick, or quits, or wants to go on vacation, it's not just George who can make the next change.
Stupid job ads, weird spam, occasional insight at
Most of the 'worst' bugs i've come accross are down to bas systems design, before a single line of code is written.
If a system is designed well then you should have far fewer bugs, even if you are using code monkies who don't know a quick sort from a n^2 bubble.
Design you systems well, know your people, Bill's good at that kind of thing and likes it(but crap at ui's say),
Jess loves doing data imports, (may not be that quick, but always does them well).
Fread always designes and produces good/fast systems cores.
Get your developers talking and sharing knowlage, 'I'm, having a bit of a problem' , or 'Who knows how to', are good things for people to be saying, so incorrage them to own up to the inadiquacies, and they won't have them for long.
If you can manage that then your productivity and bug counts should drop dramaticly, and the bugs you do have should be easier to fix.
thank God the internet isn't a human right.
Testing is necessary but not sufficient. There must be a way to capture requirements, convert requirements to design, convert a design to implementation, and finally test. At each transisition it is a good idea to make an assessment of how well you accomplished your task.
Skipping any of these steps and putting if off until test is pure folly. An extremely false economy.
Just generating random data and trying to load it caught a lot of bugs, but even more effective was to take a valid image and modify the bytes in it at random, and then try to load it.
Of course, the reason this was so effective, is that the loaders would get mostly what they expect, and then suddenly something illegal. This is the kind of thing you tend to forget about when you write code.
Since it is so easy to attack your program with random data, this kind of testing gives you a lot of bang for the buck, but on the other hand, the bugs it find may not always be those that are likely to occur in practice.
Programming is the same way. What kinds of bugs are you finding? Are they just stupid bugs, like buffer overflows or off-by-ones (good design, bad implementation), or are they unhandled errors, or are they API mis-matches or faulty algorithms (bad design)?
Have you made any effort to go back and say "Gee, we are getting a lot of off-by-one errors. OK folks, we need to think about our loops."?
And when you find one type of bug, do you go back and identify anyplace else a similar bug may exist?
If you are hitting high and right, and you never adjust your sights, you will NEVER hit the target consistently. If you never feed back the CAUSE of the bugs, you will never eliminate them.
www.eFax.com are spammers
I don't let customers dictate how programs should work. I make them tell me what information they have to enter, and what they want to get back out. I decide on mostly everything in the middle.
Maybe this is a topic for another discussion, but I don't expect business user to supply me with specs. They aren't software guys, they are business users. I expect them to supply me with business requirements and business process definitions, from which we will together develop the software process definitions from which the specs evolve.
Yes, we use formal methods to design the programs before we start to write anything. We are an SEI Level 5 shop, but most of you guys would hate to program for us. Once the requirements are approved, the programmer has virtually no freedom. However, the programmers are intimately involved with the requirements review process. Half of us involved in producing the Shuttle Flight Software have never written a single actual line of code at the terminal.
For instance, I was reviewing some code a few months ago that said to compute an array of three vectors. Well, we were actually only going to use the third element of each array, so the developer only stored the third elements.
That was OK, but we also ended up documenting that in the detailed design spec, which is thousands of pages in itself.
Most desktop apps can't afford our process and probably don't need it. Embedded software in avionics equipment probably do need this process and can't afford not to use it (I don't know if they all do).
- "Sweet merciful crap!" Homer J. Simpson
I need "insert feature here" implemented in "insesert program here" done yesterday. I promised "insert very large client here" that we would be able to add these new features and still be on time with delivery.
Sad but true.....
Boss: Go write some code that does some stuff
Me: Well what about this? I need this info.
Boss: Well just start working and we will fill that stuff in laster.
How the heck can I write good when I am hardly told what the application is supposed to do? So I write something, it doesn't take into account the missing details that I asked about. Those get defined two weeks after the thing is supposed to be done. The app turns out terrible and then the powers that be want to know why it has problems. It is incredibly frustrating.
To Do: 1. Take over world 2. Pick up Milk and Bread on the way home
A lot of the problem may rely on what methodology you are using to code the program, whether it is the traditional waterfall method, or the sprial method, or perhaps M$'s old sync-and-stabilize method. Whatever methodology you use will drive how you should be testing.
With the waterfall model, you really need to know way ahead of time that what you are coding is what will be desired in the end product. It forces you to have a clear picture in your model of what you are trying to build and with each step in the process, you must develop testing procedures that address that level of the code. For example, at a high level, you may say, let's build a compiler, and following that decision, you need to devise a test that proves that the compiler works. The next phase, you may say, let's build an assembler to produce machine code for the compiler. Then you need to build tests that prove that the assembler works. This methodology continues right down to the smallest module of code, and when all of the pieces have been written, integration testing begins, and you make sure that each larger piece can correctly function based on the output of the smaller piece.
However, in the spiral model, it allows for a well-defined core code to be produced with tons of modules that evolve as the spiral expands. Integration is a function of the spiral, and testing occurs within each iteration of the spiral loop. Code produced with the spiral model also tends to be somewhat more difficult to test in later stages, IMHO, due to the nature of the testing that occurs at each cycle in the loop. Testing becomes more critical in later stages as the previous stages become more nested into the core of the program.
Well, enough Software Engineering for one day. Back to work....
Rule #1 -- Politics always trumps technology.
I don't let customers dictate how programs should work. I make them tell me what information they have to enter, and what they want to get back out. I decide on mostly everything in the middle.
Then you aren't writing particularly complex software. If your users need software that does sophisticated processing, mathematical or otherwise, then the programmer probably isn't the best person to work out how it should do it. This is true whether you're working on software for pricing derivatives, or for tracking shipments in a supply chain, or for controlling manufacturing machinery. That's why there are notations like UML, so that functional experts can communicate unambiguously to the software developers what a system should be doing. A good programmer knows about programming, a good analyst knows about business processes, some people are both, but only with years of experience, and even then, only within a single industry.
The requirements, specification and alanlysis process is what separates software engineering from "hacking".
Thank you for your interesting and insightful treatise.
Clearly, with your keen grasp of the history of programming, and your understanding of VB and HTML, you will someday make a fine salesman or mid-level manager. You already exhibit the skill and insight used by management at most companies to plan their projects, procedures, timelines, and budgetary requirements.
Many programming issues would clearly benefit from simplification, and perhaps you are on to something. By reducing the number of tools a language attempts to implement, it clearly decreases the number of distractions for the programmer. If you wish to pursue this concept further, you may perhaps wish to research another foundational language called Logo.
Good luck in your next class!
Billbert, PhB, TFIC
Redmond, WA
P.S. - A side note regarding the original article:
A firm understanding of your process for code development, and a clear design for your testing procedure are essential.
If you spend some time in advance on planning, your code will benefit. Also, a good test plan should include a measure of the relative impact of a 'bug' or 'defect' to help determine priority of response by your programmers.
By focusing on your programming objectives from the beginning, and maintaining that focus throughout your entire design lifecycle, you should be able to identify underlying problems in your current development model and use them to improve your entire process, with a goal of helping prevent errors in design, and catching errors in code before going to test.
Peer review at each stage prior to testing (planning, functional design, algorithm design, coding, and test design) will also help catch errors in advance, and lead to the development of much better code. It may sound like it will take longer and cost more, but it saves time and money in the end in terms of not having to rewrite and maintain poorly implemented code.
Cheers.
At the time that you are coding, every assumption is going through your head. This is time to write it down, either on paper, in a document, or in comments in the code. The mental state you are in when designing test conditions cannot come close to the state of mind you are in when coding (if you are concentrating :) . You are mentally closer to a problem when you are coding than when you are designing, and you can take the shortcomings of the platform you are working on and pair it with the shortcomings of the design.
Any consideration you have during the writing of a single line of code is gold. And like a great dream, if you don't get it down when you think it, you will lose it in a day or two.
My 2 cents.
http://pcblues.com - Digits and Wood
developers are nowadays too full of false pride as well, thinkin: I am the lead coder, analyzing bugs is the job of trainees. In my opinion the situation should be (atleast in some cases) completely opposite
I don't know who your company is hiring but where I work things are as they should be. It's the college kids who are hired for the experience are the one's with cocky attitudes while the advanced developers are trying to push management to get better processes in place rather than just "Fly to the moon by next thursday."
... Then wednsday comes and it's "No wait, fly to Mars instead. But still do it by tommorow".
At the moment our company isn't as structured as it should be. We don't have a QA team or a testing team. It's just management pushing the developers to get thigns done. But the developers are pushing back to say "hey, we need a process here. It's not just writing code. We need to design it first. And that takes time. We also need to implement code review and pick someone who's got the experience to decide what goes in CVS."
So my point is that in my experience it's the inexperienced developers who want to just jump in and write code thinking "it's not my job to fix bugs". I think this has a lot to do with wanting to get that advanced status. But as they grow they realize that they're going about it wrong and smarten up with regards to processes.
--
Garett
How can you be sure you are 'Properly Testing Your Code'?
Actually you can do this by adding more bugs, yes adding them, The technique is called bebugging and the is basicly:
1) Produce code, it contains an unknown number (N) of bugs.
2) Programmer (or bebugger) seeds the code with a number (B) of known new bugs, the number and type of bugs should be determined from bugs found in previous debugging cycles.
3) Code is submitted to testing and some bugs are found (F).
3) The bugs found are examined and categorised as either real bugs (FN) or bebugs (FB).
4) Number of real bugs (N) can be found as the ratio of found bebugs (FB) to unfound bebugs (F).
5) Don't forget to remove all the bebugs.
...and say, "Developers should write their test suites BEFORE they write their code."
We have a fairly large open source project with contributors coming in and going out all the time (well, not a lot going out; but any number is a problem there). Our experience shows that if you can't write a test suite you're not ready for anything more than a crude prototype. The problem with test-after-coding regimes is the testing gets short-circuited. You've already got working code. You "know" it works. You're just proving it works. So you test the obvious stuff that proves this.
Since we have instituted this policy, coding efficiency has actually improved. Coders who have tried to devise a complete set of tests have formalized their understanding of the requirements in a sense which the most complete requirements doc will never do. We include the test suite in CVS. Nobody commits until their update passes the entire test suite. This results in an enormous (but complete) test of everything done so far. But you can't imagine the thrill of seeing your patch pass that many tests the first time.
All of which is completely separate from what a QA process is for.
Eternal vigilance only works if you look in every direction.
I beg to differ. This is how most developers test their code as well, though manually.
If you're just testing to make sure your code does what it is supposed to do you are likely in BIG, BIG trouble. Users (and black hats) do just the opposite.
Focus just as much on making sure your code doesn't do things it WASN'T designed to do. Or risk a CERT or Security Focus advisory...
I'm certain someone has already said this, but over 80% of defects come from crappy requirements. Forget about your design & analysis, your coding practices, inpsection techniques, debugging and testing abilities - if your requirements are not CLEAR, CORRECT, ATOMIC, UNAMBIGUOUS, and CONSISTENT, you might as well start burning money.
NASA correlated a $1 cost to correct a "defect" in the requirements stage (here a defect can be any requirement that does not meet all 5 attributes I listed above) to several hundred to thousands of times over when addressing the same defect at the testing stage. Crappy requirements and crappy specifications are a big part of what makes your code buggy and expensive.
LA Times posted a study last year that showed that the average US programmer only coded for 51 days a year. 51 days!! One fifth of your working year spent writing new code. The rest of the time? DOING REWORK.
Biggest cause of rework?
UNCLEAR AND AMBIGUOUS REQUIREMENTS.
Spend the time and effort to beef up your requirements gathering and management processes. You'll get your ROI in ONE project cycle.
"Content's a bitch."