Properly Testing Your Code?

lowlytester asks: "I work for an organization that does testing at various stages from unit testing (not XP style) to various kinds of integration tests. With all this one would expect that defect in code would be close to zero. Yet the number of defects reported is so large that I wonder how much testing is too much? What is the best way to get the biggest bang for your testing buck?" Sometimes it's not the what, it's the how, and in situations like this, I wonder if the testing procedure itself may be part of the problem. When testing code, what procedures work best for you, and do you feel that excessive testing hurts the development process at all?

Comments

the best way to test code...

[Anonymous Coward]

... is to not make the mistake in the first place. This may sound kind of stupid, but it's true. Don't skip on sleep - so you may stay properly awake, don't run yourself on Coca/Pitr Cola, eat good food, go for walks, and you'll find yourself making far fewer mistakes and producing better quality stuff. And _double_check_ everything.

Re:the best way to test code...

[eam]

I agree. The whole concept is flawed. Ultimately the problem with too many bugs is not a "testing" problem, but a "design" and "implementation" problem.

The flaw in the thinking is the assumption that all bugs are inevitable. You accept as given the idea that the bugs have to be found and corrected. It actually is possible to avoid introducing the bugs in the first place.

The sad thing is, it is likely true in every case that *avoiding* the bugs is cheaper than *correcting* the bugs. Yet we keep introducing bugs & assuming they will be found & corrected later.

Re:the best way to test code...

[sql*kitten]

... is to not make the mistake in the first place. This may sound kind of stupid, but it's true. Don't skip on sleep - so you may stay properly awake, don't run yourself on Coca/Pitr Cola, eat good food, go for walks, and you'll find yourself making far fewer mistakes and producing better quality stuff.

The question is what type of mistake. Is your program crashing a lot? Then see the above poster. Is your program generating the wrong results? Then the problem is that you have not specified rigorously enough. With good engineering specs, the actual code is just data entry.

Re:the best way to test code...

[Hater's Leaving, The]

Better than double checking everything is to have an external eye code review everything. It's probably a 10% overhead when it comes to the coding side, but a >50% decrease in the debugging side. Well worth it.

I'm currently on sabbatical, but I consult 1 day a fortnight for a couple of small local companies who can't afford me full time - all I do there is code review, and they are of the opinion that I more than double the effectiveness of their less experienced programmers.

THL.

Re:the best way to test code...

[Anonymous Coward]

You both miss the point. If the compiler picks up on it (or Lint), then its hardly a bug. A bug is something which passes testing and simulation, and plonks your $10 Billion space craft in the middle of the Atlantic Ocean.

Re:the best way to test code...

[Xentax]

I agree -- our own company suffers from giving less effort on code reviews than most of us know we should. People try to save time by under-planning for code reviews, but that saved time is always lost at least twofold in uncaught bugs, extra time for optimization, and so on -- all things that would be identified in a solid code review.

Identify the people in your company that have the best "critical eye" for each language you develop in -- and see to it that they get the time they need to really critique code, either during implementation or at least before starting integration testing (preferably before unit testing, actually). It may be hard to convince management the first time around, but if you account your hours precisely enough, the results WILL speak for themselves, in terms of hours spent overall and on testing/integration.

Xentax

Re:the best way to test code...

[protonman]

No that's gravity.

Re:the best way to test code...

[lrichardson]

"The best way to have your program compile the first time is to stop putting bugs in them"

Heh! Unfortunately, most of the 'problems' I deal with turn up later ... as in, whether an add/modify or new, the business side suddenly starts screaming "That's not right!"

The appropriate quote is "It's just what we asked for, but not what we want!"

I don't think this kind of 'bug' can ever be removed. Despite an understanding of the 'business' side of things, my experience has been that the overwhelming majority of specs suck ... whether it's incomplete definitions, contradictions, or questions about what order various rules things should be in. Coding errors should be few and far between. To have them occur generally means that the writing went too fast ... although, to be fair, given the "I want those changes yesterday!!!" attitude of the modern business world, this situation seems to occur with more frequency now than it did a decade back.

Re:the best way to test code...

[AJWM]

the overwhelming majority of specs suck

That is, assuming you have a spec at all beyond some vague handwaving and perhaps some marketing viewgraphs (er, power point presentations).

Specifications almost invariably need to go through a step that I refer to as "debugging the spec" -- looking for internal inconsistencies and contradictions, vagueness, ambiguity, etc. Of course many projects aren't given the time to do that -- meaning that additional time is wasted on the back end fixing all the design and code problems that lack of a good spec caused.

The projects I've been involved with that went fastest and smoothest all started with a good spec that had gone through several iterations of "debugging" and review with the end users. In one case that "spec" was the Nth draft of the user manual -- we "prototyped" the thing on a whiteboard in front of a small group who would be the first users. (Of course you need a pretty experienced designer/developer involved so as not to commit to something that can't be implemented in the available time.)

Of course you need to make sure that the business side signs off on the spec so that they can't later say that it wasn't what they asked for. (It may not be what they meant, but that's their problem -- although it is the analyst/designers responsibility to double check if that's what they really meant if there seems to be an obvious problem.)

And a good spec means that the test cases can be written in parallel with the actual code.

Re:the best way to test code...

[Ungrounded Lightning]

the best way to test code is to not make the mistake in the first place.

But the way to make solid code is to get each bug out as soon as you put it in.

Over my thirty-five years of professional programming I developed a coding/testing method that produces extremely solid code - to the point that one of my collegues once commented that [Rod] is the only person he'd trust to program his pacemaker. B-) It goes like this:

Design ahead of coding.

Of course! If you haven't spent more time designing than you eventually spend coding, you likely haven't yet understood the problem well enough.

This doesn't mean you have to understad every nitty-gritty detail before you write your first line of code - you'll discover things and change your design as you go. But you should know where you're goiong. And as you go, map the road ahead of you.

Not coding until you understand where you're going is VERY scary to administrators. But it gets you to your destination MUCH sooner than striking out randomly.

Get the modularity right.

Think of the final solution as a knotted mass of spaghetti threaded through meatballs inside an opaque plastic bag. Squeeze it around until you find a thin place, where few strands (representing flows of information) connect the halves of the solution on each side of the bag. Cut the bag in half here and label the strands: You've defined a module boundary and interface. Repeat with each of the two smaller bags until the remaining pile of bags each represent an understandable amount of code - about a page - or a single irreducable lump of heavily-interconnected code (one "meatball"). Then tear them open one-by-one and write the corresponding code.

Debug as you go:

This is the key!

Program top-down like a tree walk, stubbing out the stuff below where you're working. As you write the code, also write one or more pieces of corresponding test-code that produces output that is a signature of the operation of every bit of the code you write, and an expected-output file that is hand-generated (or computer-generated by a different tool, preferably written by someone else, or at least in a different language and style if you're alone).

Use a system tool (like diff or cmp) to compare the results (in preference to writing programs to "check" it, so you don't have to worry whether the test is passing beacuse the code is right or the test is broken.)

Run the test(s) every time you make a change or add code. Make a single change at a time between test runs, get it working and tested before you move on. (This is easy for procedural modules and subroutines. For instance: You can build the running of the test into your makefile, and fail the make if the test fails. GUI stuff is tougher, and I didn't have to deal with it myself. But tools are now available to perform similarly there.)

The result is that your bugs will generally be confined to the changes you just made, drastically limiting your search space and shortening your debugging time.

Do COVERAGE testing, and TRACK it.

Don't move on until you have exercised every bit of code where you were working. "Exercise" means your tests have been updated to execute every line or component of a line, driving it to its edge and corner cases and extracting a signature that shows they're doing their job correctly.

Automated coverage tools are an inadequate kludge to try to do this after the fact. Unfortunately, they pass code once all the branches have been executed, but have no idea whether they did the right thing. They may test that you hit the edge conditions - but can they tell if the edge is off-by-one? Human intelligent, with its knowlege of intended function, is required.

I developed a style of marking listings to document coverage, which is why I use hardcopy even in this day of glass terminals.
- a vertical bar beside a line that is completely working. Cross-mark across its top (T) at the first of a set of working lines, across the bottom of the last of a set.
- an "h"-shaped mark beside one that represents a partially-tested branching construct (if, for, do, "}", ...} with a cross across the right if the branch case is untested, across the left if the through case is untested. Switch to vertical bar when fully tested (including hitting the edge from both sides if applicable)
- For compounds (i.e. "for( ; ; ) {" or " ? : " underline the portions fully tested, put an "h" with crossbar through those partially tested.
- Declarations "pass" when you've checked that the type is right for the job and at least one hunk of client code uses them.
- Comments "pass" pretty much automatically when you think they're right.
- A place where code is not yet present, or where the code above and below is tested but the flow across the gap is not, gets a break in the vertical line, with crossbars, as if there were an untested blank line "in the cracks". (But there should be a comment in there mentioning it. I start such comments with "MORE", so I can find any that are left with an editor. Similarly a MORE comment marks where I've stopped coding for now.)

When the code is done-and-coverage-tested there's a vertical slash beside all of it. (Sometimes you have to add test code temporarily to make something visible externally, but #ifdef or comment it out rather than removing it when you're done.)

The result is like growing a perfect crystal, with flaws only in the growing surface. When you've tested a part you're usually DONE. You never have to return to it unless you misunderstood its function and have to change it later, or if the spec changes.

DOCUMENT!

Co-evolve a document as the project develops if there's more than one on it, or if it has to be handed off to others later. If you're alone, you can get away with heavy comments.

Put in the comments even if you have the document.

Keep the comments up to date.

Comment heavily even if it's just you and always will be. When you come back to the code (especially if you're following this methodology and only get back to it MUCH later) you'll have forgotten what you were thinking. So put it all down to "reload your mental cache" when you get back to it.

The document should be a complete expression of the intended operation of the code - but in a very different and human-understandable form. (Especially not just pseudo-code for the same thing, or "i = i+1; add one to i". Use pseudo-code only for an ilustration, not an explanation.) Remember: Testing can NEVER tell if it's RIGHT. It can only tell if two different descriptions match. "Self-documenting code" is an untestable myth - all that can be tested is whether the compiler worked correctly.

There's more but I have to go now. I'll try to continue later. The above contains the bulk of the key stuff.

Re:the best way to test code...

[smithmc]

That's insulting to programmers everywhere. The business requirements state why the software is needed. The specification addresses what the software will do. That still leaves the question of how the software will be implemented so as to meet the specification. In any non-trivial programming language, there are many, many ways to write a piece of code that will perform a given function. Even with excellent, detailed specs, there's still a lot of thought that goes into how to write a good program. To say "it's just data entry" implies to me that you've never actually done programming on a project of any significant magnitude.

Re:the best way to test code...

[tshak]

Very good post. This is very typical of American culture though. Look at our health care. I just posted a rant a week ago about how we spend several orders of magnatude more on trying to cure preventable diseases (AIDS, many cancers, etc.), then trying to research how we can prevent them in the first place. Obviously we need cures, just like we need software testing, but that should be the absolute LAST line of defence. The company I work for is too small to have a QA or a Software Test Engineer, so we essentially "are not allowed to code bugs" :-). This attitude has contributed to applications that have very few bugs, albeit we generally code Intranet apps so we don't have to worry about multiple platforms, OS versions, etc.

Re:the best way to test code...

[soft_guy]

What you are describing is a lack of good Program Management. A good Program Manager can head off these problems and prevent you from ever coding something that isn't what the customer wants. Such a person invariably has to have excellent communication skills and a technical (i.e. programming and maybe QA) background. They also should understand business and have a strong fear of Murphy's Law.

Such a person is not a magic bullet, but can eliminate 90% of your problem, decrease your time to market, and shield developers from the insane headaches described.

Re:the best way to test code...

[eam]

Unfortunately, I live in the same world you live in. I'd like to live in a world where the effort went into avoiding the bugs rather than finding them after the fact, but I am also faced with unreasonable demands that force inferior code out the door.

> Too bad there's no such thing as a spell
> checker built into most compilers. Just simple,
> as you type, error notification would be cute.

Even that is too late. By the time you start typing, every line of the program should already be set in stone. Every line of code should have been evaluated and approved long before it was typed. It doesn't require a "perfect world". It could happen in this one (& does in rare cases). However, you have to believe that it is possible, and then you have to choose to do it.

> Not all bugs are code, many 'bugs' are exactly
> what the designer intended... but the users
> hate it.

Again, this is a failure in the planning & design. Why would the designers plan & code something that the users don't want (or won't use, or can't use, etc.). This is one of the bigger problems I've seen. People design applications without finding out what the users want it to do. Example:

Radiology image viewing workstation. Vender designs the application to display the images with a wide black border around the image. Radiologists don't want to waste the screen. They want to display the image as large as they can, with as much detail as the hardware can display. However, it does look prettier with the border ;-)

Re:the best way to test code...

[Pfhreakaz0id]

Hmmm, I've been programming for 5 years and I just want to know ... what are these "spec" and "test cases" you speak of?

Re:the best way to test code...

[plumby]

An even better approach, if you can convince the management that it does actually use less resource in the long run, is pair programming. I am an experienced developer, who likes to believe that I develop software in the 'right' way, but until I'd worked on a pair-programming project I never realised how often I manage to convince myself to take a shortcut rather than do things properly. With someone beside you asking "why are you doing that?", you can't duck out and take the shortcut, and when you're sick of their nagging, you get the pleasure of swapping over and badgering them to do it properly.

It's also amazingly useful to have someone keeping an eye on the big picture of what you're trying to achieve that day while you worry about the minutiae of your language syntax. The system that we developed may have taken slightly more overall man-power than if it had been developed without pair programming, but it's a complex multithreaded system integrating with a badly documented 3rd party system, and has been running 24x7 for the last 6 months with not a single minute spent on bug fixes or support caused by errors in the system. The non pair-programmed systems either side of it (including the 3rd party app) have frequent failures and regularly suck up resource for support and maintenance.

Re:the best way to test code...

[DunbarTheInept]

By the time you start typing, every line of the program should already be set in stone.

Why waste everyone's time reviewing source code that doesn't even compile without syntax errors yet? FIRST you type it in, THEN clean up the obvious things you can find right away, like syntax errors, THEN take that cleaner version to your first code review. There's a reason they call them text *editors*, you know - it's because after you write one first draft, you can *edit* the file later if you like. It's a rather nice concept.

Re:the best way to test code...

[DunbarTheInept]

The purpose of a lot of these software engineering techniques is to make the design of the *algorithm* independant from the typing of the code. The goal is to make someone with more experience write out the design in a rather pseudocode fashion and then someone else can convert that to actual compiling code. In theory this means the experienced developers lead the newbies through the process, with the experienced ones doing the design, and the newbies doing the muckwork typing. That actually makes a lot of sense in theory, but in practice, it didn't seem the work that way at the job I first got out of college (where this approach was used). The deep, deep problem with this approach is that the psuedocode has to be phrased in such a way that everyone involved in the meeting can understand it, and that means no high-falutin' computer science terminology or any o' that techie stuff. So your design has to be at the level a PHB can handle, and since the coders aren't allowed to get too creative with their implentation of the design, the actual code ends up being just as simple-minded as the design. Thus time and time again we ended up with slow wasteful brute-force approaches to problems, not because the coders and designers didn't know any better, but because that's all that was expressable at the high-level meetings where the design was being hammered out.

Then the problem got worse after several years of that situation ended up making people believe that design work doesn't require a programming background, and then we started having designs made by people who only have a cursory understanding of programming (and by deisgns here, I'm talking about stuff that dictates the algorithm that must be used, not just high-level 'what does it look like' interface design.)

Another big problem is that after spending a year or two doing the "design but don't code" job, even someone who once knew what he was doing starts getting rusty and making dumb algorithmic decisions.

I don't know what the solution is, but it certainly doesn't involve putting a programmer into a position where he never programs again in his lifetime before he can affect the design.

Re:the best way to test code...

[RFC959]

I agree in part. But the critical word in your statement is "good" program management. That's the tough part. In my experience with IT projects (not necessarily programming), either the project manager is "one of the techies" or he isn't. If he is, he usually takes the techies' side and demands proper specs, testing, etc., and then the rest of the business whines to upper management that he's being a "bottleneck", and then he gets replaced. If he isn't a techie, he usually fails to understand why the techies are always griping about the lack of meaningful specs, time pressure, etc, and he exists only to help the business side push harder. Finding someone who crosses between the two sides (which you do mention) is damn tough, and such a person is worth a lot.

Re:the best way to test code...

[DunbarTheInept]

I used to know someone who always wrote his c-code equals conditionals such that if one operand was an lvalue and one was an rvalue, he'd put the rvalue on the left, where it isn't allowed for assignment statements, like so: if( 0 == myFileSize )
{ // TO-DO: print some error and die
}
Instead of the more human-readable: if( myFileSize == 0 )
,,,etc,,,

The theory was, if he screwed up and put in a '=', instead of a '==', then the compiler would catch it for him that way. An interesting idea, but I figure if you expend the mental effort to remember to type it in backward, you are already thinking about the difference between '=' and '==', and so at that point you aren't likely to make that mistake anyway.

Re:the best way to test code...

[DunbarTheInept]

That would have made sense if the statement was "each *algorithm* must be set in stone before you begin typing", but that's not what the post said. It said each *line of code* must be set in stone, which is saying that there must be some form of the entire verbatim program in existence somewhere before you even begin typing (which would preclude using an editor to actually edit the code rather than just type it in). Now, maybe that's not the meaning you meant to get across, but that *is* what your words said. I would have assumed that that's not really what you meant except that you later also mentioned all the peer reviews being done before you type, and that, to my mind, includes code reviews as well as design reviews.

The most effective way...

[rattler14]

before each compile, one should make a small sacrifice to the debugging gods and ask them to forgive you for your syn(tax).

Just use the Microsoft way...

[cliveholloway]

They use Beta testers - the have the largest group of Beta testers out there - otherwise known as their customer base :)

boom boom

cLive ;-)

Re:Just use the Microsoft way...

[spongman]

joking aside, Microsoft does use external beta testers, I believe the win95 beta program included about 500,000 testers.

I worked on the Visual Studio team a while back and we used three main methods of formal testing during product development:

1. manual testing. each feature team had a parallel team of testers that were responsible for testing that team's part of the product. the testers wrote manual test scripts driven both by the 'spec', requests from developers and regressions. these test scripts were used for generating the next two testing methods:
2. 'sniff' tests. the sniff tests were a small suite of automated tests - sometimes using point-and-click automation, sometimes using internal code hooks or exposed automation (eg COM) interfaces - that were a requirement for check-in. ie before checking new code back into the source control system (microsoft's internal 'SLM', or later, SourceSafe) the developer had to run the sniff tests and ensure that they passed. they weren't particularly thorough (they had to run pretty quickly), they basically ensured that the product was usable and that other developers/testers would not be blocked by breakage. if the sniff tests did not pass, it was the responsibility of the developer to either fix his code, or grab the tester responsible for the test in question and update the test to reflect the altered functionality.
3. BVTs (build verification tests). these were a much more thorough superset of the sniff tests that run on the build-lab's machines after the nightly build had completed. for example the compiler BVTs included iterations and combinations of self-building the compiler and libraries with the new code. the developers had little interaction with these test, unless they failed, of course.

For me as a developer the sniff tests were the most important, I could come in in the morning, sync up, build and continue working without worrying about whether or not some guy down the hall in a completely unrelated part of the product had broken the 'file open' dialog, for example.

Testing

[CoderByBirth]

I think JUnit-style testing works great, and I plan to start using it more often.
Testing is good to verify that your code does exactly what you think it does; a lot of the time I produce code that I "think" works, using JUnit allows me to verify that it actually does.
Check out junit.org.

For those of you who are sceptic about unit-testing, you should try it. Setting up the tests are not as tedious as one might think, they force you to think your problem through, and maybe most of all: they make your build look cool :)

Trying to fit in implicit restrictions

[korpiq]

One wonders how your development has been organized. Everybody here should know the basics of software engineering, including but not limited to:

1) document APIs exactly, including definitions of legal and illegal data sets

2) separate test group from programmers

3) separate quality assurance from both API testing and programmers.

Well, that's the theory. I've never worked in a place where that would have been implemented. Instead, people trying to bring this in have been kicked out. In practice, maybe one should try to get a feeling of each API: how is it supposed to be used? Use each piece of software only in the implicit limits of it's programmer's idea to keep the number of bugs down. Not to mention the obvious coding style mantras.

Re:Trying to fit in implicit restrictions

[Twylite]

This is excellent advise. In my experience, the most stable code comes from pragmatic design followed up by pragmatic coding.

Design your system thoroughly. Identify every component, and the minimum interface required for that component. Carefully document that interface (API) - use Design By Contract (preconditions, postconditions and invariants) if possible.

Moving targets mean that the API will almost certainly have to be extended - documentation on the design and intent of the component/API as a whole will reduce the pain of this process. The responsibility for this documentation is shared between the design and implementation phases. Pay careful attention to documenting assumptions made within the code, e.g. ownership of member/global variables.

When it comes to coding, start with a skeleton. Put in the API function/method as defined, then check/assert every pre/post condition. Think about how any parameter could be out of range, or violate the assumptions you make. Once you are happy you're checking for all illegal use, you can go on to code the internals.

When coding internals, remember that you cannot trust anything (with the possible exception of other code in the same component). Check/assert the return values (and in/out parameters) of all calls you make. Have a well-defined system-level design for error handling, that doesn't allow the real error (or its source, if possible) to get lost.

As for testing, I'm all for the XP method: write your test cases first. This helps you to think about what you API is doing, how you are going to actually use it, and what you can throw at it that may break it (helping you to lock down the pre/postconditions).

You must use regression tests! Testing is useless if its done one, but the code is modified afterwards. Have a library of test cases, and use all of them. Every time a bug is found, add a test case for that bug, and ensure it is regression tested every time.

Code audits can detect and solve a lot of common implementation bugs. Use them to look for unchecked pointer/buffer use, assuming return values or success/failure of functions, and that asserts are correctly and accurately used.

In my experience most bugs do NOT come from implementation errors, but from developer misunderstanding, especially late in a project or in maintenance, or even during bug fixing! A developer must fully understand the code (s)he is working on, and all the assumptions it makes. Never adjust a non-local variable without first checking all other functions that use or modify that variable, and understanding the implications. Never use a function or method without understanding all the side effects (on parameters and scope/global state). This is why all of this information should be documented, and audits performed to ensure that the documentation is accurate.

Re:Trying to fit in implicit restrictions

[Grab]

Re (2) and (3), having separate groups is a *really* good way to get the old "them-and-us" battle going. The testers hate the programmers bcos they see themselves as covering up for the programmers' lack of skills; the programmers hate the testers bcos the testers keep telling them that they're screwing up; and everyone hates QA for telling them how to do their job.

The problem is that very few bugs occur at the purely code stage, and most of them are easy to trace. The real problem is design bugs - they're the killers. The solution at our company is to (a) review, (b) separate development, and (c) follow a V-model quite strictly. We have one group of engineers, where everyone writes, codes, reviews and tests, and for each section, everyone will have a different role so no-one gets stuck just being the tester.

Someone writes a requirements spec, and also writes the system test spec by which they'll prove that the thing works as expected. Writing the test spec forces you to put numbers to your requirements, basically making you self-review the requirements for errors, and it find a lot of bugs. Someone else checks that the requirements spec makes sense, and also that the system test spec matches up with the requirements spec. And typically we spend over a quarter of our time at the requirements stage, bcos it's dead easy to edit a single line in a Word document but it's a helluva lot more difficult to change a zillion C files, test specs, etc.

If the system is simple, we'll go straight to code. But if the system is complex, we do a detailed design (usually in Simulink and Stateflow these days, since we're coding for embedded control systems). The person who does the design MUST NOT be the same person who wrote the requirements - this effectively gives us another review of the requirements, to catch any oddities which can't be implemented. And someone will review that the design is actually meeting the requirements.

The same person who's done the detailed design will also write a test spec to say how to test the code. This will cover all boundary conditions, so any time there's a comparison, for instance, we'd check that it gets ">=" instead of just ">". We've got an in-house tool which allows us to write test specs in Excel and run them on code automatically. And someone will review this to make sure it matches the design and covers all cases.

Then someone writes the code. The coder MUST NOT be the designer - as with the requirements/design separation, this gives us a free review. They'll put that through Lint to check for obvious problems (we follow the MISRA coding standard, with a few exceptions where MISRA didn't get it right), and then run their code through the test spec. If it fails, they'll look for the bugs in the code. Sometimes they find bugs in the test spec; in that case the test spec gets modified. Having an automated test spec means we can run tests on code with zero overhead for repeating the tests.

And then the code will be run against the system level tests, and hopefully everything passes. If it doesn't, the system level test has found a bug in the design, where the design isn't meeting the requirements. Rinse and repeat.

It's worth saying that we're writing code for automotive control systems. How much testing to do is really a trade-off of the cost of testing against the cost of failure, and in our case, failure is not an option!

Grab.

Need more data...

[joto]

It's impossible to tell what's wrong in your case, since all you've said so far is akin to "we find lot's of errors, should we test less?"

And the answer to that is of course: "No, you should test more, and fix the bugs". And of course, looking over your development model to see why you have so many errors might be a good idea (such as formalizing who can commit code, if you've got lot's of programmer at various skill-levels).

But in real-life, many bugs are not that important, and time-to-market and development cost is more important.

So unless you provide us with more data, such as ...

development process, type of product, size of product, complexity of product, severity of bugs, age of most bugs found, bug-review process, testing schedule, testing process, how you manage your version control tree, whether you are on or behind schedule (and if behind, how much), how management deals with the problem, etc...

...I don't think anyone will be able to give any good advice either.

Re:Need more data...

[pyrrho]

I assume the poster thinks most of the bugs are not serious, and they are willing to just not see them all. I mean, if these are fatal bugs then there is no question, you need to find them.

So I say, be happy if you have a list of thousands of bugs... just prioritize them and possibly decide which ones will be "In the Shipping Version".

Code Review, Code Review, Code Review

[JohnT]

My experience has shown that the number one way to find defects is code reviews performed by other developers who can read the code and also understand the intended functionality. This will catch 90% of all defects before they are even released to QA.
For more information, the developers bible (IMHO) Code Complete (available on Amazon and elsewhere) has some good information on testing strategies and some hard numbers on effectiveness of testing. Good luck.

Re:Code Review, Code Review, Code Review

[Lumpish Scholar]

My experience has shown that the number one way to find defects is code reviews performed by other developers who can read the code and also understand the intended functionality.

Violent agreement, for the following reasons:

Study after study has confirmed: Code reviews find more defects, per staff hour, than any other activity, including all kinds of testing.

Aside from that, the benefits of having more than one person aware of each change to the code are significant. If George is sick, or quits, or wants to go on vacation, it's not just George who can make the next change.

Re:Code Review, Code Review, Code Review

[ssclift]

Again, violent agreement. Why? Testing is basically just writing the code again, only in a more restricted form. You take a known input, and then program the output expected (rather than derive it another way) and then compare the two implementations.

Inspection, on the other hand, compares the program with it written in another form: human language. Since human language is generally too vague to execute automatically, the only way to test the equivalence of the two is to inspect.

By far the best inspection book is Software Inspection by Tom Gilb. His very generous web site contains a ton of supplementary material.

Remember, proving two statements are the same is the halting problem, and is NP-complete (i.e. you must check all possible solutions). Testing is a measure of code against code, inspection a measure of code against requirements. Together they kill a lot of bugs because they find different discrepancies between three statements of the same problem.

Re:Code Review, Code Review, Code Review

[Zathrus]

We need to do a code review in my shop, since we're approaching release on our project, but there's a slight issue... there are two coders on the project (myself and the senior coder), and we're the only ones in house that know C++ very well.

What do you do in that case? Self-reviewing the code is of questionable value -- since you tend to skim over the parts you wrote "because you know it works!".

Re:Code Review, Code Review, Code Review

[Kombat]

I strongly disagree. It may have been true in the past, but you have to consider that languages, compilers and tools have evolved considerably since the idea of "code reviews" was first introduced.

Designers should rely on their tools. We're a Java shop here, and we employ everything from JUnit testcases for whitebox testing to JProbe for memory leak testing. We also use JTest for lint-like output. This, combined with selective code reviews of trouble spots gives you the most bang for the buck.

You must remember that code reviews will NOT catch the most insidious and costly mistakes, such as general architecture design flaws, race conditions, and memory leaks. You may catch a few, but peoples' eyes glaze over pretty quickly, and these are tricky bugs to catch.

In short: USE THE TOOLS! That's what they're there for. They've come a long way. They're very good at what they do. And there's no substitute for firing the product up and just plain using it for a while.

Re:Code Review, Code Review, Code Review

[Digital_Quartz]

Well, no actually.

Requirements review and Design review find more defects per person hour than code review, especially when you consider that a single requirement defect will result in multiple design defects, and a single design defect in multiple implementation defects.

But yes, code review is a good plan all the same.

Re:Code Review, Code Review, Code Review

[broody]

Buy this book, Handbook of Walthroughs, Inspections, and Technical Reviews by Daniel Freedman and Gerald Weinberg.

Re:Code Review, Code Review, Code Review

[bluGill]

Maybe, if you code reviewers are any good. Here the process won't let any code be commited until it has been code reviewed. However the following code would pass code review:
// This function is invoked when a mistle launch is initiated to make
// sure that two different users have both turned their key in the lock.
// The locks status is stored at location 0x23236341 and 0x65a3e222, and
// follow the format of the electronic lock (document number 235235)
// It returns TRUE if both locks have been turned, otherwise false.
bool lockCheck()
{
return TRUE;
}

The code review would flag only that missle is spelled wrong in the comments.

So if you can get quality code reviews, more power to you, I'll agree they are worth doing. If your code reviewer isn't very good, then it is a waste of everyone's time to do them.

Re:Code Review, Code Review, Code Review

[dubl-u]

there are two coders on the project (myself and the senior coder)

Do you already review each other's code? If not, try that. Also consider trying pair programming, which gives you a continuous code/design review.

Otherwise, consider hiring an outside contractor to do the code reviews. Or even a couple of them to give you different perspectives.

we're approaching release on our project

This is not the best time to start code reviews. Why? At this point you've written a bunch of code; you're also facing release pressure. The only things most people would be willing to fix at this point are clear bugs that can be fixed without changing the design. Anything else you'll likely be forced to ignore (or hack around).

The earlier you catch a problem, the cheaper it is to solve. In your next design cycle, start code and design reviews early!

Re:Code Review, Code Review, Code Review

[dubl-u]

Designers should rely on their tools.

I think you may be setting up a bit of false opposition here. I agree that tools are good; I use many of those same tools myself. But when you say

[...]code reviews will NOT catch the most insidious and costly mistakes, such as general architecture design flaws[...]

you go too far. If there's a tool out there that will catch design problems, I've never met it.

Software development is an expression of human desires in languages meant simultaneously for human and machine interpretation. Good tools can help with the machine interpretation side, but only inspection by other humans can catch problems relating to human desires or human interpretation.

Design walkthroughs, code reviews, and pair programming can address problems for which there are as yet no good tools.

Deciding whether to use a code inspection or a profiler depends entirely on nature of the problem causing the poor software quality.

Re:Code Review, Code Review, Code Review

[laserjet]

Isn't your comment kind of redundant? Of course if your code reviewer sucks, it will be a waste of time!

That's like saying, "Unless you buy good beer, your beer will taste like crap."

In either case, it pays to hire someone who is qualified. Why hire someone who will not help?e

Re:Code Review, Code Review, Code Review

[Grab]

Depends. If ppl go straight to code, then the code *is* the design. So review the code for all of this.

If they do a detailed design first, then you should be taking the next logical step of reviewing the design against the requirements. Then the code gets reviewed against the design.

The thing is, no tool can catch the classes of bugs that are thrown up by these review steps. No tool EVER will, bcos what these steps are doing is taking a concept expressed in some customer's Word document and over a bunch of meetings and turning it into a design document and some code. Believe me, if you've found a tool that can do this, you've cracked natural languages, AI and the Turing Test in one go!

That's not to say that tools aren't useful. If you can have a tool manually search for "if(x=0)" then that's useful, so there's a good chance that you can at least catch most of the typos that way. But no tool will ever replace a review by a human.

Grab.

Re:Code Review, Code Review, Code Review

[angel'o'sphere]

Your post has some points but your terminology is wrong!

Remember, proving two statements are the same is the halting problem, and is NP-complete (i.e. you must check all possible solutions).

Thats wrong.
C:
int a = 1;
int b = 2;
int sum = a + b;

Pascal:
a, b, sum : interger;
a := 1;
b := 2;
sum := a + b;

Different statements. Semanticaly equvivalent. You prove this by using a meta language which has a regourus semantics or in this case by looking at the assembly codde. The compiler output is in both cases the same. No halting problem involved.

Its more complex if you use a functional language and have a = 1, b = 2, sum = (add a b).

Now you likely have no assembly output to compare but you can define for all languages a simple pseudo code with a defined semantics and mappings form your language to the pseudocode.

Often you transform the procedural code INTO a functional language and just evaluate it to a final expression ... if both expressions are the same the code is equivalent.


Testing is a measure of code against code,
No idea what you mean with that ... the test is performed WITH code but usualy you test expected output versus real output for a given set of input.

inspection a measure of code against requirements. Thats wrong.
Inspection is going through a check list and "inspecting" wether your code matches formal criterias which have absolutely no relation to the intend of the code.

A typical point on a check list for an inspection is: for all if statements involving the character "=" in an expression inspect wether the character only appears once and report: Likely bug as it is an assignment!!
Another one:
For all if statements involving the character "=" in an expression inspect wether the left side is "const", if not report: Likely bug as it COULD be an assignment!!

If you use C++ a typical check point on the check list could be:
For all parameters to a member function or function check wether they are modified inside of the method body and if NOT then check if the parameter declaratin is either const, or const reference or pointer to a const object. IF NOT; report that you have fragile code as the the code will behave in its context of use totaly different if a single line inside of the function is changed and by that a parameter gets changed.

Inspections and most other ways of code reviews are a formal way to check wether a piece of code adheres to a set of coding guidelines.

The coding guidelines are set up to avoid typical bugs.

Ah ... the most typical check on top of the check list: are all local variables initialized?

This has absolutely nothing to do with the intended calculations the code should do. You never look on the requirements while making inspections. Probably you do it if you do make a walk through.

angel'o'sphere

Start at the beginning

[yogi]

Make sure that testing starts with each developer, so that they attempt to break all of their code before it goes anywhere.

If you look at the guys with really low bug rates, like the NASA guys running the Shuttle control software, they have very separate test and development teams, and a competitive attitude. The test team "wins" if it finds a bug, and the devlopers don't want to look silly.

Some Extreme Programming techniques, such as paired coding may help too.

Re:Start at the beginning

[cheeto]

Yes, we use formal methods to design the programs before we start to write anything. We are an SEI Level 5 shop, but most of you guys would hate to program for us. Once the requirements are approved, the programmer has virtually no freedom. However, the programmers are intimately involved with the requirements review process. Half of us involved in producing the Shuttle Flight Software have never written a single actual line of code at the terminal.

For instance, I was reviewing some code a few months ago that said to compute an array of three vectors. Well, we were actually only going to use the third element of each array, so the developer only stored the third elements.

That was OK, but we also ended up documenting that in the detailed design spec, which is thousands of pages in itself.

Most desktop apps can't afford our process and probably don't need it. Embedded software in avionics equipment probably do need this process and can't afford not to use it (I don't know if they all do).

Quality is Designed In...

[reptar64]

... NOT tested in. If the product is poorly engineered, there should be no surprise at the vast number of bugs, no matter how much testing you do. Crap is crap.

Wrong problem..

[psycho_tinman]

The main thing: testing does absolutely nothing to minimize the number of defects in a particular application.. There are lots of other things that are as important.. ie: are these defect reports being seen by the appropriate developers and are they being acted on, what types of procedures and communication actually exists between the developer and the QA persons (assuming that they are not the same folk)..

The last point isn't as bizarre as it sounds, I've seen lots of places where a QA person enters bugs, but the developers silently reject them ("its not a bug, that's how the program works")

Testing just tries to discover the presence of defects, by itself, it cannot ensure that your product works perfectly (for an application of even moderate complexity, there may be an exponential number of cases and paths to check, most test cases are written for a percentage of those only).. Because of this, if you feel that you're spending too much time testing, perhaps you need to check if your test cases are appropriate to the situation and stage of development..

Another point is that tests can be automated to some degree or the other, perhaps a scriptable tool might assist in lowering some of the drudgery associated with actually assuring the quality of your software...

rant mode = on...Excessive testing ONLY hurts if it takes people away from development at the early or even middle stages of a project and forces them to run tests on incomplete sections of code.. otherwise, there is NO such thing as too many things...

Irrational users

[AVee]

"I wonder how much testing is too much"

Well, it's never to much, until you've done every possible thing. Thats one of the advantages of open source development. A lot of people are working, playing, coding with the beta version wich means a lot of things get tested. In my experiance most errors show up with unexpected things, like missing error checks etc. A good point to start testing is to test agains the developer. Ask him everything, what if i do this, what if i do that. If you think crazy enough (like normal users), you will recieve a dozen of 'euh, don't know' answers. Thats where your errors are ;-) Most tests are based on what the system is supposed to do, not on what one can do. Given a few users anything can be done to the program, things you likely didn't test because you understand the program to good. Try putting your mother behind the prog ;-)

Testing is just part of good design/implementation

[GnomeKing]

Its all in the same package...

If you have a well thought out and well structured design... and if every single function is documented atleast mostly before coding begins... and if all of the theory of interactions fit in... then only a small amount of testing is required

The main problems which require more testing come when corners are cut over the initial design stages, or their not done as fully as they should be doing, or when people dont actually think about what input users can give

For one of my projects I did ages ago for college, half of the class (which was split to give the same coding ability spread in both halfs) were each asked to write this program and the other half were asked to design it, review the design, make a proper testing strategy, document and THEN start writing it

The second half took a little longer to get their program working, but the first half had more bugs, and spent more time testing ... My bet is that the second half would spend less total time on that program and bug fixes in a real world situation (which was the whole point of the exercise)

For me

[pkplex]

Well.. showing your work to someone important always brings up bugs, every time :)

I suppose its just a case of using the product like an actual user who knows nothing about it would, right from the first step.

Its been said before, but...

[tgd]

If you really want to generate bug free code, you have to keep one rule in mind at ALL times. A bug occuring in the code is a failure in the methodology you are currently using to avoid them. Sounds very basic, but a lot of companies forget that.

When you have a problem with bugs, you need to figure out where in the process the problem happened. Was the unit spec wrong? Documentation? Implementation? Unit testing procedures? Was it a correctable problem caused by the engineer involved?

If you really want to be bug-free, every time one shows up you have to figure out why it happened, and change things.

Personally, I think the biggest one is to make engineers work 9-5. Not 9-7, or 11-9. Tell them to go home at 5, even if they're in the middle of something. Software engineering is a very complex task that takes a lot of energy and concentration to do right. Just like Doctors who work long hours make mistakes (resulting, often, in people dying), engineers who work too long make mistakes too.

Being in the "zone" is often the death of good code. You get lots of cool code written, but none of it is double-checked, none of it is verified to match spec, and it often ends up afterwards difficult to understand.

Now don't get me wrong, I don't do any of this, and my crap is FULL of bugs, but thats what you need to do if you really want to help it. Writing buggy code is like a public works program for QA people. Who wants a hundred thousand unemployed anal-retentive QA people nitpicking something else like your car's inspection or your tax forms? Better to keep them in the software where they can't do any harm ;-)

Design by Contract

[Yousef]

Another good way to reduce errors is to follow the principals of Design by Contract.

State using Assertions what is expected of the code. Pre Conditions and Post Conditions.
If any of these fail, then throw an appropriate Exception.

Eat your own dogfood....

[T-Ranger]

and force feed it to others.

Mathmatical proofs and unit testing asside make sure that your program dosent crap out when its being used or abused. So get some regular people to bang on it..

Never..

[kb]

... trust the programmers when it comes to testing. You may find some obvious buigs in your own code, but when it comes to runtime testing most programmers tend to emphasize on "correct" user behaviour or maybe the few wrong input sets they've taken care of and completely neglect the pervert fantasy of lusers *g*

Dedicated testers OTOH are perfect in letting a program die in the most obscure ways. Did you know that you can crash a Commodore 64's BASIC interpreter by just typing PRINT 5+"A"+-5 ? ;)

programming and debugging are the same thing

[dikappa]

IMHO, programming and testing should be done at the same time in the development stage.

While programming and "bugging" happen at the same time, programming and de bugging/testing should happen at the same time too.

It is very well explained in Bruce Eckel's Thinking in Java . You should just test everything in the code itself, even if it happens to add some overhead. Once called that function, you want that <something> happens.. so check it in the code.

I know this is not the usual way procedural programming happens. It seems much more straightforward to drop the code as it comes and then check if it behaves correctly.

But if you do so you will often discover that that tests made afterwards ara not comprehensive of all possible situations.

And so you discover that testing and debugging are just unfinished tales, and it is even worst if testers are not the programmers who did the work.

Plus, I hate testing, so I force myself to do the work well and let the code (as long as possible) test itself, even if it makes development slower and boring.

Umhh... i'll preview this post 10 times, hoping it's free from bugs :)

Obviously my code contains no ewwows ;)

Re:programming and debugging are the same thing

[interiot]

The test code could well be three times the size of the normal code (IMHO, it could be much larger than that, but since I only have experience with the more standard way of debugging, I'll tone it down a bit). Doesn't this debugging code have to be bug-free as well? If the debugging code needs to go through the most of the whole process (at my place, the relevant steps would be design / code / unit-test / inspect), then it has the potential for doubling or tripling the time spent coding. It will reduce the number of bugs found later, but the number of bugs that pop up due to lack-of-regression-tests are pretty small overall. Perhaps it's not worth it to do this?

Re:programming and debugging are the same thing

[lpontiac]

The test code could well be three times the size of the normal code (IMHO, it could be much larger than that, but since I only have experience with the more standard way of debugging, I'll tone it down a bit). Doesn't this debugging code have to be bug-free as well?

Testing a solution for correctness, or probable-correctness, is usually easier than figuring out the solution in the first place. This means the test code will be smaller and less bug-prone than the code that does the actual work.

An example: sorting a list of numbers. An efficient sorting implementation can be rather complex, but making sure it works is pretty simple, even in C: min = list[0]; for(i = 1; i < length; i++) assert(list[i-1] <= list[i]);

Literate Programming

[SWroclawski]

This may not address your current situation, but literate programming can often help reduce bugs and clean up errors found later.

When a programmer is simultaniously coding and documenting thier code, at both the high and low levels, the larger "thought" bugs will decrease in number and severity.

Even if you don't use a literate programming system, often documenting the system before you write it can help make the code more clear.

- Serge Wroclawski

Too many people perhaps?

[tetranz]

I don't know how relevent this is but I read somewhere long ago that newspapers ended up with more errors if they had multiple people proof reading the same text because nobody was really taking responsibility. Even if it is not intentional, there is always a feeling of 'the other guy will pick that up'.

I vaguely remember something similar being said about the space shuttle disaster.

Re:testing...

[AVee]

... is the most annoyin part of dev

That's mostly true, but it gets better when you let developers test eachothers code. They will do anything to make it go wrong ;-)
And they are very descriptive testers most of the time.

Simple sata test and regresion

[oliverthered]

Good design is always the best way to avoid buggy, hard to fix code.
But for testing it depends what you testing,
a good general test process for data processes (most functions can be though of as dataprocesses) is

Generate some input test data,
Work out what the results should be by hand.
this is your first stage regression.

Now run the input test data through the application/function

Diff the results against your hand generate file.

Any descrepancies should be resolved as,
A bug in the application/function
or
A Bug in the hand generated files.

Fix all the hand-job problems

Repeat until you test files are perfect.

You now have a second stage regression test,
Known good inputs, and known good outputs

Use the correct test files to fix the application bugs.

If a bug is found that isn't in you second stage regression, then generate test files for it.

Fix the bug in the application(using the test files).

Then run you second stage regression and check that any differences are down to the bug that was found (and been corrected).

Following this process you application should always get better, and a you should soon be able to build up a fairly large sample of test data.

The test harnis is simple enough (just a diff on the files and a bit of code to wrap up the functions.), to prevent artifacts caused by the testing process.

Any-how that's more-or-less what I do for most of my testing, bug fixing.

Re:Simple sata test and regresion

[NetWurkGuy]

One problem that usually arises with this approach is that an very large quantity of data must be hand created to test all the paths in the function. This is not IMO a flaw in the procedure you recommend. It points to a design weakness: it means that higher level logic has failed to adequately sort different cases to be dispatched to different functions. This can be a result of abusing the notion of "hiding messy details" at the upper levels and pushing the problems down to the bottom. There must be a balance of hiding/deferring and exposing/processing of situations at every level to keep the complexity burden reasonable at each level.

Understand your real goals

[James Youngman]

Well, I'm assuming that your systems start with analysis & design, followed by coding, redesign and more coding, go through unit testing (with more redesign if you are unlucky), followed by integration testing once unit testing is complete, and perhaps acceptance testing once integration testing is complete. This is more or less the traditional waterfall cycle when the deliverable is the finished code.

This strategy works - lots of shops use it all the time. However, the real premise of the process is that you want to get through client acceptance testing as soon as possible, as long as the result is not dissatisfaction on the part of the client with the software after they've accepted it. As you have noticed this strategy doesn't actually produce bug-free code.

This is not surprising. What you achieve is after all pretty much determined by what your goal was. You (shops in general) need to think hard about what your actual goal is. If your goal is nearly-zero-defects, then the traditional process isn't doing the right things for you. If however, your goal is to obtain milestone payments from your client, then it's pretty good. This is an area where the business goals determine the software engineering processes.

Let's put another hat on and think about what the negative affects of this strategy might be (negative is really defined in terms of what your goals are, but let's be vague about that for a moment).

• If your goal is not "zero bugs" then you will stop work before there are no bugs left
• Your software is delivered to the customer with bugs in it, which the customer will find
• Your development team will partly move on to other areas, probably leaving a smaller number of people to deal with the remaining bugs
• Maintenance programmers are typically less skilled than some of the original team - because some of the original team have been pulled off to activities which are more important to your business (e.g. delivering another set of code in order to meet a payment milestone somewhere else).
• Skills evaporate over time - after N years it gets very difficult to find authoritative information on why something works like it does.
• As the code is fixed, it gets brittle. The emphasis is on just fixing bugs and making low-risk changes in order to avoid breaking the production code - hence refactoring is rare.

All of the above factors are unpleasant for those left to maintain the code. Many of them also limit the longer term flexibility of the product and hence the useful life of the software. This feeds back into development processes because limited product lifetimes mean that there is less incentive to change your process to produce software which can persist (i.e. why make the effort to ensure that the system is flexible enough to last through 20 years of changing requirements when you expect the system to be retired after only 7 years?)

You mentioned XP - it offers a lot of techniques that resolve these problems:-

• Programming in pairs - this makes for very efficient skills transfer, hence you limit the extent to which the expertise boils off
• XP testing aims for automation - this encourages more testing and the stronger testing ability allows you to contemplate high-risk-high-return activities like refactoring.
• Refactoring - which prevents your code getting old, brittle and hard to change
• Many more I'm sure (I'm not an XP practitioner)

However, XP is best adapted to projects where a single team makes multiple frequent deliveries of code, can work closely with the client, and where the development project continues in the medium to long term. These characteristics allow many of the XP techniques, and this means that techniques taken out of XP may not help projects of a different style.

Having said this, the automated testing angle is a real strength. If testing is done manually, it's time consuming and expensive. Hence people don't do it as much as they might otherwise thing is appropriate. Maintenance deliveries often just undergo regression testing, and faults can creep in which might have been caught by the original unit or integration tests. Automated testing has many advantages :-

1. Automated tests are faster, so people actually do it!
2. You can redo all the tests after every change if you want.
3. Automated testing allows you to refactor without danger
4. Being able to re-run all the tests really does keep out the new bugs which would otherwise have been introduced during maintenance
5. Your testing coverage grows with time (since bew tests are introduced but tests are only retired when the relevant functionality is changed or dropped)
6. You don't fail to spot errors (quite often with manual testing regimes errors can go unnoticed because the tester doesn't spot a small bit of incorrect behaviour that the original team might just have spotted).

Just as a data point, I work on some software that has an automated test suite. The suite contains between 500 and 1000 test cases; the test suite conducts those tests in under 5 minutes on a very old machine. To do these tests manually would take one full-time person at least a week.

The summary is :-

• Understand your business's real goals
• Cherry-pick techniques that will help achieve those goals (you might even be able to adopt a whole methodology if its processes are designed to achieve the kinds of goals that your business actually has).

Missing in the discussion

[HiQ]

What I miss in this discussion is something about the persons performing the tests. In my companyh we have a test team, consisting mainly of people who don't know the first thing about coding, who cannot read sources and who can only test 'through the UI'. And yet the system we work on has thousands of sources, a percentage of which has a UI (20%). Testing of all the underlying objects is a lot harder, and my experience is that with this many sources the total amount of possible 'paths' in the system is so large that tests using the UI take too much time, and therefore is never done properly. So now the developers are constantly asked to provide methods by which the testers can perform the tests.

Biggest bang per buck: case specific testing

[jukal]

I believe that most bang per buck can be achieved if the organisation is not too fixated to one or two standard testing procedures. Projects differ a lot. Using 30 percent of testing budget for testing the testing plan might well be worth the effort. If your company is making a set of applications for a fixed platform using fixed components and fixed architecture and these basics have been previously thoroughly tested, then ofcourse what was said above might not be true.

Get back to programming basics.

[Anonymous Coward]

Having been on teams producing 24 X 7, bullet proof code for communication servers and credit card processing I have an idea about the increasing number of bugs found. In the Old Days(tm), we wrote every line of code ourselves and used time tested libraries (C language). I quit using microsnot when their libraries stared having bugs in their rush to C++. Now most coders use massive OOP libraries from who knows where built by slackers, and GUI app builders that generate code and perform all sorts of actions under the hood. When something goes TU it is often hard to find all the conflicts.

Even when using one of these app builders I read through all the code and put tests and logging into the generated code. Funny that these tools are supposed to make us more productive. My coding and testing every line still beats total time spent on a project since I don't have to go back and redo it later. When it's done, it's done. Next project. I've had comm programs run for over 5 years error free servicing 1000s of users per day. One specialized delivery, billing, and inventory system I wrote was used over 6 years error free and caused the owner to stay with hardware that was compatible with the software (not M$) because the programs always worked. And not a damn bit of it was OO or came from some automated builder tool.

In short, the closer you get to the metal and the more familiar you are with the code that is executing, the better your chances of producing error free programs. Takes longer to market, but then you don't have to redo it forever until the next bug ridden version comes out. Saves time and coders to work on the next version and the customers are always pleased. Get back to the basics. Try it, you'll like it.

Re:Get back to programming basics.

[jukal]

I agree 100%

The only perverted problem with this approach is that in case you are selling your code to some 3rd party company for example, and you are competing with a set of other companies you might have terribly hard time trying to make the customer understand why your development takes 3 times longer than what is promised in other proposals. This is not so much problem when you have a proven track record clearly stating that the complete time wasted is less using your approach. Still, in some cases, the customer might be stuck to some bizarre outsourcing process with inherently excludes all proposals that exceed the shortest estimate by some magic percent.

The point being that many of current customers for software support this bizarre approach of first jumping to lake and then seeing if you hit a stone or not. Ofcourse its not easy job to decide to pay $100 for a candy, if you can get the same thing with just some dog dung added for $33. Especially if the software provider does not say that the dung is included in the offering.

Re:Get back to programming basics.

[scrytch]

In short, the closer you get to the metal and the more familiar you are with the code that is executing, the better your chances of producing error free programs

I would claim that the first two conditions are often mutually exclusive. I spend far too much time chasing down fencepost errors because some nitwit used a C array and a for loop instead of a STL list and iterator. Said fencepost errors are known to lead to root compromises when performed on the stack (but usually it's on the heap and tends to cause heap corruption ... sometimes, in ways that are hard to detect). And often I wonder why an app that spends 99% of its time waiting for user input is being written in C++ anyway, but that's a whole other story.

Basically, I get mired in some stupid piddly implementation detail instead of being able to think about the system, because someone decided to program to the machine instead of to the spec.

No, libraries and frameworks are good. Just don't get seduced by frameworks that claim to do everything, because they usually do so ... poorly. Use the right one for the job, and one that's been around the block a few times, and you'll find yourself producing code that works, and even getting over that uneasy feeling that it shouldn't actually work the first time

Re:Get back to programming basics.

[tshak]

Having been on teams producing 24 X 7, bullet proof code for communication servers and credit card processing...Now most coders use massive OOP libraries from who knows where built by slackers

Although there's some truth to your point, I think your first statement is what really matters. You worked for a company that made software for credit card processing. This means that you A) probably didn't have to continually debate the importance of software quality and B) probably had a budget to allowed for decent programmers and reasonable project timelines.

Although MFC was originilly pretty bad (what a surprise), it really has stabalized. Win2K is objectively a stable (read: not secure, stable) OS from a software standpoint and it's full of software with these so-called "unstable OOP libraries using GUI app builders". Also, from my personal experience, the base class library of .NET is also extremely stable (albeit it hasn't been around too long). The Java libraries have also gotten a lot more stable and are used in many mission critical e-commerce or financial systems. Really, there's no point in reinventing the wheel unless we are concerned about the wheels quality. Five years ago I may have agreed with you more, but times have changed and class libraries have matured greatly.

Re:Get back to programming basics.

[jfengel]

I think that your success may have as much to do with the sorts of programs you've been asked to wrote as your programming style.

A comms program does the same thing, over and over. If it runs for one day, it'll probably run for five years. The number of novel scenarios it encounters is relatively small.

While delivery, billing, and inventory systems are driven by many complex algorithms and sets of rules, they, too, tend to do the same thing over and over. Once they work, they tend to go on working; six years is the same thing as a single day to a piece of software.

You're talking about a specialized system here. If you'd written a general inventory system, which has to deal with every single customization that the end user will try to come up with, is a vastly more complicated piece of machinery and is much harder to test.

It sounds like you're used to working on small systems, with a single programmer working from scratch. The larger the system, the harder it is to examine every single line yourself, and track every single one of the dependencies. OOP systems are one way of encapsulating the work so that the dependencies are reduced and the task becomes manageable even when done by a large team.

It is without a doubt true that "the more familiar you are with the code that is executing, the better your chances of producing error free programs." However, demanding 100% understanding of the code puts an important limit on the sorts of programs that can get written.

Hurt yourself?

[richie2000]

do you feel that excessive testing hurts the development process at all?

Testing should always be a part of the development process. The wording here implies that testing somehow is considered to be outside the scope of development and I suspect this mindset is causing a lot of bugs to remain undetected.

It's just like documentation or support, those are also (or damned well should be) integral parts of the development process. Sometimes I think that most programmer's believe that the development process consists of the steps hack, compile, ship instead of the tedious iterative process of analyze, design, code, test.

So what, then, is excessive testing?

Well, as long as you find bugs doing it, it's not excessive.
If your projections predict a bug or two in a specific piece of code and your tests fail to find them, then testing (provided that the test method isn't flawed) gave you a much desired quality assessment of that piece of code - meaning that the testing still wasn't excessive.
Running the same tests over and over on the same code with the same data, now that's excessive, not to mention stupid.

Wanna know a secret?

[Kanasta]

When someone is told to implement feature A, they spend a little time sifting thru the 20yr old code, and do the minimum to get it done.

They write test cases to test A, unit, system, etc. Their team leader approves it, and of course all the tests pass before it goes out.

In the end, there's always some obscure way feat A interferes with feat B, but you're not going to write tests for every combination of keystrokes possible.

If you have user testing (u should), they'll find a score of bugs u didn't. Of course, the users too have a deadline to get stuff shipping and they too want to do the minimum possible.

In the end, nobody involved has a personal incentive to make it perfect. With proper testing procedures in place, everyone has a piece of paper with someone's signature on it which says they passed. They don't feel (too) guilty when there's a bug, (hey, my TL approved it!).

Anyway, how are you going to ask the customer for $20k extra so you can test for an extra week?

Suggestion

[ceeam]

I guess you should try to spend a part of your testing budget on improving your design and programming practices.

Close the loop

[edhall]

The object of finding bugs isn't to result in fewer bugs by fixing them. It's to result in fewer bugs by not writing them in the first place. The developers need to review found bugs on a regular basis, with the objective of changing development methods to avoid them in the future.

It's all fine and good to say "don't write buggy code in the first place," but this sort of feedback is the only way to get there. What makes this so hard in many organizations -- aside from the usual disrespect many developers have for QA people -- is that developers fear that this process is some sort of performance evaluation. As soon as this happens, the focus shifts from finding better processes to defending existing processes: "It's not really a bug," "There isn't really a better way of doing that," "We just don't have time to do it the 'right' way," and so on.

This is why the feedback needs to be direct from QA to the developers, who are then tasked to categorize bugs and develop recommendations for avoiding them. It's the latter that is the "product" required by management, not a list of bugs with developer's names on them. Management should otherwise get the hell out of the way.

-Ed

Re:Close the loop

[jukal]

> This is why the feedback needs to be direct from QA to the developers

I agree. For some reason (maybe it's just me) developers are nowadays too full of false pride as well, thinkin: I am the lead coder, analyzing bugs is the job of trainees. In my opinion the situation should be (atleast in some cases) completely opposite, only veteran coders can make correct assumptions and define pre-cautions for future and fix this particular case in the correct way. Otherwise it might just lead to a decline of the original code - making things even worse.

Working with bugs is a tough job, do it with pride! *with the allbugfixers unite anthem playing gently on background*

Re:Close the loop

[garett_spencley]

developers are nowadays too full of false pride as well, thinkin: I am the lead coder, analyzing bugs is the job of trainees. In my opinion the situation should be (atleast in some cases) completely opposite

I don't know who your company is hiring but where I work things are as they should be. It's the college kids who are hired for the experience are the one's with cocky attitudes while the advanced developers are trying to push management to get better processes in place rather than just "Fly to the moon by next thursday."

... Then wednsday comes and it's "No wait, fly to Mars instead. But still do it by tommorow".

At the moment our company isn't as structured as it should be. We don't have a QA team or a testing team. It's just management pushing the developers to get thigns done. But the developers are pushing back to say "hey, we need a process here. It's not just writing code. We need to design it first. And that takes time. We also need to implement code review and pick someone who's got the experience to decide what goes in CVS."

So my point is that in my experience it's the inexperienced developers who want to just jump in and write code thinking "it's not my job to fix bugs". I think this has a lot to do with wanting to get that advanced status. But as they grow they realize that they're going about it wrong and smarten up with regards to processes.

--
Garett

Re:best way to test is to use automated testing

[Cpt_Corelli]

Of course properly written functionality test scripts (doing what the user does) will find most bugs. The downside is that it is boring to follow test scripts manually.

My company has been successful implementing automated functionality tests with Rational Robot (part of teamtest). If you just take the time to define proper test scripts you can easily redo all functionality tests on various platforms (if you use VMWare or similar sw to simulate different platforms) at the click of a button.

This saves time every release as the developers can focus on finding the really tough bugs instead of running boring functionality tests again.

Developer testing...

[fcrick]

I've worked on both ends (dev and test), at M$ and other places, and I've come to one conclusion (I'm sure its not the only correct one).

Developers must test their code.

With a test team backing you up, it becomes too easy to change something, run it once (if at all), and then push it into the next build so the test team can catch your errors. I've found that as a tester, a huge proportion of bugs are simply features implemented where the developer just forgot something stupid. I end up wasting 5 minutes writing a report, my manager assigns the bug back to a developer (hopefully the one who made the mistake but not always), and the developer comes back to the code a week later, spending 20 minutes just trying to figure what s/he wrote a week back.

My point: this wastes 30 minutes of people's time for every little stupid mistake. Pressure your developers to really give a thorough test to the code they write before the check it in, especially if you have a test team, because you just end up wasting more people's time.

Speaking from a non programmer point of view

[q-soe]

I often think the time lines have become so compressed in terms of expectations that it becomes harder and harder for companies to write clean code and get it out the door in order to meet the expected cycle of upgrades - and this is something i find common to all companies and even open source software. It seems we as consumers have come to expect an upgrade to this or that every year and so the dev cycle becomes one continuous thing - coders who are exhausted or working long hours write buggy code.

I think many people would be happy to wait a bit longer for better products but the industry has brainwashed them into thinking that its almost easy to bung a new version out.

The best way to avoid mistakes is not to make them in the first place but thats not so easy when working on a compressed cycle with management on top of you - its not just programmers who deal with it - its network designers, SOE architects etc etc

Bugs exist - they always will - but minimising them requires time and time is a commodity not readily found. I dont know what the solution is - as i say its just my thoughts.

Product or process problems ?

[PinglePongle]

You say you're finding too many bugs - it sounds like there is something fundamentally wrong with either the product or the development process.

First thing to do : look in your bugtracking software ( you DO use bug tracking software, right ?) , and try to isolate hot spots. Is there a particular piece of code that generates more bugs than others ? Is there a common pattern to the bugs (ie. memory not being freed, of-by-one errors etc.) ? Are they _really_ bugs or mis-interpretations of the requirements or the design ? In my experience, the 80/20 rule applies to bugs in spades - it is just hard to find the patterns.

If you need to, make the bug categorisation in your bug tracking software more specific. Once you get an idea of what your hotspot is, you can work at fixing the cause of the bugs.

If it's a particular piece of code, make sure it's reviewed by the best developers/architects you have, and consider refactoring it. At the very least, insist that it is reviewed and tested thoroughly before chec-in to the source code control system, and consider adding a hurdle to jump prior to check in (e.g. get the manager to sign it off).

If the code was written by one developer, consider swapping them out and giving it to someone else - it may be they're in over their head.

Make sure you increase the number of test cases for this piece of software, and check for "edge cases" religiously - if the code is broken at all, it is likely to be broken in more ways than you realized.

If it turns out that the problems tend to have a common cause (memory leaks, of-by-one errors,etc.) consider a structure which forces developers to concentrate on those issues before checking in code; again, consider the hurdle ("software must be signed off by the off-by-one guru prior to check in"), and hone your tests to check for these kinds of errors if possible.

If the bugs stem more from misunderstood requirements or designs, beef up those areas. Work on your requirements and analysis processes; consider training courses for the developers to get them up to speed on interpreting these nebulous documents, and look at improving the review process by having designers present. Frequent "mini-deliverables" (another concept stolen from XP) will help here too - get your team to deliver a working piece of code - it need only be a minimal sub-system - and get it reviewed by designers and analysts. If the bugs tend to occur on the boundaries - i.e. invalid API calls, invalid parameters etc. - consider design by contract or aspects.

Finally, there's a bunch of hygiene stuff :

• coding standards (ideally with tools to check for compliance) help by allowing you to specify how to deal with commonly bug-prone operations
• invest in a decent bug tracker, request tracker and source code control system - it really helps if you can trace a bug back to a particular requirement or previous bug fix if you're trying to find out where all your bugs are coming from
• seating arrangements - I worked on a project once where the bug rate was halved when the developers were moved to sit in the same room as the database team. Get everyone in the same room if at all possible
• automated test tools - already mentioned - help to ease the pain of testing. If practical, insist that the developers run the automated tests before checking in code
• religiously review all work products - designs, specifications, documentation, project plans, code, tests, the works - before declaring them "ready for use". It's the single cheapest way of finding problems, and a great way to spread good practice.

N

no one size fits all process

[jilles]

There's no one size fits all process for testing. How much effort you need to spend on testing depends on a lot of factors including but certainly not limited to: code size, amount of developers, customer requirements, life cycle of the system etc.

That being said, here are some remarks that make sense for any project:

In general a testing procedure that gives you no defects just indicates your testing procedure is bogus: defect free code does not exist and no test procedure (especially no automated procedure) will reveal all defects.

The XP way of determining when a product is good enough: write a test for a feature before you write code. If your code passes the test it is good enough. This makes sense and I have seen it applied successfully.

A second guideline is to write regression tests: when you fix a bug, write an automated test so you can avoid this bug in the future. Regression tests should be run as often as possible (e.g. on nightly builds). All large software organizations I've encountered do this. Combined with the first approach this will provide you with a growing set of automated tests that will assure your code is doing what its supposed to do without breaking stuff.

Thirdly, make sure code is reviewed (by an expert and not the new script kiddie on the block) before it is checked in. Don't accept code that is not up to the preset standards. Once you start accepting bad code you're code base will start to deteriorate rapidly.

Testing is a start

[shoppa]

Testing is a start, but not enough. The results from testing (e.g. "300 buffer overflows") have to be identified and fed back into the organization to change how future software is developed for the better.

Testing/debugging is like finding and putting out existing fires. If the organization can use test results to prevent future fires, then you're a step above. And probably more advanced than 90% of the software houses, too :-)

How I Test

[Apreche]

I program mostly in object oriented languages. So I have seperate files, which have seperate classes. I start at the bottom of my UML and work my way up testing each class as if it were its own program. When I know they all work individually, I can be certain that, despite the fact there had to be a few bugs I overlooked, that all bugs are due to the way they interact. It takes awhile, but in the end I'm mostly bug free.

Buggy degsign and planning

[oliverthered]

Most of the 'worst' bugs i've come accross are down to bas systems design, before a single line of code is written.
If a system is designed well then you should have far fewer bugs, even if you are using code monkies who don't know a quick sort from a n^2 bubble.

Design you systems well, know your people, Bill's good at that kind of thing and likes it(but crap at ui's say),
Jess loves doing data imports, (may not be that quick, but always does them well).
Fread always designes and produces good/fast systems cores.

Get your developers talking and sharing knowlage, 'I'm, having a bit of a problem' , or 'Who knows how to', are good things for people to be saying, so incorrage them to own up to the inadiquacies, and they won't have them for long.

If you can manage that then your productivity and bug counts should drop dramaticly, and the bugs you do have should be easier to fix.

Hold on! Don't use testing to fix a poor process

[SledgeHammerSeb]

If your testing is resulting in an inordinate amount of bugs, then there is probably bigger problems than you think.

Testing is necessary but not sufficient. There must be a way to capture requirements, convert requirements to design, convert a design to implementation, and finally test. At each transisition it is a good idea to make an assessment of how well you accomplished your task.

Skipping any of these steps and putting if off until test is pure folly. An extremely false economy.

Assault it with random data

[Sandmann]

The image loaders in the gdkpixbuf library included with gtk+ 2.0 were tested with random data. This caught a lot of bugs, including some in the standard image manipulation libraries.

Just generating random data and trying to load it caught a lot of bugs, but even more effective was to take a valid image and modify the bytes in it at random, and then try to load it.

Of course, the reason this was so effective, is that the loaders would get mostly what they expect, and then suddenly something illegal. This is the kind of thing you tend to forget about when you write code.

Since it is so easy to attack your program with random data, this kind of testing gives you a lot of bang for the buck, but on the other hand, the bugs it find may not always be those that are likely to occur in practice.

proper design + inspections ?

[Jeppe Salvesen]

I am having a bit of a QA problem myself. After reading up (Steve McConnel, etc), I'm looking to spend more time in pre-code, and also implement inspections (a code review technique).

The disadvantage to testing is that you detect errors, but need to spend time finding the source. If you avoid the error by detecting it at design-time or during code review, you will spend less time dealing with it, since you will know more about the root cause to begin with.

101 little bugs in the code

[NeilArrow]

Just to emphasise how good design is the key to avoiding most bugs, not testing - there's a song that often gets sung at my place of work...

Hundred and one little bugs in the code
Hundred and one little bugs in the code
Fix the code, compile the code
Hundred and two little bugs in the code

When was the last time...

[barzok]

most of us actually got good specs? Been close to 3 years now for me. With half-assed specs derived from business users who A) don't know what they want and B) don't know when they're out of their league when talking about how something should work you're pretty much screwed from the beginning.

Re:When was the last time...

[scott1853]

I don't let customers dictate how programs should work. I make them tell me what information they have to enter, and what they want to get back out. I decide on mostly everything in the middle.

Re:When was the last time...

[dwarfking]

Maybe this is a topic for another discussion, but I don't expect business user to supply me with specs. They aren't software guys, they are business users. I expect them to supply me with business requirements and business process definitions, from which we will together develop the software process definitions from which the specs evolve.

Re:When was the last time...

[zaphod110676]

Sad but true.....

Boss: Go write some code that does some stuff

Me: Well what about this? I need this info.

Boss: Well just start working and we will fill that stuff in laster.

How the heck can I write good when I am hardly told what the application is supposed to do? So I write something, it doesn't take into account the missing details that I asked about. Those get defined two weeks after the thing is supposed to be done. The app turns out terrible and then the powers that be want to know why it has problems. It is incredibly frustrating.

Re:When was the last time...

[sql*kitten]

I don't let customers dictate how programs should work. I make them tell me what information they have to enter, and what they want to get back out. I decide on mostly everything in the middle.

Then you aren't writing particularly complex software. If your users need software that does sophisticated processing, mathematical or otherwise, then the programmer probably isn't the best person to work out how it should do it. This is true whether you're working on software for pricing derivatives, or for tracking shipments in a supply chain, or for controlling manufacturing machinery. That's why there are notations like UML, so that functional experts can communicate unambiguously to the software developers what a system should be doing. A good programmer knows about programming, a good analyst knows about business processes, some people are both, but only with years of experience, and even then, only within a single industry.

The requirements, specification and alanlysis process is what separates software engineering from "hacking".

Re:When was the last time...

[scott1853]

My original post was a little ambiguous. My company provides software for construction companies for different phases of their business operations. So yes, there are complex mathematical formulas. On the other hand, complex is an ambiguous term in itself. Something is only complex when you can't understand it.

But I was referring more to the actual user interface and db support. The formulas are standard and fairly easy to write and debug. But any time a user requests that they have a button somewhere I try to cut them short and just ask them what they want the end result to be. But if they want some custom analysis then yes they need to let us know what the formula is though.

Re:When was the last time...

[plumby]

But if they want some custom analysis then yes they need to let us know what the formula is though.

And if they don't? or they forget to mention an important detail of it? I've lost count of the amount of times I've gone through detailed requirements gathering, got them to agree that in this situation X happens, developed software that does X, and then been told, after they've actually started using the the software, that they always want it to do X unless Y is true - something that they've never mentioned before. The problem is that most users don't actually understand their requirements, think that "almost always" is the same as "always", that error handling is a software issue not a business process issue, and that detail is not important.

Re:When was the last time...

[pthisis]

How the heck can I write good when I am hardly told what the application is supposed to do? So I write something, it doesn't take into account the missing details that I asked about. Those get defined two weeks after the thing is supposed to be done. The app turns out terrible and then the powers that be want to know why it has problems. It is incredibly frustrating.

Requirements change. Users don't know how to specify things. They often don't want what they think they want, and even when they do it may be obsolete or of secondary importance before you're done implementing it.

Your #1 job as a programmer is to write useful code--code that helps your customer/business/department--which usually (in my experience) means that your skill in determining requirements and working with the customer to refine them is far more important than your actual programming chops. Relying on an external project manager to gather requirements and define scope is usually disastrous.

This is a major part of the XP ethos and especially the agile programming philosophy ("Welcome changing requirements, even late in development" is one of their primary principles...). The sooner you develop softare practices that realize this and plan for it, the better off you'll be.

see e.g. the Agile Alliance article repository for stuff to think about.

Sumner

What kinds of bugs are you finding?

[wowbagger]

When I was an undergrad, one of the out-of-major classes I took was archery (I needed a PE credit, and I was interested in it). In archery (and in any other kind of marksmenship) the trick is

• Be consistent
• Measure your error
• Identify the cause of the error
• corrent the cause
• repeat

Programming is the same way. What kinds of bugs are you finding? Are they just stupid bugs, like buffer overflows or off-by-ones (good design, bad implementation), or are they unhandled errors, or are they API mis-matches or faulty algorithms (bad design)?

Have you made any effort to go back and say "Gee, we are getting a lot of off-by-one errors. OK folks, we need to think about our loops."?

And when you find one type of bug, do you go back and identify anyplace else a similar bug may exist?

If you are hitting high and right, and you never adjust your sights, you will NEVER hit the target consistently. If you never feed back the CAUSE of the bugs, you will never eliminate them.

Re:What kinds of bugs are you finding?

[p3d0]

A third kind of bug is harder to put a catchy name on, but it happens when you constantly say "oh, I didn't think of that case" or "I made a change in these six places but forgot that one". These indicate design problems of a fundamental nature. It means your code is cumbersome and inelegant.

Re:What kinds of bugs are you finding?

[bluebomber]

I'd actually consider these separate types of bugs. They're both design problems. The first is of the type that can often be remedied by either a) forming your requirements better or b) performing peer design reviews.

The second is what the XP-people would call a "code smell". I've found Fowler's Refactoring book to be quite useful in figuring out ways to fix the design such that the "I made a change in these six places but forgot that one" (and other) bugs go away.

Evaluate your test suite for coverage

[pfdietz]

One thing I've found invaluable is to compile your program with a translator that inserts code to detect when branches have been followed. Then run the test suite and see that all the code was executed. Any code that was not executed has not been tested.

It's amazing how poor coverage can be with a naively written set of tests. Ideally you want to write the tests so that the coverage comes out good, but in practice you may have to patch the tests with more tests to cover the parts you missed. You may also have to change the code to make it easier to cover.

Rare error cases (like malloc failures) can be hard to cover.

Re:Evaluate your test suite for coverage

[bluGill]

That sounds good until your realize the uncovered code path is only executed when the machine is on fire as a last message before it dies... A cheap machine is no problem of course, but management frowns physically destroying a million dollar computer for single test, especcially if there are several different cases like that.

If there is a bug it is worse yet, becuase you have to fix the bug, and destroy anouther machine to test. Code reviews and careful programing are a must in this case. (See my other posts for how bad my code reviews are though...)

Fortunatly things are not normally that bad, I have got a special board that generates bad parity, install that board, and see what happens. However there are over 100 different hardware problems I'm responsible for catching, and there isn't money for 100 boards exach destroied in some way. I only got a parity error board because that board went bad at a customers site and it wasn't detected correctly.

PC board testing analogy

[ortholattice]

A number of years back I wrote test programs for printed circuit boards. First you created a model for the board that simulated the logic circuits. You then wrote test patterns that were applied to the board's inputs, and the simulator model predicted the board's outputs. The inputs together with the predicted outputs were applied to a real board that you wanted to test, and if this test program passed you assumed that the PC board was good with a high degree of probability.

One mode of the simulator allowed you to simulate faults that might occur on the board. The simplest kinds of faults were physical IC pins "stuck-at-zero" and "stuck-at-one" (these were the most common faults in real life), and if you wanted to be thorough you could also simulate "internal" faults down to the gate level.

I worked in a contract test programming house, where the contract with the customer required us to produce a test program with a specified minimum level of fault coverage, usually just at the physical IC pin level to minimize cost of developing the program. This ranged from say 90% for cheaper commercial work to 99%+ for certain government contracts. With >95% coverage, the "real life" fault coverage was maybe one or two "dog pile" boards out of 1000 would pass the test program but fail a system test.

The point of this is in that business, there was a clear objective measure of a test programs "quality". The measure wasn't perfect, but it was far better than just blindly writing a test program based on a "gut feel" for how the board should work. In addition, the test programmer had a clear, objective goal.

I think a useful tool in the software business would be a measurement of the percent of lines of code that were actually run during the QA process, along with a log of those lines that were not run and not run. Often there are big chunks of code that only get triggered by very special conditions, and there is no way QA can guess those strange conditions. The standard QA process is very subjective; there is no objective measure of any kind as to how thorough the testing was, other than just documenting a list of features that were (often superficially) exercised.

A more sophisticated tool could go beyond lines of code and into log the various logic combinations exercised in "if" statements, etc.

Several years ago I wrote an experimental tool that did this for a specialized database programming language. Basically it rewrote the program with a logging call after each statement (and yes, the "QA version" ran very slowly). The results were quite eye-opening, revealing chunks of "dead code" and conditions no one ever thought of testing. Unfortunately the project kind of died.

Many languages have "code profilers" that are mainly intended to analyze performance, but many of them could be easily adapted to become QA quality measurement tools.

Do these kinds of tools exist, and if so why aren't they more widely used?

Re:PC board testing analogy

[bluebomber]

Excellent analogy!

Do these kinds of tools exist, and if so why aren't they more widely used?
Yes, they're called "coverage analyzers" or something similar. A couple of tools that I've used in the past (for c/c++):

• PureCoverage from Rational. It's been a few years since I've used this, but it was handy to have. Basically, you compile the code with special instrumentation and then run the PureCoverage tool which brings up a window showing (among other possibilities) functions, lines, and number of times each was called/executed.
  
• gcov (part of gcc). This provides the same basic functionality as PureCoverage but doesn't give the pretty windows (duh).
  

These types of tools can also be useful for performance tuning (with gcc, gperf is also handy).

Evolve the Tests

[4of12]

Sure, build a big suite of tests to run and check for things to go wrong. Every bug fixing process suggests it own test.

Then you find out that you don't have the time and resources to run all the tests everytime someone makes a change to the codebase.

So, use smaller suites of the faster tests and weed out some of the ones that have been ironclad passes for the last 5 dozen code checkins. For frequent testing it makes sense to only shake what's new and rickety, not what's stood through 10 hurricanes.

Run the exhaustive complete test suite infrequently, say when a release is imminent, or as often as you can afford to spare the resource cycles.

Architecture Problems

[peterdaly]

If you test the code as much as you says you do, and are testing for the correct thing (which I do not know you are doing) the problem may be the architecture.

Code which is "forced" into a paper architecture is sometime worse that code with no architecture at all. In many of my projects, parts of my architecture change part way through so that the code will work better. Sometime not everything can be thought of before hand. OO programs have a lot of information to fit in a human barin at one time, problems are bound to show through. I don't have any "high eng tools" to help with the architecture either, which doesn't help.

Also, the architecture itself may suck.

What kinds of problem are you having? I think you need to design test routines geared towards not letting the types of problems you currently have through. It is hard to have any specifics, since the post was so vague.

-Pete

very large quantity of data

[oliverthered]

The idea is to build up the level of test data you have,
You could have somthing that generats loads of test data , and use a simple script to check that it's data is ok, and even see what happens when bad data is fed in.

If someone one finds a bug, they generate test cases(and variants).

UAT testing should generate loads of data.

An live envoriemts give you loads of data for full-cycle developments.

Book Recommendations?

[scott1853]

You seem to know what you're talking about, so are there any good books that cover the software design process? A book that covers what should be flowcharted and how detailed it needs to be, as well as writing good specifications and what should be contained in them?

Re:Book Recommendations?

[scott1853]

I didn't say I consistently write code that's 99.9% bug free. I just believe I can if I'm awake and focused and not getting interrupted every few minutes. Of course isn't hubris one of the three programmer's virtues?

But the problem I face at my company is that nothing is fully planned ahead of time, which means that down the road we may run across some fundamental design flaw that require completely changing several modules. Which in turn usually results in bugs. That's what I'd like to avoid.

I think the biggest problem is convincing the boss to sit down and fully plan his ideas before asking software to be written for it. So maybe a book on upper management mental manipulation would be better.

Re:Book Recommendations?

[dubl-u]

Please note that you are asking a question similar to "Are there any good books that cover the process of living? You know, what it's all about, and particularly what rituals to follow for success?" You are likely to get religious answers to religious questions.

For a good overview of development practices, though, pick up McConnell's Rapid Development . The main thing to come along since then is the various agile processes like XP. For the scoop on that, pick up Extreme Programming Installed . (Don't get Beck's book, Extreme Programming Explained to start with; it's a manifesto, not a practical book.)

As with religion, you should probably ignore the people who claim to have The Answer ("Use UML!" "Use XP!" "The word of the Architect shall be as law, and you verily must smite the coders who stray from it." "Only infidels use COBOL/VB/C++/Java!" "There shall be no editor but Emacs!").

Instead look at it more like home maintenance or martial arts. Is a screwdriver better or worse than a hammer? Is a punch better than a kick? Those questions can be entertaining, but they are never helpful. The right questions are more subtle: "Given the problem I'm facing, what tools might help? How should I adapt them to the situation? And how can I tell when they help rather than hinder?"

Re:Book Recommendations?

[dubl-u]

But the problem I face at my company is that nothing is fully planned ahead of time [...] Which in turn usually results in bugs. That's what I'd like to avoid. [...] I think the biggest problem is convincing the boss to sit down and fully plan his ideas before asking software to be written for it.

That's one possible solution. If your development process can't handle changing requirements, either you can a) suppress all changes after the first requirements are given, or b) change your development process to handle mutating requirements well.

Both can work, but they have different cost/benefit profiles. You might chat with your boss about how much business value could be gained from reducing time spent in the analysis phase, and adapting more quickly to the business environment. Then you can balance this against the costs of using a more flexible process.

I can do either, but personally I enjoy the flexible approach more. I get so tired of telling users no.

Methodology and testing

[southpolesammy]

A lot of the problem may rely on what methodology you are using to code the program, whether it is the traditional waterfall method, or the sprial method, or perhaps M$'s old sync-and-stabilize method. Whatever methodology you use will drive how you should be testing.

With the waterfall model, you really need to know way ahead of time that what you are coding is what will be desired in the end product. It forces you to have a clear picture in your model of what you are trying to build and with each step in the process, you must develop testing procedures that address that level of the code. For example, at a high level, you may say, let's build a compiler, and following that decision, you need to devise a test that proves that the compiler works. The next phase, you may say, let's build an assembler to produce machine code for the compiler. Then you need to build tests that prove that the assembler works. This methodology continues right down to the smallest module of code, and when all of the pieces have been written, integration testing begins, and you make sure that each larger piece can correctly function based on the output of the smaller piece.

However, in the spiral model, it allows for a well-defined core code to be produced with tons of modules that evolve as the spiral expands. Integration is a function of the spiral, and testing occurs within each iteration of the spiral loop. Code produced with the spiral model also tends to be somewhat more difficult to test in later stages, IMHO, due to the nature of the testing that occurs at each cycle in the loop. Testing becomes more critical in later stages as the previous stages become more nested into the core of the program.

Well, enough Software Engineering for one day. Back to work....

Re:best way to test is to use automated testing

[Flamesplash]

Maybe when we have quantum computers we can test every single user scenario in parallel

-flamesplash

Let the User's Do It

[John Hasler]

"When testing code, what procedures work best for you,..."

Make sure it compiles and runs and then upload it to Debian/unstable.

(Yes, I'm joking).

"...and do you feel that excessive testing hurts the development process at all?"

If didn't hurt why would you label it "excessive"?

testing throughts

[bons]

The difference between testing expected inputs and possible inputs is that reality doesn't limit itself to expected inputs. Heck, Sometimes it doesn't even limit itself to possible inputs.

Larger tests don't test more. What the large tests do is make sure everything works together. You need the small tests to make sure each piece actually works.

The bigger the test, the more likely that your testing platform doesn't resemble production.

There is a big difference between getting your test to run sucessfully and having bug free code.

Chances are the test cases have nothing to do with hiow the users actually use the program. Chances are the programmer has never actually seen how a user uses the program. Chances are, the first time he does he'll go back to his computer and start cursing the user for not doing things the "right" way.

If something breaks and you don't add that something to the test case you're asking for it to break again.

Testing deserves powerhouse machines and sadistic maniacs who like to break code. People who want the tests to be sucessful and don't run them that often are obviously not going to be as nasty as the real world. Even the sadistic manic has a hard time being as nasty as the real world.

Tests are less expensive than that production errors. But only if they find errors. Tests that prove the code works perfectly usually don't.

No programmer likes to be told he made a mistake. On the other hand they love a challege. Make testing fun and brutal and it'll be much more productive than if you make it boring and painless.

The point of testing is to FIND the bugs

[alanjstr]

If you're not finding the bugs, then you're not doing a good job testing.

I find that..

[MarvinMouse]

When I code programs that are used by the general public. I find double-blind testing, and black-box testing works best. With software that means life or death or something severe I will also do white-box testing.

double-blind testing is when you give the code to a willing party and just let them work with it like they normally would for business purposes, without letting them know it is a beta testing. You have to also include some type of bug report that people can fill in if they wish, but try to encourage them not to cause bugs, and just work with the program as if it was normal. This allows you to see if any of the normal functions that people use everyday would be buggy.

Black-box testing works great to Just test the programs function calls and modules. When I do BBTesting I usually give it to another party with instructions as to how the functions are called and utilized. This party knows how to test the extremes and the common values and give me the best testing.

White-box testing is testing that involves intricate knowledge of the code. When I do this it is usually in development. At the end, if I feel like I enjoy pain I will do a through white-box testing suite for the program, but that has only happened once or twice.

In expenses, the cheapest form of testing is BB testing, followed by Double Blind, and then WB. Since white box testing takes a long time to design run and analyze the results I find.

There's some thoughts for you though.

More testing != Better Quality

[Digital_Quartz]

There are two subjects I want to discuss here. First of all, I'm going to present the "jelly bean model" of defect discovery, then I'm going to talk about why the "testing to improve quality" model is fundamentally flawed.

The Jelly Bean model goes like this: Let's suppose you have a big vat of red and blue jelly beans. Your objective is to remove all the blue beans. You do this by reaching in, grabing a hand full of beans, throwing away all the blue ones, and dumping the red ones back in.

At the begining, it will be very easy to find the blue beans (assuming the blue-bean density is high), and towards the end, it will be very difficult (since the blue-bean density will be low). If you graph the cumulative number of blue beans you remove each day, you'll get a exponential curve; quite steep at the begining (high rate of discovery) and which flattens out as you approach total bean removal.

Software defect discovery follows this model exactly. Defects are easy to find at the begining if there are a lot of them, and hard to find towards the end. This means that if your defect discovery rate is pretty much constant (with respect to the number of hours of testing you've done) then you're probably still way down in the very first part of the curve, and your number of defects is probably very high.

Here's the important thing to remember though; the quality of your product has nothing to do with how many defects you find and fix during testing. The quality of your product is determined by the number of defects remaining! If you find and fix 10,000 problems, you might think you're doing very well, but if there are 10,000,000 defects remaining, your product is still crap.

You can estimate the number of defects remaining by trying to fit the number of defects you've found so far onto that exponential graph I mentioned above. The most popular method to use a Weibull curve, or Quadradic Regression.

Now, why is testing to improve quality a bad plan?

Let's say you worked at Ford, and roughly 50% of the cars you turned out had something wrong with them. You get lots of unhappy customers demanding their money back. Is your problem:

a) That you have a design defect in your car.
b) That you are introducing defects in production.
c) That you are testing cars insufficiently.

Most people realize that to test every car as it comes off the line is futile. There's too many of them, with too many potential points of failure. There's no way you can test them all. The root cause of the problem has to be in either a or b, and if you're looking to improve the qulaity of your cars, this is where you would spend your money. This isn't to say that Ford doesn't test their cars, I'm sure they do, but testing should be a means of verifying quality (IE, 1/1000 cars tested had a defect, our goal was 1/500, so therefore we can stop spending money on finding design and production faults), and not a means of improving it.

It's so easy to see this when we're talking about cars. Why does everyone get it backwards when we start talking about software?

Not only is it impossible to test every possible combination of inputs to most software, it's also very expensive to find and fix problems this way. If you find a problem in design review, or code inspection, then you have your finger on it. You know EXACTLY where the defect is, and how to fix it. On the other hand, when you say "Microsoft Word crashes when I try to select a paragraph and make it Bold", you have no idea where the fault is. Any one of several thousand lines of code could be the problem. It can take literally days to track down and fix the defect.

Your testing should not be a means of finding faults, but a means of verifying the quality of your product. Testing is not part of the development process.

Fascinating!

[MatthewDunbar]

Dear Egg,

Thank you for your interesting and insightful treatise.

Clearly, with your keen grasp of the history of programming, and your understanding of VB and HTML, you will someday make a fine salesman or mid-level manager. You already exhibit the skill and insight used by management at most companies to plan their projects, procedures, timelines, and budgetary requirements.

Many programming issues would clearly benefit from simplification, and perhaps you are on to something. By reducing the number of tools a language attempts to implement, it clearly decreases the number of distractions for the programmer. If you wish to pursue this concept further, you may perhaps wish to research another foundational language called Logo.

Good luck in your next class!

Billbert, PhB, TFIC
Redmond, WA

P.S. - A side note regarding the original article:

A firm understanding of your process for code development, and a clear design for your testing procedure are essential.

If you spend some time in advance on planning, your code will benefit. Also, a good test plan should include a measure of the relative impact of a 'bug' or 'defect' to help determine priority of response by your programmers.

By focusing on your programming objectives from the beginning, and maintaining that focus throughout your entire design lifecycle, you should be able to identify underlying problems in your current development model and use them to improve your entire process, with a goal of helping prevent errors in design, and catching errors in code before going to test.

Peer review at each stage prior to testing (planning, functional design, algorithm design, coding, and test design) will also help catch errors in advance, and lead to the development of much better code. It may sound like it will take longer and cost more, but it saves time and money in the end in terms of not having to rewrite and maintain poorly implemented code.

Cheers.

Automated testing does indeed work

[Junks Jerzey]

Though remember, this is Slashdot :) Automated testing is common in embedded systems programming, and all but non-existant for any kind of Open Source desktop applications (gcc is an exception).

You write test cases as you go. You make sure you can run an automated regression test at any time. If you don't do this, then any time you change code you might break old code and you won't realize it. Just doing spot checks at the keyboard isn't good enough. And the programmers need to be writing these test cases first, and they need to be kept separate from tests written by external groups.

Tried Cleanroom?

[CyberGarp]

My personal recommendation is the "Cleanroom" methodology. You create a functional specification with a mathematical guarantee of completeness and consistancy. Auditable correctness is also a part of the process. Then when it comes to testing you generate test cases that cover all states, all arcs and then do statistical test case generation based on a usage model. The overall cost of this process is a bit more up front, but studies have shown that the process far more than pays for itself in greatly reduced maintenance/debugging costs.

So to answer you question is that to generate a decent set of test cases, you really have to understand the problem space and have mapped out the state-space in some manner. Trying to derive this without a methodical approach and ones testing will be spotty. The worst I've seen so far was a random state-space walker (ala Brownian motion). Statistically this approach avoids all the difficult cases in the far corners of the state-space.

Now for the bad news: Cleanroom is quite tedious for the programmer. The enumeration phase takes seemingly forever and can be mind-numbingly boring.

Here's the amazon link on the layman's book on Cleanroom: Cleanroom Software Engineering: Technology and Process by Stacy J. Prowell, Carmen J. Trammell,Richard C. Linger, Jesse H. Poore

And now for the shameless self promotion bit with a long winded sales pitch for executives on Cleanroom: my own Cleanroom company: eLucidSoft.

Just chant over and over: "Hire eLucid, play golf."

Document your thought process

[marko123]

At the time that you are coding, every assumption is going through your head. This is time to write it down, either on paper, in a document, or in comments in the code. The mental state you are in when designing test conditions cannot come close to the state of mind you are in when coding (if you are concentrating :) . You are mentally closer to a problem when you are coding than when you are designing, and you can take the shortcomings of the platform you are working on and pair it with the shortcomings of the design.

Any consideration you have during the writing of a single line of code is gold. And like a great dream, if you don't get it down when you think it, you will lose it in a day or two.

My 2 cents.

zerg

[Lord Omlette]

MSDN used to have a column called Stone's Way or something, and in one of them they discussed user case testing: set up a video camera to record the user as he/she uses the program OR masquerade as a nondeveloper and spy on the user as he/she uses the program.

If you're just looking for regular bug testing, assume it's a given that the user will not report bugs to you and have the program automatically email you a core dump and/or stack trace and/or any appropriate data if an unhandled exception occurs.

What's your methodology

[anomaly]

I work for a large company with a large number of internally developed applications.

I am shocked at how frequently our developers don't have a good understanding of their architecture, or sometimes even the problem that they are trying to solve. As a result, when they go do do "testing" they are frequently performing tests that are not valid.

For example, they might create a new build and test that build only on their development workstation before full deployment of the application.

Naturally the development box has different resources from that of a standard production machine. Many developers don't seem to understand this.

Another example - frequently boundary conditions, or interfaces to other applications are not fully tested.

Using bad methodology, all of the time that you spend testing is wasted.

Management tends to feel that testing time is wasted because their experience is that the time that they have invested in the past has been fruitless.

Please develop:
valid test cases,
valid test plans, then

execute them,
find gaps, then

use the gaps to learn how not to make the same mistakes in the future!

Phooey.

Anomaly
PS - God loves you and longs for relationship with you. If you would like to know more about this, please contact me at tom_cooper at bigfoot dot com

Re:Metrics and Code Reviews

[DecoDragon]

I heard an interesting presentation at the Baltimore Chapter of the Association for Women in Computing, given by Dr. Linda Rosenberg - Chief Scientist for Software Assurance at NASA's Goddard Space Flight Center. The presentation is online. In the middle of the presentation was a discussion of some of the tools the assurance team uses. There's a density test, which might be of particular interest to the original poster. The density test looks at how many errors one would expect to find based on lines of code, and I think complexity but I'm a bit fuzzy on the exact details. Anyway, part of it is that there is that based on measurement of past debugging there is a curve of how many errors you would expect to find over time. When the testing team says they are done, the results are compared against the curve, to judge whether they fit the pattern one would expect to be done. That is, how far along on the curve are they? There's another tool the assurance team has to judge whether requirements are meaningful, by evaluating the requirements document to make sure it's complete. Using key phrases to judge whether a requirement statement is definitive or fuzzy or incomplete (that is does it say "to be determined later") and giving a percentage of how many of these types of statements you have compared to the rest of the document. Maybe not perfect, but a starting place to judge.

Neither programming or software assurance are areas I have knowledge in, so take the above with a grain of salt and check it out yourself. It all seemed rather interesting.

Software Inspections

[dwheeler]

If you want to get rid of defects, you need to review your software. There are a lot of quantitative studies that show that software inspections (a particular way to do a peer review of software) are very effective at finding defects.

Here are some experience reports showing software inspections to be effective: (1) G.W. Russell's "Experience with Inspection in Ultralarge-Scale Developments" (IEEE Software, Jan. 1991), (2) E.F. Weller's "Lessons from Three Years of Inspection Data" (IEEE Software, Sept. 1993), and (3) Grady and Van Slack's "Key Lessons in Achieving Widespread Inspection Use" (IEEE Software, July 1994).

There are many other such articles; I'm highlighting the IEEE Software articles because they're easy to get. Full disclosure: I co-edited a book on software inspections, titled "Software Inspection: An Industry Best Practice" (IEEE Computer Science Press, 1996; edited by David A. Wheeler, Bill Brykczynski, and Reginald N. Meeson, Jr.). That book reprints the most interesting articles on software inspection of the time, many of which are quite hard to get hold of, as well as additional material not found elsewhere that gives you the "big picture." Unfortunately, that book is now out of print and getting increasingly hard to get, but you might be able to get a used copy (or convince IEEE to reprint it).

If you dont want to find any bugs in your code...

[LordZardoz]

Then dont test it. The problem isnt that the testing is reporting too many non-bugs. The fact that your finding a large number of bugs means simply that a large number of bugs exist. Less testing does not reduce the number of bugs in a program, it simply reduces the bug count.

I will admit that bug testing often results in many non-bugs being reported. But that does not mean that testing is a bad thing. If anything, testing should be started as early as possible, and be carried out frequently. The earlier that bugs are found, the easier it is to fix them.

END COMMUNICATION

Re:Priorities?

[bluGill]

Unfortunatly i've come to agree that some bugs are not important enough. I have some in my code that the customer will never see just by following directions exactly, and even if seen it is a minor problem to the customer. To fix the problem though would require a major design change, and when we want to ship in 3 months there isn't time for that level of change to the code.

Do XP Unit Testing

[sohp]

It has to be said -- maybe if you did do XP-style unit testing you'd get better results?

Which level of bugs?

[bluGill]

My code comes back from test with three levels of bugs.

Problem 1: A crashing processor doesn't show up on the GUI. Turns out that I told the GUI that a porcessor crashed, in testing I saw something come out on the command line, and didn't notice the mispelling, but the GUI didn't know what to do with it. Took longer to open the file than to fix the bug. These should all be fixed, unfortunatly they are all minor enough that if caught late in the test cycle they are defered.

Problem 2: After causing failure A, failure B wasn't detected, it turns out the code to detect failure B is the same as failure A, and once A occures the code stops watching for B, even though the two are not related. This is a fundamental design problem, and can only be fixed by a re-write. (My excuse: someone else wrote the code and quit, I maintain it, but I have to impliment feature gamma before I can fix this problem...)

Problem 3: Tester pulls the ethernet cable between two nodes, and the complains that we said the node broke instead of the ethernet cable. This can be fixed, but we need some other way of determining that the other node is still operational we just can't communicate with it.

the first one is easy to fix, the second is solvable, but takes a lot of time, and the third can't be solved. When you come across the third, I hope you have better luck that me with people noticing the bold letters in your documentation noting that additional hardware is needed to solve that problem.

Testing is essential, BUT...

[William Tanksley]

From experience, we know that testing is essential for maintainable code, but the practice which REALLY catches defects is peer review, not testing.

Two effective ways, both tried and true:

1. Take your code to your coworkers (at most two at a time) and have them go over it with a checklist of common defects and goals. Then explain the code (while they still have the checklist, so they can make additions), then have them explain the problems they found. Look up "formal reviews" for more info.

2. Pair program at all times. Follow the basic XP guidelines for this: be sure to rotate teams. This is usually harder to arrange than #1 (it takes more management support), and it doesn't provide recorded numbers for analysis, but it squashes a similar number of bugs, and has other benefits (for example, knowledge about every chunk of code is present with at least two people on the team, rather than just one).

-Billy

Design smart, Code smart, Test Smart

[SloppyElvis]

How could testing less possibly solve your problem!?

Test smarter, not less. Many of the above posts have shrewdly indicated good method of preventing bugs (an ounce of prevention, anyone?), and below I have some ways you may want to consider in testing your software. Sadly, the poster didn't give a whole lot of information to go on, so this offering is a bit vague, I'm afraid.

At our shop, we invest a good amount of time in developing test modules for all interfaces in the project, almost as much time as it takes to write the modules themselves. Every night, an automated build machine builds and installs the latest version from source control, and initiates a test script that invokes the test driver suite which loads and tests all public interfaces. Every function is tested with expected input versus expected output, and further by sending unexpected input (like that of a malicious user, or failed communication). All exceptions, assertions, and incorrect output is checked as well, as any memory leaks, by the C-runtime debugging utilities, and Bounds Checker will soon be in use as well.

This approach serves 3 purposes:

1. It readily finds crashes that human testers may not find, since it checks every public function with varietal input.
2. It aids regression testing for modules that have been changed.
3. It checks for leaks, which can cause problems that may be evident only after extended execution times (doesn't apply if your language is garbage collected)

Further, by having an automated build every night (that emails everyone on the project, including the project manager, ehhem), developers are careful not to "break the build", and thus pay attention to details before committing their code.

This method is only a preliminary test; however, and it cannot find every bug that may or may not exist. All code is audited to ensure it meets coding standards. You'd be surprised how failing to meet standards often evidences a piece of code that was written in haste, and as such, is a good place to looks for bugs. Also, we employ code testers who run the software through use cases (which are sequences of actions determined to be general ways in which users want to use the software). These testers also put the software through non-use cases, in which the actions do not follow expected sequences.

I hate to point out the obvious, but it is important to test software freshly installed on "clean machines", not development machines.

In the end, if it is at all possible, it is nice to have users who are in some way affiliated with the software company to give a "field test" so-to-speak on alpha and beta versions of the software. In any complex piece of software, there will be a chance for hard to find bugs (relative to the developer) to occur. Tolerant users are the best and last resort to finding any obscure bugs before the general public gets their click-happy hands on the software.

Conclusion: design smart, code smart, follow standards, test smart, test everything, and track all bugs (as if that is news to anyone).

bebugging

[Martin Spamer]

How can you be sure you are 'Properly Testing Your Code'?

Actually you can do this by adding more bugs, yes adding them, The technique is called bebugging and the is basicly:

1) Produce code, it contains an unknown number (N) of bugs.
2) Programmer (or bebugger) seeds the code with a number (B) of known new bugs, the number and type of bugs should be determined from bugs found in previous debugging cycles.
3) Code is submitted to testing and some bugs are found (F).
3) The bugs found are examined and categorised as either real bugs (FN) or bebugs (FB).
4) Number of real bugs (N) can be found as the ratio of found bebugs (FB) to unfound bebugs (F).
5) Don't forget to remove all the bebugs.

Re:bebugging

[regen]

Almost Right...
4) Number of real bugs (N) can be found as the ratio of found bebugs (FB) to unfound bebugs (F)

Should read:

4) Estimate of number of real bugs (Estimate of N) can be found as the ratio of found bebugs (FB) to unfound bebugs (B) times the number of real bugs found (FN).

4a) Testing can be considered complete when number of real bugs found (FN) equals Estimate of number of real bugs (Estimate of N).

Re:bebugging

[Crispy Critters]

Parent said

seeds the code with a number (B) of known new bugs, the number and type of bugs should be determined from bugs found in previous debugging cycles.

In other words, you are introducing bugs that you already know your testing can find. But you are assuming that any real bug is equally as easy to find as the bebugs. This may work if your app is full of really brain-dead errors, like buttons that don't do anything or menu selections that crash the app. It doesn't tell you anything about whether subtle errors are being found.

Beware a fool armed with an equation.

testing

[jacobm]

Our testing philosophy:

1. Test every branch. (This goes hand in hand with making sure you have small enough functions that they only have one or two branches apiece.) Sounds like overkill, but when you're writing a function it's easy, if a little dull, to write test cases for each branch. It's much harder to fix a bug in a given section of code if all you know is that one of the sixteen functions it calls has a bug, and that bug is in one of seven functions that code calls, etc.

2. Maintain your test suite. Every test we've ever written is still in our test suite, and every time a bug comes in, that bug gets its own set of tests and goes into the suite.

3. Run your test suite regularly.

This works well for a lot of things; the big area in which it doesn't work is very complicated 'mathy' algorithms for which you don't really know what the correct answer ought to be other than by just running your program and seeing what pops out and stochastic algorithms. Stochastic algorithms in particular are a total bitch to test. But, for the other stuff, there's really no better solution.

Code Review AND testing

[mughi]

My experience has shown that the number one way to find defects is code reviews performed by other developers who can read the code and also understand the intended functionality. This will catch 90% of all defects before they are even released to QA.

Overally, good testing catches perhaps 63% of all defects. Code inspections alone catch about 63%. Combined on a project, they catch about 95+% of all defects. That's the key. (My copy of Code Complete is at the office and I'm still at home, but that has the exact numbers and study).

And remember, a good testing regiment will include all kinds of testing. Unit tests and integration tests are both needed (Usually it's only the latter that happens in QA). And it's quite handy to have started the unit tests before you start coding the units.

Test cases are useless

[bstrahm]

Outside of creating an automated regression suite, I don't see much use in test cases. I mean so you test 10, 100, 1000, 100000000000000 things that the user might do... That leaves a lot of room for creativity on the end user. I actually had a manager who tried to say since a bug wasn't covered in a test case, it didn't need to be fixed. WOW.

I think there should be about 20% of time dedicated to running testcases/regression tests 60% of the time to automating the above tests, and 20% of time dedicated to allowing testers to just beat on the product... Get creative and think like a user...

I would go one step further...

[freeBill]

...and say, "Developers should write their test suites BEFORE they write their code."

We have a fairly large open source project with contributors coming in and going out all the time (well, not a lot going out; but any number is a problem there). Our experience shows that if you can't write a test suite you're not ready for anything more than a crude prototype. The problem with test-after-coding regimes is the testing gets short-circuited. You've already got working code. You "know" it works. You're just proving it works. So you test the obvious stuff that proves this.

Since we have instituted this policy, coding efficiency has actually improved. Coders who have tried to devise a complete set of tests have formalized their understanding of the requirements in a sense which the most complete requirements doc will never do. We include the test suite in CVS. Nobody commits until their update passes the entire test suite. This results in an enormous (but complete) test of everything done so far. But you can't imagine the thrill of seeing your patch pass that many tests the first time.

All of which is completely separate from what a QA process is for.

Testing considered harmful

[Bastian]

I have to agree about the code reviews. There have been plenty of studies showing that frequent code reviews just work better - think of it as a way to suck up a large number of the advantages of pair programming without actually doing it.

Speaking of pair programming, it's also been shown that you'l save a lot of time and effort if you use pair programming to do the complicated or difficult chunks of code. Yeah, there's the cutting-your-productivity-in-half argument, but that really only holds true if you don't know how to use pair programming. If done correctly, you'll save more than enough time to justify the cost later on when you don't have to put nearly as much effort into debugging that code.

As for testing, it's overrated, almost worthless. I realize you gotta do it, but it does so much to distract from the best way to keep bugs out of your program (which is to not put them there in the first place), that I wonder if testing doesn't in some wierd indirect way actually create more bugs than it discovers.

Heck, with code reviews, your programmers will probably start writing better code just so they don't hvae to stuffer the embarassment of having someone notice a particularly stupid algorithm design flaw in the middle of a code review.

Re:best way to test is to use automated testing

[caferace]

Of course properly written functionality test scripts (doing what the user does) will find most bugs.

I beg to differ. This is how most developers test their code as well, though manually.

If you're just testing to make sure your code does what it is supposed to do you are likely in BIG, BIG trouble. Users (and black hats) do just the opposite.

Focus just as much on making sure your code doesn't do things it WASN'T designed to do. Or risk a CERT or Security Focus advisory...

Or...

[krow]

Perhaps you just load up your software on a populor website and wait for bug reports? You know if your audience is addicted enough to the site this just might work.

Detachment from your code...

[sterno]

One of the big problems that comes up with developing is that the people writing the code have two big strikes against them for testing their own code:

1) They know how it is supposed to be used
2) They want it to work

The side effect of this is that frequently the code will give every impression of working perfectly until somebody who isn't familiar with the code tries to play with it. Then suddenly they are doing unexpected things, entering blank spaces, wierd characters, and other things that can be expected to happen in the real world.

So, in any system you are developing it is very useful to have the developers try to break eachother's code. People don't want to break their own code, so generally they don't.

Testing don't get no respect

[Wansu]

So, it isn't done well, if at all. QA isn't taken seriously. Regression tests, if they exist, are poorly maintained. Then people wonder why something wasn't caught before it went out.

Put the QA department in charge of nightly builds, testing, docs and shipping. Allocate sufficient time for a thorough testing. Give them the power to hold up a release. Hire good people to do this, people who are good developers themselves. Pay them well and listen to them. Don't second guess and overrule them.

Regression tests don't work !!

[matsh]

OK, this may come as surprising to many, and it sure did to me, but my experience is that regression testing doesn't work, i.e. it doesn't find bugs.

I had this grandiose idea that we would run all our unit tests as one big regression test every night, and it would alert us when our server broke. It didn't turn out that way. Instead, when the server was changed, we ended up having to rewrite the unit tests, and in many cases that turned out to be a royal pain. So we stopped maintaining the unit tests. When one, and later two, unit tests were constantly failing, then nobody cared any more about broken regression tests.

There are benefits with unit tests, though:

1. If you write the unit tests before you write the code, then you will force yourself to specify (in code) how your system will be used, which is way better than stupid UML diagrams. Yes, I don't like UML.
2. Testing your code before you ship it makes perfect sense, of course. It will catch many bugs.

So, my experience is that stringing unit tests together into a regression test suite is not worth the effort. Sorry JUnit, I also loved the idea when I first read about it, but I don't think it works.

Mats

Re:WHAT?!

[bluGill]

Okay, the example was a little obvious, and likely would be flaged. I wanted an obvious example that even non-programers are likely to understand. 20 lines of comment and 10 lines of code with a subtile logic error that work 99% of the time through all brances except when some unchecked condition exists are really hard to write on the fly, much less so that everyone can understand why *foo would not be null and yet have invalid data...

Many bugs is good.

[bluGill]

I was once on a project where very few bugs were found in testing. Eventially we shipped and discovered that the testing group wasn't doing a good job of testing.

Requirements

[andymac]

Hi there --

I'm certain someone has already said this, but over 80% of defects come from crappy requirements. Forget about your design & analysis, your coding practices, inpsection techniques, debugging and testing abilities - if your requirements are not CLEAR, CORRECT, ATOMIC, UNAMBIGUOUS, and CONSISTENT, you might as well start burning money.

NASA correlated a $1 cost to correct a "defect" in the requirements stage (here a defect can be any requirement that does not meet all 5 attributes I listed above) to several hundred to thousands of times over when addressing the same defect at the testing stage. Crappy requirements and crappy specifications are a big part of what makes your code buggy and expensive.

LA Times posted a study last year that showed that the average US programmer only coded for 51 days a year. 51 days!! One fifth of your working year spent writing new code. The rest of the time? DOING REWORK.

Biggest cause of rework?

UNCLEAR AND AMBIGUOUS REQUIREMENTS.

Spend the time and effort to beef up your requirements gathering and management processes. You'll get your ROI in ONE project cycle.

Downside to Code Reviews

[Raedwald]

Many others have pointed out that studies consistently show that formal reviews (especially of specifications and designs) are the most cost effective ways of removing defects. Others have provided references to the classic books on the subject. Anyone considering doing formal reviews should read them. I personnally like Tom Gilb's books.

There is a downside to consider, however, which is little mentioned, even in the formal review literature. Formal reviews require a particular type of company culture, and not all companies have or want that kind of culture. Trying to introduce formal reviews in a company that has an incompatible culture will be some mixture of painful, counter productive and political suicide.

The idea that a company would, in any way, be opposed to using the most cost effective way of removing defects seems bizarre. The truth is, not all companies care about product quality. Sure, everyone will say they care, but words are cheap. To find out what a company really cares about, see what decisions they make under pressure. See what they sacrifice, and what they keep.

• Do they cut testing to release the product on time? That means schedule is more important than quality.
• Do they refuse to return an error riddled specification to the customer so the errors are corrected before work even begins? That means saying yes (a 'can do' attitude) is more important than quality.
• When engineers start pointing out defects, are the whistleblowers labelled as 'troublemakers'? That means internal politics or a harmonious working environment are more important than quality.

The difficulty is, that before you can introduce formal reviews into an organisation, that organisation must already be highly committed to quality. Quite simply, many organisations have to introduce other, fundamental, improvements before they can use the advanced technique of formal reviews. The Capability Maturity Model (CMM) produced by the Software Engineering Institute (SEI) is a useful way of prioritising these improvements. I recommend it; I've used it in a project, and ISO 9001/9000-3 in another, and I conclude that CMM is the better of the two. They have a website.

Re:Do it mathematica-style

[DumbSwede]

Some people will now be able to guess the identity of DumbSwede, but
I was really shocked to see the Wolfram testing method mentioned, because
I invented it (for Wolfram at least), back in 1989. Stephen himself gave me
credit for this type of testing numerous times, at various conferences, when I was still with the company.

I didn't invent random testing in general, but with a mathematically based
language like Mathematica, many operations are symmetric algebraically.
If I generate a random algebraic expression, Integrate it, then Differentiate it,
then it should then match (after simplification or variable substitutions) the original
Integrand. This was the first type of random test that I designed for Wolfram, and
man did it find slews of problems with V1.0 versions of Mathematica. Back when I worked part time
and was an undergrad at U of I (more clues to who DumbSwede is).

I worked developing scores of other math identity verification procedures for Wolfram
over a 10 year period, then sadly, moved on (I use to really enjoy pounding on that code).

Even things like word processors can be tested in this fashion. If you randomly add 10 characters, then
delete those 10 characters, should get you the original file back.

Operations do not always have to be symmetric, but they have to have some testable property or
identity after a series of events.

While this kind of testing doesn't replace other types of testing, I guarantee it takes you into
a whole new bug-space you didn't even guess existed with your software, requiring a more mathematically
consistent way of handling data.

On a related note, for GUI based apps, having a way to script equivalent events is a must.
Where I work today, we use Android to play back GUI events, but I pre-process the Android scripts,
(that I have peppered with replaceable tags) to do things like random testing or repetitive testing over
many inputs, rather than have a static GUI test, that can only simulate one very narrow set of events.

Correctness...

[richieb]

Building correct software from requirements is as easy as walking on water. As long as they are frozen. :-)

Code Reviews

[cant_get_a_good_nick]

In my opinion, one of the best ways to get rid of bugs is code reviews. I've been in a couple organizations, and we did code reviews at both. Both helped getting better code out, and had other side enefits.

Plusses

• Analysis doesn't just catch bugs, but design decisions. No amount of QA of compiled code will show you a bad design, yet this is where a lot of problems occur.
  
• Careful analysis finds bugs that are hard to test, like when the world is crashing around you. How often do you simulate low memory situations?
  
• It helps disseminate knowledge in the organization. Someone has a technique or an algorithm. It comes up as a suggestion in a code review, and now others know of it too. I learned a few things in code reviews I never knew to ask. Some developers will never ask because of ego issues, but will hear of them in code reviews.
  
• Fosters a sense of responsibility in the code. If someone is going to see tis, then the code has to be of better quality. I can't just write garbage and hope it gets by QA.
  

Problems? of course...

• Managing egos, easily the biggest problem. Have to reinforce that this isn't a witch hunt, just to get better code. The code review leader can't let himself get caught in an argument, cause it will disrupt the meeting and foster resentment in the developer.
  
• Time. Developers hate meetings. It's best to have a few developers in the meeting, the many eyes approach, but how do you get some guy on a deadline with his stuff to look though someone else's code? Need to foster a team feeling, hard to do sometimes. Also, make sure you limit the time in the meeting, you get diminishing returns after a while, an hour, maybe two. Anything more is a waste of time.
  

No one should run ANY tests.

[Sax Maniac]

Seriously. Manual tests are waste of time. The only test worthy of running is an automated test. If it's not automated, then somebody can forget to run it. If somebody can forget to run it, then you're not really testing it, are you?

If you have to have X people run tests at cost Y, then adding more tests makes it more expensive! What kind of screwed-up logic is that, when a computer can do better at 4am while you sleep?

Any tester worth his salt is not a button-pusher/bug-report-writer. A real QA person writes automated tests and checks them into the code base so that it runs automatically when building. Flame to death anyone who checks in code that breaks a test. The optimum situation is that all developers are testers: they write tests and code and check them both in simultaneously.

If you're finding that tests are showing up lots of bugs, you're finding the symptom. It might be you're finding bugs and fixing them will reduce the amount. However, if you find that your development team creates bugs faster than it can fix them, then it means your organization doesn't know to code it's way out of a paper bag and shouldn't be programming. No amount of tests will fix it. The only thing that will work is refactoring, and most managers in such places erroneously think refactoring is the devil.

Not all software is equally testable. You have to write to so that is testable. If you want to read a great book on how to test your software, I recommend John Lakos' Large Scale C++ Software Design. Testing techniques apply to most other languages, not just C++. Personally, I've unscientifically found that every hour writing automated tests pays back at least 10 in saved future effort. YMMV will vary on the complexity of your project.

Orthogonal Array Based Robust Testing

[yokimbo]

I'm sorry, but I didn't have time to read all the other responses. The replies I did read were mostly questions back at you and clarifications to other replies. So, here is my attempt to answer your questions.

do you feel that excessive testing hurts the development process at all?
Yes, of course it does. You could, theoretically, test code for one program for the rest of your life and still not discover all the bugs. That would be excessive testing and would definitely be a bummer to the development process. I think what you really mean is how do you determine how much testing is enough. For this I refer you to a few good testing books because frankly speaking, people have made careers out of this sort of thing :)

Books: The Art of Software Testing (hard to find and a little expensive) by Glenford J. Myers; The Complete Guide to Software Testing by William Hetzel; Code Complete : A Practical Handbook of Software Construction Steve C McConnell. These are some good options to get you started.

What is the best way to get the biggest bang for your testing buck?
I would take a serious look at Orthogonal Array Based Robust Testing. A method of testing developed by Taguchi and Konishi, and using orthogonal arrays to determine test cases. I don't have enough room here to get into details, but basically this type of testing guarantees detection of atleast 1st and 2nd order defects with the minimal amount of test cases. Madhan S. Phadke's Quality Engineering Using Robust Design mentions this type of testing. Also Bell Labs has been so kind as to publish online some fairly heavy strength Orthogonal arrays, so you don't have to calculate them. My employer uses this type of testing on many of its projects and it's a huge time saver. I just learned about it in an onsite class by our top tester and am going to pitch it to my project soon.

Good luck, sorry for being so vague in places, and finally, if you have more questions about Orthogonal Array Based Robust Testing, please let me know: redundant_pleonasm@hotmail.com

Laws of Software Testing?

[Peahippo]

From my experience testing at PictureTel and DEC in Mass., I found out that the usually-understaffed test team runs into the Laws of Software Testing: Law #1: Most of the time, you are not testing. You are obtaining (beg, plead, cajole) equipment and software, and you are configuring and fixing software, just so that the environment is ready for the software testing to occur. Law #2: When testing, most of the time you are not testing anything that matters. Sure, scripts are great, but they are very narrow, and bugs are sneaky. Running a limited variety of scripts across some clients and servers only gives the impression of coverage ... kind of like the concept of "busy work". Law #3: When you find a bug, most of the time it can't cross the political barrier. After all, bugs are rated and prioritized, and that is the domain of management, who is overwhelmingly concerned about release dates. Law #4: The bugs you don't find will reach the customer and will return as the highest-priority bugs that will usurp other bugs in the ongoing process. Fixing customer problems is of course a priority, but it landslides into the current product's production.

Re:Code Review

[Xentax]

Yeah -- our numbers can definitely highlight where review time correlates with testing/support time. But, as you said, breaking that down into whether the code's robustness vs. support/maintainability is a little trickier.

And I agree about reusability -- hand-in-hand with that is using 3rd party solutions when appropriate. Forget solving the problem twice or even ONCE if someone else has done it for you (and you can legally and ethically use their solution, of course, e.g. we found using ACE on a C++ project to be well worth it).

Luckily, we have a leg up on most places for these issues -- we can generally use Java and C++, which lend themselves to maintainable, supportable code far better than C or assembly, and we make the effort for our code to be readable and understandable if at all possible.

Xentax

Re:Do it mathematica-style

[DumbSwede]

If you will look two posts up in this same thread (also by myself, DumbSwede), you will see that I flesh out the verification process, as the original author of this method for Wolfram. With out automatic and reliable verification, this method only uncovers catastrophic error conditions like crashes (which can be a valuable exercise in itself). When thing go wrong, you have to be able to trace back the series of events, so you keep updating a log of every action performed.

As explained the first time, many operations are symmetric, especially in the case of Mathematica, which is why it works so well at Wolfram. In some rare cases a symmetric operation may make and then erase a potential bug find, but it is very, very rare. Most often a bug is amplified when doing a series of symmetric operations. For a simple example, take factoring a polynomial expression, subtracting the original polynomial from the factored one and simplifying should give 0. It won't tell you that the factoring was done correctly, but it will tell the attempted factorization was not equivalent algebraically. More than this, it will tell you if your simplification algorithm was strong enough to find the two expressions equal to Zero after subtraction. One trick to this method is finding non-trivial ways to generate good mathematical examples randomly, for which I spent years coming up with a bag of tricks to do. The other trick is finding a testable pass condition. In the case of symmetric operations, the pass is that you got the original back, or Zero after a Subtraction (if you are testing something mathematical in nature). For the many Matrix operations, you test for properties that hold after a transformation, such as knowing what the determinate should be, or performing a series of events that will eventually lead to an identity matrix.

But don't just think Math here, think any testable property that data should have after a known series of events, even if you generate those events randomly. Hint, you may have generated the events randomly, but you know what they where, and can factor that into creating additional deterministic operations, that lead to a testable property - a property that doesn't need eyeballs to test.

A final note here. The real power of random testing is not that any one test is better than one test by a knowledgeable human - it is that it can do MILLIONS of more tests than a knowledgeable human in a given time frame. Most of the tests will pass, most combinations absurd in their utility, many will be repeated in trivial combinations, BUT if you only find one bug after 1000 tests random tests, you could still potentially find 100 to 1000 bugs a day, remember you only have to eyeball the Failures. Some days I came close to that hundred figure, with 20,000 bugs reported in my tenure at Wolfram. Bugs, you won't see in Mathematica, but most likely would have, without random testing.

OK, A final-final note.
We even mused about a creating a utility for customers to use that would constantly delve Mathematica for obscure hidden bugs during computer idle time, very much like SETI@HOME. The powers to be, didn't want to give the impression to customers that there would be any bugs in the product that should be looked for.