Slashdot Mirror
Properly Testing Your Code?
lowlytester asks: "I work for an organization that does testing at various stages from unit testing (not XP style) to various kinds of integration tests. With all this one would expect that defect in code would be close to zero. Yet the number of defects reported is so large that I wonder how much testing is too much? What is the best way to get the biggest bang for your testing buck?" Sometimes it's not the what, it's the how, and in situations like this, I wonder if the testing procedure itself may be part of the problem. When testing code, what procedures work best for you, and do you feel that excessive testing hurts the development process at all?
In MO you can never test too much...although it is advisable to actually run user enviroment test in addition to debug test.
Moderation: +4. Modded 70% Funny and 30% Overrated. 100% Saturated.
... is to not make the mistake in the first place. This may sound kind of stupid, but it's true. Don't skip on sleep - so you may stay properly awake, don't run yourself on Coca/Pitr Cola, eat good food, go for walks, and you'll find yourself making far fewer mistakes and producing better quality stuff. And _double_check_ everything.
May be it is the motivation that drives you while testing. While the optimal motivation IMO should be "I wanna clean up the code, find and fix potential and actual bugs", I can observe many developers (and don't wanna exclude me) which assume, that anything is corrent.
In our minds, we should look for mistakes, simply imagine we were engeneers building bridges...
before each compile, one should make a small sacrifice to the debugging gods and ask them to forgive you for your syn(tax).
my last sig was too controversial... now, a new and improved useless sig!
or wolfram-style, if you prefer.
Put your program on many computers (local to you fif possible) and have them generate random input and drive it to the program to. Or if that is not possible, betatesters, try to make your program die.
The most certain way to weed out bugs (before others, that is...others that we don't want to find out i.e. users who bought the program, etc) is to use it a lot yourselfes beforehand.
In my mind this will work very well...
And no, there can never be too much testing, unless your code is like one line, which i doubt...for any full blown application, there is not "too much"
Looking for people to chat about multicopters, coding, music. skype: gtsiros
Remember that testing, and test implementation is mainly a human activity, and must be tested, and corrected...
boom boom
cLive ;-)
-- Trinity in high heels carrying a whip: The donimatrix - there is no spoonerism
If it compiles, its fine! VB's great isn't it?
I never did learn what segmentation fault meant at college.
Invoicing, Time Tracking, Reporting
I think JUnit-style testing works great, and I plan to start using it more often.
:)
Testing is good to verify that your code does exactly what you think it does; a lot of the time I produce code that I "think" works, using JUnit allows me to verify that it actually does.
Check out junit.org.
For those of you who are sceptic about unit-testing, you should try it. Setting up the tests are not as tedious as one might think, they force you to think your problem through, and maybe most of all: they make your build look cool
One wonders how your development has been organized. Everybody here should know the basics of software engineering, including but not limited to:
1) document APIs exactly, including definitions of legal and illegal data sets
2) separate test group from programmers
3) separate quality assurance from both API testing and programmers.
Well, that's the theory. I've never worked in a place where that would have been implemented. Instead, people trying to bring this in have been kicked out. In practice, maybe one should try to get a feeling of each API: how is it supposed to be used? Use each piece of software only in the implicit limits of it's programmer's idea to keep the number of bugs down. Not to mention the obvious coding style mantras.
I think, therefore thoughts exist. Ego is just an impression.
Linux is incomplete and I am finally out of Linux.
Bye.
And the answer to that is of course: "No, you should test more, and fix the bugs". And of course, looking over your development model to see why you have so many errors might be a good idea (such as formalizing who can commit code, if you've got lot's of programmer at various skill-levels).
But in real-life, many bugs are not that important, and time-to-market and development cost is more important.
So unless you provide us with more data, such as ...
...I don't think anyone will be able to give any good advice either.
Offer the product for free to a sizable corporation who could use it. It will not only give you bragging rights ("as used by Whateversoft") but will also give you reviews and suggestions from the best consultants of all: your future customers.
My experience has shown that the number one way to find defects is code reviews performed by other developers who can read the code and also understand the intended functionality. This will catch 90% of all defects before they are even released to QA.
For more information, the developers bible (IMHO) Code Complete (available on Amazon and elsewhere) has some good information on testing strategies and some hard numbers on effectiveness of testing. Good luck.
If you look at the guys with really low bug rates, like the NASA guys running the Shuttle control software, they have very separate test and development teams, and a competitive attitude. The test team "wins" if it finds a bug, and the devlopers don't want to look silly.
Some Extreme Programming techniques, such as paired coding may help too.
... is the most annoyin part of dev, but it needs to be done, i find gettin a bunch of people who are reliable and don't mind downloading builds in matter of minutes/hours is the best way, also make sure they are good at explaining the problem (eg not 'i clicked a button and it went wrong'), the more descriptive they are the faster you can find the problem.
instant messaging, a decent webhost/ftp, descriptive testers are the things you need
... NOT tested in. If the product is poorly engineered, there should be no surprise at the vast number of bugs, no matter how much testing you do. Crap is crap.
There are numerous issues to be considered when testing your code.
What are you testing for?
Who is doing the Testing?
What kind of Error/Exception Handling is in place?
What processes are in place to manage project?
How have you defined "BUG"?
Just because you have tested does not mean that your application will be bug free.
Does your bug crash the system, or do you have correct error/exception handling in place to deal with any and all bugs?
I'd suggest that you invest the time to develop a robust Exception handling, Logging and Debugging system.
Flag all possible errors (Database interactions, NullPointers etc) and deal with them specifically. Decide which errors must notify the user, and which errors are for the system log. Which errors should cause the system to halt, and which can be ignored as expected or minor.
Make your error messages meaningful. If you use Java, then try Apache log4j to handle your logging.
The best rule to follow is:
1. More things can go wrong with the Software System than go right.
2. User actions are always unpredictable.
3. Expect the unexpected/worst - then deal with it.
-- "To ask a question is to show ignorance; Not to ask a question means you'll remain ignorant."
The main thing: testing does absolutely nothing to minimize the number of defects in a particular application.. There are lots of other things that are as important.. ie: are these defect reports being seen by the appropriate developers and are they being acted on, what types of procedures and communication actually exists between the developer and the QA persons (assuming that they are not the same folk)..
The last point isn't as bizarre as it sounds, I've seen lots of places where a QA person enters bugs, but the developers silently reject them ("its not a bug, that's how the program works")
Testing just tries to discover the presence of defects, by itself, it cannot ensure that your product works perfectly (for an application of even moderate complexity, there may be an exponential number of cases and paths to check, most test cases are written for a percentage of those only).. Because of this, if you feel that you're spending too much time testing, perhaps you need to check if your test cases are appropriate to the situation and stage of development..
Another point is that tests can be automated to some degree or the other, perhaps a scriptable tool might assist in lowering some of the drudgery associated with actually assuring the quality of your software...
rant mode = on...Excessive testing ONLY hurts if it takes people away from development at the early or even middle stages of a project and forces them to run tests on incomplete sections of code.. otherwise, there is NO such thing as too many things...
"I wonder how much testing is too much"
;-)
Most tests are based on what the system is supposed to do, not on what one can do. Given a few users anything can be done to the program, things you likely didn't test because you understand the program to good. Try putting your mother behind the prog ;-)
Well, it's never to much, until you've done every possible thing. Thats one of the advantages of open source development. A lot of people are working, playing, coding with the beta version wich means a lot of things get tested. In my experiance most errors show up with unexpected things, like missing error checks etc. A good point to start testing is to test agains the developer. Ask him everything, what if i do this, what if i do that. If you think crazy enough (like normal users), you will recieve a dozen of 'euh, don't know' answers. Thats where your errors are
It's very common that a set of test cases covers only a little part of the real bugs in a product.
And it's a normal and clever wish to prepare a larger set of test cases. This can surely lead to a better bug-coverage. But this must be done wisely: all the time you receive from a release product a bug which wasn't covered by your tests, you shold look at the cases to see how they must be changed/expanded so that, if the tests are repeated that bug will be (dis)covered.
Bottom line: use bug report (also) as a feedback to improve the way you deisgn test cases. And remeber that also the testing needs to be planned and well designed, pay to it as much attention as you give to your code. Cheers.
(yeah, my sig is wrong, so what?)
667 The Neighbour of the Beast
Its all in the same package...
... My bet is that the second half would spend less total time on that program and bug fixes in a real world situation (which was the whole point of the exercise)
If you have a well thought out and well structured design... and if every single function is documented atleast mostly before coding begins... and if all of the theory of interactions fit in... then only a small amount of testing is required
The main problems which require more testing come when corners are cut over the initial design stages, or their not done as fully as they should be doing, or when people dont actually think about what input users can give
For one of my projects I did ages ago for college, half of the class (which was split to give the same coding ability spread in both halfs) were each asked to write this program and the other half were asked to design it, review the design, make a proper testing strategy, document and THEN start writing it
The second half took a little longer to get their program working, but the first half had more bugs, and spent more time testing
Most of them have nothing to do with poor coding, they have to do with poor understanding of the original problem. Which leads to writing a solution for the wrong thing. No amount of debugging will fix a wrong assumption.
Well.. showing your work to someone important always brings up bugs, every time :)
I suppose its just a case of using the product like an actual user who knows nothing about it would, right from the first step.
If you really want to generate bug free code, you have to keep one rule in mind at ALL times. A bug occuring in the code is a failure in the methodology you are currently using to avoid them. Sounds very basic, but a lot of companies forget that.
;-)
When you have a problem with bugs, you need to figure out where in the process the problem happened. Was the unit spec wrong? Documentation? Implementation? Unit testing procedures? Was it a correctable problem caused by the engineer involved?
If you really want to be bug-free, every time one shows up you have to figure out why it happened, and change things.
Personally, I think the biggest one is to make engineers work 9-5. Not 9-7, or 11-9. Tell them to go home at 5, even if they're in the middle of something. Software engineering is a very complex task that takes a lot of energy and concentration to do right. Just like Doctors who work long hours make mistakes (resulting, often, in people dying), engineers who work too long make mistakes too.
Being in the "zone" is often the death of good code. You get lots of cool code written, but none of it is double-checked, none of it is verified to match spec, and it often ends up afterwards difficult to understand.
Now don't get me wrong, I don't do any of this, and my crap is FULL of bugs, but thats what you need to do if you really want to help it. Writing buggy code is like a public works program for QA people. Who wants a hundred thousand unemployed anal-retentive QA people nitpicking something else like your car's inspection or your tax forms? Better to keep them in the software where they can't do any harm
Another good way to reduce errors is to follow the principals of Design by Contract.
State using Assertions what is expected of the code. Pre Conditions and Post Conditions.
If any of these fail, then throw an appropriate Exception.
-- "To ask a question is to show ignorance; Not to ask a question means you'll remain ignorant."
Mathmatical proofs and unit testing asside make sure that your program dosent crap out when its being used or abused. So get some regular people to bang on it..
... trust the programmers when it comes to testing. You may find some obvious buigs in your own code, but when it comes to runtime testing most programmers tend to emphasize on "correct" user behaviour or maybe the few wrong input sets they've taken care of and completely neglect the pervert fantasy of lusers *g*
;)
Dedicated testers OTOH are perfect in letting a program die in the most obscure ways. Did you know that you can crash a Commodore 64's BASIC interpreter by just typing PRINT 5+"A"+-5 ?
I know it sounds a bit cheesy, but the trick is to make sure there are very few bugs in the product before you enter test. Try things like code reviews, spend time on removing defects early on in your process. Read a few books on how to achieve high quality programs. There are lots of books on software quality and process stuff, Rapid Development by Steve McConnel is one of my favourites.
Testing won't make a bad product good no matter how long you test it, you will only find lots of bugs. Testing a good (high quality) product will confirm that it is OK. Testing is a very inefficient way to find defects. Mind you that doesn't mean to get rid of testing!
Thomas
See ya, Thomas
IMHO, programming and testing should be done at the same time in the development stage.
While programming and "bugging" happen at the same time, programming and de bugging/testing should happen at the same time too.
It is very well explained in Bruce Eckel's Thinking in Java . You should just test everything in the code itself, even if it happens to add some overhead. Once called that function, you want that <something> happens.. so check it in the code.
I know this is not the usual way procedural programming happens. It seems much more straightforward to drop the code as it comes and then check if it behaves correctly.
But if you do so you will often discover that that tests made afterwards ara not comprehensive of all possible situations.
And so you discover that testing and debugging are just unfinished tales, and it is even worst if testers are not the programmers who did the work.
Plus, I hate testing, so I force myself to do the work well and let the code (as long as possible) test itself, even if it makes development slower and boring.
Umhh... i'll preview this post 10 times, hoping it's free from bugs :)
Obviously my code contains no ewwows ;)
:dikappa
It's a well known fact that source inspection, massive beta testing with a huge number of beta testers and proper bug reporting system is the most effective way to debug your software..
Open source systems can be good at all points (but it's usually not the case).
Microsoft scores only on the number of beta testers...
Software is immune from defect liability anyway so why bother? I worked as a 'lowlytester' once too, and it's one step below building custodian. The marketing brings in the sales, so they're the crown princes. And the software engineers surely don't appreciate someone pointing out their defects, how rude, especially from someone who just installs and runs it, and doesn't appreciate all the effort it takes to code. Testers are worse then customers. At least the engineers are shielded from customers by tech support. Plus, if a commercial software company actually did publish quality software, it would destroy the whole software cash cow ecosystem that's been carefully evolved over the last 25 years. As long as everyone emits buggy products, it's a level playing field, customers are kept on the perpetual upgrade tredmill, and unit sales keep the gravy train going. But sell a good, working product? Give me a break, that would ruin everything! Sure it would look good on the current balance sheet, but in the long run it would just put competitors out of business, then customers would lose the incentive to try out the next version, in vain hopes that "maybe THIS one will do what we want" and boom, that's the end.
Yesterday we rolled out Symantec WinFaxPro 10.02 and guess what, it breaks Word 2K! Isn't that great? Now you might think, "What, does Symantec think nobody uses fax software and word on the same PC" but that's not the point. The good part is that I got to go around to all these workstations and delete macros and reboot a couple of times, which kept me from dealing with something productive like purchasing a notebook for a field tech. Now that's job security, knowing which files to delete to make Word work again. It's all about the benjamins dude, and bugs and workarounds are an essential part of customer control we leverage to get them. We will not tolerate some lowly 'tester' out to destroy our jobs.
try { do() || do_not(); } catch (JediException err) { yoda(err); }
Microsoft's massive use of Beta Testers in the user community was at the time called the largest beta test in History. They had thousands of people working for free, for quite some time.
Perhaps this model should be more widely adopted....
especially considering that the result was an effeciently designed product which was practically bug free....
Oh, wait, maybe it didn't work out EXACTLY that way..
-- -- Warning. Do not stare directly at the sun.
When a programmer is simultaniously coding and documenting thier code, at both the high and low levels, the larger "thought" bugs will decrease in number and severity.
Even if you don't use a literate programming system, often documenting the system before you write it can help make the code more clear.
- Serge Wroclawski
I don't know how relevent this is but I read somewhere long ago that newspapers ended up with more errors if they had multiple people proof reading the same text because nobody was really taking responsibility. Even if it is not intentional, there is always a feeling of 'the other guy will pick that up'.
I vaguely remember something similar being said about the space shuttle disaster.
*Unless* you never stop testing your program.
Define acceptence criteria for the program eg: must process square widgets sucessfully, and test, using every valid and invalid situation you can think of (does it reject triangles correctly?) until you meet *all* those criteria.
There's always going to be issues, you just need to be sure that the cost of resolving those issues will be within acceptable tolerances.
Of course, making sure your system is designed and specified properly in the first place will eliminate most defects and (IMHO) make the ones that show up just all that much easier to fix.
btw: http://www.fastcompany.com/online/06/writestuff.ht ml is a great article on how they test the software that goes into the Space Shuttle.
One thing I find that adds bugs is when the management feel that an extra 'feature' or two is the most useful thing you could do. It's not - every thime the spec changes, you're having to hack little extra bits into the system.
I suppose a modular design is useful, as the 'plugging in' of new modules will not make huge, bloated classes.
The way we test things is to get the developer to test his own work, then a tester tests the work. The amount of bugs found by the tester is noted, and the developer fixes them, then back to the next test iteration.
It's about sharing the responsibility for when things go wrong - the amount of bugs found by the tester relate to how well the developer is doing, but when the tester finally says a system has 'passed', they also become responsible. I guess it's fear motivating people to produce quality software.
Just a few thoughts.
Adrian Hill @ zendog.co.uk.
---
"An eye for an eye leaves the whole world blind" - Gandhi
Testing is mainly done to find errors and bugs, so the test cases must be designed in a way to probe the limits of the system that you are testing. For example its not a proper test for a web server if it reacts to the request for http://foo.bar but it could be a nice test to send request with an URL that is terribly long and so we try to provoke a buffer overflow.
One other major mistake is that often the test is just "trying the thing out". A good test needs a good plan of WHAT you want to test, HOW you are going to perform the test and WHAT criterias are used to decide if a test fails or passes. Especially the last point is something that people want to "discuss away" by saying "Oh, that is just a minor issue, but the test at all passed" and so on.
And last but not least: To test something you also need a specification to know what the thing should do or shouldn't do. I've once been assigned to test a software that was completely without documentation and they told us to play around with it to find out what it does. That was really a nice base for a test setup...
Just the 2 cents from a guy that has worked long in software development and is now working in a PC hardware test lab.
I don't think that you should worry about finding 'too many' bugs in your testing. If you find them the customers will not, and that is good. Here are a few of my core beliefs about testing. * You can never find the last bug in your system. There have been many studies about this. It is impossible to prove that your system is bug free. Don't argue, it's true. Impossible. What you must do is decide what level of testing to perform based on the cost of that testing against the cost of your customers finding bugs. * You must categorise the bugs that you find and record discovery & clean up rates over time. This allows you to see whether you are improving or not and what trends are emerging. Joel Spolsky's site has good stuff on defect tracking. * The best testing is balanced and I strongly believe in a three-cornered tension. Separate analysis, development and test teams allow the refinement of each of those processes. Analysts and testers in conjunction will identify bugs and also trends and faults in the development process. Developers and testers in conjunction find faults in the analysis and analysis process. Developers and Analysts in conjunction, over time, refine the test process. * Insert defects that you know about at various stages in your process to measure the effectiveness of the test process. This works well for reviews and pretty well for code as long as you tell your boss. * Finally and most importantly, post pompously to discussion groups at all times.
Good design is always the best way to avoid buggy, hard to fix code.
But for testing it depends what you testing,
a good general test process for data processes (most functions can be though of as dataprocesses) is
Generate some input test data,
Work out what the results should be by hand.
this is your first stage regression.
Now run the input test data through the application/function
Diff the results against your hand generate file.
Any descrepancies should be resolved as,
A bug in the application/function
or
A Bug in the hand generated files.
Fix all the hand-job problems
Repeat until you test files are perfect.
You now have a second stage regression test,
Known good inputs, and known good outputs
Use the correct test files to fix the application bugs.
If a bug is found that isn't in you second stage regression, then generate test files for it.
Fix the bug in the application(using the test files).
Then run you second stage regression and check that any differences are down to the bug that was found (and been corrected).
Following this process you application should always get better, and a you should soon be able to build up a fairly large sample of test data.
The test harnis is simple enough (just a diff on the files and a bit of code to wrap up the functions.), to prevent artifacts caused by the testing process.
Any-how that's more-or-less what I do for most of my testing, bug fixing.
thank God the internet isn't a human right.
as XP says... The prupose of testing a program is to find problems in it, not to verify that a program runs correctly. A test that reveals a problem is a success. A test that did not reveal a problem is a waste of time. However, if the design has flaws, your program will be sure to have too many bugs for your own good. Ever tried reviewing the designs with an experienced tester? They might find some nice flaws and possible bugs even before you have written a single line of code.
We're all out there, somewhere, waiting to happen.
This strategy works - lots of shops use it all the time. However, the real premise of the process is that you want to get through client acceptance testing as soon as possible, as long as the result is not dissatisfaction on the part of the client with the software after they've accepted it. As you have noticed this strategy doesn't actually produce bug-free code.
This is not surprising. What you achieve is after all pretty much determined by what your goal was. You (shops in general) need to think hard about what your actual goal is. If your goal is nearly-zero-defects, then the traditional process isn't doing the right things for you. If however, your goal is to obtain milestone payments from your client, then it's pretty good. This is an area where the business goals determine the software engineering processes.
Let's put another hat on and think about what the negative affects of this strategy might be (negative is really defined in terms of what your goals are, but let's be vague about that for a moment).
All of the above factors are unpleasant for those left to maintain the code. Many of them also limit the longer term flexibility of the product and hence the useful life of the software. This feeds back into development processes because limited product lifetimes mean that there is less incentive to change your process to produce software which can persist (i.e. why make the effort to ensure that the system is flexible enough to last through 20 years of changing requirements when you expect the system to be retired after only 7 years?)
You mentioned XP - it offers a lot of techniques that resolve these problems:-
However, XP is best adapted to projects where a single team makes multiple frequent deliveries of code, can work closely with the client, and where the development project continues in the medium to long term. These characteristics allow many of the XP techniques, and this means that techniques taken out of XP may not help projects of a different style.
Having said this, the automated testing angle is a real strength. If testing is done manually, it's time consuming and expensive. Hence people don't do it as much as they might otherwise thing is appropriate. Maintenance deliveries often just undergo regression testing, and faults can creep in which might have been caught by the original unit or integration tests. Automated testing has many advantages :-
Just as a data point, I work on some software that has an automated test suite. The suite contains between 500 and 1000 test cases; the test suite conducts those tests in under 5 minutes on a very old machine. To do these tests manually would take one full-time person at least a week.
The summary is :-
Testing, like documentation, is not improved by sheer quantity. If you are doing 1,000 crap tests then 10,000 crap tests are not going to catch many more bugs.
Look at your bugs and ask yourself: at which stage of testing should this bug have been caught? Then find out why it was not.
If the bug couldn't have been caught by a component of your testing framework then you will have to look at the framework.
Quality audited tests as what you want, you may find that most of your tests are testing for things that never could happen and thus flooding you with worthless reports and giving you a false sense of confidence.
As an aside, I have in the past created useless tests to calm panicy project leaders. "5% of our tests failed" sounds much better then "50% of our tests failed" and it as all achieved by adding unnecessary tests. But at least I knew which ones they were.
What I miss in this discussion is something about the persons performing the tests. In my companyh we have a test team, consisting mainly of people who don't know the first thing about coding, who cannot read sources and who can only test 'through the UI'. And yet the system we work on has thousands of sources, a percentage of which has a UI (20%). Testing of all the underlying objects is a lot harder, and my experience is that with this many sources the total amount of possible 'paths' in the system is so large that tests using the UI take too much time, and therefore is never done properly. So now the developers are constantly asked to provide methods by which the testers can perform the tests.
I can read your sig, but it would be more interesting if it said something, rather than being random letters.
When it comes to software, lots of the errors are caused by bad design or a people who have not been thinking. Often many errors are of the same basic type.
The real way to test is, that when you find a few errors of one type, you call it a design error, and sends it back to the developers so they canfix this type of error all over their software. This is much more efficient than having the developers focus on 10 single instances of the same error.
This type of test really ought to be done internally over a few iterations , but most often the people you use for testing don't find the,.
A good way to fix bugs is to at least have someparts the debugging done by groups of 2 people, letting person1 explain for person 2 what the code does. This will force 2 people to think, and will give better code quality. This is called Extreme Programming these days, but I have used it years before it was invented, and it works perfectly for debugging.
first is that the program works properly, eg if it was a finance program put in some likely scenarios, with large and small numbers, make sure everythin works as its supposed to
secondly is enter in as much crap as you can, do your best to crash it, act like a complete and utter retard to the program, put in the opposite of what it would normally want
I believe that most bang per buck can be achieved if the organisation is not too fixated to one or two standard testing procedures. Projects differ a lot. Using 30 percent of testing budget for testing the testing plan might well be worth the effort. If your company is making a set of applications for a fixed platform using fixed components and fixed architecture and these basics have been previously thoroughly tested, then ofcourse what was said above might not be true.
Having been on teams producing 24 X 7, bullet proof code for communication servers and credit card processing I have an idea about the increasing number of bugs found. In the Old Days(tm), we wrote every line of code ourselves and used time tested libraries (C language). I quit using microsnot when their libraries stared having bugs in their rush to C++. Now most coders use massive OOP libraries from who knows where built by slackers, and GUI app builders that generate code and perform all sorts of actions under the hood. When something goes TU it is often hard to find all the conflicts.
Even when using one of these app builders I read through all the code and put tests and logging into the generated code. Funny that these tools are supposed to make us more productive. My coding and testing every line still beats total time spent on a project since I don't have to go back and redo it later. When it's done, it's done. Next project. I've had comm programs run for over 5 years error free servicing 1000s of users per day. One specialized delivery, billing, and inventory system I wrote was used over 6 years error free and caused the owner to stay with hardware that was compatible with the software (not M$) because the programs always worked. And not a damn bit of it was OO or came from some automated builder tool.
In short, the closer you get to the metal and the more familiar you are with the code that is executing, the better your chances of producing error free programs. Takes longer to market, but then you don't have to redo it forever until the next bug ridden version comes out. Saves time and coders to work on the next version and the customers are always pleased. Get back to the basics. Try it, you'll like it.
One of the best ways to find out non-obvious bugs is to, if you can, get normal people to use the program as they would. People with no prior knowledge of the software, apart from what they want it to do, will use the software in a way that people who have worked on the code and know the internal structure wouldn't, and hence find things that coders wouldn't. Nothing is idiot proof because idiots are so damned clever.
Help the scientists free the world from the evil curse of the dracula
If a project is worked on with the assumption that bugs can be worked out during testing, then usually a greater number of bugs are introduced when the project reaches the testing stage. Instead, a project should be engineered for quality; it may take slightly longer, but there will be fewer bugs introduced to the project, and less time will be needed to squish bugs to an "acceptable" level.
Firstly, I strongly recommend a peer-review system where other developers eye-ball the code to make sure it all looks ok. This should catch most bugs.
The other thing we use is a system that peeks at the code and data structures and look for common mistakes. The code can't be checked in unless this automatic test has been run and the recommendations followed. Automated systems are great for picking up easily recognisable (and often silly) problems before it ever gets to QA.
The great thing about rules based systems is that you can often automatic checking that the rules have been followed.
...you're kinda vague about what exactly most of the bugs are in.
Assuming that everyone is working on a different branch of the program that you're building (Coder (or group of coders) X is responsible for writing a nice new algorithm, Coder (or group of coders) Y is responsible for writing a GUI for it, Coder (or group of coders) Z is responsible for the making it run on a network) then the most important issue is to make sure everything works right together. Comment lots (and in a standard format - i.e. If you're using an OOP language, make sure every object has an accurate description of what it does, and what'll make it break), and communicate lots, so that everyone knows exactly where everyone else is going.
After that, make sure every coder (or coder group) runs their code through white box, black box and whatever other testing methods whose names are locked away in the back of my memory. Beta test every 'module', before you even start to think about throwing them together.
Also, someone mentioned above that the health of the coders was important. He's/She's/They're right. Offer lots of free snacks (or at least free perked coffee), give them a couple cots to lie down on if they get tired, give them an hour a day to do some kind of physical activity (yes, make them excercise) and refuse to let them work more than 50 hours a week (even that's extreme). You'll find their productivity increases, and when productivity goes up, mistakes go way down.
Testing should always be a part of the development process. The wording here implies that testing somehow is considered to be outside the scope of development and I suspect this mindset is causing a lot of bugs to remain undetected.
It's just like documentation or support, those are also (or damned well should be) integral parts of the development process. Sometimes I think that most programmer's believe that the development process consists of the steps hack, compile, ship instead of the tedious iterative process of analyze, design, code, test.
So what, then, is excessive testing?
Well, as long as you find bugs doing it, it's not excessive.
If your projections predict a bug or two in a specific piece of code and your tests fail to find them, then testing (provided that the test method isn't flawed) gave you a much desired quality assessment of that piece of code - meaning that the testing still wasn't excessive.
Running the same tests over and over on the same code with the same data, now that's excessive, not to mention stupid.
Money for nothing, pix for free
Well, at least partially.
Write code and test it as you write it, a combination of bottom-up and top-down development and incremental testing. At this level, the test cases should be trying to break everything, particularly if you're dealing with any kind of external input (i.e. anything NOT in memory that you didn't put there yourself). Remove files, fill them with binary, make them unreadable, etc. This sort of stuff can be the basis or even be the regression test suites that can be used for automated testing.
If I write a particularly tricky bit of code (which sometimes you just have to, for performance or something), I separate it and write it completely separately so I can test as much of it as possible before it gets into a bigger system. Faster turn-around with changes helps with more-or-less prototype code like this.
If your api says 'you can't pass a null here', or 'must be between 1 and 7', put an assertion inside that invocation, let the computer test it for you at runtime.
Tools like gprof and quantify let you know if you're test cases are touching all the cases. If you have exception cases, try and write test cases that will touch all different exception cases at least once. A bunch of bugs i come across are stuff that wouldn't normally happen that does. Sure, the program might be going down the toilet already, but its better it keeps going down than floods the bathroom floor.
As the scale of the part of the system under test grows (i.e. from unit to module to system testing), its usually harder to exercise individual cases, so the tests change focus a bit. e.g. to usability (just because dialogue a works great, and dialogue b works fine, what happens if a opens before b and you close a first, etc), scalability - 10 things might be fine, 1000 might be unusable, functionality - did you hook up a menu to run function y?
Some things you can fully automate, others you need someone to operate.
If its something that needs to run in multiple environments, test in as many of them as you can. Before you beta test. Just as its bad to get a low-level api type bug exposed by your qa testing team, its much worse for it to reach your beta testers.
And yeah, the best way is to write and maintain quality *DESIGN* from the start. It makes finding and fixing bugs easier, and adding new features isn't like walking a tightrope with glass slippers on.
_
\\/ are accustomed' - First Lensman
When someone is told to implement feature A, they spend a little time sifting thru the 20yr old code, and do the minimum to get it done.
They write test cases to test A, unit, system, etc. Their team leader approves it, and of course all the tests pass before it goes out.
In the end, there's always some obscure way feat A interferes with feat B, but you're not going to write tests for every combination of keystrokes possible.
If you have user testing (u should), they'll find a score of bugs u didn't. Of course, the users too have a deadline to get stuff shipping and they too want to do the minimum possible.
In the end, nobody involved has a personal incentive to make it perfect. With proper testing procedures in place, everyone has a piece of paper with someone's signature on it which says they passed. They don't feel (too) guilty when there's a bug, (hey, my TL approved it!).
Anyway, how are you going to ask the customer for $20k extra so you can test for an extra week?
Of course you should stop testing! There is no such thing as flawless code. You must choose a limit for testing: life-support systems need more testing and QA than an ordinary Information System.
A neet technique to find how many bugs are you missing in your tests, is bug seeding. It consists in create bugs artificially and submit that code to the test team. Then you count the number of seeded bugs the test team found.
Lets try an example: suppose you seed 10 bugs and the test team found 8 of them. Then you know that you are finding at most 80% of the bugs (in a given confidence interval, depending of the total number of founded bugs). 80% can be very acceptable in the classical Information System, in slashcode, but it't completely unnacceptable for NASA Shuttle control software.
--
Senador
I guess you should try to spend a part of your testing budget on improving your design and programming practices.
You can't test your code. When you do this you always try to use it the same way you thought when you were coding.
I suggest you ask your grandma or someone that doesn't know much about computers to do it. They can always find bugs.
The object of finding bugs isn't to result in fewer bugs by fixing them. It's to result in fewer bugs by not writing them in the first place. The developers need to review found bugs on a regular basis, with the objective of changing development methods to avoid them in the future.
It's all fine and good to say "don't write buggy code in the first place," but this sort of feedback is the only way to get there. What makes this so hard in many organizations -- aside from the usual disrespect many developers have for QA people -- is that developers fear that this process is some sort of performance evaluation. As soon as this happens, the focus shifts from finding better processes to defending existing processes: "It's not really a bug," "There isn't really a better way of doing that," "We just don't have time to do it the 'right' way," and so on.
This is why the feedback needs to be direct from QA to the developers, who are then tasked to categorize bugs and develop recommendations for avoiding them. It's the latter that is the "product" required by management, not a list of bugs with developer's names on them. Management should otherwise get the hell out of the way.
Of course properly written functionality test scripts (doing what the user does) will find most bugs. The downside is that it is boring to follow test scripts manually.
My company has been successful implementing automated functionality tests with Rational Robot (part of teamtest). If you just take the time to define proper test scripts you can easily redo all functionality tests on various platforms (if you use VMWare or similar sw to simulate different platforms) at the click of a button.
This saves time every release as the developers can focus on finding the really tough bugs instead of running boring functionality tests again.
I've worked on both ends (dev and test), at M$ and other places, and I've come to one conclusion (I'm sure its not the only correct one).
Developers must test their code.
With a test team backing you up, it becomes too easy to change something, run it once (if at all), and then push it into the next build so the test team can catch your errors. I've found that as a tester, a huge proportion of bugs are simply features implemented where the developer just forgot something stupid. I end up wasting 5 minutes writing a report, my manager assigns the bug back to a developer (hopefully the one who made the mistake but not always), and the developer comes back to the code a week later, spending 20 minutes just trying to figure what s/he wrote a week back.
My point: this wastes 30 minutes of people's time for every little stupid mistake. Pressure your developers to really give a thorough test to the code they write before the check it in, especially if you have a test team, because you just end up wasting more people's time.
Your signatures belong to me.
I often think the time lines have become so compressed in terms of expectations that it becomes harder and harder for companies to write clean code and get it out the door in order to meet the expected cycle of upgrades - and this is something i find common to all companies and even open source software. It seems we as consumers have come to expect an upgrade to this or that every year and so the dev cycle becomes one continuous thing - coders who are exhausted or working long hours write buggy code.
I think many people would be happy to wait a bit longer for better products but the industry has brainwashed them into thinking that its almost easy to bung a new version out.
The best way to avoid mistakes is not to make them in the first place but thats not so easy when working on a compressed cycle with management on top of you - its not just programmers who deal with it - its network designers, SOE architects etc etc
Bugs exist - they always will - but minimising them requires time and time is a commodity not readily found. I dont know what the solution is - as i say its just my thoughts.
I refuse to argue with Anonymous Cowards - if you want a discussion get an account....
They told us to put a potted plant infront of the keyboard and see if the program would still function properly.
I believe that the existence of women is proof that God loves us and wants us to be happy
execessive testing never can hurt the code - at worst it is a waste of time.
perhaps the most usefull tests for you right now is to just make tests for all the known bugs to ensure they do not pop up again. So at least you won't regress.
While I am not certain exactly what you are programming for it sounds like the problems are comming because you are making lots of changes to your current code base. I would also suggest setting an expectation of the amount of bugs that pop up/the amount of changes that are made to the code. Becoming more consciousness of how changes in code relate to new bugs can be very helpfull in overall stability as it can help you predict how many new bugs will occur and also show you the strengths/weaknesses in the design of your project in terms of changability.
The Point of Testing is NOT to find bugs in your code, but rather in your software engineering process! if testing discovers a problem with your code that means your software engineering process is flawed and needs to be changed so as to ensure that this kind of bug (1) will never be able to occur again and (2) cannot hide in all your old code.
it sounds like your SW engineering process is totally crap an urgently needs to be changed.
striving for error free testing sounds complicated at first and requires discipline but it really pays off, economically and otherwise!
www.sei.cmu.edu/
i would refuse to work for shops that don't strive for and regularly achieve 100% error free software at the point before testing.
Testing input/output, or the application's functionality if you will, is important, but do not forget to test your application on all the different platforms the users will run this on. You may run into missing DLLs, incompatible versions of ODBC drivers. Your application is only part of a greater whole, and often you have only limited control over components outside your application. This is a great source of bugs for programs that are meant to run on many workstations.
Indeed. Yet, many a time have I seen projects where seemingly rigorous testing was applied: Unit testing, integration testing, FAT, and SAT are all carried out diligently, but upon closer examination all these tests were testing the same things under the same conditions. In short, a huge waste of time and effort. If your project can afford one, get a test specialist or test manager, and have him make sure each test is conducted at the proper level.
If construction was anything like programming, an incorrectly fitted lock would bring down the entire building...
Have a kid go at it for an hour or two, they're cheap and efficient.
Seriously, this is the best thing to do. You want to see how your product performs out of the test lab? Then go outside the lab and give a copy to your friends and their friends. People who know how the product is supposed to work seldom breaks stuff, it's when you let the inexperienced join the game the fun begins.
Anataka suki desu. Itsumo. Itsumademo.
Is that it liberates programmers from the responsibility for getting it right in the first place. After all, any problems will be found in testing, right?
Wrong, actually. Testing tends to find the surface problems. Sometimes in the process of fixing the surface problems the deeper problems are revealed, but just as often they become even more deeply hidden.
There is no substitute for reading the code and making sure it makes sense. In an ideal world, this would be done by someone other than the original programmer, but doing it yourself is better than nothing. Yeah, it's really boring. That's what those paychecks are for.
use constant PERL_IS_BROKEN => $] >= 5.006;
By 'a large number of defects' do you mean larger than expected? The "average developer" creates between 50 and 150 bugs per KLOC (thousand lines of code) of new code. Is your error rate higher than this? What other metrics do you have in place to indicate that the error rate is high? How many errors are they creating in this release versus the previous release? How many did you expect? The phrasing of your question implies that these basic metrics are not known and thus your statement about the number of errors is not meaningful. If you want to improve software quality, one place to start is to make some basic measurements and predictions so you can track your progress and detect anomalies as they occur.
Secondly, developers, not testing/QA, are solely responsible for code quality. You cannot test quality into the system; it must be designed and constructed into the system from its inception. If the developers are creating too many errors, examine the development process. It must include formal code reviews (80-90% of all bugs will be found in review) and adequate time to refine the requirements, develop a sensible design and build the software. Pick up a copy of "Code Complete"; it'll help alot.
public void karmaWhore(String url){addSlashdotComment(fetchContent(url));}
The point of testing is to find bugs. If you're not finding bugs then you're not testing hard enough because ALL software contains bugs.
I once heard of a place where the testing people were actually paid a small bonus for each bug they found, and the programmers bonus was based on the number of bugs found in their code, and the time it took them to write it in the first place.
The job of a programmer is to write code, not to find bugs, so you write code that seems to work and send it off for testing. It then comes back with a list of bugs as long as your arm, mostly to do with integration testing, and you fix'em. You send it back for more testing and it comes back with a short list of bugs caused by your previous fixes and you fix 'em. Eventually someone decides that bugs are sufficiently rare, or unimportant (ie minor glitches which could be difficult to fix quickly) that it is time to release the software. Then you get a long list of bug reports from your customers when they blatently and wantonly start using your software in a totally different way to which you had thought they would use it thus breaking things you thought worked.
You can't win Darth. If you mod me down, I shall become more powerful than you could possibly imagine
Hi, I am a software designer for a large corporation (as well as free-lance) I have found that there are people in this world who have a natural knack for crashing computers and finding bugs in software. It's kind of like the light/dark side of the force.
Some people - such as myself- can simply enter a room with a 'mis-behaving' computer, and it will all of the sudden start working properly. Likewise, there are people who can just LOOK at a computer and it will start to malfunction.
I have three people (natural crashers) that I have beta test all of my software that are incredible testers. By the time each one of them sees three or four revisions, the software is nearly flawless.
Don't be fooled, some people are just plain idiots. Sometimes it is hard to tell the difference between an idiot and someone with the crashing-gift. For example: A natural crasher AND an idiot may try to put text in a numeric-only field, but the idiot will want you to change the database format to allow text - whereas the 'natural crasher' will tell you that it doesn't make sense to turn that field into a text field - but he wanted to see what would happen if he tried it.
Once you stumble onto one of these amazing people, your software design will improve 10-fold.
You still find many bugs? Maybe you wouldn't find them if you weren't doing unit test? What if they'd be there, you just wouldn't detect them.
Testing is important. I remember reading that testing and debugging a program takes as much ressources (time, money, manpower) than designing and writing it in the first place. I found it to bne close to that. The most common mistake is to have many programmers and designers and just one or two people doing QA. Your efforts should be on par on both sides. It is also good to involve the developement staff in testing and debugging (even tough they'll hate you for that) at least once because they'll be more conscious about their designing and programming and next projects (or itterations) and the potential bugs hidden and issues that might come up in QA.
Unit testing is a double edged sword. Since you are testing units you might be tempted to test everything about a specific module. And then overlook integration tests. That's a mistake. Integration tests are critical because they are the ones bearing the maximum level of complexity. Also tests should always be defined according to some kind of user story or scenario (many of them, at the very least one per feature, use case templates are really not a bad start).
First thing to do : look in your bugtracking software ( you DO use bug tracking software, right ?) , and try to isolate hot spots. Is there a particular piece of code that generates more bugs than others ? Is there a common pattern to the bugs (ie. memory not being freed, of-by-one errors etc.) ? Are they _really_ bugs or mis-interpretations of the requirements or the design ? In my experience, the 80/20 rule applies to bugs in spades - it is just hard to find the patterns.
If you need to, make the bug categorisation in your bug tracking software more specific. Once you get an idea of what your hotspot is, you can work at fixing the cause of the bugs.
If it's a particular piece of code, make sure it's reviewed by the best developers/architects you have, and consider refactoring it. At the very least, insist that it is reviewed and tested thoroughly before chec-in to the source code control system, and consider adding a hurdle to jump prior to check in (e.g. get the manager to sign it off).
If the code was written by one developer, consider swapping them out and giving it to someone else - it may be they're in over their head.
Make sure you increase the number of test cases for this piece of software, and check for "edge cases" religiously - if the code is broken at all, it is likely to be broken in more ways than you realized.
If it turns out that the problems tend to have a common cause (memory leaks, of-by-one errors,etc.) consider a structure which forces developers to concentrate on those issues before checking in code; again, consider the hurdle ("software must be signed off by the off-by-one guru prior to check in"), and hone your tests to check for these kinds of errors if possible.
If the bugs stem more from misunderstood requirements or designs, beef up those areas. Work on your requirements and analysis processes; consider training courses for the developers to get them up to speed on interpreting these nebulous documents, and look at improving the review process by having designers present. Frequent "mini-deliverables" (another concept stolen from XP) will help here too - get your team to deliver a working piece of code - it need only be a minimal sub-system - and get it reviewed by designers and analysts. If the bugs tend to occur on the boundaries - i.e. invalid API calls, invalid parameters etc. - consider design by contract or aspects.
Finally, there's a bunch of hygiene stuff
N
It's all very well in practice, but it will never work in theory.
There's no one size fits all process for testing. How much effort you need to spend on testing depends on a lot of factors including but certainly not limited to: code size, amount of developers, customer requirements, life cycle of the system etc.
That being said, here are some remarks that make sense for any project:
In general a testing procedure that gives you no defects just indicates your testing procedure is bogus: defect free code does not exist and no test procedure (especially no automated procedure) will reveal all defects.
The XP way of determining when a product is good enough: write a test for a feature before you write code. If your code passes the test it is good enough. This makes sense and I have seen it applied successfully.
A second guideline is to write regression tests: when you fix a bug, write an automated test so you can avoid this bug in the future. Regression tests should be run as often as possible (e.g. on nightly builds). All large software organizations I've encountered do this. Combined with the first approach this will provide you with a growing set of automated tests that will assure your code is doing what its supposed to do without breaking stuff.
Thirdly, make sure code is reviewed (by an expert and not the new script kiddie on the block) before it is checked in. Don't accept code that is not up to the preset standards. Once you start accepting bad code you're code base will start to deteriorate rapidly.
Jilles
Part of the coding process should be 'quality' coding to begin with- with proper planning details and what not.
However, in the web world - most projects are rushed, and pushed, and not allowed the proper time to do so (at least in my experience in the dot.com and post dot.com world) - hence, the continued cycle of carnage.
Under these conditions - if you can get some semi-decent QA people to help out - you can pinpoint problems that way a lot quicker. Why? They are not attached to the code. In fact, most QA people these days, can't even code. However, what they should excel in is being either a) A dumb ass user or b) following the logic train.
Lets start with the 'dumb ass user' view point. They will try to put in letters to numbers, numbers to letter, insane characters to whatever fields that you've never intended. They will try to break the code under different environments, doing things that you never quite expected. Basically - a fresh perspective. Ideally, let them mess with it (if that is possible), or see if you could integrate the item in question into one of their enviroments to see what else it could possibly toast while they are doing their job. While you're coding for a particular thing, and are aware of what it is supposed to affect - its usually when it's thrown into a total environment that things start to break, and break quickly.
Now, onto the follow the logic train argument - if you, as a developer, can explain to a QA person who cannot code, but who understands the logic, in simple english - AND - they understand what you're talking about - you're 80%+ of the way there already. Again, the key thing here is a fresh perspective on the item in question. Kinda of like stepping away and them coming back to it.
Testing/debugging is like finding and putting out existing fires. If the organization can use test results to prevent future fires, then you're a step above. And probably more advanced than 90% of the software houses, too :-)
I program mostly in object oriented languages. So I have seperate files, which have seperate classes. I start at the bottom of my UML and work my way up testing each class as if it were its own program. When I know they all work individually, I can be certain that, despite the fact there had to be a few bugs I overlooked, that all bugs are due to the way they interact. It takes awhile, but in the end I'm mostly bug free.
The GeekNights podcast is going strong. Listen!
At a previous company, we had seperate Test Engineers that never (or very seldom) did any Software Engineering (although they had same qualifications). Oh, the whole company was very into Best Practices, ISO9000, TQM, procedures and all that sort of thing. So that wasn't just a fancy-sounding job title, these people actually made it their jobs to design thoroug tests, keep up to date with latest developments in their field, etc. It was a pain, but it delivered the goods.
There is a lot of research available on the Web (and trade literature) on best practices in testing. Just entering some random but legal data and saying "Ok, this works!" is NOT testing. A combination of techniques like boundary testing (input data is close to the dividing line between legal and illegal input - from both sides of the line), error injection (modify code to intentionally generate an error then analyse how this is handled), branch analysis, unit + system tests, peer reviews etc. etc. would probably be best.
One rule of thumb you would want to keep in mind, is that the earlier in your development lifecycle the bug is caught, the cheaper it is to fix (substantially). So it pays off to have testers involved with appropriate consistency checks, peer reviews, and what not, right from the requirements gathering phase. (Yes, "lifecycle" is a bit bigger than code-compile-test :-) So it doesn't help much to have a good test team bnut the development team don't know their stuff.)
Also, it is assumed that Zero Defect is very hard to achieve, therefor when addressing bugs, first go for the 20% that cause 80% of the trouble (ye olde 20/80 rule-o'-thumb).
Free, as in your money being freed from the confines of your account.
The questions were with regards to bang per buck and could you have too much testing.
In terms of bang per buck the single best test instrument we have at our disposal is the human mind, and given that it's also always best to try to catch defects at as early a stage as possible. Then in my experience the one thing that makes the biggest difference is formal peer group inspection (== testing with brain) of all project deliverables, at all stages of the project; be that requirements, HLDs, LLDs, code or test cases. It's a pain to implement, but it works far better than any other form of testing at preventing bugs reaching customers.
As to too much testing, yes, under certain circumstances you can have too much testing. Testers who've tested the same thing a hundred times will tend not to see the problems on the 101st iteration. Mindlessly running automated tests is unlikely to buy you much other than again boring the same human testers who have to check the results at the end of the day. However, in general you can't have too much testing, as I've already mentioned get in very early and test deep.
Most of the 'worst' bugs i've come accross are down to bas systems design, before a single line of code is written.
If a system is designed well then you should have far fewer bugs, even if you are using code monkies who don't know a quick sort from a n^2 bubble.
Design you systems well, know your people, Bill's good at that kind of thing and likes it(but crap at ui's say),
Jess loves doing data imports, (may not be that quick, but always does them well).
Fread always designes and produces good/fast systems cores.
Get your developers talking and sharing knowlage, 'I'm, having a bit of a problem' , or 'Who knows how to', are good things for people to be saying, so incorrage them to own up to the inadiquacies, and they won't have them for long.
If you can manage that then your productivity and bug counts should drop dramaticly, and the bugs you do have should be easier to fix.
thank God the internet isn't a human right.
I find that, when debugging code I write, I tend to try to verify that the code properly handles exceptions I've anticipated. But that's the problem with my process: if I've anticipated an exception, I've already handled; if I haven't anticipated an exception, I won't think to test for it.
Ultimately, I think a lot of software problems result from the assumption on the part of the coder that users will try to use the program properly, that users won't be lazy and sloppy, and that users will RTFM. That, and many programmers just don't think things through, don't consider all the possible combinations (especially true in parser code!).
It's important to have people that aren't familiar with the expected functional specifics test the code. And when testing, you have to deliberately try to break the program.
Testing is necessary but not sufficient. There must be a way to capture requirements, convert requirements to design, convert a design to implementation, and finally test. At each transisition it is a good idea to make an assessment of how well you accomplished your task.
Skipping any of these steps and putting if off until test is pure folly. An extremely false economy.
Just generating random data and trying to load it caught a lot of bugs, but even more effective was to take a valid image and modify the bytes in it at random, and then try to load it.
Of course, the reason this was so effective, is that the loaders would get mostly what they expect, and then suddenly something illegal. This is the kind of thing you tend to forget about when you write code.
Since it is so easy to attack your program with random data, this kind of testing gives you a lot of bang for the buck, but on the other hand, the bugs it find may not always be those that are likely to occur in practice.
I am having a bit of a QA problem myself. After reading up (Steve McConnel, etc), I'm looking to spend more time in pre-code, and also implement inspections (a code review technique).
The disadvantage to testing is that you detect errors, but need to spend time finding the source. If you avoid the error by detecting it at design-time or during code review, you will spend less time dealing with it, since you will know more about the root cause to begin with.
Stop the brainwash
Hello Gentlemen,
I'm a first year programming student at an Ivy League school and I've just finished my Visual Basic classes. This term I'll be moving onto C++. However I've noticed some issues with C++ that I'd like to discuss with the rest of the programming community. Please do not think of me as being technically ignorant. In addition to VB, I am very skilled at HTML programming, one of the most challenging languages out there!
C++ is based on a concept known as Object Oriented Programming. In this style of programming (also known as OOPS in the coding community) a programmer builds "objects" or "glasses" out of his code, and then manipulates these "glasses". Since I'm assuming that you, dear reader, are as skilled at programming as I am, I'll skip further explanation of these "glasses".
Please allow me to make a brief aside here and discuss the origins C++ for a moment. My research shows that this language is one of the oldest languages in existance, pre-dating even assembly! It was created in the early 70s when AT&T began looking for a new language to write BSD, its Unix Operation System (later on, other companies would "borrow" the BSD source code to build both Solaris and Linux!) Interestingly, the name C++ is a pun by the creator of the language. When the first beta was released, it was remarked that the language would be graded as a C+, because of how hideously complex and unwieldy it was. The extra plus was tacked on during a later release when some of these issues were fixed. The language would still be graded a C, but it was the highest C possible! Truly a clever name for this language.
Back to the topic on hand, I feel that C++ - despite its flaws - has been a very valuable tool to the world of computers. Unfortunately its starting to show its age, and I feel that it should be retired as COBOL, ADA and Smalltalk seem to have been. Recently I've become aquainted with another language that's quite recently been developed. Its one that promises to greatly simplify programming. This new language is called C.
Although syntactically borrowing a great deal from its predecessor C++, C greatly simplifies things (thus its name, which hints at its simpler nature by striping off the klunky double-pluses.) Its biggest strength is that it abandons an OOPS-style of programming. No more awkward "objects" or "glasses". Instead C uses what are called structs. Vaguely similiar to a C++ "glass", a struct does away with anachronisms like inheiritance, namespaces and the whole private/public/protected/friend access issues of its variables and routines. By freeing the programmer from the requirement to juggle all these issues, the coder can focus on implementing his algorithm and rapidly developing his application.
While C lacks the speed and robustness of C++, I think these are petty issues. Given the speed of modern computers, the relative sluggishness of C shouldn't be an issue. Robustness and stability will occur as C becomes more pervasive amongst the programming community and it becomes more fine-tuned. Eventually C should have stablity rivalling that of C++.
I'm hoping to see C adopted as the de facto standard of programming. Based on what I've learned of this language, the future seems very bright indeed for C! Eventually, many years from now, perhaps we'll even see an operating system coded in this langauage.
Thank you for your time. Your feedback is greatly appreciated.
Egg Troll
Just because you are using unit tests doesn't mean you'll be bug-free.
Your tests should be good enough to catch any errors, and I've found that too often people don't cover too many possible bugs in their tests.
Review your tests. When you write a test, think about "what are all the problems that I could have here? What should be the intended behavior of the system?" And when you change your API, carefully check your tests again. Believe me, if you do it right, it works!
I've worked on a project using unit tests, and it was just great -- usually, the tests would catch several bugs before we commited the code to CVS.
But just like any other powerful tool, you need to use it right, or it just won't work.
There really isn't enough information to provide a specific response...but the "our testing is finding too many bugs, so maybe we should test less" seems a bit, well, PHBish.
Are the tests actually silly (e.g., is a low comment/code ratio a "stop-ship error") or is the problem really despair at the inability to develop good software?
BTW, in my experience, a non-competitive "we're all on the same team, we all want the same thing" relationship between testing and engineering works very well. I've never worked in an environment where the testers rub the engineer's noses in any bugs they find, nor do I care to.
TEST AS YOU GO!!!!
I write this in caps as 90% of coders write a thousand lines of code and then test. The way to do it is:
1) Write your methods from your specification (you do have a specification don't you???)
2) Read the code to make sure it does exactly what the specification says it should
3) Test the method with every type of value it can recieve.
4) fix the bugs in the method
5) re-read the code to make sure it still meets the spec.
I say method, because testing should be on the method level. A method is AT MOST 20-30 lines of code (from O-O point of view) any more and there is too much code and it's to confusing (GUIs are exceptions).
The theory is that if the methods are correct and the specs are correct, the program will work together, but still do unit testing in gradually increasing sizes.
ALSO (in O-O programming (Java,C++,C#,etc)) NEVER EVER EVER EVER EVER USE GOTO, EVER
If your spec needs a goto statement, re-writeit. I can't stress this enough. GOTO is evil, it's bad, it's taliban! It breaks the flow of the program. Java does not have goto, so you can do without it!
Also don't use pointers in C#!!!!!
Just to emphasise how good design is the key to avoiding most bugs, not testing - there's a song that often gets sung at my place of work...
Hundred and one little bugs in the code
Hundred and one little bugs in the code
Fix the code, compile the code
Hundred and two little bugs in the code
most of us actually got good specs? Been close to 3 years now for me. With half-assed specs derived from business users who A) don't know what they want and B) don't know when they're out of their league when talking about how something should work you're pretty much screwed from the beginning.
Programming is the same way. What kinds of bugs are you finding? Are they just stupid bugs, like buffer overflows or off-by-ones (good design, bad implementation), or are they unhandled errors, or are they API mis-matches or faulty algorithms (bad design)?
Have you made any effort to go back and say "Gee, we are getting a lot of off-by-one errors. OK folks, we need to think about our loops."?
And when you find one type of bug, do you go back and identify anyplace else a similar bug may exist?
If you are hitting high and right, and you never adjust your sights, you will NEVER hit the target consistently. If you never feed back the CAUSE of the bugs, you will never eliminate them.
www.eFax.com are spammers
I could swear that said biggest bong.
just wishful thinking I guess.
Next, unit- and integration-testing only detects so many bugs.
How many steps are there in the developement process?
An exampe of steps to take:
And, of course, for each document (specification, code as well as test code and its result) produced, a FORMAL REVIEW of that document also counts as a step in the process.
One rule-of-thumb I've learnt is that each step in the process cuts the number of remaining bugs to half. (A bug found in the specification might cause 10000 bugs to be detected in testing).
Thus, assuming that a 64kloc program must contain less than one bug, at least 16 steps must be involved.
In my experience the best way to develop stable code is over time. There is no way you can ship something 100% bug free the first time. Deploy the code in the real environment and weed out the bugs one by one. So never expect to make something bug free. You can make something 95 % bug free if you plan and execute really good before you deploy, but the last 5% will cost you as much as the first 95% belive me. For some companies the last 5% is important enough to justify the cost (any rocket engineers here?). The problem is that a test with 1 person is not the same with 10 dumb users doing totaly insane things to your program.
One thing I've found invaluable is to compile your program with a translator that inserts code to detect when branches have been followed. Then run the test suite and see that all the code was executed. Any code that was not executed has not been tested.
It's amazing how poor coverage can be with a naively written set of tests. Ideally you want to write the tests so that the coverage comes out good, but in practice you may have to patch the tests with more tests to cover the parts you missed. You may also have to change the code to make it easier to cover.
Rare error cases (like malloc failures) can be hard to cover.
One technique that I have found very useful is to write test programs before you write the main program. Verify the test program via code review, and ensure that it catches as many problems as reasonably possible.
Example: suppose you have a file format A, and your program is supposed to work on it and produce a file in format B. Write a program that takes files of format B and ensures that they make sense. Then also write a program that compares files of format A and B against each other and makes sure they are "equivalent" in whichever way that makes sense.
Next, develop a collection of files in format A and you have a good test bed. If your program is doing something wrong, chances are good that it will be caught soon by the test program.
A number of years back I wrote test programs for printed circuit boards. First you created a model for the board that simulated the logic circuits. You then wrote test patterns that were applied to the board's inputs, and the simulator model predicted the board's outputs. The inputs together with the predicted outputs were applied to a real board that you wanted to test, and if this test program passed you assumed that the PC board was good with a high degree of probability.
One mode of the simulator allowed you to simulate faults that might occur on the board. The simplest kinds of faults were physical IC pins "stuck-at-zero" and "stuck-at-one" (these were the most common faults in real life), and if you wanted to be thorough you could also simulate "internal" faults down to the gate level.
I worked in a contract test programming house, where the contract with the customer required us to produce a test program with a specified minimum level of fault coverage, usually just at the physical IC pin level to minimize cost of developing the program. This ranged from say 90% for cheaper commercial work to 99%+ for certain government contracts. With >95% coverage, the "real life" fault coverage was maybe one or two "dog pile" boards out of 1000 would pass the test program but fail a system test.
The point of this is in that business, there was a clear objective measure of a test programs "quality". The measure wasn't perfect, but it was far better than just blindly writing a test program based on a "gut feel" for how the board should work. In addition, the test programmer had a clear, objective goal.
I think a useful tool in the software business would be a measurement of the percent of lines of code that were actually run during the QA process, along with a log of those lines that were not run and not run. Often there are big chunks of code that only get triggered by very special conditions, and there is no way QA can guess those strange conditions. The standard QA process is very subjective; there is no objective measure of any kind as to how thorough the testing was, other than just documenting a list of features that were (often superficially) exercised.
A more sophisticated tool could go beyond lines of code and into log the various logic combinations exercised in "if" statements, etc.
Several years ago I wrote an experimental tool that did this for a specialized database programming language. Basically it rewrote the program with a logging call after each statement (and yes, the "QA version" ran very slowly). The results were quite eye-opening, revealing chunks of "dead code" and conditions no one ever thought of testing. Unfortunately the project kind of died.
Many languages have "code profilers" that are mainly intended to analyze performance, but many of them could be easily adapted to become QA quality measurement tools.
Do these kinds of tools exist, and if so why aren't they more widely used?
Sure, build a big suite of tests to run and check for things to go wrong. Every bug fixing process suggests it own test.
Then you find out that you don't have the time and resources to run all the tests everytime someone makes a change to the codebase.
So, use smaller suites of the faster tests and weed out some of the ones that have been ironclad passes for the last 5 dozen code checkins. For frequent testing it makes sense to only shake what's new and rickety, not what's stood through 10 hurricanes.
Run the exhaustive complete test suite infrequently, say when a release is imminent, or as often as you can afford to spare the resource cycles.
"Provided by the management for your protection."
If you test the code as much as you says you do, and are testing for the correct thing (which I do not know you are doing) the problem may be the architecture.
Code which is "forced" into a paper architecture is sometime worse that code with no architecture at all. In many of my projects, parts of my architecture change part way through so that the code will work better. Sometime not everything can be thought of before hand. OO programs have a lot of information to fit in a human barin at one time, problems are bound to show through. I don't have any "high eng tools" to help with the architecture either, which doesn't help.
Also, the architecture itself may suck.
What kinds of problem are you having? I think you need to design test routines geared towards not letting the types of problems you currently have through. It is hard to have any specifics, since the post was so vague.
-Pete
Soccer Goal Plans
make QA independant from both development and operations, if you run the code in-house. Developers always SUCK at testing code, and they will rush stuff through Qa in order to meet deadlines.
Most of the developers out there don't understand proper testing methodologies. Sad to say, most of QA doesn't understand either.
Take for instance... load testing. A lot of places I have worked for didn't have the proper environment setup to even come close to doing load testing. And how is "developer testing" and "taking more walks" going to help this???
This attitude is now starting to creep into other areas - other than software companies - and into real-time or 24/7 type systems. Bad mistake.
This is an Ask Slashdot?
Wasn't there a fragging story about this yesterday which stated exactly why code sucks? And didn't that story contain the answer?
The idea is to build up the level of test data you have,
You could have somthing that generats loads of test data , and use a simple script to check that it's data is ok, and even see what happens when bad data is fed in.
If someone one finds a bug, they generate test cases(and variants).
UAT testing should generate loads of data.
An live envoriemts give you loads of data for full-cycle developments.
thank God the internet isn't a human right.
Get a pack of users stupid enough to do the testing for free... And leave them to it... At least then you are testing for bugs with the 5% of people who do the things stupid enough to cause the esoteric ones...
[root@GRIFFIN root]# rpm -e coffee-1.22.3-1a.i386.rpm
error: removing these packages would break dependencies:
You seem to know what you're talking about, so are there any good books that cover the software design process? A book that covers what should be flowcharted and how detailed it needs to be, as well as writing good specifications and what should be contained in them?
In an ideal world, testing should be budgeted an equal amount of time to development. If a project will take 10 man-weeks to develop, 10 man-weeks of testing time should be budgeted. When you take into account the planning, execution, documentation, retesting, user acceptance testing and any load/performance testing applicable, this time is soon eaten up. The main obstacles stopping this from happening are: a) The project planners are often naive to the testing process. They see testing a s a quick one-day job to prove that the product works! (I have seen project plans that allocate 1 day of testing for 2 weeks of development work). b) The customer cannot justify the expense of testing. This is very common. Mr Customer will often ask: "Why do we need so much testing? Are the developers not capable of writing quality code?". c) Testers are often not viewed as a valued resource. I have often worked on projects where testing time has been added to increase the billable amount on a project, instead of ensuring the code works and conforms to the specification. d)As highlighted by the some of the writers here, some developers honestly believe that testers are not necessary if the developer writes good code in the first place...........hello? I have worked with some very disciplined and talented developers. Every one of them makes silly mistakes. There is always some undocumented issue which causes some integration to fail. There is always some misinterpretation of the specification. It happens. You cannot avoid it
What follows is a not an comprehensive list of points, but merely a few pointers for how to ensure code is tested properly:
Ensure enough time is budgeted
Ensure the customer is aware of development process and the importance of testing. If they are not willing to pay for so much testing, bill it under development (It is part of the same process, why bill separately at all?)
If the code differs from the spec, document and communicate the changes. I have experienced many delays and problems during testing due to undocumented changes.
Ensure that the developers communicate with the testers. I have always found that working in the same area as the developers works best. It also helps avoid any 'them and us' situations.
Project managers: Be serious about testing. Your company and your job will be judged on the quality of your product. The low quoted price and the fact you met your deadline will mean nothing if the product is unstable and has not met the customer's specification. If the project will over-run, let it over-run. Don't shave off testing time to meet the deadline. It will come back and bite you on the ass very quickly!
Be honest with your customer. let them see the bug lists. Help them understand the issues. The customer will respect you more and you will be far more likely to reach that inevitable deadline extension in an amicable manner!
How do you determine how complex a software system is? Are a dozen relatively simple routines a complex system, or is the complexity weighed by how they are employed?
The complexity of a piece of software deeply affects how it should be tested. Most software is modular and must thus be tested modularly. It can pay to start by using an automated testing system which requires functions to declare their modus-operandi, specifying constraints and boundary results, such that the testing software can determine a functional hierarchy in which to test program functionality.
But if you're not talking commercial, highgrade custom development, then the designers and developers need to keep testing in mind before, during and after their stages of the process. If you are one of those developers blessed with the ability to go straight from concept to code, testing is just one more system component you have to mentally track: the testing stack, or some such device.
Testing an entire system won't work - it will generate huge quantities of bugs - unless you have a testing methodology that is based on the model used to actually code the system. If your word processor works in bytes and somehow gets into writing two bytes at a time instead of one, all kinds of other problems will ensue.
While most testers do understand the concept of modular testing, its the notion of aligning the modularity of testing with the modularity of the code that most developers *and* most testers fail to recognise. The solution: design, develop and implement with testing in mind.
-- A change is as good as a reboot.
Code reviews are fine as long as you have somebody looking for bugs and quality of code and not some anal retentive jerk who thinks their way is the only way. I agree that code reviews can help eliminate errors by newbie programmers, but that tend to cause general disgruntlement among those who know what they're doing. The reduction in cost for bug fixes can be offset by the increase in cost from your experienced people sitting around angry.
No matter how much testing you do, the best testers are customers. I've worked on projects that passed testing by myself and three other levels, only to have it go squirrely at the customer site, usually because somebody doesn't know what they're doing, or that particular customer is the one company in the industry to do things just a little bit differently. Real customer data is sometimes the only thing that will cause a bug. For those of you in support, think of the number of times you've had to work for a week to replicate a bug reported by a customer.
There's also the definition of "bug". If a programmer gets a requirement document, and his code does what the document said, then erroneous results are due to the "garbage in, garbage out" rule, not due to faulty code. This was all too common at my last job. We did insurance software (some still do), and between all the tech folks and the customers was a thin layer of insurance professionals who knew NOTHING about software and refused to learn. It's hard to get good code without good requirements gathering, and it's hard to get good requirements gathering without people who understand the importance of it.
Well,
:-) you would have NEVER thought of.
:-)
actually the best way (but admitted overkill) would be, to formal proof every algorithm is correct. Then again, this would consume so much time, nobody really does it. Clean specifications, Proper planing etc. help alot (as mentioned by others). Linear Regressions Tests, good test samples (Which should be done by other people than programers, cuz progrmers NEVER think of ALL the strange DAU minds, which might wreck the software. This leads us to betatesters, I personally experienced that letting a complete idiot use your program shows you ways of use (abuse?
Many people think, software developing is a cycle of programing followed by debugging, if debugging is meant to remove errors, programing obviously is for ????
A common problem is: Hack the code, see if it compiles, see if it crashes, fix it. Actually one should avoid this attitude and in the first place think about the algortihms, check em, etc. before one starts the implementation.
People seem to think that good testing means fewer bugs. That is a myth. Even perfect testing that reveals every bug could still result with highly faulty software if the vendors decide THIS BUG ISN'T IMPORTANT ENOUGH TO FIX. The priority of shipping seems to press harder than the priority to fix problems. Ship it now and provide a patch tomorrow. Though when tomorrow comes, oops, we're too busy on the next version.
Proper testing is vital, but only when the bug database is actually emptied as fixes are made should we expect a rise in software quality as a result to even better testing. Assuming all reported bugs are fixed before new deliveries are made, software would still be bad but not nearly as bad as it is now. (There are just too many bad programmers who actually think they did learn to program in 14 days.)
Next, we need to teach customers to stop demanding new products that don't work, from demanding them before they're ready, and from expecting the impossible. The weak economy is fuel for the customer to be calling the shots, and companies may be agreeing to deliver software on a timeline that just isn't feasible, simply to keep business rolling. Naturally, even without great insight, it's easy to forecast the quality of such endeavors. C-R-A-P
> What is the best way to get the biggest bang for your testing buck?
t tp://libre.act-europe.fr/
The first step to reduce testing efforts is to use a good design and a good language.
Since I switched from C/C++ to Ada 95 my development time reduced to about the half. Ada 95 educates to disciplined programming and the excellent GNAT compilers find many semantic errors which helps to reduce the need for testing efforts.
http://www.adapower.com
http://www.gnat.com
h
No, it's not magic, but we have found it makes a HUGE difference in code quality. In combination with regression test scripts, it helps a lot.
The thing about programming in pairs is that it dramatically improves general code design/implementation, and that has a big effect on bugs.
I need "insert feature here" implemented in "insesert program here" done yesterday. I promised "insert very large client here" that we would be able to add these new features and still be on time with delivery.
First, I'm willing to bet that your development organization is too large. The best code is usually produced by a small team of developers. Too many cooks spoil the soup kind of thing.
Second, yes, there is such a thing as too much testing. QA Managers have a real problem to deal with - they can catch 85% of bugs with x amount of effort. However, getting that last 15% becomes exponentially harder. QA Managers often have to make the decision to let the last 10% go, because the effort to find them just is not cost effective (i.e. to test a build to 85% takes three weeks, to 99% takes a year kind of thing).
Without knowing jack about your situation, I'd guess the problem is in your process somewhere - too much of it most likely. Too many developers, too bureaucratic of a QA process, could be many things. just my $.02
Doing unit tests "XP Style" is a good way to solve the How-much-is-too-much problem because you'll write most of you unit tests before actually coding the implementation. Your tests will reflect the specification/requirement of the users, not more, not less.
And because "XP Style" unit tests are generated automatically on a regular (and frequent) basis. It will prevent you from reintroducing "old bugs" as you change the actual code. That will help you to keep your defects in code closer to zero.
A lot of the problem may rely on what methodology you are using to code the program, whether it is the traditional waterfall method, or the sprial method, or perhaps M$'s old sync-and-stabilize method. Whatever methodology you use will drive how you should be testing.
With the waterfall model, you really need to know way ahead of time that what you are coding is what will be desired in the end product. It forces you to have a clear picture in your model of what you are trying to build and with each step in the process, you must develop testing procedures that address that level of the code. For example, at a high level, you may say, let's build a compiler, and following that decision, you need to devise a test that proves that the compiler works. The next phase, you may say, let's build an assembler to produce machine code for the compiler. Then you need to build tests that prove that the assembler works. This methodology continues right down to the smallest module of code, and when all of the pieces have been written, integration testing begins, and you make sure that each larger piece can correctly function based on the output of the smaller piece.
However, in the spiral model, it allows for a well-defined core code to be produced with tons of modules that evolve as the spiral expands. Integration is a function of the spiral, and testing occurs within each iteration of the spiral loop. Code produced with the spiral model also tends to be somewhat more difficult to test in later stages, IMHO, due to the nature of the testing that occurs at each cycle in the loop. Testing becomes more critical in later stages as the previous stages become more nested into the core of the program.
Well, enough Software Engineering for one day. Back to work....
Rule #1 -- Politics always trumps technology.
Use these rules:
1. Write less code. Line not written is line without bugs. Don't use languages that make you write lot low level code.
2. Write programs that write code. Write your onwn program specific language or make language framework that generates the code you need. Less human written code is good.
Investments into good code generators and frameworks pays off.
3. Use the most highest level language possible. Pyhton, Lisp, Haskell, others.
Use many languages if needed. Only 1-10% of code should be human written written in C, Java, C++.
4. Code bottom up. First code utilities, libraries, language-extensions, languages etc. until you can write your applications with these building blocks easily and with less code.
5. Work with the best! Don't hire code monkeys. Hire only from the top (1-5%) programmers. Less is more.
Hire 50% less programmers. Pay them 300% more!
Most employers want medicore programmers writing medicore and verbose code that looks nice with UML. That's because they want to be able to replace programmesrs easily and they can hire lots of them. They feel secure that way.
Dont work with programmers who can't read code! This is really common problem nowdays. People just don't have concentration to read code trough, line by line.
Dyslexics have more fnu.
Make sure your company keeps an archive of already tested code. This way, you don't have to retest everything you release (which takes up a lot of time). In a "get it done yesterday" market like advertising, this can be the difference between success and failure.
Go here for teh [sic] funny.
Maybe when we have quantum computers we can test every single user scenario in parallel
-flamesplash
"Not knowing when the dawn will come, I open every door." - Emily Dickinson
If possible have the testers be different people than the developers. As mentioned in other posts cross developer checking helps also. In small organizations, this may not be feasible (due to limited staff).
If you have separate testing and developers, it may make sense to try the follosing:
"When testing code, what procedures work best for you,..."
Make sure it compiles and runs and then upload it to Debian/unstable.
(Yes, I'm joking).
"...and do you feel that excessive testing hurts the development process at all?"
If didn't hurt why would you label it "excessive"?
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
The difference between testing expected inputs and possible inputs is that reality doesn't limit itself to expected inputs. Heck, Sometimes it doesn't even limit itself to possible inputs.
Larger tests don't test more. What the large tests do is make sure everything works together. You need the small tests to make sure each piece actually works.
The bigger the test, the more likely that your testing platform doesn't resemble production.
There is a big difference between getting your test to run sucessfully and having bug free code.
Chances are the test cases have nothing to do with hiow the users actually use the program. Chances are the programmer has never actually seen how a user uses the program. Chances are, the first time he does he'll go back to his computer and start cursing the user for not doing things the "right" way.
If something breaks and you don't add that something to the test case you're asking for it to break again.
Testing deserves powerhouse machines and sadistic maniacs who like to break code. People who want the tests to be sucessful and don't run them that often are obviously not going to be as nasty as the real world. Even the sadistic manic has a hard time being as nasty as the real world.
Tests are less expensive than that production errors. But only if they find errors. Tests that prove the code works perfectly usually don't.
No programmer likes to be told he made a mistake. On the other hand they love a challege. Make testing fun and brutal and it'll be much more productive than if you make it boring and painless.
No Zen is good zen
Since no programmer is capable of producing perfect code, debugging/testing is a neccessary process. And as many people have already said, there is no such thing as too much testing. In the world of computers, testing/debugging is an always on going process. Code is only released to the public when it is determined that serious problems and/or when all of the known issues are dealt with. (Hence the nickname "undocumented feature" for bugs.)
So possibly the best and most accurate response to how much testing you should do is one that relates to how good you think your programmers and QA processes are as well as how good they think they are. You will always see bugs and problems being reported when you release a piece of software to the public. The reason is that you have a huge amount of people looking at, playing with, and testing the limits of your code. The secret to creating a good QA and code testing process is not smacking your developers every time a bug comes in (although that does relieve some stress) or changing how much testing you do without regards to processes. The secret is to find out exactly how the person found the bug and incorporate that into your testing strategies.
Also keep this little quote in mind that someone once told me, "A program is only as good/smart as the person(s) who wrote it." The reason you should keep this in mind is that programs, computers and anything else you can think of that interacts with your software was created by a human being. Thus, they are all subject to the flaws of human intelligence. Nothing and no one is perfect. It's when the imperfections mount to form a large mountain of problems that restructuring is in order.
Summation: If it isn't broke, don't fix. But tweaking is always a good idea.
The Uber
http://www.tulg.org/
http://devurandom.livejournal.com/
Your development team should not be writing code for a single task. Code for a specific task / requirement is by its very nature, bug filled code. All code should be written to be re-usable code. Once the re-usable, library routines are written and debugged, simply reuse that code as it will not introduce additional bugs into the system. The only part of the application that would not be library routines is the "glue" code that maps the interface to those routines.
This was a real question from a job interview! Q: What area of programming do you consider yourself not to be good in?
"A common mistake that people make when trying to design something
completely foolproof was to underestimate the ingenuity of complete fools."
- Douglas Adams, Mostly Harmless, 1992
Mr. Adams is right, you can't completely avoid errors !
But a good approach to (near) failsafe software is to let non software engineers/pros test your solution, because they allways end up doing something completely foolish and crash the app.
Jake.
In order to form an immaculate member of a flock of sheep one must, above all, be a sheep.
If you're not finding the bugs, then you're not doing a good job testing.
for automation of some parts of the verification process in hardware see:
http://www.verisity.com
software and hardware verification share much in common. Actually the product is used to test itself.
(disclaimer!! I was an employee of verisity , before turning to other areas, and continue to hold her stock )
Working for necessity's mother.
A good testing approach does not cure automagically bad code.
If you guys have means to tie defects not only back to the parts of the system, but also to specific builds then you should be able to link bugs with teams or persons. This info could be used to establish e.g. peer pressure with the goal of fostering an environment where dilligence and low defect rate is valued and their advantages can be seen (e.g. less time wasted on last minute integration bugs).
Be glad that you have a good QA process in place. Now you need to integrate the results of the QA process back into the development process.
What's that?
How does that fit into the Big Ball of Mud programming?
When I code programs that are used by the general public. I find double-blind testing, and black-box testing works best. With software that means life or death or something severe I will also do white-box testing.
double-blind testing is when you give the code to a willing party and just let them work with it like they normally would for business purposes, without letting them know it is a beta testing. You have to also include some type of bug report that people can fill in if they wish, but try to encourage them not to cause bugs, and just work with the program as if it was normal. This allows you to see if any of the normal functions that people use everyday would be buggy.
Black-box testing works great to Just test the programs function calls and modules. When I do BBTesting I usually give it to another party with instructions as to how the functions are called and utilized. This party knows how to test the extremes and the common values and give me the best testing.
White-box testing is testing that involves intricate knowledge of the code. When I do this it is usually in development. At the end, if I feel like I enjoy pain I will do a through white-box testing suite for the program, but that has only happened once or twice.
In expenses, the cheapest form of testing is BB testing, followed by Double Blind, and then WB. Since white box testing takes a long time to design run and analyze the results I find.
There's some thoughts for you though.
~ kjrose
There are two subjects I want to discuss here. First of all, I'm going to present the "jelly bean model" of defect discovery, then I'm going to talk about why the "testing to improve quality" model is fundamentally flawed.
The Jelly Bean model goes like this: Let's suppose you have a big vat of red and blue jelly beans. Your objective is to remove all the blue beans. You do this by reaching in, grabing a hand full of beans, throwing away all the blue ones, and dumping the red ones back in.
At the begining, it will be very easy to find the blue beans (assuming the blue-bean density is high), and towards the end, it will be very difficult (since the blue-bean density will be low). If you graph the cumulative number of blue beans you remove each day, you'll get a exponential curve; quite steep at the begining (high rate of discovery) and which flattens out as you approach total bean removal.
Software defect discovery follows this model exactly. Defects are easy to find at the begining if there are a lot of them, and hard to find towards the end. This means that if your defect discovery rate is pretty much constant (with respect to the number of hours of testing you've done) then you're probably still way down in the very first part of the curve, and your number of defects is probably very high.
Here's the important thing to remember though; the quality of your product has nothing to do with how many defects you find and fix during testing. The quality of your product is determined by the number of defects remaining! If you find and fix 10,000 problems, you might think you're doing very well, but if there are 10,000,000 defects remaining, your product is still crap.
You can estimate the number of defects remaining by trying to fit the number of defects you've found so far onto that exponential graph I mentioned above. The most popular method to use a Weibull curve, or Quadradic Regression.
Now, why is testing to improve quality a bad plan?
Let's say you worked at Ford, and roughly 50% of the cars you turned out had something wrong with them. You get lots of unhappy customers demanding their money back. Is your problem:
a) That you have a design defect in your car.
b) That you are introducing defects in production.
c) That you are testing cars insufficiently.
Most people realize that to test every car as it comes off the line is futile. There's too many of them, with too many potential points of failure. There's no way you can test them all. The root cause of the problem has to be in either a or b, and if you're looking to improve the qulaity of your cars, this is where you would spend your money. This isn't to say that Ford doesn't test their cars, I'm sure they do, but testing should be a means of verifying quality (IE, 1/1000 cars tested had a defect, our goal was 1/500, so therefore we can stop spending money on finding design and production faults), and not a means of improving it.
It's so easy to see this when we're talking about cars. Why does everyone get it backwards when we start talking about software?
Not only is it impossible to test every possible combination of inputs to most software, it's also very expensive to find and fix problems this way. If you find a problem in design review, or code inspection, then you have your finger on it. You know EXACTLY where the defect is, and how to fix it. On the other hand, when you say "Microsoft Word crashes when I try to select a paragraph and make it Bold", you have no idea where the fault is. Any one of several thousand lines of code could be the problem. It can take literally days to track down and fix the defect.
Your testing should not be a means of finding faults, but a means of verifying the quality of your product. Testing is not part of the development process.
Thank you for your interesting and insightful treatise.
Clearly, with your keen grasp of the history of programming, and your understanding of VB and HTML, you will someday make a fine salesman or mid-level manager. You already exhibit the skill and insight used by management at most companies to plan their projects, procedures, timelines, and budgetary requirements.
Many programming issues would clearly benefit from simplification, and perhaps you are on to something. By reducing the number of tools a language attempts to implement, it clearly decreases the number of distractions for the programmer. If you wish to pursue this concept further, you may perhaps wish to research another foundational language called Logo.
Good luck in your next class!
Billbert, PhB, TFIC
Redmond, WA
P.S. - A side note regarding the original article:
A firm understanding of your process for code development, and a clear design for your testing procedure are essential.
If you spend some time in advance on planning, your code will benefit. Also, a good test plan should include a measure of the relative impact of a 'bug' or 'defect' to help determine priority of response by your programmers.
By focusing on your programming objectives from the beginning, and maintaining that focus throughout your entire design lifecycle, you should be able to identify underlying problems in your current development model and use them to improve your entire process, with a goal of helping prevent errors in design, and catching errors in code before going to test.
Peer review at each stage prior to testing (planning, functional design, algorithm design, coding, and test design) will also help catch errors in advance, and lead to the development of much better code. It may sound like it will take longer and cost more, but it saves time and money in the end in terms of not having to rewrite and maintain poorly implemented code.
Cheers.
Though remember, this is Slashdot :) Automated testing is common in embedded systems programming, and all but non-existant for any kind of Open Source desktop applications (gcc is an exception).
You write test cases as you go. You make sure you can run an automated regression test at any time. If you don't do this, then any time you change code you might break old code and you won't realize it. Just doing spot checks at the keyboard isn't good enough. And the programmers need to be writing these test cases first, and they need to be kept separate from tests written by external groups.
My personal recommendation is the "Cleanroom" methodology. You create a functional specification with a mathematical guarantee of completeness and consistancy. Auditable correctness is also a part of the process. Then when it comes to testing you generate test cases that cover all states, all arcs and then do statistical test case generation based on a usage model. The overall cost of this process is a bit more up front, but studies have shown that the process far more than pays for itself in greatly reduced maintenance/debugging costs.
So to answer you question is that to generate a decent set of test cases, you really have to understand the problem space and have mapped out the state-space in some manner. Trying to derive this without a methodical approach and ones testing will be spotty. The worst I've seen so far was a random state-space walker (ala Brownian motion). Statistically this approach avoids all the difficult cases in the far corners of the state-space.
Now for the bad news: Cleanroom is quite tedious for the programmer. The enumeration phase takes seemingly forever and can be mind-numbingly boring.
Here's the amazon link on the layman's book on Cleanroom: Cleanroom Software Engineering: Technology and Process by Stacy J. Prowell, Carmen J. Trammell,Richard C. Linger, Jesse H. Poore
And now for the shameless self promotion bit with a long winded sales pitch for executives on Cleanroom: my own Cleanroom company: eLucidSoft.
Just chant over and over: "Hire eLucid, play golf."
I used to wonder what was so holy about a silent night, now I have a child.
At the time that you are coding, every assumption is going through your head. This is time to write it down, either on paper, in a document, or in comments in the code. The mental state you are in when designing test conditions cannot come close to the state of mind you are in when coding (if you are concentrating :) . You are mentally closer to a problem when you are coding than when you are designing, and you can take the shortcomings of the platform you are working on and pair it with the shortcomings of the design.
Any consideration you have during the writing of a single line of code is gold. And like a great dream, if you don't get it down when you think it, you will lose it in a day or two.
My 2 cents.
http://pcblues.com - Digits and Wood
MSDN used to have a column called Stone's Way or something, and in one of them they discussed user case testing: set up a video camera to record the user as he/she uses the program OR masquerade as a nondeveloper and spy on the user as he/she uses the program.
If you're just looking for regular bug testing, assume it's a given that the user will not report bugs to you and have the program automatically email you a core dump and/or stack trace and/or any appropriate data if an unhandled exception occurs.
[o]_O
I work for a large company with a large number of internally developed applications.
I am shocked at how frequently our developers don't have a good understanding of their architecture, or sometimes even the problem that they are trying to solve. As a result, when they go do do "testing" they are frequently performing tests that are not valid.
For example, they might create a new build and test that build only on their development workstation before full deployment of the application.
Naturally the development box has different resources from that of a standard production machine. Many developers don't seem to understand this.
Another example - frequently boundary conditions, or interfaces to other applications are not fully tested.
Using bad methodology, all of the time that you spend testing is wasted.
Management tends to feel that testing time is wasted because their experience is that the time that they have invested in the past has been fruitless.
Please develop:
valid test cases,
valid test plans, then
execute them,
find gaps, then
use the gaps to learn how not to make the same mistakes in the future!
Phooey.
Anomaly
PS - God loves you and longs for relationship with you. If you would like to know more about this, please contact me at tom_cooper at bigfoot dot com
But Herr Heisenberg, how does the electron know when I'm looking?
I think bounds checking is the main part.
1) Find out what the program is supposed to do.
2) Find out the exteremities for values.
3) Test the extremeties.
4) Test outside the extremities.
For example. If a program asks someone to enter their age, you need to check for:
1) something
2) that is a number,
3) that qualifies as an age.
4) The lower boundary for an age is 0,
5) the upper boundary should be unlimited.
Test for the following.
1) Entering nothing at all.
2) Entering various non-number entries.
3) Entering non-whole numbers.
4) Enter 0, and negative numbers.
5) Enter insanely large numbers.
One failure of coders, is that they assume that entering data is only done through cannonized ways of the program. For example, they expect a user to use the keyboard, but don't take the clipboard into consideration. So, in case 3 above, the program may rely on a control to not accept keypressed decimals. And thus the program is "safe". However, if you can get it there from the clipboard, the program may break.
So, when testing, you need try your hardest to enter bad data in order to test how the program deals with it.
Another example. If the program is actually a web page, it may rely on a combo box to have only specific values. However, I can rewrite the page myself, and add any value I want. And, it gets even easier if the next page allows GETs.
On another point, the test routines should be written *before* coding is done. If afterwards, you may only test what you already thought of fixing. If you force yourself to think of tests before the program was written, you have a good change of testing more.
Have you read my journal today?
Instituting a program of code reviews (best if done for everything including minor bug fixes, at least for major functionality additions) will catch a lot of stuff up front. But nothing gives a bigger bang for the buck that developing/purchasing an automated test facility that can run test suites that you develop. Still, those tests will only be as good as your test suites so use a code coverage tool from time to time to make sure you are doing thorough testing.
If you want bug-free software, you won't get there just by testing. According to "DeMarco. Controlling Software Projects, Management, Measurement, and Evaluation. Yourdon Press, 1982", at least half of all bugs cannot be identified no matter how much testing is done. Another classic reference is "Thayer, Lipow, and Nelson. Software Reliability, A Study of Large Project Reality, North-Holland, 1978", which studies the residual bugs in a large programming project and concludes that even full path and parameter testing would not have identified about 30% of the bugs that were experienced in the field.
Here are some experience reports showing software inspections to be effective: (1) G.W. Russell's "Experience with Inspection in Ultralarge-Scale Developments" (IEEE Software, Jan. 1991), (2) E.F. Weller's "Lessons from Three Years of Inspection Data" (IEEE Software, Sept. 1993), and (3) Grady and Van Slack's "Key Lessons in Achieving Widespread Inspection Use" (IEEE Software, July 1994).
There are many other such articles; I'm highlighting the IEEE Software articles because they're easy to get. Full disclosure: I co-edited a book on software inspections, titled "Software Inspection: An Industry Best Practice" (IEEE Computer Science Press, 1996; edited by David A. Wheeler, Bill Brykczynski, and Reginald N. Meeson, Jr.). That book reprints the most interesting articles on software inspection of the time, many of which are quite hard to get hold of, as well as additional material not found elsewhere that gives you the "big picture." Unfortunately, that book is now out of print and getting increasingly hard to get, but you might be able to get a used copy (or convince IEEE to reprint it).
- David A. Wheeler (see my Secure Programming HOWTO)
Then dont test it. The problem isnt that the testing is reporting too many non-bugs. The fact that your finding a large number of bugs means simply that a large number of bugs exist. Less testing does not reduce the number of bugs in a program, it simply reduces the bug count.
I will admit that bug testing often results in many non-bugs being reported. But that does not mean that testing is a bad thing. If anything, testing should be started as early as possible, and be carried out frequently. The earlier that bugs are found, the easier it is to fix them.
END COMMUNICATION
I think testing should considered at the very beginning of the software development cycle. When we decide "what" the software should do, we need to identify a way to prove to ourselves it happened and it was right. (Requirements.) When we decide "how" we'll implement our requirements, we need to identify a way to prove that design is working. (Design.) When we code, we need to identify the procedure we'll follow to make sure all of the above is proved correct.
Personally, I prefer automated tests because testing always occurs at the end of the test cycle when deadlines are bearing down, and the pressure to put a fork in it is high. People under pressure are wont to do the least possible to get the job done. If testing is automated, it's relatively painless and quick, thus more testing occurs for unit time.
Of course, if you want to automate testing of some GUI oriented program, you have to think about how you'll do that (scripting anyone?) at the early requirements and design phases. I try to put a scripting interface in every app I write so I can run a test batch file.
If my characterization of the s/w devel process sounds too waterfallish, I'm sorry, I've been doing this for 20 years and I'm stuck in a rut.
steve poling
grand rapids, MI
While I am a developer, my wife is a graphics artist who did a lot of ad & copy writing for various catalog and advertising firms.
She is still stupified at the idea of a programmer testing their own code. It was a firing offense at her firm to proofread your own copy and approve it. You always had to have another pair of eyes look over your work, since if you wrote it wrong, you would read it wrong when looking it over.
Programs working fine on test data the programmer wrote, then crashing horribly, are so common as to be legendary.
I have recently applied XP write tests first techniques to COBOL programs, of all things, and have been pleasantly surprised at how well it has worked.
Methodologies like Test First have been gaining popularity, along with their supporting tools like JUnit and NUnit, to help solve the testing dilemna. Normally when you're coding strictly against the compiler, and perhaps some defined design, the only errors you'll see are compile time, and in the case of java or (insert scripting language here), run time errors. Logic and design errors are revealed through testing your software, usually in the form of unit and/or integration tests, figuring out if your software does what it's actually supposed to do.
.NET (C#, VB, etc) code. Because of the complexities involved, I doubt a generic CUnit or C++Unit could ever be engineered for generic distribution. The best support I've found is just to get the developers to think within the guidelines of test first.
t est_first/n et
The Test First methodology actually takes this process one step further, elevating logic errors to the level of obviousness that compiler errors have. This is key, because it helps break the mindset of the assuming programmer that if it compiles, it must be fine.
Unfortunately, the tools I've seen that support test first only work in the context of testing Java or
I personally think this methodology, like any other should be viewed as a possible tool to use to solve some problem, in your specific project environment and schedule and what have you. But thinking along the lines of trying to test your software before you build your software will help you to write more robust code.
-ds
References:
http://www.junit.org/news/article/
http://junit.sf.net
http://nunit.sf.
Look at how complex mechanical products, like cars, are designed and tested. There are are vast number of components and subsystems in a car. Each component and is specified, and tested to its spec. If the component fails, redesign it, and test it again.
Each subsystem is made up of tested components. Test the subsystems, to find any "bugs" that did not show up at component level. These are often specification bugs. Finally, put it all together, and test the whole car. At this stage, almost all the debugging has been done, so you do not get "millions of bugs".
I think the software testing problems described in the question arise from doing all the testing at "whole car" level, instead of incrementally, as the code is written.
By the way, I am an electronics engineer more than a software designer. I applied the above approach to a little C++ project for my own interest, and I found that applying electronic engineering methodology to the design and testing worked well for me.
Glyn
Pardon the shouting, but I mean it. Testing as is mostly done in the coding shops is the detection of what bugs got introduced in the code. Quality code production doesn't mean finding the flaws, but keeping them from occurring in the first place. There is only one way to accomplish this, and that is to make testing a design criterion. There is no substitute for specifying design invariants and coding to assure them. This is where specification languages like Z come in handy, and why the Eiffel wankers are so in love with their language.
All locating failures does is locate failures; it does not ensure success. The only way to ensure success of a system of any complexity is to define what correct operation is and ensure it. There are brainiacs from the planet Smartron-Five that are good enough at their formal logic to actually prove whether a piece of code is correct or not. That's great if you can do it, otherwise, you're stuck with ASSERTs or JUnit or somesuch. But you can't do any of that meaningfully without a rigorous understanding of what it truly means when you say "it works".
--- The reclining dragon deeply fears the blue pool's clarity.
It has to be said -- maybe if you did do XP-style unit testing you'd get better results?
The halting problem is not NP-complete. The halting problem is undecidable which means no solution can exist (and problems in NP have a solution by definition).
:)
Proving two statements are the same is not equivalent to the halting problem in general. You can prove two statements equivalent either exhaustively (only if there's a finite number of cases!) or axiomatically as long as you don't run afoul of the Incompleteness theorem. However, the number of statements found to not be axiomatically deriveable is quite small and I've yet to see one in a computer program
The goal of testing is to find bugs. If you're finding lots of bugs, then the testing is being done at least passably.
If the number of legitimate bugs is so large that you think your testing procedure is broken, the actual problem is that your development procedure is broken.
The only case that your testing procedure is broken is if you want to ship bugs. There are cases where this is desirable, but it's really hard to get management to admit it in clear language.
I'd seriously reconsider your development process first. At a bare minimum, read at least 3 of the following books:
- Rapid Development by Steve McConnell
- Mythical Man-Month by Fred Brooks
- Peopleware by DeMarco & Lister
- Code Complete by Steve McConnell
An eXtreme Programming book wouldn't hurt either, but the XP recommendation is little more than "follow all the recommended practices religiously".The first thing you should learn from those books are that bugs cost an order of magnitude more to fix as they past through each stage in development. Further, bug fixes often introduce more bugs. This can easily leads to a project killing feedback cycle.
After those notions are beat into your head, they'll clue you in to moe really useful truths. Here are a few examples:
- The more people read the code, the more errors they'll catch -- meaning stepping through your own code in a debugger, code reviews, or pair programming.. this isn't telling you to open source your project.
- Design things to be testable -- I've thrown out numerous designs just because they'd be hard to test.. Places where you think you need multithreaded or reentrant code are especially good things to reconsider. This can be extended to design things to past tests as in XP.
- Programming requires concentration -- I learned long ago that any code I write past 8 hours in a day might as well be burnt later.
- Programming is done in a chaotic world -- Virtually nobody has decent requirements up front, programmers get demotivated with distant goals, and people make design errors... all of this leads to emphasis on iterative development of some sort.
If anybody in the Chicago area is in an organization that understands all of this, please let me know: gruschow_resume at yahoo.comMy code comes back from test with three levels of bugs.
Problem 1: A crashing processor doesn't show up on the GUI. Turns out that I told the GUI that a porcessor crashed, in testing I saw something come out on the command line, and didn't notice the mispelling, but the GUI didn't know what to do with it. Took longer to open the file than to fix the bug. These should all be fixed, unfortunatly they are all minor enough that if caught late in the test cycle they are defered.
Problem 2: After causing failure A, failure B wasn't detected, it turns out the code to detect failure B is the same as failure A, and once A occures the code stops watching for B, even though the two are not related. This is a fundamental design problem, and can only be fixed by a re-write. (My excuse: someone else wrote the code and quit, I maintain it, but I have to impliment feature gamma before I can fix this problem...)
Problem 3: Tester pulls the ethernet cable between two nodes, and the complains that we said the node broke instead of the ethernet cable. This can be fixed, but we need some other way of determining that the other node is still operational we just can't communicate with it.
the first one is easy to fix, the second is solvable, but takes a lot of time, and the third can't be solved. When you come across the third, I hope you have better luck that me with people noticing the bold letters in your documentation noting that additional hardware is needed to solve that problem.
I like to get something that compiles & runs ASAP, test what I have, add a little code & test that, add some more code & test that. That way, when something breaks, I know it's probably the last bit of code I added.
Slow down, cowboy! It has been 4 hours since you last posted. You must wait another few hours.
It's been long been understood that your biggest variable is PEOPLE. Process, is in comparison, a much weaker factor in the software development equation. A QA process that includes multiple-levels/stages of testing will only be as good at uncovering bugs as the quality of the test cases written (breadth and depth), the people that exercise them, and the tools that automate the task. But note, testing isn't meant to find bugs. V&V is intended to test adherence to requirements and conformance to specs.
People possess/display varying levels of skill, motivation, knowledge, and self-discipline. If you want better software, then get better people. What do popular authors like McConnell and DeMarco write about? Why in there increasing demand for certification and professionalism?
Create defect free software. Too many people (developers, PMs and Clients) that I have worked for as a consultant think that. It doesn't matter how good your testing procedures are if there are no good procedures up to that point. I agree with those who have said that most defects are a result of poor communication, I've got the numbers to prove it. I also agree that sometimes developers are put on the defensive when testing takes place. Here's how I get around it: Create a logical flow of data from the people who create the requirements to the testers. Support it with documentation. (Use Cases, Object Models, Interaction Diagrams, etc) Build accountability into the process for everyone involved. This way everyone is clear who has what deliverable. Continuously check back with the business owners to make sure that you have a clear understanding of what they want, or at least they have repeatedly said that you do. (CYA!) Track each requirement though the process, from specification through testing and sign off. If your organization is small enough, let the developers sit in on the meetings where the original requirements are developed. I can't stress enough how well good communication reduces the number of defects. The fact of the matter is that defects will happen. They will not be as frequent, however, if you are able to track down the reason easily, as all involved parties will be getting feedback crucial to their development as a team.
1. Use a compiler with all warnings turned on.
2. Always have a default case in a switch statement, with a syslog or other line in it stating there is an error. Never rely on a default case either.
3. Do in-range checking on the passed in variables BEFORE you act on them. Is the passed in variable too large? etc.
4. string code always needs double checking (strcpy, strcat, strlen). Overwritting is a COMMON mistake.
5. Be careful about simple things that should make sense but does not. sizeof a string returns 4 because its a pointer, but strlen of a string returns the length.. but the length of a string is actually one more than strlen tells you. So that means to allocate the correct memory you need strlen of the string +1. Simple, but serious errors happen due to this.
Okay.. now there are common error URL's out there.. search them out and look at them.
I can program myself out of a Hello World Contest!!
How well does this product work? Does it involve a lot of configuration? Do you have to devote someone to maintaining this? We use Rational Rose, but not this product and I would appreciate some feedback.
Excessive testing is never bad for your defect rate. If you have unlimited time, test for as long as you need to. However, improper testing is time wasted. If you spend 99 hours testing, and 98 of them were done according to an ineffective testing policy, then you've really only spent 1 hour testing.
In short, no, your problem isnt that you're testing too much, it's that you're testing wrong. Testing should be finding these defects, if it is not, your testing procedures are not sufficient.
Good solid code isn't just about testing. It's about each engineers belief and effort to code to the highest standard. Without engineers that place quality as the first priority, you're never going to get bug free applications.
From experience, we know that testing is essential for maintainable code, but the practice which REALLY catches defects is peer review, not testing.
Two effective ways, both tried and true:
1. Take your code to your coworkers (at most two at a time) and have them go over it with a checklist of common defects and goals. Then explain the code (while they still have the checklist, so they can make additions), then have them explain the problems they found. Look up "formal reviews" for more info.
2. Pair program at all times. Follow the basic XP guidelines for this: be sure to rotate teams. This is usually harder to arrange than #1 (it takes more management support), and it doesn't provide recorded numbers for analysis, but it squashes a similar number of bugs, and has other benefits (for example, knowledge about every chunk of code is present with at least two people on the team, rather than just one).
-Billy
Test smarter, not less. Many of the above posts have shrewdly indicated good method of preventing bugs (an ounce of prevention, anyone?), and below I have some ways you may want to consider in testing your software. Sadly, the poster didn't give a whole lot of information to go on, so this offering is a bit vague, I'm afraid.
At our shop, we invest a good amount of time in developing test modules for all interfaces in the project, almost as much time as it takes to write the modules themselves. Every night, an automated build machine builds and installs the latest version from source control, and initiates a test script that invokes the test driver suite which loads and tests all public interfaces. Every function is tested with expected input versus expected output, and further by sending unexpected input (like that of a malicious user, or failed communication). All exceptions, assertions, and incorrect output is checked as well, as any memory leaks, by the C-runtime debugging utilities, and Bounds Checker will soon be in use as well.
This approach serves 3 purposes:
- It readily finds crashes that human testers may not find, since it checks every public function with varietal input.
- It aids regression testing for modules that have been changed.
- It checks for leaks, which can cause problems that may be evident only after extended execution times (doesn't apply if your language is garbage collected)
Further, by having an automated build every night (that emails everyone on the project, including the project manager, ehhem), developers are careful not to "break the build", and thus pay attention to details before committing their code.This method is only a preliminary test; however, and it cannot find every bug that may or may not exist. All code is audited to ensure it meets coding standards. You'd be surprised how failing to meet standards often evidences a piece of code that was written in haste, and as such, is a good place to looks for bugs. Also, we employ code testers who run the software through use cases (which are sequences of actions determined to be general ways in which users want to use the software). These testers also put the software through non-use cases, in which the actions do not follow expected sequences.
I hate to point out the obvious, but it is important to test software freshly installed on "clean machines", not development machines.
In the end, if it is at all possible, it is nice to have users who are in some way affiliated with the software company to give a "field test" so-to-speak on alpha and beta versions of the software. In any complex piece of software, there will be a chance for hard to find bugs (relative to the developer) to occur. Tolerant users are the best and last resort to finding any obscure bugs before the general public gets their click-happy hands on the software.
Conclusion: design smart, code smart, follow standards, test smart, test everything, and track all bugs (as if that is news to anyone).
How can you be sure you are 'Properly Testing Your Code'?
Actually you can do this by adding more bugs, yes adding them, The technique is called bebugging and the is basicly:
1) Produce code, it contains an unknown number (N) of bugs.
2) Programmer (or bebugger) seeds the code with a number (B) of known new bugs, the number and type of bugs should be determined from bugs found in previous debugging cycles.
3) Code is submitted to testing and some bugs are found (F).
3) The bugs found are examined and categorised as either real bugs (FN) or bebugs (FB).
4) Number of real bugs (N) can be found as the ratio of found bebugs (FB) to unfound bebugs (F).
5) Don't forget to remove all the bebugs.
We're actually going through the same sort of questions right now where I work. In the process of testing a new piece of software, my small group has been finding defects in a piece of software that has been in our system for more than 4 years. That piece of software also controls our prime interface, the one that is pretty much the reason for our system. My group has decided that the only reason we're finding these bugs is that the the developers didn't do thorough enough testing the first time around. What happens is that someone writes some test cases, then only tests those cases. If the cases work, then the software passes and goes into the system. The problem is that test cases are never thorough enough, and never catch the really obscure bugs (like the ones we're finding).
The trick is to not stick to test cases. Test everything you can possibly test in the time allowed. We've told management a number of times recently that we're trying not to stop until we have it as accurate as we think it can be. We're still under a fairly strict deadline, but they're allowing some slippage in order for us to get it right.
"In case of emergency, break glass. Scream. Bleed to death."
and let your customers do the testing/debugging for you!
One thing that I've found to be useful in general is this: don't confuse the goal with the metric.
It's fine to have a methodology in place to try and minimize the number of completely obvious faults and mistakes. But don't become a slave to the methodology. The user does not care that you have followed every rule in the book if in the end the program does not work.
Mistakes can be costly. Guidelines help prevent some of them. But institute an elaborate framework to idiot-proof your development process and the world will build a better idiot.
One project I was working on made it pretty much impossible for the number of bugs to go down. The codebase was about 15 years old, about half a million lines of code, and maybe 10,000 global variables. Testing would identify a bug. A programmer would find the problem and fix it. Then three other features, wriiten years ago assuming that the bug was standard behavior, would break. The people fixing these bugs were often ignorant of the efforts of the last programmer, so as often as not the would go in and re-enable the old bug. Even if they didn't, their fix would break something else. Spaghetti code such as is common in some of these old code bases is often harder to debug than it would be to rewrite.
Tell the users that they are simply "Undocumented Features"
(Ok, someone had to say it...)
"Whoever would overthrow the liberty of a nation must begin by subduing the freeness of speech."--Benjamin Franklin
I actually do introduce, or leave in, certain bugs. It's not so much to get a definite percentage of bugs found or anything. I do it to get a feel for how extensive and how detailed the testing was performed.
Amazingly, I was once asked to help fix a production application so I decided 'kick the tires' a bit. The very first button on the very first screen was broken...
I'm glad none of *MY* (Lotus Notes) applications have these problems
(If a bug is embedded in a system, and no user ever triggers it, is it really a bug ?)
"Whoever would overthrow the liberty of a nation must begin by subduing the freeness of speech."--Benjamin Franklin
What exactly is non-XP style unit testing? Unit testing is unit testing, regardless of your development methodology.
Our testing philosophy:
1. Test every branch. (This goes hand in hand with making sure you have small enough functions that they only have one or two branches apiece.) Sounds like overkill, but when you're writing a function it's easy, if a little dull, to write test cases for each branch. It's much harder to fix a bug in a given section of code if all you know is that one of the sixteen functions it calls has a bug, and that bug is in one of seven functions that code calls, etc.
2. Maintain your test suite. Every test we've ever written is still in our test suite, and every time a bug comes in, that bug gets its own set of tests and goes into the suite.
3. Run your test suite regularly.
This works well for a lot of things; the big area in which it doesn't work is very complicated 'mathy' algorithms for which you don't really know what the correct answer ought to be other than by just running your program and seeing what pops out and stochastic algorithms. Stochastic algorithms in particular are a total bitch to test. But, for the other stuff, there's really no better solution.
-jacob
Overally, good testing catches perhaps 63% of all defects. Code inspections alone catch about 63%. Combined on a project, they catch about 95+% of all defects. That's the key. (My copy of Code Complete is at the office and I'm still at home, but that has the exact numbers and study).
And remember, a good testing regiment will include all kinds of testing. Unit tests and integration tests are both needed (Usually it's only the latter that happens in QA). And it's quite handy to have started the unit tests before you start coding the units.
Outside of creating an automated regression suite, I don't see much use in test cases. I mean so you test 10, 100, 1000, 100000000000000 things that the user might do... That leaves a lot of room for creativity on the end user. I actually had a manager who tried to say since a bug wasn't covered in a test case, it didn't need to be fixed. WOW.
I think there should be about 20% of time dedicated to running testcases/regression tests 60% of the time to automating the above tests, and 20% of time dedicated to allowing testers to just beat on the product... Get creative and think like a user...
...and say, "Developers should write their test suites BEFORE they write their code."
We have a fairly large open source project with contributors coming in and going out all the time (well, not a lot going out; but any number is a problem there). Our experience shows that if you can't write a test suite you're not ready for anything more than a crude prototype. The problem with test-after-coding regimes is the testing gets short-circuited. You've already got working code. You "know" it works. You're just proving it works. So you test the obvious stuff that proves this.
Since we have instituted this policy, coding efficiency has actually improved. Coders who have tried to devise a complete set of tests have formalized their understanding of the requirements in a sense which the most complete requirements doc will never do. We include the test suite in CVS. Nobody commits until their update passes the entire test suite. This results in an enormous (but complete) test of everything done so far. But you can't imagine the thrill of seeing your patch pass that many tests the first time.
All of which is completely separate from what a QA process is for.
Eternal vigilance only works if you look in every direction.
I have to agree about the code reviews. There have been plenty of studies showing that frequent code reviews just work better - think of it as a way to suck up a large number of the advantages of pair programming without actually doing it.
Speaking of pair programming, it's also been shown that you'l save a lot of time and effort if you use pair programming to do the complicated or difficult chunks of code. Yeah, there's the cutting-your-productivity-in-half argument, but that really only holds true if you don't know how to use pair programming. If done correctly, you'll save more than enough time to justify the cost later on when you don't have to put nearly as much effort into debugging that code.
As for testing, it's overrated, almost worthless. I realize you gotta do it, but it does so much to distract from the best way to keep bugs out of your program (which is to not put them there in the first place), that I wonder if testing doesn't in some wierd indirect way actually create more bugs than it discovers.
Heck, with code reviews, your programmers will probably start writing better code just so they don't hvae to stuffer the embarassment of having someone notice a particularly stupid algorithm design flaw in the middle of a code review.
> and do you feel that excessive testing hurts
> the development process at all?
No. What the original poster described doesn't sound like excessive testing. At one of my former jobs we did unit testing and automated black box (compared output with diff, crude but it worked) and white box (tested C API w/C test harness) testing. It was a lot, and kept three or four folks really busy, but it kept defects reasonably low for the amount of hacking that was going on.
I don't think excessive testing means you miss defects. I think high numbers of defects reported by QA means you're doing something wrong on the development side of things, and high numbers of defects reported by users means you're doing something wrong in QA - maybe not testing like a user would.
If its software intended for interactive use,
stick a person that never seen the program before in front of it. Make him do whatever he wants, by
the end of the day, you'll have dozens of bugs to fix, and even more usability problems to sort out.
An experienced developer, whether thru a stated methodology or a more holistic approach, will have the objects, data structures and procedural program flow set up so that bugs are less likely. This can save a lot of time. There are working practices as simple as taking a little extra time to enforce naming standards and coding styles that significantly decrease the bug count at every stage. These vary by area but usually involve structural choices like one table versus many, not overloading objects too much, etc. It does make a difference and it's hard to measure.
Also it really pays to front-load a project (spend a lot of time designing) and very important, prototyping. Remember, and this was researched pretty thoroughly by IBM at one point: $1 to fix a bug in the design phase, $10 in development, $100 in testing, and $1000 in implementation.
So design very carefully and you won't have to spend as much time later... but it's not usually done that way (sigh). And be willing to redo the design process as carefully when modifying as in the initial design phase. If it takes longer, let it; you'll make it up later.
The simple fact of the matter is that no amount of testing does anything to reduce the defect rate of software. If your software has lots of defects when it reaches QA, then it is due to a failing in design and implementation. To fix
such problems usually requires going back to basics: developing robust specifications, robust
designs and robust implementations. Testing
in the ideal merely verifies functionality. In
the best of all real worlds, it uncovers bugs in implementation. In the typical/worst case
scenario, it uncovers bugs in design.
There is much pleasure to be gained in useless knowledge.
Testing cannot show the absence of defects, it can only show that software defects are present.
We cannot test quality into a product, we have to design it in.
I beg to differ. This is how most developers test their code as well, though manually.
If you're just testing to make sure your code does what it is supposed to do you are likely in BIG, BIG trouble. Users (and black hats) do just the opposite.
Focus just as much on making sure your code doesn't do things it WASN'T designed to do. Or risk a CERT or Security Focus advisory...
Doing this won't prevent defects in the code, but it will make it more robust out at a customer site. It also doesn't seem to add significantly to project completion time as the heavy duty hard-to-fix bugs are reduced.
If I had a choice between getting developers to code this way or increasing the testing of the code, I'd prefer the developer solution.
Welcome to the net of 1000 lies. Upgrades are scheduled soon that should bring us to the 10,000 lies mark.
You are so right........sooooooo right.
So far, I've been involved on 3 major projects and I have four years of experience as a programmer.
At first all I wanted was to code, code and code again.
The hell with the analysis, once in the code, all is fine.
Man was I wrong, I think I'll put that in my life's lessons bag.
While on my first 2 projects, analysis was being done properly. Meeting clients, white-board designing, use cases, sequence diagram, class diagrams then again.....Meeting clients...etc.
always putting the work back on the bench before even writing a single line of code.
When we started coding, it didn't went all so smoothly, we had our share of technical problems but we knew exactly what we were aiming and why we were. If the programming seemed too hard then we'd go back to design thinking that if its so hard its because we didn't think it right.
Ironically, now that I have a good understanding of the need to design, design, design ever and ever again I find myself on a pretty big project where the boss does not seem to understand the needs for design.
And man....how many time did I say : "Designing an architecture is not simply designing a database structure".
Often, too often, we have to come back on decisions we took because we forgot this or that business rule.
...and every time I keep "whining" for more design and he keeps saying that this is not how things works nowadays.
So now....we have a working product. All patched up to make up for the things we forgot. I did the best I could with what we had while respecting my boss's decisions.
...but, I don't even want to my name associated with it.
Honestly? I'd be ashamed if my boss would be in front of a crowd and said : "Look, its thanks to him if we have a system now"
Oh well...sorry for the rant, didn't mean to but hey....I simply wanted to tell just how right you are.
If you look like your passport photo, you're too ill to travel. - Will Kommen
Perhaps you just load up your software on a populor website and wait for bug reports? You know if your audience is addicted enough to the site this just might work.
You can't grep a dead tree.
Only if the reviewers are:
a: Lazy. They didn't really read the code before hand and just ripped through it in the review meeting itself. Been guilty of this a few times myself.
b: Morons.
Any code that disagrees with its comment is a problem. Whether you need to change the code or the comment is something that needs to be decided on a case-by-case basis. THIS case is pretty obviously a code flaw.
I think your hypothetical example is doing a disservice to your coworkers. VERY FEW people would let that slide if they actually looked at it.
Fooz Meister
One of the big problems that comes up with developing is that the people writing the code have two big strikes against them for testing their own code:
1) They know how it is supposed to be used
2) They want it to work
The side effect of this is that frequently the code will give every impression of working perfectly until somebody who isn't familiar with the code tries to play with it. Then suddenly they are doing unexpected things, entering blank spaces, wierd characters, and other things that can be expected to happen in the real world.
So, in any system you are developing it is very useful to have the developers try to break eachother's code. People don't want to break their own code, so generally they don't.
This sig has been temporarily disconnected or is no longer in service
Where I work we do code reviews. Small changes are reviewed by at least 1 person, and large changes are reviewed by many people. Because the reviewer is also responsable for any bugs this code creates we spend a great deal of time reviewing the code in depth.
We catch really picky stuff like variable names that aren't descriptive, doing the same thing in both the if and the else case instead of once outside the condition, taking locks a few lines earlier than necessary, indentation, etc. I've had code rejected for mispelling in a comment.
Because we pay such careful attention to details really serious things that would actually be bugs rarely make it past us. It still happens, but usually the really subtle bugs are all that remain. The kind that out of your thousands of customers one will have occur every month or so of constant run.
We do all the testing too. Heck, we have a huge testing organization. They even find bugs. But they miss a higher percentage than code review.
Anyone who cannot cope with mathematics is not fully human.
So, it isn't done well, if at all. QA isn't taken seriously. Regression tests, if they exist, are poorly maintained. Then people wonder why something wasn't caught before it went out.
Put the QA department in charge of nightly builds, testing, docs and shipping. Allocate sufficient time for a thorough testing. Give them the power to hold up a release. Hire good people to do this, people who are good developers themselves. Pay them well and listen to them. Don't second guess and overrule them.
Wansu, th' chinese sailor
Type: Grammar/Syntax
Severity: 2
Area: Subject Line
Detailed Description: "Let the User's Do It" contains the following error: "User's" should be "Users". This bug seems to be a case of apostrophe misuse, one of the more common comment bugs.
Suggested solution: Comment writer re-education along with a moratorium on all unreviewed use of said apostrophes.
[Score: -1, Not even funny]
"It's tough to be bilingual when you get hit in the head."
As many developers here have said, code reviews are the most effecive testing techniqe.
Another approach that I swear by is writing debugging code into the program and designing code that is testable. There is a good book on these techniques called "Writing Solid Code". At the very least your developers should be putting asserts into their code.
Many shops take this to the next level. Some examples:
When writing the Excel recalculation engine Microsoft wrote two versions of the engine. The first version was lightning fast written in hand coded assembler. The second version was written as simply as possible in C++. When a recalculation occured, the two versions of the engine would both do the calculation independently and compare their answers at every step. If they ever disagreed then error messages would appear. For the production release the slow engine was disabled. Techniques such as this are a bit more work up front, but they pay off in spades down the road.
Another example: It is easy to look at a datamodel and see what the valid values and constraints are for the various fields. It is also easy to write an "integrity checking engine" that runs on a background thread and traverses the data model every few seconds looking for inconsistancies. If anything is inconsistant then errors messages appear. Such a checking engine will find most bugs shortly after they are introduced.
Look at your test scripts and data constraints to see which checks can be automated. Automating as much as possible will save an enormous amount of manpower down the road.
I had this grandiose idea that we would run all our unit tests as one big regression test every night, and it would alert us when our server broke. It didn't turn out that way. Instead, when the server was changed, we ended up having to rewrite the unit tests, and in many cases that turned out to be a royal pain. So we stopped maintaining the unit tests. When one, and later two, unit tests were constantly failing, then nobody cared any more about broken regression tests.
There are benefits with unit tests, though:
So, my experience is that stringing unit tests together into a regression test suite is not worth the effort. Sorry JUnit, I also loved the idea when I first read about it, but I don't think it works.
Mats
I've worked for years in large and small companies. And I have experienced various levels of SEI/ISO process conformance.The fact of the matter is that the whole conformance issue is a fraud; done primarily, hence cosmetically, to win contracts. As such, tailoring the process to make it _effective_ is never properly carried out.
I am disappointed to see so many people on
As for code reviews, only incompetent parrots see significant ROI on the time spent on code-level reviews. To illustrate, the fact of the matter is that most/all projects simply do not have the overlap of expertese it takes to assess someone else's work -- let alone the issues of needed time, and constructiveness of the feedback. As such, most review results boil down to cosmetics; at best, trivial returns on the time spent.
And don't even get me started on Metrics.
Many, many years ago I worked with a developer who had a cartoon taped to his file cabinet. In it, a mid-level-suit-looking guy was standing in front of a bunch of developer-types. The caption said, "Start writing code. I'll go find out what they want." This summarizes the vast majority of projects I've worked on. And it despite all the talk about why software sucks, it seems to be getting worse lately.
Just thought I'd drop a few links that those of us using Borland's fantastic Delphi should definitely check out:
That should be enough to get your code into shape, get cracking :)
Yes it's possible to test too much. For example, if it takes 15 minutes to fix a bug but a week to run the required regression tests for touching that area of code, that's way too much. There's a tradeoff between correctness and turnaround time.
A really bad way to write a test is to take an existing test, copy it, turn on your feature in the copy, then add the copy to the existing test. Or equivalently, call the same test twice with different parameters. It doubles the running time of the test and doesn't give you significantly better coverage. 100 iterations of that gives you a test that wouldn't finish in a trillion years.
A really good way to test orthogonal features is to write a test that turns on half the features all at once. Choose the half at random when writing the test, but avoid any combination that raises an error. Then write another the same way, and another. 50 such tests will cover 99.9% of all triples of features. Much faster than 2^^#features tests, or (#features choose 3) tests. If you add a feature, add it to half the existing tests. That means the test changes over time. That's OK because they were blackbox tests to begin with.
Error cases and bug testcases are different. They're whitebox, you know what you're testing for. Those testcase can't change over time. Don't copy them (exponential growth!). Use the simplest testcases possible.
Finding an abnormally large number of serious defects, however, does speak to some failure in your software development proceses. What is precisely wrong in your case can only be determined by you, or your colleagues. You need to do some analysis to determine what the source of the defect was.
That is, did the defect occur because requirements were poorly defined? Did the defect occur because of poor technical design? Or are most of the defects just simple "typos" in the code?
After you have done this analysis, it should be clear where changes are required. There is no magic process that a Slashdot reader can identify for you that will instantly give you results. You'll have to do the investigation yourself.
I've seen a lot of people who code empirically. They rely on a fast compile run cycle and just make changes until the code does what it is supposed to do. Perl coding is especially susceptible to this problem. What's wrong with this style of programming is that you often don't really understand what the code is doing. This is a good way to introduce bugs.
Make sure you understand the problem space before you write the code that navigates through it. And don't code blindly. If you don't understand how a given function or class works, write a test program that explores how it works.
Four fifths of all our troubles in this life would disappear if we would just sit down and keep still. -C. Coolidge
I have been developing software for 29 years now.
I once (only once, never again) took a job as one of more than 100 of testers of a huge product.
We were all very busy sending memos with charts indicating testing percentage coverage and bickering about responsibilities for testing sub-components of the product. With a huge headcount dedicated to testing, it appeared that less and less actual testing was being done as the various test groups (System Test1, System Test2, Integration, Certification, Packaging etc,.) were all consumed in administration and self-justification.
We managed to ship a version of the product which DIDN'T WORK AT ALL. No-one felt individually responsible.
I carefully write and thoroughly test my own code. I don't release code until I am satisfied, despite the pressures of the business to deliver. I take great pride in my work. I regard each bug found by others as a personal failure and examine how it could have escaped my attention. I submit my designs and code to my peers for review because I want to get it right.
My attention to detail and pride in my work has paid off financially for me.
If you know you are doing a good job, you will be happy. I just want to be happy. Every year, I am testing my code more thoroughly than ever.
Now all you developers get back there and stop skimping on the testing (self included). You'll feel much happier.
by Steve Maguire is probably the best book I've read (ok... I only read about half of it yet) on bullet proofing your development process. The book is a bit focused on C, but most of it's techniques can be adapted in C++ or any other programming language.
Anyone who comments on how bad this book is (because it's written by Microsoft) that hasn't at least flipped through it: YOU ARE A TROLL!
Hope that this helps.
Loren Osborn
Sound like you might be on one of my former projects :-)
If you have multiple levels of testing, (e.g. unit testing, integration testing, acceptance testing, performance testing, etc.), then you are in a large, complex environemt. This environment probably has had several hundred developers working on a few thousand different programming modules at different release levels. Unlike a program like a browser that may have 50 or so basic functions that it can perform, this environment has probably several thousand function points that span an enterprise.
Each level of testing is designed to catch different types of errors. A program that passes a proper unit test should have bug free code, i.e. no core dumps or segmentation violations and proper exception handling.
A set of programs are tested together in integration testing to verify that the programs are calling each other properly and are performing basic business functions, e.g. adding a new user, maintaining account info, routing calls, etc.
Acceptance testing is designed to ensure that the user of the system can perform his mission critical business tasks with the new system. Acceptance testing will also allow the end users to verify that what they are getting is what they asked for. Every business case scenario should be performed during acceptance testing by the ultimate users of the system.
Performance testing is used to identify bottlenecks when running the system against a fully loaded database/system. When a programmer is building the initial program, the test bed used is only a fraction of the size of the ultimate database to handle tens of millions of customer accounts.
There are several problems that manifest in these environments. First and foremost is sheer incompetence, from management to designers to programmers to database and system administrators.
These large, complex environments allow for incompetence to go unfettered under the cloud of technology. Errors that are made often go unchecked and when they are discovered, there's no accountability to the individual that made the error.
Programmers rush to meet deadlines and don't properly test their code. A designer may not include certain business functions that are required to complete a business scenario. Management may make decisions based on trade magazines or a sales pitch and those decisions could introduce instability into the product (i.e. use MS in a clustered 24x7 SQL Server environment). You get my drift.
Second is empowerment. A programmer may know of buggy code and even identify the bug, but code is locked down and has a defined migration path that has to be adhered to. Applying a quick fix and throwing the code to a location available to all programmers is not facilitated. Most of these enviornments have not made the paradigm shift from a traditional version control tool like SCCS or PVCS to the open source approach of CVS. With CVS a programmer can't lock the code in a project.
Anyway, these levels of testing are essential in delivering enterprise wide solutions. Unfortuanately, the levels of testing in themselves do not ensure success. Without proper quality assurance (e.g. peer reviews of test plans, detailed designs, technical architecture approach documents), good communication and life-cycle accountability (sure you completed it on time and within budget, but it was crap - you're fired!), buggy code will prevail.
Cheers
jpg
I was once on a project where very few bugs were found in testing. Eventially we shipped and discovered that the testing group wasn't doing a good job of testing.
The approach that works best for me is writing small pieces of code at a time, and then testing those. Additionally, I always have others (both hackers and laymen) test the program from a user's perspective -- sometimes they find obvious bugs that I overlooked. One caveat here: I usually write very small programs, my experience with large projects is very minimal.
One very good piece of advice that cannot be stressed enough is STAY ALERT. Don't just keep going, but take breaks often, and get some exercise. Physical exercise is the only thing that really clears all the programming stuff from my mind, so I can make a fresh start again. This drastically reduces the number of bugs made. Another thing that helps is design everything in a modular fashion -- that way you won't break any pieces you have already written.
Please correct me if I got my facts wrong.
In other words, you've got nothing but faith that some kind of test harness will shake out the bugs. Faith sucks. Far better to do code reviews or Cleanroom style development. Takes only a bit longer, and with far fewer bugs.
As Ron Grimes rhetorically asked in 1994, "If you don't believe it's correct before you start testing, what could possibly convince you?"
I'm certain someone has already said this, but over 80% of defects come from crappy requirements. Forget about your design & analysis, your coding practices, inpsection techniques, debugging and testing abilities - if your requirements are not CLEAR, CORRECT, ATOMIC, UNAMBIGUOUS, and CONSISTENT, you might as well start burning money.
NASA correlated a $1 cost to correct a "defect" in the requirements stage (here a defect can be any requirement that does not meet all 5 attributes I listed above) to several hundred to thousands of times over when addressing the same defect at the testing stage. Crappy requirements and crappy specifications are a big part of what makes your code buggy and expensive.
LA Times posted a study last year that showed that the average US programmer only coded for 51 days a year. 51 days!! One fifth of your working year spent writing new code. The rest of the time? DOING REWORK.
Biggest cause of rework?
UNCLEAR AND AMBIGUOUS REQUIREMENTS.
Spend the time and effort to beef up your requirements gathering and management processes. You'll get your ROI in ONE project cycle.
"Content's a bitch."
First, there are two types of tests: Unit Tests and Functional Tests. The first type, unit tests can be measured and rather easily implemented as part of your coding process. This is, to test every function of your libraries, and if you're up to it, you should even write these tests before implementing the functionality to be sure it measures up to your expectation of what it should be doing. These tests must be run before putting anything on the source code database by the programmer and should also be automatically run periodically to asses that the code is still perfect. The idea is NOT to test everything, but to test the limit and probable problems that you think you could encounter. As you find programming but, you should ADD a test to make sure that the bug never crawls back in your code. That being said, it does not make your code full proof and neither TESTED! It only makes you, the coder, more confident that your code "works". That is a great achievement though... how many times has your boss ever asked: "So, does it work?" and you say...: "Huh... well, it worked when I tried it..." now you can tell him/her... I've tested it within reason and it works perfectly fine. That should never take the place of Functional Testing. Some stuff cannot be tested programmatically, for example, you cannot really test much of anything having to do with communication ports and communication errors. These should be done by hand by somebody who is from the QA department (if you're lucky enough to have one ;-) If there are major bugs at this stage of testing, you probably had a bogus design to begin with as most of these bugs should be faily minor.
just my thoughts...
Many others have pointed out that studies consistently show that formal reviews (especially of specifications and designs) are the most cost effective ways of removing defects. Others have provided references to the classic books on the subject. Anyone considering doing formal reviews should read them. I personnally like Tom Gilb's books.
There is a downside to consider, however, which is little mentioned, even in the formal review literature. Formal reviews require a particular type of company culture, and not all companies have or want that kind of culture. Trying to introduce formal reviews in a company that has an incompatible culture will be some mixture of painful, counter productive and political suicide.
The idea that a company would, in any way, be opposed to using the most cost effective way of removing defects seems bizarre. The truth is, not all companies care about product quality. Sure, everyone will say they care, but words are cheap. To find out what a company really cares about, see what decisions they make under pressure. See what they sacrifice, and what they keep.
The difficulty is, that before you can introduce formal reviews into an organisation, that organisation must already be highly committed to quality. Quite simply, many organisations have to introduce other, fundamental, improvements before they can use the advanced technique of formal reviews. The Capability Maturity Model (CMM) produced by the Software Engineering Institute (SEI) is a useful way of prioritising these improvements. I recommend it; I've used it in a project, and ISO 9001/9000-3 in another, and I conclude that CMM is the better of the two. They have a website.
Ne mæg werig mod wyrde wiðstondan, ne se hreo hyge helpe gefremman.
Have a plan for whatever you are going to code. (In other words, design your program.)
Test and document your code as you are going along. Double-check everything. This will make your coding somewhat slower but it will be more deliberate.
Don't just trust any external data. Check it. Always think about any exploit you can when your code deals with external data. Think about user input, data files, external programs, remote hosts (which could be bogus, unreachable, or hax0red), etc. Consider external data untrustworthy.
I recommend giving nice verbose error messages when something goes wrong. (Example, "loadfile.c, line 299: couldn't load file bogus/path/to/blah.cfg specified in the $BLAH environment variable. file stats: etc etc")
Stack traces are also good.
Always keep resource limits in mind.
Testing quality into a product is extremely unrewarding
1) review early and often
requirements, architecture, code, etc. - it's cheaper to catch defects before they escape the phase in which they are generated
gather metrics on how long it takes to fix a bug, and the impact on testing and release schedules to prove this to yourself and others
1.5) review bugfixes - the "cure" may be worse than the disease
2) assuming your regression and new-feature testing is reasonably complete, the rate at which you're finding defects is probably closely linked to the number of defects left in the product
you probably need historical data from comparable projects to be able to make good predictions, so start gathering data
as for your current project, try to plot your error find rate to get an idea of "quality"
...richie - It is a good day to code.
I've tried literate programming, XP, and several flavors of traditional strategies, all of which purport to be the "answer" to your needs. It's ALL a load of crap. If there's any one answer, it's this: there is no one testing methodology that outperforms any other in EVERY testing situation. In my own personal development, I have come to rely on several beliefs/prtactices that have sustained the test of time over my 20 years of programming (and have made life much easier when coupled with a reliable testing methodology):
I second this opinion.
From my experience, the benefits of a test suite increase dramatically the closer you get to full coverage. Some tests are always better than none at all: 25% coverage is better than completely untested. But 50% coverage is much better than 25%, and the closer you get to 100% the greater the benefits you will encounter.
There are obvious benefits of high coverage, such as the ability to refactor with confidence. But high coverage has unexpected benefits. For example, reorganizing hard to test code often has the benefit of reducing unnecessary dependencies.
Unfortunately, getting high coverage requires discipline. Test first coding is a good guideline to achieve higher coverage.
Also, INHO, the testing program should start from the developers pov, not from the clients. The business problems of programming originate in the programming process, not idiot clients. I believe you get a better value by improving the programming process, then adding other QA processes as needed.
i.e., test first coding will pick up a large percentage of your qa issues. Issues that are not picked up by test first coding should be addressed by other processes. However, in many common small scale efforts test-first coding will take care of enough issues that another formal process is not needed.
So if you're working on a database program, you make an automated tester that puts in random types of information in random orders, if you're working on a console game you write a program that generates random controller inputs (i know a guy who did that last one.) This random generator will come up with combinations of input that you as a programmer would never have thought to test for.
You let the random input generator run overnight, and come back in the morning. If the program crashed, you've got a serious problem you need to deal with. If it didn't crash (and even if it did) you go back through the logs and see if it got into any weird states or gave any bad output, in which case you check what inputs generated that, and protect against that type of case.
It probably won't catch _all_ the errors, but it will catch a great many of them, leaving the testers to look for the more subtle errrors.
This Space Intentionally Left Blank
It is by far the best generalized programming book I have ever seen, and after almost 10 years writing code, I have gone through two copies of my own and bought it with money out of my own pocket for at least 6 people I have worked with. The best way I have heard it described is as a giant compendium of common sense, none of which anyone pays attention to until it's all laid out in front of them.
If there is anything I wish people on Slashdot would get religious about it would be writing excellent code and understanding what exactly makes their code excellent. If they did, surely Code Complete would be their bible. I know it's mine. Keep preaching, maybe you'll get a convert or ten!
http://www.technologyreview.com/articles/mann0702. asp
MIT's site gives a quick discussion about the current problems of buggy software. It is a fairly good read.
For those of you are engineers you do wear that ring for a reason.
Plusses
Problems? of course...
Such tools certainly do exist, including the more advanced variants you suggest. For example, Cantata++ provides not only the basic statement coverage you suggest, but also decision coverage (%age of true/false branches executed for all decisions, including ?:), call-pair coverage and boolean effectiveness coverage. (Disclaimer: I work for this company so you'll want to do your own research.)
These sorts of tools are infact fairly widely used. When doing unit testing the tools are normally used as part of an automated unit test script which checks the coverage and fails the test if insufficient coverage is achieved. They are probably the best way of measuring the effectiveness of unit tests.
Some coverage tools (<shameless-plug>including Cantata++</shameless-plug>) can also be set to simply log coverage as the program is run normally (e.g. through a user acceptance test) and the thoroughness of this test can then be evaluated (or checked, if the principle measure of thoroughness is requirements driven) by examining the coverage data gathered. Obviously this impacts on the performance and memory footprint of your acceptance test - so it doesn't work in all cases. When developing for targets with limited memory, tests are often run with coverage instrumentation in a development host environment and then re-run without coverage on the target to prove correct execution.
Is that my boss, how reads Slashdot, doesn't have a clue as to how to implement any of the great suggestions put forth by y'all. And isn't interested in being told how to do it.
the number of three tier systems we work with these days. We use MQ Series and C++, some Java, etc within a pretty big point-of-sale application where I work. Testing the integrity of each piece and then the integration of all of the pieces can be a huge headache.
When you add to it the fact that we get periodic updates from IBM for MQ Series with bug fixes, compiler changes, class library patches it is sometimes hard to know if the bug exists in our code or some of the middleware. We have had to bounce some of our servers periodically for wierd message response timeout situations and now it looks like it may be an MQ Series bug...
Whoever writes the song sometimes pays the singer in unexpected ways....
As mentioned by a heap of posts above, code reviewing can make a huge difference to software quality, in an addition to unit tests. Check out http://codestriker.sf.net as an example code reviewing tool.
i do bugger all (ha ha!!) testing and produce hardly any bugs
One way testing can hurt is if developers assume that it's the tester's jobs to catch the bugs.
Developers who don't have testers to lean on have (potentially) more of an incentive to figure out how to introduce fewer bugs in the first place.
ha. what a silly bunt! of course, i meant to say "we don't need no stinking testing!" pity my english isn't as free of bugs as my code!
Nothing can replace .so doesn't cut it. You need the full package for testing from start to finish.
1) Well written requirements
2) careful design
3) peer reviewed design
4) peer reviewed code
5) manually built regression test scripts (insert favorite testing tool here)
6) Building often on every platform (nightly!)
7) Perform user scenario testing with memory tools (Purify is my favorite)
and
8) Running the regression tests for every build.
8a) The build isn't complete until the installation packages are created for every platform just as if you were going to ship it to a customer. Copying a newly linked DLL or
"We don't have time for all that" is a poor excuse. If even 1 bug is experienced in the first week by a customer, you've lost their trust. Good reputations are easy to lose, but very difficult to get back.
I've just unlocked the secrets to successful software development. That will be $20k please.
1) The bugs created are made during the programming and compiling phase, and prevention is better than cure. I develop some small progs in bsd and do a cc -Wall -ansi -pedantic -O to make sure things are good, and this is after a thorough lint stops giving errors and warnings.
2) Modular progs should be preferrably compiled etc on similar or the same machine.
3) And probably the most important, the programming style. Be very clear about what you're programming and decrease the places where the prog can break to a zero. Use libraries that have proven to be resilient etc and the final application no matter how complex should give little or no errors.
PS dont forget a thorough script based testing system to test the app and leave it for a few nights to test all possible inputs/loads etc.
Hope this helps.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
If you have to have X people run tests at cost Y, then adding more tests makes it more expensive! What kind of screwed-up logic is that, when a computer can do better at 4am while you sleep?
Any tester worth his salt is not a button-pusher/bug-report-writer. A real QA person writes automated tests and checks them into the code base so that it runs automatically when building. Flame to death anyone who checks in code that breaks a test. The optimum situation is that all developers are testers: they write tests and code and check them both in simultaneously.
If you're finding that tests are showing up lots of bugs, you're finding the symptom. It might be you're finding bugs and fixing them will reduce the amount. However, if you find that your development team creates bugs faster than it can fix them, then it means your organization doesn't know to code it's way out of a paper bag and shouldn't be programming. No amount of tests will fix it. The only thing that will work is refactoring, and most managers in such places erroneously think refactoring is the devil.
Not all software is equally testable. You have to write to so that is testable. If you want to read a great book on how to test your software, I recommend John Lakos' Large Scale C++ Software Design. Testing techniques apply to most other languages, not just C++. Personally, I've unscientifically found that every hour writing automated tests pays back at least 10 in saved future effort. YMMV will vary on the complexity of your project.
I can explanate how to administrate your network. You must configurate and segmentate it, so it can computate.
The idea behind testing in the corprate environment is kind of messed up. The people who test applications are PAID to test applications. This is their job. Hence you have whole departments, in some cases, who's job is to test software. Don't let them do it!
The problem lies with the general public, and the people who market the software.
Reason the general public is a problem - they have a tendancy to do stuff the creator of the application, or module expected. To fully test software, it must either be done 'in the wild' in userland, or in a controlled userland where the participants are as educated as they typically are when they start using an application - this is little to no education. Basically sit them in a room with the software and watch what they do, and see if it breaks.
If you wrote the software and THINK its broke, check it! If you think it works, then let the users tell you if it does or not. This does two things, it allows you to test the code in a real environment, and also allows you to see if the user interface is as 'intuative' as you say it is.
This is the second problem - MARKETING. Why do you put out BETA releases of software? To test it out in the wild to find out if something is broken, or will break other things. But the probelm with this is MARKETING. You put a company with a cool new program (which is really a BETA release) and they marekt it as open to the genral public, put it on websites all over the place, etc. and people expect it to work. Even if it is BETA, or they expect it not to cause problems on their system. When they do they complain to the people who made it, to the manufacture of the PC, etc.
Make good software, test it, and when you release a BETA version, change the public understanding of what BETA really means.
I'm sorry, but I didn't have time to read all the other responses. The replies I did read were mostly questions back at you and clarifications to other replies. So, here is my attempt to answer your questions.
:)
do you feel that excessive testing hurts the development process at all?
Yes, of course it does. You could, theoretically, test code for one program for the rest of your life and still not discover all the bugs. That would be excessive testing and would definitely be a bummer to the development process. I think what you really mean is how do you determine how much testing is enough. For this I refer you to a few good testing books because frankly speaking, people have made careers out of this sort of thing
Books: The Art of Software Testing (hard to find and a little expensive) by Glenford J. Myers; The Complete Guide to Software Testing by William Hetzel; Code Complete : A Practical Handbook of Software Construction Steve C McConnell. These are some good options to get you started.
What is the best way to get the biggest bang for your testing buck?
I would take a serious look at Orthogonal Array Based Robust Testing. A method of testing developed by Taguchi and Konishi, and using orthogonal arrays to determine test cases. I don't have enough room here to get into details, but basically this type of testing guarantees detection of atleast 1st and 2nd order defects with the minimal amount of test cases. Madhan S. Phadke's Quality Engineering Using Robust Design mentions this type of testing. Also Bell Labs has been so kind as to publish online some fairly heavy strength Orthogonal arrays, so you don't have to calculate them. My employer uses this type of testing on many of its projects and it's a huge time saver. I just learned about it in an onsite class by our top tester and am going to pitch it to my project soon.
Good luck, sorry for being so vague in places, and finally, if you have more questions about Orthogonal Array Based Robust Testing, please let me know: redundant_pleonasm@hotmail.com
... may be your programmers are using wrong language? Too low level, like Java or C++? May be better to switch them to Prolog, PHP or TCL/incrTCL? They will produce less lines of more functional code then...
IFF, you can relate future costs back to the original development.
When I do code reviews, I'm often looking at:
You need to account for you time very well, and with a good tie back to the original development to truly realise your saving.
Or, you need to do a complete switch for at least 6 months, and see how much better you get overall.
Read more of this story at Slashdot.Read more of this story at Slashdot.Read more of this story at Slashdot.
a little overkill, but i usually make a very tiny program to test every function in my project. tell it what to give the function and what the function should do back. lets you control the testing alot. now, if i had a team, id have someone take the code documentation and do all this, so i can code more. but this works for me.
Question
http://www.ironfroggy.com/
Another slashdot stories which says
"Jeez, maybe I should have trained in the stuff before trying to do it as a job"
From my experience testing at PictureTel and DEC in Mass., I found out that the usually-understaffed test team runs into the Laws of Software Testing: Law #1: Most of the time, you are not testing. You are obtaining (beg, plead, cajole) equipment and software, and you are configuring and fixing software, just so that the environment is ready for the software testing to occur. Law #2: When testing, most of the time you are not testing anything that matters. Sure, scripts are great, but they are very narrow, and bugs are sneaky. Running a limited variety of scripts across some clients and servers only gives the impression of coverage ... kind of like the concept of "busy work".
Law #3: When you find a bug, most of the time it can't cross the political barrier. After all, bugs are rated and prioritized, and that is the domain of management, who is overwhelmingly concerned about release dates.
Law #4: The bugs you don't find will reach the customer and will return as the highest-priority bugs that will usurp other bugs in the ongoing process. Fixing customer problems is of course a priority, but it landslides into the current product's production.
[also misbehaves on Kuro5hin as Peahippo]
Until companies and individuals are held accountable, any testing, other than 'getting it to work', may be "too much.
/# lines or by department or programmer. Often, looking for why a bug exists or who wrote the code is often seen as "searching for someone to blame". What is important is fixing them. We are not to care why they exist. Speaking of someone's bug-laden code is 'bad-mouthing' them and considered 'unprofessional'.
:-(
That's the message I have gotten over the years. Some managers believe it 'fine' to call a project 'done' that hadn't been given any SMP, boundary or stress testing. It might not meet the original design specs, but still be acceptable if it was shippable. I ported some third-person code to a new OS. I found built-in kludges, one of the heinous being a check for a data-object stack overflow that quietly stopped keeping state and kept only 'nesting level' beyond a several-item hard-coded limit. This effectively hid imbalances of pushes-pops in the code something not readily observable unless you
did thorough testing for expected output and then only if rare routines were called.
During my work, I found and reported problems and fixed them as part of the porting process. This stopped as the manager reprimanded me for not staying on target and getting the port done. He compromised by allowing me to file the bugs in the bug database but still counted much of that as wasted time.
The result: the original coder was 'great' and I was a someone who rocked-the-boat and didn't strictly follow orders.
Other managers have chosen to ignore bugs in hopes that they won't be found because the cost-to-fix was 'too high' (manager might not make his 'schedule'). "Bugs", however, are expected and folded into schedules later on -- they aren't regarded as a negative against the manager or programmer -- they just "exist" like thunderstorms and wind -- random events beyond our control. Keeping schedules, however, is a metric that can be used to evaluate performance.
It's rare to see metrics on bugs/program or project or
What is one to do when market pressures care about who's first -- who's the market leader? Who provides feature "X" first? Who is "innovative" -- and who can get their hands into customer pockets first?
Often programs are outdated within a few years, so if you can get it out fast and do a few patches for high-profile bugs, that is often more cost effective than more thorough testing up front. Often customers are migrated to 'upgrades' before fundamental flaws in the old
software become a problem.
Customers can be forced into new versions by the company just declaring the old version is no longer supported -- and then you have to upgrade to a new version which may not work as well as the original. Then the company stalls for months while working on a patch that, sometimes, you have to pay extra for, via a support contract!
No *real* warrantee other than maybe being offered a refund -- but if you wanted a working program? Sorry, that's sometimes not available at any price.
I have had to do some off-the-wall things to reproduce bugs found by our test team. Sometimes the attacks the testers launch against our system are just ridiculous, especially for a desk-top app with no connectivity. If the user rapidly clicks his mouse in a concentric pattern at 11:59:59 while humming the star-spangled banner and pouring Mountain-Dew on the keyboard - focus may be set to the wrong field on the UI - sorry but it's going to happen. The problem is that we are compelled to fix all these bugs and fix them quickly. We won't refactor designs or redesign at this point so everything is a hack - we are just making more bugs! I think Hack Attacks are just the way that testers survive their boring mundane jobs. Test the boundaries and stress test but realize that the development team had to meet dead-lines. I have seen test teams test code for 4 months when I only had two weeks to write the code. Guess what? They found bugs.
This is online community at its best. I bookmark disussions such as these because they help me with my job.
If only I could cite Slashdot to my skeptical PHB. Such is life.