Slashdot Mirror
Too Darned Big to Test?
gManZboy writes "In part 2 of its special report on Quality Assurance (part 1) Queue magazine is running an article from Keith Stobie, a test architect in Microsoft's XML Web Services group, about the challenges one faces in trying to test against large codebases."
Maybe they can take a clue from other large code projects and open source it?
So this will be Microsoft's latest excuse, then? ;)
they should just hire a lot of monkeys to test their software.
Besides, in this way the IQ of the later user and the testers arent differing too much.
to Keep It Simple, Stupid.
Shouldn't that be too darned bloated to test? It shouldn't be hard to test the individual subcomponents for functionality and at boundary conditions. Of course, you can't fully test something as complex as the system in the article. No reasonable sized program can ever be fully debugged -- the possibilities are too many to explore. However, it is possible to fully verify the smallest components, and build large components from them and fully verify those as well. Obviously, the complexity increases greatly with each new layer, but when one is working with fully verified components, any errors that occur must be in the local logic. Granted, this is much more labour intensive, but as long as each component follows precise specifications, it's more than feasible. I'm amazed that many prominent software projects still use largely monolithic testing...
Be relentless!
For those who didn't RTFA, it is basically saying that exaustive (?sp) testing can't be done on a large codebase, and random testing is all you can use, to which most coders will say bull.
If a piece of code is too big to test exaustivly, it's time to refactor it into bits that can be.
After you've tested each part to make sure it works, you test a super set of parts, thus testing the interactions between the smaller parts, lather rinse repeat until you've tested th whole application.
Correct use of unit testing will always outstrip random testing.
This is just an excuse for badly designed code bases.
Ask 8 slackers a question, get 10 awnsers (a citation, but I can't remember from who)
The article just says what everyone knew ..
* code coverage != proper testing
* clever inputs are needed to test
* few programmers test concurrency
Ending with - "ECONOMY IN TESTING" (ever heard about "Good Enough Isn't")
Essentially apologetic about the lack of testing. Test driven development is not a philosophy, it's a way of doing. In a perfect company environment, you'll never be blamed for breaking someone's code - but in most places the idea is "he made me look bad". Peer reviews never work out properly. This is why FOSS is turning out more secure and clean code.
Quidquid latine dictum sit, altum videtur
I recently had a problem with ordering from Amazon that illustrates the problem with testing and all the possible permutations of user actions. I was checking out when I noticed that high shipping cost from one vendor, went back to order from a different vendor and hosed the order. Apparently, there was only one of the item in stock and it was now committed to the pending, partially checked-out order. There was no way to clear the partially complete check-out process and no way to checkout with the item in my shopping cart -- it would only complain that I was trying to order TWO of the item and pull the ONE instance of the item from the cart.
Amazon is not the only e-commerce site with this problem (although I expected better from Amazon). Many sites fail to test for user action sequences other than the straight-through order process. I'm not suggesting that developers test for all possible sequences (that's impossible), but they should test for more plausible ones that a simple linear execution of the process.
When I did software testing (a task that I hated), I quickly broke an RDBMS application with just a simple series of adding and removing items from a user-manipulable working set of data objects. Moreover, I even broke the UI layer and dumped myself into a lower level of the RDBMS shell that was supposedly inaccessible to users. The developers grew to hate me so much for finding bugs in their code and the RDBMS vendor's code that I was moved to another job (YAY!).
The point is that it is often too easy to break code because the developers have created overly simple linear use cases that are then used in testing.
Two wrongs don't make a right, but three lefts do.
It is possible to build immense and complex code bases that are incredibly well tested and robust. Look at any Linux distribution and this is what you have.
The key is that the code base is structured so that it can evolve over time as many independent layers and threads, each using an appropriate technology and competing in terms of quality and functionality.
The problem is not the overall size of the code base, it's the attempt to exert centralised control over it.
To take a parallel from another domain: we can see very large economies working pretty well. The economies that fail are invariably the ones which attempt to exert centralised planning and control.
The solution is to break the code base into independent, competing projects that have some common goal, guidelines, and possibly economic rationale, but for the rest are free to develop as they need to.
Not only does this make for better code, it is also cheaper.
But it's less profitable... and thus we come to the dilema of the 2000s: attempt to make large systems on the classical model (which tends towards failure) or accept that distributed cooperate development is the only scalable option (and then lose control over the profits).
Sig for sale or rent. One previous user. Inquire within.
"Yo' codebase's so fat, when it get in a lift it has to go down!"
"Yo' codebase is so bloated, it's got its own dialling code!"
"Yo' codebase's so big, NASA includes it in orbital calculations!"
Etc. etc., ad nauseam et infinitum...
Software rewrites may be considered harmful, but at which point do you declare that enough is enough and start again, breaking it down into smaller, easily tested modules? Big, old projects (like, say, OpenOffice.org) can get so appallingly baroque that there must be vital areas of code which haven't been modified (or, more importantly, understood) in years - how do you test those?
Tedious Bloggy Stuff - hooray?
We have little time to code much less test! Who else codes, sends to QA, gets bugs, codes, sends to QA, get bugs...............
I do a lot of programming with visual output. It is impossible to have a computer check that the font got outlined correctly in the PDF, say.
When you combine this with user input and then rare-case branching logic, you can end up with a nightmare of unfollowed paths. Unfollowed, to some extent, means untested.
Just one extra branch can be disasterous because of factorials involved depending where it is placed in the branch pipeline. One minute, everything working, next minute some new code and
things that need to be eyeballed.[% slash_sig_val.text %]
Mark, incentive is spelt with a c.
I was tempted to send you a text on your phone to tell you - are you sure you want to leave that on your website to be harvested? You might regret it in a few years.
... automatically performed by OTS:
Finally, testers can use models to generate test coverage and good stochastic tests, and to act as test oracles. A fundamental flaw made by many organizations (especially by management, which measures by numbers) is to presume that because low code-coverage measures indicate poor testing, or that because good sets of tests have high coverage, high coverage therefore implies good testing (see Logical Fallacies sidebar). One of the big debates in testing is partitioned (typically handcrafted) test design versus operational, profile-based stochastic testing (a method of random testing). Current evidence indicates that unless you have reliable knowledge about areas of increased fault likelihood, then random testing can do as well as handcrafted tests.[4,5]
For example, a recent academic study with fault seeding showed that under some circumstance the all-pairs testing technique (see Choose configuration interactions with all-pairs later in this article) applied to function parameters was no better than random testing at detecting faults.[6]
The real difficulty in doing random testing (like the problem with coverage) is verifying the result. A test design implication of this is to create relatively small test cases to reduce extraneous testing or factor big tests into little ones.[9]
Good static checking (including model property checking). If you know the coverage of each test case, you can prioritize the tests such that you run tests in the least amount of time to get the highest coverage. First run the minimal set of tests providing the same coverage as all of the tests, and then run the remaining tests to see how many additional defects are revealed. Models can be used to generate all relevant variations for limited sizes of data structures.[13,14] You can also use a stochastic model that defines the structure of how the target system is stimulated by its environment.[15] This stochastic testing takes a different approach to sampling than partition testing and simple random testing. Code coverage should be used to make testing more efficient in selecting and prioritizing tests, but not necessarily in judging the tests. Test groups must require and product developers must embrace thorough unit testing and preferably tests before code (test-driven development).
Singularity: a belief in the "God" idea with the "demiurge" relation inverted.
My own research group works on methods to reduce this burden in a number of ways. One, my personal work, is on "semi-random" testing (we call it Adaptive Random Testing) which, we claim, detects more errors with fewer tests and reduces the problem that way. Another is "metamorphic testing" which tackles the oracle problem more directly by a slightly more sophisticated form of sanity checking assertions. You test the program with two (or more) related inputs, and check whether the outputs have the relationship you'd expect based on the inputs.
Unfortunately, the boss has an, um, slightly behind-the-times attitude to putting papers on the web; but if you search the DBLP bibliography server for T.Y. Chen you can get references for most of them.
However, I'd be the last to claim that we have a complete solution to the oracle problem; there will of course never be one. But it is a problem that will continue to make automated testing a challenge.
Any sufficiently advanced technology is indistinguishable from a rigged demo
--Andy Finkel (J. Klass?)
...such is the outcome of not doing test-driven development. Test the functions as you write them, and just leave the tests there until you release, makes sure everything works. When will these people learn!
Yes. I am far too big to test. Oh you meant the code ...
...
Quote from an (otherwise) intelligent guy at work today:
"Yeah I got a bit heavy with the delete key whenever I saw the word 'test' but I got annoyed that it wouldn't compile"
sigh
Unrelated note: why do people pretend to swear ("Darned")? Either swear (Damned at least) or don't - but why pretend? I don't get it.
Peer reviews never work out properly. This is why FOSS is turning out more secure and clean code.
Isn't that a contradiction in terms? Or by "peer", do you mean something other than "a comparable programmer"?
It's official. Most of you are morons.
I've seen this debate before, and the part I always wonder is "why not both?" At least, when you are starting from scratch. You can verify your components do what they are supposed to and then check for bizarre situations no one thought of with random testing (sometimes you will expose obscure bugs in the software stack itself, not just your code - but remember no code stands alone, and all crashes look the same to the end user no matter what the root cause.)
Particularly on large, old projects one has inherited, random testing can really help because you have absolutely no clue what you are looking for. There are so many discrete components to the system that could be tested it would be the work of ten years to set it up, so you are forced to (as much as possible) assume that things work and find the cases where they don't. Then, you gradually begin to fix things over the long haul while fighting fires.
GCL and the other free Lisp implimentations are a good example of testing - we have a very dedicated individual who has been creating tests of ANSI behavior from the spec and testing a wide variety of implimentations - indeed many non-standard behaviors have been corrected because of these tests. He has also created a "random tester", which I like to call "the Two Year Old Test." It is a code generator which generates random but legally valid Lisp code and throws it at the implimentation. It has exposed some very obscure bugs in GCL which probably would have otherwise hidden for years. Anybody who has been around small kids knows they will introduce you to all sorts of new failure modes in just about everything you own, so I always think the Two Year Old Test should be administered as a final check whenever possible. (Granted this works particularly well for compilers.) Newbies are very useful for this kind of stuff as well, because they will use the software in ways you never thought to.
"I object to doing things that computers can do." -- Olin Shivers, lispers.org
LMAO
Is anyone doing independant QA audits of linux, outside of the development sources/bug report/linus ruling on high loop?
If you resist reading what you disagree with, how will you ever acquire deeper insights into your own beliefs?
They call their monkies "End users" and charge them big bucks for the testing, and on top of that have them accept EULA's that take away their rights. ;-)
"However, it is possible to fully verify the smallest components"
Absolutely not true, and because of that the rest of your argument isn't true either.
Let me provide you with one example of why it's not true...Imagine when that new printer model comes out and your printing code that worked fine on all the models that existed at the time of original testing no longer works correctly for this new printer (perhaps it's a bug in the printer drivers). Either way, it's something that just can't be controlled.
Here's an interesting article to read:
http://www.kaner.com/pdfs/impossible.pdf
Some points covered in the article are:
1. We can't test all the inputs to the
program.
2. We can't test all the combinations of
inputs to the program.
3. We can't test all the paths through the
program.
4. We can't test for all of the other
potential failures, such as those caused
by user interface design errors or
incomplete requirements analyses.
If you still don't believe it, ask yourself why certain bugs slip past the team even though they pronounced that it was "fully tested". Why did that happen? How did they escape the entire development and QA team? Was human behavior part of it?
This fact alone is enough to dispense with programming languages that attempt to use large numbers of low quality programmers by inhibiting polymorphism with static type declarations. Compile time assertions are only one kind of test and do a lot less for quality assurance than allowing flexibility in choosing the test points on a more succinct body of code.
Seastead this.
[TFA] Another great way to target testing is based on actual customer usage.
This is a really good idea.
The crash feedback systems in Mozilla exhibits this model of testing.
I think more of the casual user applications I run on the desktop should be compiled with debugging and a simple transparent mechanism for returning information to the developers about problems.
Nothing mandatory, no hidden information sent back to the mother ship, just a text file showing back traces, etc. that the user can see contains no sensitive information.
Thus all users become beta users that can feedback to the developer which bugs really matter.
Taken to the next step of optimization and UI design, developers can find out which code paths really matter in terms of real life usage if the application is instrumented with profiling turned on and the option for the user to feedback information this way. IIRC, some compilers have options to take advantage of run-time statistics to better compile the second time around.
"Provided by the management for your protection."
Back in the old days, a common way to write a program was to make code that can be used in many different places from within the program. Routines that are similar would be considered a bad thing, so you make routines that are designed to handle the differet situations that need similar code.
The problem with Microsoft is that they have forgotten or never learned how to design a program before their people have started to write anything. As a result, we see 384k patches from Microsoft that take several minutes to install on some systems.
Another problem is that there is a LOT of duplicate code that is in use even within common libraries.
The people who suggest that there are too many features are almost correct, but the problem isn't with the number of features, it's the way those features are added to programs.
Also, there is only so far you can take a given design while you add features before things start to break due to design. If you start with a good DESIGN, then implement that design in code, it becomes a LOT easier to debug.
Microsoft needs to come up with a NEW OS that isn't an extension of Windows NT or Windows 3.0(95/98/ME are still based on that old code in many ways). Windows NT was the right idea back when it was first developed. Toss the old design, start from scratch, and you end up with a better product. The only problem that Windows NT really had was that compatability wasn't written into the core design of the OS, it was a layer added on top, which means you need a "translator" to handle that. If it's in the design, then you figure out how to do the emulation of the old system in a way that is compatable with the "new" way of doing things. Today, it's not as difficult as it used to be back in those early days of Windows NT. We have enough processing power to make virtual machines that can handle just about anything if they are coded properly. The only problem is that the emulation of the old DOS environment or Windows environment hasn't been implemented by Microsoft.
But I've gone off topic a bit. The key to easily debugged code is to design in a way to make things properly modular. Almost all features within Windows should be TIGHT code. To open a file probably has 200 different versions of that code within the Windows XP code base scattered through all the programs that come with Windows XP or 2003. Think about that, and wonder why it's hard to debug.
If you claimed an income tax return too big to audit for accuracy, or better yet, too big to file.
The problem is that you aren't supposed to test huge chunks of software all at once. This is insanity! You will never be able to write an OS and then test it all after it's done. What you do is write the program piece by piece, then test every individual function independantly of the rest of the program. Give the function hell! Send it bad data, wierd data, data out of it's range, etc. Once it's fireproof, move on to the next function and continue. Sure, this will take a while with a big program, but you will get a really stable program. A company as big as Microsoft surely has the resources to do this.
...Had this been an actual emergency, we would have fled in terror, and you would not have been informed.
Instead of trying to test huge code bases, why not write decoupled systems and test small pieces of code? Oh wait, that requires effort.
I've worked on a number of projects (that borderline on huge) which have a thorough set of unit tests. Each test sets up pre and post conditions and checks the output against what we expect. (Duh!) It's not difficult, it just requires planning and careful attention to detail.
If you've ever built Perl from source, you'll notice that the entire code base gets tested during the process.
I have to say that it's not about theory or speculation, it's just about hankering down and doing it.
Testing, fundamentally is not that hard. I think the real problem is developers often trying to find excuses to either put it off or worse yet, not do it at all. Added to the problem are badly designed architectures where most components have tight dependencies with others. This prohibits running them in isolation and hence limits testability. Naturally, it's always more complicated than this (budges on time and money) but the root of the problem is lack of motivation or ignorance to the benefits of having easily and hence well tested code.
Join Tor today!
i worked for an HP lab that was designing a rather large ASIC that was previously a large chipset used for coherency control between the four procs on a cellboard and the other cellboards as well. We had one hallway of design engineers pounding out the code/layout/etc for the chip and an *entire* other hallway of guys writing test fixtures (unit and system) at the same time. My task at the time was to write an in-house, unit testing language translator to fit with the system testing language and engine. We had 20 guys working on perfecting the tests and catching chip problems way before the chip ever thought about making it to tapeout.
... and there still remained the real-world testing when the chip would finally make it back to the fab in hardware form. Interesting process, i'll tell you that much.
So yeah, tons of resources went into just testing the damn thing
RFP
specs/design docs/...
Oversight; customer - managers - qa/test - test validation - test verification - peer review ...
...you get the idea. It's not theory, though it's practice (more later).
The idea being; make it clear what failure is by using severity/priority levels (IEEE 12207, 1012). Look for problems before deployment. Know what fails. Deploy by manager's directive but know what doesn't work as it should.
That way, you know all failure points and management takes responsibility...like they should anyway. If something fails that you didn't know about, the finger pointing is not effective unless someone truely deserves blaim.
I've worked in environments where they neither had the will nor the budget to follow these steps. In these cases, I act as if the people around me should be acting as I've described above. I ask for documents semi-formally and descibe problems as I encouter them mostly to CYA. This alone makes people take what they are doing more seriously and also make lists.
If the managers up the chain can't provide the right environment to make reliable software, it's unreasonable for them to demand it.
In the original paper, "Managing the Development of Large Software Systems" by Winston Royce, describing the waterfall model, Royce actually points to the flaws of that process. He actually says that it "is risky and invites failure".
It appears that at some point some PHB's saw the paper, looked at the pictures (instead of reading) and decided "we should all use the waterfall development process".
As for iterative development, I couldn't agree with you more. And its also what Royce was really at where each "phase" provides a feedback to the one before it. And if a project follows the steps you outlined for each iteration, as well as doing some refactoring for leasoned learned, you will generally see the bugs go down and the code quality steadily increase.
OTH, trying to suddenly apply a test scafolding around an existing large codebase can be a very painful process. Many time its actually easier to just rebuild using the original app as the basis for the new test cases. And the NIH syndrome weighs heavily.
----- If communism is a system where the government owns business, what do you call a system where business owns govern
You actually bring up a better question. How do we deal with big pieces of steaming ****, I mean spaghetti that get handed to us to maintain.
There are all kinds of processes and theories that if you religiously follow you can be sure to prevent a project from becoming crap. But, always in the life of a project there comes a PHB determined to turn the code bad.
I think we need a lot more attention on how to deal with code thats already in bad shape. We've got refactoring and Code Reading but, little else on the subject of improving existing code bases. A result of this, most dev's tasked with maintenance simply do as little as possible when modifying the code from fear of unwanted consequences. Thus, code rot sets in.
----- If communism is a system where the government owns business, what do you call a system where business owns govern
The article just says what everyone knew ..
What every devloper know.
Now you've got a paper to send your boss to reinforce what you've been telling him for years. Ever notice how things seem more authoritative if they come from outside the company walls.
----- If communism is a system where the government owns business, what do you call a system where business owns govern
In Soviet Russia, software tests you.
#include <signature.h>
The trick for beta Testing is to get as many eyes on it as possible who know that this isn't a completed or stable product. and are able to try funky things to break it.
I agree with that statement.
However, what you've described takes an enormous amount of time and effort and [background] knowledge, to the extent that "try(ing) funky things to break it" could very well become a full time job. Hell, just spending the time necessary to read the documentation [and surf the web looking for "gotchas"], solely for the purpose of figuring out how to INSTALL a piece of software, is d@mned near a full time job.
But in the real world, people get paid to do full time jobs - in fact, they even get paid to do part-time jobs. And if their job title is something like "Senior Testing Engineer for Quality Control", then they get paid sh1tl0ads of money.
When I did software testing (a task that I hated), I quickly broke an RDBMS application with just a simple series of adding and removing items from a user-manipulable working set of data objects. Moreover, I even broke the UI layer and dumped myself into a lower level of the RDBMS shell that was supposedly inaccessible to users. The developers grew to hate me so much for finding bugs in their code and the RDBMS vendor's code that I was moved to another job (YAY!).
The fundamental problem here is that you were more intelligent than your developers [which, in turn, seems to have been because your developers were morons].
Microsoft is very concerned about its ratio of developers to test engineers, and is doing what it can to REDUCE the number of test people on any specific project. The ratio was two test to one dev, and the project over the last 18 months has been to reduce that ratio to about 1.2 test to one developer.
So, we can see, Microsoft is creating yet more complex untestable software while at the same time reducing the amount of testing they can actually do.
The 'theory' is that they can automate a lot of the testing. The practice seems to be to push the release button and call it good.
A most overlooked advantage to owning a computer is if they foul up there's no law against wacking them around a bit.
Bug free, cheap, on time, works. Pick two.
(BTW thanks for the informative viewpoint.)
~REZ~ #43301. Who'd fake being me anyway?
"It is possible to build immense and complex code bases that are incredibly well tested and robust. Look at any Linux distribution and this is what you have."
Insightful my eye. I was using Pan earlier on Suse, and a delayed CRON job started up. X then proceeded to die. So I wouldn't be using Linux in the same sentence as "incredibly well tested and robust".*
The code you see in situations were life and limb are on the line, are an example of "incredibly well tested and robust".
*A pet peeve. A lot of Linux apps are "brittle". Change one line in their config file for example, and it breaks.
"The solution is to break the code base into independent, competing projects that have some common goal, guidelines, and possibly economic rationale, but for the rest are free to develop as they need to."
Two problems with your "Evolution is king". One much like "back in the day". People forget all the blind-ends that evolution finds, and only see it's successes. The other is that evolution takes a long time to get desired results. That's why all evolution theories require a very large timeframe.
If the user base has all the features they want in those mature apps, why would they buy the new "elegant/Perfect" software design?
You're not giving them anything new, so economically, you will fail since you won't attract enough interest.
Reason why there is hope for the future generation #364:
"I wish my grass was emo so it could cut itself."
From the article...
"You can test square root for all the 32-bit values in a reasonable time,[1,2] but we are now moving to 64-bit machines with 64-bit values (and even longer for floating point). Assuming a nanosecond per test case, it would take 584 years to test all the values. Sure, you could scale to, say, 1,000 processors, but you would still need six months for just this one test case."
Does he really think this is what software testing involves? If I write a function to calculate the square root of a number, there is no way my unit test is going to run through every 32 bit number and make sure they're all correct! I'll check a few random positive numbers to make sure they work. Then I'll check a negative number to make sure it throws an error (provided I'm not dealing with complex numbers). The same thing goes on a 64 bit system. Good testing is smart testing, not exhaustive testing, that's just common sense.
I hate to rant on them (again), but they have to test their product. The easy way to test extremely large code bases, is 1. first make sure the code is M*O*D*U*L*A*R, so that each piece can be tested to ensure it meets the criteria. There is no excuse for not testing software other than 1. "Were too lazy to fully test the software." 2. "We're too cheap to fully test the software." 3. "Why test the software when our customers will test the software for us?" ...All 3 answers are bad, bad, bad! The announcement 'Our codebase is too large!" ...is junk. Many OSS projects are very large (X windows/Gnome desktop):170 million lines, Linux kernel 40 million lines, OpenOffice.org: 75 million lines, etc. It's true that testing is done in parallel, but it has to be done.
"...which is why open source works. The philosophy of OSS apps has always been to make small programs that do one thing very well, then join them together to get good funcionality for more complex tasks."
However the testing problem isn't in the "well written, does one thing" module, but in the exponential explosion of code paths, and interactions between the code modules. F/USS addresses the code module, but it doesn't deal as well with the interaction aspect. It's kind of like expecting one kind of physical laws for people in the US, and a different type of physics in say Europe.
"And not through specific design, but throught adaptation and tinkering."
Evolution is basically the "Drunken Tree Walk" and it has it's downsides, of many dead-ends, and long seek times. Plus it only really addresses the "nodes" of the code graph. The paths don't do as well.
"Have you tried putting a bounty on the bug you want resolved, such as cash?
Complaining about bug-fixing in volunteer maintained software is like complaining that no one picks the litter up in front of your house."
Sounds to me like the F/OSS development model isn't the panacea that everyone makes it out to be. You have to "pay cash" to get Gnome optomized, and get rid of bugs. What good is a "thousand eye" if you have to pay them all?*
*Plus closed-source is a "pay for results" methadology too. But we hate closed-source.
The title to this story on the main page, "Developers: Too Darned Big to Test?" seems to imply that developers are too corpulent to test. For a little while, I though I was about to read a story about developer obesity.
I had a good laugh at that one. Anyone else misinterpret that as well?
The given reason is that you have to check the result or its no use, but that applies to any form of testing.
The real problem with using code coverage as an indicator is that a given line of code can be executed with many states and what you need is coverage of the cartesian product of statements and states.
A good example is an optimization in a compiler where you may spend many lines of code checking if it is possible and only 1 or 2 actually doing the optimization. So if you never do the optimization you could easily have close to 100% code coverage but you have 0% testing of the optimization.
So as a measurement it is falacious, as there is no correlation between percentage tested and percentage covered for less than 100% coverage but you can see that you can even have 100% code coverage and still miss bugs from the following statement:
if (size > limit) size = limit - 1;
which fails if size == limit (you intended to write >=) - i.e. there is one value of the state that fails.
Squirrel!
Oh wait, they mean the code, not the developers themselves.
"Teacher, I can't fit in the exam chair. I'm too obese to test. Can I be excused?"
Table-ized A.I.
Just takes the will to do it. It means getting quality personnel involved from the beginning--in the design phase of the project.
It means allocating resources to each project module (and if the project isn't modular, it's crap). The quality people don't have to be geeks, in most cases. But they do have to be smart, diligent and detail-oriented.
It means adopting a professional testing approach and sticking with it.
It means demanding that programmers do their own unit testing before submitting their work to the code base and having acceptance testers who confirm that before a build is released into the general quality process.
I was a white-box tester because I had a programming background. I would generate a list of test cases from the code for which I was responsible, but I also believe that testing a product means using it as it is intended to be used, if possible.
There's no excuse for not expending the time, money and effort on assuring a quality product. Better not to do a project at all than to do it badly.
Absolute rubbish. Not testing != good, clean code. I spoke on this topic at linux.conf.au a few years ago. I noted then, and I'll note it now, that OSS authors are better than average coders, but awful testers. Do this exercise:
1. Grab any C or C++ foundry tarball from Sourceforge.
2. TypeWhat happens? I reckon in more than 99% of occasions, absolutely nothing. Not testing is not good enough.
For a good example of this in action, phpBB is free and open source software, licensed under the GPL. It has security bugs coming out of its ears. What it needs is a good quality code review.
I do these for a living, and they are definitely worthwhile. The software I help maintain (XMB) used to be like phpBB - buggy, insecure, and slow. Now it's just somewhat buggy and a bit slow.
Code Reviews and testing make all the difference.
Andrew van der Stock
Maybe this is why OSS is generally good? Instead of one giant monolithic codebase (ex: Windows), we have hundreds of little packages where people actually know WTF all the code does, or at least have a team of people who can cover the entire codebase in terms of knowledge.
When you look at the state of the world, how can you not become a radical, liberal anarchist?
Two groups:
...
1. Linux distributors (defined loosely): IBM, Novell, RedHat,
2. IT Security companies that actually make money by finding security bugs (e.g. Secunia)