Slashdot Mirror
Test Coverage Leading You Astray?
An anonymous reader writes "Are your test coverage measurements leading you astray? Test coverage tools bring valuable depth to unit testing, but they're often misused. This article takes a closer look at what the numbers on the coverage report really mean, as well as what they don't. It then suggests three ways you can use your coverage to ensure code quality early and often."
But for any reasonable size program the combinations are just so high you can't sit and write unit tests to cover and execute each and every posibility. Either physically or financially.
What you can do, is use tools like PolySpace (www.polyspace.com) to ensure you won't have any array overruns, out of range errors, access through dangling pointers, etc. You can then run unit tests on the 'working' code in working scenarios to ensure it does what it should.
The technique of unit testing is good, and catches many errors, and code coverage is a very good companion in finding out what you haven't tested. Unlike what some posters above have indicated, this is generic, and has nothing to do with the programming paradigm used, nor the programming language. There are two major problems, however. 1. With unit testing, you're only testing that the unit does what you expects it to, given its interfaces (the API, global variables, whatever...) If a bug is a misunderstanding of the specs, you won't catch it, unless the person who wrote the unit test is the one who wrote the specs. 2. You won't discover errors in situations you haven't tested for, and if the code is written poorly enough, it'll give you very good coverage numbers. Example, code that has no error handling what so ever, and a test suite that doesn't subject it to error situations. These problems doesn't make unit testing, and code coverage analysis bad. It's far better than not even trying. But you have to be aware of them and scrutinise the test suite to see what it *doesn't* test, especially if code coverage numbers are really high.
What can be better than cricket?!
Get your own free personal location tracker
The code was designed, exercised, tested, and executed properly from what you're saying. The display failed due to hardware problems.
In what way is that hardware failure related to code coverage or any other form of software testing or QA metric?
I do not fail; I succeed at finding out what does not work.
Functions lie like dogs. You can test the output of functions until you're blue in the face, but until you take a holistic view of the application and what it does, unit tests are more a salve for management's mind than a boon to developers.
And the solution is... "holistic" unit tests.
While it's true that unit tests have a hard time making that last little yard (mostly in the form of hardware output, like graphics on the screen or your example), you're not writing your unit tests correctly. It's a rare unit test for me that is the equivalent of checking that adding two numbers work correctly, and while those are useful in development, they very, very rarely ever break later. Pure arithmetic function are the easiest to write, in general, and they correspondingly have the smallest need for continuous automated testing. (Not zero, of course, just the smallest. And when they do break, boy howdy...!)
In your other example, you ask:
Can you verify that the input data was correctly injected into the database...
(and I cut the rest of this question off as it posits an incorrect approach.)
The answer to this is yes, although you need a good database and a good understanding of how they work. (Not "great", just good.) I have thousands of tests that verify that certain code correctly manipulates the database, and that verifies calling certain webpages correctly manipulates the database. It's only marginally harder than testing a traditional function. The key here is to do everything inside a transaction; perform the task, do your verification, then roll the entire transaction back. Then it doesn't affect your database (which should normally be the "test" database, of course), and as a side effect under all but the "READ-UNCOMMITTED" transaction level, allows you to have any number of copies of the same test(s) running against the exact same database.
I can't imagine writing a distributed database-based application without such tests. Well, I can, but it's no fun.
In a lot of database-based applications, since the database is the application, this goes a long way toward testing the entire app.
Your unit tests ought to cover everything but the hardware output, which is more the exception than the rule.
Part of the problem is the number of APIs that exist with no thought for testing, making it seem as if unit testing them is impossible. For example, a lot of GUI toolkits are a major pain in the ass because it's difficult or impossible to fully simulate pressing a key in them and then processing the event loop exactly once, after which you will see what happened. This is a limitation of the toolkit, though, not unit testing, one I fervently hope will someday be eliminated after my whining on Slashdot catches the eye of one of the GTK developers or something.
In other cases, you have to a little work, but it can be done. We use Apache::ASP, and it ships with a little Perl script that can run an ASP page outside of the webserver via a command line. Still not terribly useful, but I was able to take that script and turn it into something that accepts multiple requests over a pipe, and wrap another Perl module around it that manages the connection to make it easy to use. Now, in my unit tests, calling a web page looks just like calling a function. Unfortunately, the rollback idea doesn't trivially work here, but I have some other things in place to help with this. The upshot is my unit tests include whether entire web pages work. This is some damned fine testing, and it's caught plenty of bugs long before they get out to the user.
Sure, right on the periphery of some systems is hard to reach, but the vast majority of any system is perfectly managable.