Test Coverage Leading You Astray?

An anonymous reader writes "Are your test coverage measurements leading you astray? Test coverage tools bring valuable depth to unit testing, but they're often misused. This article takes a closer look at what the numbers on the coverage report really mean, as well as what they don't. It then suggests three ways you can use your coverage to ensure code quality early and often."

Comments

Non-object oriented test tools?

[iangoldby]

It's a pity the submitter didn't provide a short paragraph review of the article rather than just copy-paste the abstract.

Anyway, having had a quick look, it is all about Java.

I'd love to hear from anyone who can recommend test coverage tools for C (ie. non-object oriented). I think that just about all of the articles I've ever read about testing methodologies have been exclusively about object-oriented patterns, and pretty much only Java or .NET.

Object-oriented techniques are a good tool, but not the right tool for every job...

DO-178B

[nonsequitor]

Three types of code coverage are required for safety critical airline applications:

1) Line Coverage - Has every line been tested
2) Branch Coverage - Has every branch been tested
3) Boolean Coverage - Is EVERY possibility on a truth table for each logical operator explicitly defined

These tests alone don't certify that the code is ready for an airplane and that it is indeed "bug free." My software engineering professor said it best when he stated, you can only prove the existence of bugs, you cannot prove the non-existence of bugs. These guidelines as adopted by the FAA for the certification of safety critical code, don't prove the non-existence of bugs, but they do go a long way towards proving the existence many bugs and provide a MINIMUM standard to which code must be exercised before being allowed into an airplane.

Software Engineering is a science, methodology has been pioneered to help us ENGINEER the software we develop to be as defect free as we know how to make it. As in other disciplines of engineering, there will always be things not yet quantified. Take architecture for example, an architect would design a bridge to withstand an earthquake of a specific magnitude, winds of a specific speed. Does that mean the bridge is safe? What if the materials used weren't rated for the temperature range needed for the locale, etc...

As much as we do to ensure quality, there is no silver bullet. The company I interned at which will remain nameless made a multi-function navigational display for airforce one. It rebooted during a touch and go at 40 degrees farenheit. Wasn't it tested you ask? Of course it was, it was tested at -40 degrees and 140 degrees, but the timing on one of the buses was off at 40 and the hardware watchdog took it into a reboot at a very critical time. It was DO 178B Level A certified, had 100% code coverage of course, but there will always be bugs. Don't trust tools to tell you otherwise, because you can never prove the non-existence of bugs.

(For those who don't know, a touch and go is where the plane starts landing and takes off again)

Re:DO-178B

[OneManCongaLine]

There are also: Path coverage (extremley complex and not practical to use in most cases, but for critical sub-systems it might come in handy) Linear Code Sequence and Jump (LCSJ) There are more, but these two on top of my head is worthy of inclusion in any discussion on coverage. There are a lot of business-specific standards out there that specify use of coverage. Aerospace has one, vehicle control systems has one, pharmaceutical and nuclear system yet others. Guess wich one of these that has the _least_ strict standard for coverage? (Hint, Homer's workplace =)

Re:DO-178B

[nonsequitor]

You're right, that was technically a hardware bug, but they fixed it with a software patch to change the bus timings. Not the optimal solution of course, but a board respin costs much more than a software update. It may not have been the best example, but the point remains, you can't prove the non-existence of bugs. Anyone thats trying to tell you otherwise is either a genius or more likely an idiot.

To answer your question, I was trying to illustrate that the metrics while a good starting point are merely metrics and only look for specific types of bugs which have frequent occurance and have been well quantified. Since that specific problem ended up categorized ultimately as a software bug, the metric being applied didn't have a chance in hell of finding it. And while new metrics could be put in place to find that sort of low occurance bug, the expense of development would increase accordingly. There then comes a point where the decision has to be made what metrics to use, and how much they cost to implement across the board. This of course cuts into profits which is why I said regulations like DO-178B are a starting point, but not proof that a device will operate flawlessly under all conditions it may be rated for. Its also the reason not all avionics equipment is certified at level A, the effort to obtain that certification is much more costly than the effort required to obtain a level C certification.

The goal of course is to find faster, cheaper ways, to meet these minimum requirements. Or if your field is not regulated, implement best practices which can improve overall quality at a minimum of cost. I made my initial post because while a lot of people who read slashdot are very technically adept, there may be some who might not realize 100% code coverage is not the same as bug free code, or like my example, a defect free product.