Seeking Input for Software Verification Policies?

e8johan asks: "I currently work at a company in the automotive industry. It is importat to ensure quality and for our products we use extensive testing before, during and after the assembly. Now I have been asked to help in the creation of a company instuction for ensuring that all 'in-house' developed software works properly. I have written a short summary describing the essentials such as: "do not ignore compiler warnings", "always describe what the code is intended to do in your comments", "do not assume that all users run in the same environment", etc. I now feel that I need more input, has anyone done anything like this before? How do you test your software?"

Comments

Make the first lines in ALL of your procedures...

[hackwrench]

code to check for bounds. Too many programs don't, just figuring that the values of a byte, integer, long int are sufficient, and leaving it up to other procedures or libraries to check.

Yikes..

[Tom7]

Sounds like you are not going far enough... lives are at stake if this is code for a car, and handing a bunch of C hackers some guidelines is not going to cut it. In general, I think that when you require absolute correctness, software systems are just too complex to try to tackle by brute mindpower. I really believe that you need to develop automated systems for checking that code is correct -- or instance, I understand that BMW does model checking on their code to make sure that bad things can't happen (ie, deadlock).

Some recommendations

[Read The Fine Manual]

(I work in an academic context, so take the following with a grain of salt.)

First you should understand the difference between testing and verification. With testing, you rule out a certain number of bugs, but you can never guarantee the absence of bugs. The latter is only possible with software verification, where you prove that the software contains no bugs and fulfills its specification.

Unfortunately, software verification is very expensive. Also, software verification is still a subject of research, and AFAIK it has not been tried for large software projects so far. (My current research project aims at verifying a complete operating-system kernel -- check out the VFiasco project.)

So in most cases, verification is too expensive, and you need to ensure robustness by other means like a good programming standard and testing. (I believe that is what you are looking for, despite the title of your question including the word ``verification.'')

If your shop uses C, I recommend the following book: Les Hatton, Safer C . It explains what a good programming standard should contain, and advocates automatic enforcability. (It also explains what the standard should not contain -- in particular, issues of style such as indentation and variable naming.) One example of such a programming standard for C are the Guidelines for the Use of the C Language in Vehicle Based Software, which should be particularly relevant to you as you also work in the automotive industry.

Testing is another important aspect. In safety-related environments such as yours, a high test coverage for your code is desirable. This requirements suggests automated testing on the unit level. Automated unit testing is practical only if the code is designed to be testable, and there are a number of strategies which provide this property, such as avoiding dependency cycles (for C and C++, John Lakos' Large-Scale C++ Software Design still is the ultimate guide), or creating the unit test before the program (the Extreme Programming way). I also liked Testing Object-Oriented Systems: Models, Patterns, and Tools by Robert Binder - a huge and thorough guide to all imaginable kinds of testing, especially for object-oriented systems.

Read *this* book: The Pragmatic Programmer

[cnvogel]

Hi,

you might want to have a look at this book:

http://books.slashdot.org/article.pl?sid=00/02/15/ 1912222

http://www.pragmaticprogrammer.com/

"The Pragmatic Programmer". They have a chapter that outlines the general principles of testing your software... And all the other chapters are very well worth their money!

Comments considered harmful

[MarkusQ]

"always describe what the code is intended to do in your comments",

This can hinder the finding of bugs. Too often I have seen people read the comments, say "oh, that's what it does" and go on without actually looking at the code.

A much better idea: write clear code, with good variable names, well abstracted and well named procedures / functions / methods, so that the code itself is the "documentation."

Save the comments for out-of-band information like citations, etc.

-- MarkusQ

Seperate test from development

[Spoing]

Project leads must make sure that specifications are made before coding starts.

Test should primarily (but not entirely) be concerned with how the systems should work -- and they need to verify that the specifications are complete and usable. This means bringing test (tester/VV&T) on before coding starts.

Developers must develop to those specifications.

If something changes, the PM should OK a specification change. No unilatteral code changes. Keep in mind that this does not eliminate creativity, it does mean that if some new feature is added it gets doccumented and tested. A bad PM will not be flexible on this point. A bad developer will ignore the process. One can lead to the other.

Initially, developers will not like it -- and some parts they will never like (the extra paper work, someone finding stupid errors constantly...). In the end, few will hate the idea and most will like the benifits...even if it is annoying at times.

Can there be exceptions to any of this? There will be -- but keep the exceptions rare. A good ballence is important.