Are Regression Tests an Industry Standard?

Sludge asks: "I just finished leading a team through a software project. It was the first of it's type for our company: financial transactions were involved, and it was therefore very fault intolerant. In order to complete this, a set of regression tests were written. For example, if the amount of money collected doesn't match up to our order table, we get notified via our cellphone's text messaging as soon as the cronjob picks it up. Lots of other implementation-specific tests exist as well. My question is, how common is this for the software industry? My company had never heard of this before I came along. Is it the norm? (When you answer, also say whether or not your company does risk management.)"

Comments

Testing is very common

[PinglePongle]

As someone else has already mentioned, the process you describe is not strictly speaking regression testing.

I have worked on quite a large number of software projects, and every single one included "testing". The level to which software is tested, however, varies widely. One project I worked on was a billing application which collected the entire company's annual revenue. Yep, we tested that one pretty rigorously....But I've also worked on web site projects where the downside of getting it wrong was not so severe; we tested almost as an afterthought.

There are a lot of test "gurus", and a bunch of different methodologies to provide a testing framework. Checkout testing.com to get a feel for this...

It all boils down to the decision how much time do you spend on testing versus other quality assurance methods. Testing is the most expensive and least effective way of finding bugs except for releasing the code to your customers. Practices such as specification, design and code reviews, design-by-contract, aspect-based programming give you far more bang for your buck.

FWIW, on the billing project, we had a formal specification review to make sure that the product we built did what the business needed, a business representative to help fill in the blanks in the specification, a design review to make sure that the software we intended to build was indeed what the spefication asked for, and made sense in its own right. We produced numerous prototypes and mock-ups to get our customers to tell us we were on the right or wrong track without having to learn to read software design documents.
During the code phase, we created unit and integration tests which measured the kinds of thing you mention (e.g. order total must equal sum of order lines), and had a dedicated test resource. We ran code reviews. We also made sure we showed the work in progress to our business sponsors as often as we could.
When we thought we were done, we had a formal show-n-tell to present our work to the business; this lead to a bunch of rework, which again was tested, reviewed etc.

The software was succesful from the business point of view; with hindsight, I'd say that the code was truely awful, and I wish we'd spent more time on code reviews. How important was testing in the QA process ? It provided a useful yardstick to tell us how close we were to meeting our objectives. Would I have relied on testing without all the other stuff - reviews, prototypes, great access to the business folk ? Hell no - if you don't know that what you're testing is what the customer wants, your tests are pretty much valueless.

So, I guess I distrust any organisation that over-emphasizes testing as a QA process - there are better ways of avoiding bugs. On the other hand, you have to provide the appropriate level of testing - if you're writing nuclear missile guidance systems, you need to allocate a lot more resources to testing than if you're building a website to hail your cats' achievements as politicians.

Ne