Slashdot Mirror
Are Regression Tests an Industry Standard?
Sludge asks: "I just finished leading a team through a software project. It was the first of it's type for our company: financial transactions were involved, and it was therefore very fault intolerant. In order to complete this, a set of regression tests were written. For example, if the amount of money collected doesn't match up to our order table, we get notified via our cellphone's text messaging as soon as the cronjob picks it up. Lots of other implementation-specific tests exist as well. My question is, how common is this for the software industry? My company had never heard of this before I came along. Is it the norm? (When you answer, also say whether or not your company does risk management.)"
In fact regression tests spotted not only implementation errors, but documentation errors when the semantics of a function changed, but the docs, and hence the regression tests didn't match.
Now, strictly speaking, what you describe is more of a sanity audit rather than a regression test, unless you provide test data to trigger the potential conditions you test for.
You could've hired me.
Not having automation is one thing. It's actually quite a bit of work to get a good regression test automated. Testers, many very green fresh out of school types, generally do not have the experience necessary to write good tests that can fit into a harness framework. SDETs have the experience, but are more expensive to hire and keep around.
If management knows about automation but no such system exists in the company, it's mainly a matter of money and tuits. If management doesn't know about automation at all, then you're dealing with very inexperienced leadership. Any book that deals with software testing, even the very simple treatment of the topic by Kaner, Falk, and Nguyen discusses test automation and regression testing.
I have been pwned because my
The last place I worked did extensive regression testing at all levels of software development. I was a member of a team developing telemetry firmware for communicating with implantable medical devices, so needless to say we needed to minimize any problems. Software 'bugs' in the field are not taken lightly.
You catch doc. errors this way because when the test reveals an error it means one of three things: (a) you coded the test wrong; (b) you found an implementation error; (c) the implementation is right, but you coded to the docs, which were wrong.
Now, if you use various 4GL tools (dunno if Rational Rose lets you do this), they might be able to automate the tests for you.
Good luck.
You could've hired me.
What you describe isn't regression testing. Regression testing "is a quality control measure to ensure that the newly-modified code still complies with its specified requirements and that unmodified code has not been affected by the maintenance activity." More accurately what you've done is paranoid programming. Really, these two things are orthogonal.
This depends. Every company I've worked for has claimed to be concerned with mitigating risks both in the testing phases and post-release phases of the software development lifecycle. However, the amount and kind of testing and programming actually done have varied wildly and always ends up being determined by the industry for which the software is being built. In your case, money is the biggest factor. Organizations such as banks and other financial institutions are highly risk-averse due to the responsibilities and legal concerns related to handling others' money. It follows that these organizations regularly conduct formal testing of their code as well as "program paranoid" to mitigate screw-ups. In start-up's I've worked at in the past, this wasn't nearly the case since it was more important to get a product out the door and this sort of testing/coding always went out the window with looming deadlines.
So to answer your question, yes, regression testing (and paranoid programming) are highly common in the IT industry and their respective importance is a function of the risk aversion of the intended users/customers. My advice is to always practice good, paranoid, professional programming augmented by formal testing procedures. Vary the time spent on each to achieve the appropriate balance.
Frankly, the best way to enlighten yourself on this matter is to educate yourself in the ways of Extreme Programming. The horribly trendy name aside, this is the truly the only management fad I've seen in 10+ years that holds any merit.
We hardly ever test anything before releasing it. So, regression testing? Customers are our regression tests, they are quite fast at finding all the stuff we've broken with the new release.
True warriors use the Klingon Google
I wrote some automated test programs in my last job.
:)
Used a very expensive tool (which we already had) to perform very simple regression testing on a new software package, and found more faults in an automated run over one weekend than we normally found in a two month period testing by hand.
Saved the company and customers untold amounts of money, and when the software went live we had at most 5% of the normal faults reported in that area. Customer was delighted.
For the next project I wanted to expand the tool to cover more of the functionality. Everyone appreciated the reasoning, and I asked for a month or two to develop it. (In a pinch I could have done it in a fortnight, but I'm an engineer, so I padded my estimate
The response? 'Do it the old way - our project timescales are too tight for this'.
Even a 'give me a fortnight and let me prove the concept' fell on stony ground. This DESPITE the fact I'd proved the concept already.
Shortly afterwards, I quit - this was the final straw for me.
All the wonderful automation, test tools and experienced testers count for nothing if management have an anus/cranium interface issue...
Liked this comment? Why not buy me something nice
I know this is not a commercial project in a large corporation, I thought I mention it anyway. In the GnuGo project we have a large set of simple test scripts, and all the time we keep track of which of them succeed, and which fail. There are always some failures, but we can live with that (we sort of have to). This test set has saved us a lot of time. It has also given valuable information on the effects of a new algorithm or piece of code, or what happens when you adjust some tunable parameters. No expensive tools used, just a bit extra coding to make all the necessary information available for the tests, and a few scripts...
In Murphy We Turst
Is also a great tool, not just risk management.
there are subtle differences between the two.
As someone else has already mentioned, the process you describe is not strictly speaking regression testing.
I have worked on quite a large number of software projects, and every single one included "testing". The level to which software is tested, however, varies widely. One project I worked on was a billing application which collected the entire company's annual revenue. Yep, we tested that one pretty rigorously....But I've also worked on web site projects where the downside of getting it wrong was not so severe; we tested almost as an afterthought.
There are a lot of test "gurus", and a bunch of different methodologies to provide a testing framework. Checkout testing.com to get a feel for this...
It all boils down to the decision how much time do you spend on testing versus other quality assurance methods. Testing is the most expensive and least effective way of finding bugs except for releasing the code to your customers. Practices such as specification, design and code reviews, design-by-contract, aspect-based programming give you far more bang for your buck.
FWIW, on the billing project, we had a formal specification review to make sure that the product we built did what the business needed, a business representative to help fill in the blanks in the specification, a design review to make sure that the software we intended to build was indeed what the spefication asked for, and made sense in its own right. We produced numerous prototypes and mock-ups to get our customers to tell us we were on the right or wrong track without having to learn to read software design documents.
During the code phase, we created unit and integration tests which measured the kinds of thing you mention (e.g. order total must equal sum of order lines), and had a dedicated test resource. We ran code reviews. We also made sure we showed the work in progress to our business sponsors as often as we could.
When we thought we were done, we had a formal show-n-tell to present our work to the business; this lead to a bunch of rework, which again was tested, reviewed etc.
The software was succesful from the business point of view; with hindsight, I'd say that the code was truely awful, and I wish we'd spent more time on code reviews. How important was testing in the QA process ? It provided a useful yardstick to tell us how close we were to meeting our objectives. Would I have relied on testing without all the other stuff - reviews, prototypes, great access to the business folk ? Hell no - if you don't know that what you're testing is what the customer wants, your tests are pretty much valueless.
So, I guess I distrust any organisation that over-emphasizes testing as a QA process - there are better ways of avoiding bugs. On the other hand, you have to provide the appropriate level of testing - if you're writing nuclear missile guidance systems, you need to allocate a lot more resources to testing than if you're building a website to hail your cats' achievements as politicians.
Ne
It's all very well in practice, but it will never work in theory.
Regression testing has been going on at my office since I joined, and for many years previously.
Under our Unix-based app, we use a terminal-emulator which supports scripting to send sequences of characters to the app to simulate normal use. Very easy, and very efficient.
We're currently in the process of trying out various Windows-based regression testing packages, to test our brand-new Windows-based app (which, sadly is due to replace the Unix app), but it's proving to be a much harder thing to do under Windows than under character-based terminals, because of the mouse-driven and event-driven nature of the environment.
We are starting to get to grips with the problem, but it has been a much bigger task than we expected. If a minor detail (eg size of an input field) changes in the Unix app, no changes are needed to the test-suite; under Windows, you have to keep a much tighter control on it.
(Spudley Strikes Again!)
In order to complete this, a set of regression tests were written.
That's not really regression testing. Regression testing is making sure something that used to work still works; this is done by running tests that were originally done to qualify the original software.
If you don't have any test results from the "before" system, you cannot compare them with test results from the "after" system. And without such a comparison, you are not regression testing.
The one example test you describe is more of an internal-consistency-error-handler check. These are important too, of course. Perhaps the most difficult part of internal-consistency-error-handler testing is that you have to cause an internal inconsistency to test the error-handling code.