Slashdot Mirror
Are Regression Tests an Industry Standard?
Sludge asks: "I just finished leading a team through a software project. It was the first of it's type for our company: financial transactions were involved, and it was therefore very fault intolerant. In order to complete this, a set of regression tests were written. For example, if the amount of money collected doesn't match up to our order table, we get notified via our cellphone's text messaging as soon as the cronjob picks it up. Lots of other implementation-specific tests exist as well. My question is, how common is this for the software industry? My company had never heard of this before I came along. Is it the norm? (When you answer, also say whether or not your company does risk management.)"
In fact regression tests spotted not only implementation errors, but documentation errors when the semantics of a function changed, but the docs, and hence the regression tests didn't match.
Now, strictly speaking, what you describe is more of a sanity audit rather than a regression test, unless you provide test data to trigger the potential conditions you test for.
You could've hired me.
Not having automation is one thing. It's actually quite a bit of work to get a good regression test automated. Testers, many very green fresh out of school types, generally do not have the experience necessary to write good tests that can fit into a harness framework. SDETs have the experience, but are more expensive to hire and keep around.
If management knows about automation but no such system exists in the company, it's mainly a matter of money and tuits. If management doesn't know about automation at all, then you're dealing with very inexperienced leadership. Any book that deals with software testing, even the very simple treatment of the topic by Kaner, Falk, and Nguyen discusses test automation and regression testing.
I have been pwned because my
You catch doc. errors this way because when the test reveals an error it means one of three things: (a) you coded the test wrong; (b) you found an implementation error; (c) the implementation is right, but you coded to the docs, which were wrong.
Now, if you use various 4GL tools (dunno if Rational Rose lets you do this), they might be able to automate the tests for you.
Good luck.
You could've hired me.
What you describe isn't regression testing. Regression testing "is a quality control measure to ensure that the newly-modified code still complies with its specified requirements and that unmodified code has not been affected by the maintenance activity." More accurately what you've done is paranoid programming. Really, these two things are orthogonal.
This depends. Every company I've worked for has claimed to be concerned with mitigating risks both in the testing phases and post-release phases of the software development lifecycle. However, the amount and kind of testing and programming actually done have varied wildly and always ends up being determined by the industry for which the software is being built. In your case, money is the biggest factor. Organizations such as banks and other financial institutions are highly risk-averse due to the responsibilities and legal concerns related to handling others' money. It follows that these organizations regularly conduct formal testing of their code as well as "program paranoid" to mitigate screw-ups. In start-up's I've worked at in the past, this wasn't nearly the case since it was more important to get a product out the door and this sort of testing/coding always went out the window with looming deadlines.
So to answer your question, yes, regression testing (and paranoid programming) are highly common in the IT industry and their respective importance is a function of the risk aversion of the intended users/customers. My advice is to always practice good, paranoid, professional programming augmented by formal testing procedures. Vary the time spent on each to achieve the appropriate balance.
Frankly, the best way to enlighten yourself on this matter is to educate yourself in the ways of Extreme Programming. The horribly trendy name aside, this is the truly the only management fad I've seen in 10+ years that holds any merit.
Regression testing has been going on at my office since I joined, and for many years previously.
Under our Unix-based app, we use a terminal-emulator which supports scripting to send sequences of characters to the app to simulate normal use. Very easy, and very efficient.
We're currently in the process of trying out various Windows-based regression testing packages, to test our brand-new Windows-based app (which, sadly is due to replace the Unix app), but it's proving to be a much harder thing to do under Windows than under character-based terminals, because of the mouse-driven and event-driven nature of the environment.
We are starting to get to grips with the problem, but it has been a much bigger task than we expected. If a minor detail (eg size of an input field) changes in the Unix app, no changes are needed to the test-suite; under Windows, you have to keep a much tighter control on it.
(Spudley Strikes Again!)