Suggestions for Performing Regression Testing?

gmletzkojr asks: "The company that I am currently working at develops a fairly complex industrial controller, complete with embedded software, a GUI on the controller, and a Windows app to connect over Ethernet. On previous versions of a similar project, we have performed testing manually - ie, monkey presses button, and sees that light turns on, widget turns, GUI updates, etc. However, this is extremely time consuming (previous complete regression testing took ~3 weeks) and is error prone in itself. How do you perform complete system regression testing? Do you use shrink-wrapped packages, or build your own? How do you test features that are easy for humans to observe, but not as easy for software to detect (ie, the light came on, the GUI updated when I pressed the external input, etc)?"

Comments

Break down the problem and test components.

[noahbagels]

I work at a company with a 30+ year mainframe application that runs exclusively via line mode commands. This software is wrapped by SOAP middleware that I largely wrote or migrated our old legacy binary middleware to. This SOAP middleware is called by Tomcat - our choice of Java Servlet Container serving up dynamic web / transactional content. The application is extremely complex and results of almost any command/query change on a minute-by-minute basis - based on approx 1GB of fresh weather and other data updated asynchronously with over 20 data feeds from worldwide reporting stations.

How do we test this you ask?
Break down the problem into:
(a) components
(b) component integrations (i.e. network/middleware checks)

Automate tests for each and you're at least 80/90% of the way there. I.e. Unit test pieces such as a UI or Middleware, and also make unit tests that verify the connections between components.

If you're unlucky enough to be in a totally mixed-up environment without any seperation between GUI, logic, back-end, etc... often lumped together as MVC, but I prefer just to say "component based" you may have to do some re-architecture to get to a better scriptable setup.

Our automated tests (web environment, mind you) test for:

1. Basic web server & Tomcat / Servlet function.
2. Web/Tomcat Server -> Middleware connection (no actual request)
3. Tomcat->Middleware->Legacy System connection.
4. Tomcat->Middleware->Legacy System operations - i.e. actually do something.

Given the complex and ever changing data we operate with, we can't test everything that only a human eyeball can spot. However, these tests are re-used for live system monitoring, regression testing, and load testing. They also can find many of the "brain dead" issues - such as a total system meltdown or specific function being broken by a code change without requiring a tester to test everything out.

I strongly recommend going for automated testing as much as possible - even if you're only getting at the low hanging fruit. Breaking down the problem into managable chunks w/unit tests can also save serious hassle trying to figure out later why a tester got an error somewhere.

Mercury?

[spacecowboy420]

Mercury has a decent suite:
They have Quality Center:
Which will help you write manual test scripts and create new test cases and do defect tracking.
Quick Test Pro:
Semi decent intergration into quality center so you write your test cases, then bust it out to code. You CAN just simply record a test and run it, but the lifetime of your script may vary. You need to build in tons of logic just to ensure the script can handle what it thrown at it and handle various error conditions.

Caveats:
If you like open source, this is not for you.
If you don't like vbscript, you may find a difficult path to using other languages. There are options, but if you have a large automation team, a common language is useful. Vbscipt is sometimes the common denominator.

If you ARE familiar with Load Runner or their performance tools, you'll love it.
Even if you don't mind vbscript you may find their tool unweildy in the way it picks up objects.

All in all, probably the BEST commercial offering - simply because a monkey can use it. There are good ones in opensource if you can code, i.e. "Push To Test" if you like java.

ps: sorry there are no links, I would just google them anyway

Re:Mercury?

[lowmagnet]

I have built testing frameworks in both WinRunner and QuickTest Pro, and I have to say the QTP framework was easier to write. Using the XLS-as-database structure built in to everything COM, we were able to write tests in Excel files, and do it in about 1,500 lines of code.

Anything is simple if you break down each step into a window, object, action, input, output format. One thing we found that makes our lives somewhat easier is to forego the entirety of QTP except as an execution engine. We don't use actions in the QTP script except to call our test case, and we write the whole framework in a normal text editor, except when testing the engine itself. Our repository is stored in a database using descriptive object naming and a versioning scheme I came up with. It sits on the same SQL Server as TestDirector, too.

There are some things I'd like to do better, but the way I built the framework, it should be easy to add these features in. We have already added several functional extensions to the regular object classes, and will continue to do so in the future.

The same could be done in WinRunner, but the app doesn't support ActiveX and COM directly in code, choosing instead to use intermediary layers. For this reason we didn't go that route with WinRunner tests (We use both in-house) instead opting for direct coding of the engine. Then again, WinRunner is far from OO which makes life hell especially when talking to a VB/.Net appliacation under test.

Re:Is there an Open Src ver of MS Test?

[Ender_Stonebender]

I think you mean MS Visual Test, which was (surprisingly) sold off to Rational Software. Rational has, I believe, in turn been sold to IBM. (At least, the first result of a Google search for "Rational Visual Test" is now an ibm.com page.) It was a Rational product when I first encountered it; I thought that it was pretty decent. However, it was based on Visual Basic; I would have preferred something based on C or C++ as those were the languages I was used to writing in at the time. There was a 30-day demo available; I'm sure with some digging you could turn up a copy of it. (Provided, of course, that you've got a Windows box sitting around; which is never a safe assumption on slashdot.)

--Ender

Re:Much more importantly...

[Ithika]

Regression testing is testing for regression... as a subset of general unit testing. It's the way you make sure that bugs you fixed yesterday don't reappear as a result of today's bigfixes. To ensure that you're always moving forwards, that you don't regress.

Good luck...

[bluGill]

Good luck. I have found that regression tests often cost just as much as just paying someone to do the test. The only time they are an advantage is when you run them against the daily builds. The problem is when you do that every day one test will fail because of a new feature, and the test now needs to be changed, not a bug written up. (Of course you will catch bugs too) So you have to have a full time programmer who only job is to keep the regression tests up to date. This can be more expensive than just hiring testers. (Kids are cheap and most of them can operate a computer well enough to do your testing if you keep track of them)

Don't read too much into that downside. The more expensive programmer will in the end catch more bugs with the regression test suite, so your code has less bugs when you ship. However it is expensive.

Automatic regression tests are NOT a substitute for manual regression testing! You still need to plan on some time (though not the 3 weeks you did before) for regression testing, just to make sure the automatic tests didn't miss anything. Much of this will be done as part of your normal tests, but make sure you are watching for regressions when you test.

For Java Swing, try QFTest

[Webcommando]

I've used a product called QFTest http://www.qfs.de/en/qftestJUI/ for performing long-term stress testing and functional testing on Java Swing applications.

It is similar to many of the other products that do automated GUI testing, but will run on Linux and Windows (others are very Windows centric). It is also is scriptable using Jython.

Doesn't relate directly to your needs but I thought others working with Java might find this information useful.