Suggestions for Performing Regression Testing?

gmletzkojr asks: "The company that I am currently working at develops a fairly complex industrial controller, complete with embedded software, a GUI on the controller, and a Windows app to connect over Ethernet. On previous versions of a similar project, we have performed testing manually - ie, monkey presses button, and sees that light turns on, widget turns, GUI updates, etc. However, this is extremely time consuming (previous complete regression testing took ~3 weeks) and is error prone in itself. How do you perform complete system regression testing? Do you use shrink-wrapped packages, or build your own? How do you test features that are easy for humans to observe, but not as easy for software to detect (ie, the light came on, the GUI updated when I pressed the external input, etc)?"

Comments

Well, I did regression for a big company...

[ForestGrump]

and their name starts with an M. Microsoft maybe? I don't remember. NDAs have a way for making me forget details.

Anyway, doing regression testing is a 2 edged sword.
1. You have to program more stuff into your program to allow for automation. Now, because you're adding another feature to the entire program, that means more testing to ensure that the added functionality actually works. Lastly, you have to hire/write regression scripts. How much time does it take to write scripts? How long will they be useful for? If it takes ~the same time/money to write scripts as to hire/be the monkey...How much is that worth?

2. It can be a big convenience if you can justify #1.

Grump

Separate IO from the guts of the system

[pfdietz]

Can you cleanly separate the internals of the system from the external interfaces? That way, interaction with the internals can be scripted without having to watch GUIs or blinking lights. You do have to test the part that drives the GUI/etc., but that is presumably easier than having to do all the system tests through it.

Re:Break down the problem and test components.

[koryn]

Question: How do you test features that are easy for humans to observe, but not as easy for software to detect (ie, the light came on, the GUI updated when I pressed the external input, etc)?

Answer: I work at a company with a 30+ year mainframe application that runs exclusively via line mode commands followed by several paragraphs that have nothing to do with the original question, but instead tell how to use (yawn) unit testing.

Relevancy: None.

If you are thinking of automated testing...

[Spoing]

Keep in mind that if the UI or backend changes on a regular basis, you will also be making substantial changes to the automated tests. Part of this can be delt with by a good tool automatically, though for most substantive changes or ones that change the workflow in even minor ways that will not be the case.

Also, people tend to think that automated testing takes less time...it *CAN* though expect that on many projects it will take much more time as automated tests are detailed and implementation specific; you can't create tests at the spec level unless your specs are detailed design documents too and even then only in a limited way.

The time savings kicks in when you want to frequently repeat the tests across the whole project when even minor changes are made to the code in one place. It also allows you to be somewhat certian that only the things you expect to change do indeed change.

If you do not have the budget or time to do complete manual tests, forget about autmating it unless you are dealing with a very static project that requires excessively detailed testing.

I expect people to disagree on much of what I wrote above...when they do, keep in mind that situations can differ. These are just general rules of thumb and worked for the vast majority of projects I've been on.

Testing physical devices

[dubl-u]

How do you test features that are easy for humans to observe, but not as easy for software to detect (ie, the light came on, the GUI updated when I pressed the external input, etc)?"

For the GUI, I recommend instrumenting your app so you can programatically tell what's going on. An API is one way, but a quick and dirty way is just to keep an internal event log and then probe that. Then for free, you get a detailed log you can dump if there's an error in production.

For the physical hardware, consider building a simulator. You could do it partly in hardware, adding simulation logic to your hardware controller, but running disconnected from the machinery. Or you could build another board that connects to your production board's inputs and outputs and simulates the machinery at an electrical level. Another option is to simulate your production board entirely, leaving the embedded code out of the testing loop.

The right choices depend a lot on where you can get the best bang for your test automation buck. Unfortunately, starting with a lot of untested legacy code means you have a long slog ahead of you. Start with the modules that generate the most bugs, or the bugs hardest to find during manual testing and automate your tests of those. That will teach you a lot about good ways to test in your environment.

GUI, hunh?

[Intron]

Reminds me of HP storage. They removed a scriptable, serial port CLI in favor of an IE-only interface. Guess which is harder to use and I imagine is a bitch to test. Layering the GUI on top of the CLI would have been my choice, as either a user or a tester.