Slashdot Mirror

← Back to Stories (view on slashdot.org)

Suggestions for Performing Regression Testing?

Posted by Cliff on from the let's-verify-that-it-all-works-like-it-should dept.

gmletzkojr asks: "The company that I am currently working at develops a fairly complex industrial controller, complete with embedded software, a GUI on the controller, and a Windows app to connect over Ethernet. On previous versions of a similar project, we have performed testing manually - ie, monkey presses button, and sees that light turns on, widget turns, GUI updates, etc. However, this is extremely time consuming (previous complete regression testing took ~3 weeks) and is error prone in itself. How do you perform complete system regression testing? Do you use shrink-wrapped packages, or build your own? How do you test features that are easy for humans to observe, but not as easy for software to detect (ie, the light came on, the GUI updated when I pressed the external input, etc)?"

37 comments

  1. Well, I did regression for a big company... by ForestGrump · · Score: 2, Insightful

    and their name starts with an M. Microsoft maybe? I don't remember. NDAs have a way for making me forget details.

    Anyway, doing regression testing is a 2 edged sword.
    1. You have to program more stuff into your program to allow for automation. Now, because you're adding another feature to the entire program, that means more testing to ensure that the added functionality actually works. Lastly, you have to hire/write regression scripts. How much time does it take to write scripts? How long will they be useful for? If it takes ~the same time/money to write scripts as to hire/be the monkey...How much is that worth?

    2. It can be a big convenience if you can justify #1.

    Grump

    --
    Is it true that more people vote for the winner of American Idol, than vote for the president? -Ali G.
    1. Re:Well, I did regression for a big company... by HungWeiLo · · Score: 1

      We use c++ in our project, so the solution to your first problem is pretty simple.

      class Foo
      {
      friend class FooTester; ...
      };

      That's it. Now you can do whatever you want in your regression testing without molesting your existing code (having to "program more stuff into your program," as you say)

      --
      There are a huge number of yeast infections in this county. Probably because we're downriver from the bread factory.
  2. Break down the problem and test components. by noahbagels · · Score: 4, Informative

    I work at a company with a 30+ year mainframe application that runs exclusively via line mode commands. This software is wrapped by SOAP middleware that I largely wrote or migrated our old legacy binary middleware to. This SOAP middleware is called by Tomcat - our choice of Java Servlet Container serving up dynamic web / transactional content. The application is extremely complex and results of almost any command/query change on a minute-by-minute basis - based on approx 1GB of fresh weather and other data updated asynchronously with over 20 data feeds from worldwide reporting stations.

    How do we test this you ask?
    Break down the problem into:
    (a) components
    (b) component integrations (i.e. network/middleware checks)

    Automate tests for each and you're at least 80/90% of the way there. I.e. Unit test pieces such as a UI or Middleware, and also make unit tests that verify the connections between components.

    If you're unlucky enough to be in a totally mixed-up environment without any seperation between GUI, logic, back-end, etc... often lumped together as MVC, but I prefer just to say "component based" you may have to do some re-architecture to get to a better scriptable setup.

    Our automated tests (web environment, mind you) test for:

    1. Basic web server & Tomcat / Servlet function.
    2. Web/Tomcat Server -> Middleware connection (no actual request)
    3. Tomcat->Middleware->Legacy System connection.
    4. Tomcat->Middleware->Legacy System operations - i.e. actually do something.

    Given the complex and ever changing data we operate with, we can't test everything that only a human eyeball can spot. However, these tests are re-used for live system monitoring, regression testing, and load testing. They also can find many of the "brain dead" issues - such as a total system meltdown or specific function being broken by a code change without requiring a tester to test everything out.

    I strongly recommend going for automated testing as much as possible - even if you're only getting at the low hanging fruit. Breaking down the problem into managable chunks w/unit tests can also save serious hassle trying to figure out later why a tester got an error somewhere.

    1. Re:Break down the problem and test components. by koryn · · Score: 2, Insightful

      Question: How do you test features that are easy for humans to observe, but not as easy for software to detect (ie, the light came on, the GUI updated when I pressed the external input, etc)?

      Answer: I work at a company with a 30+ year mainframe application that runs exclusively via line mode commands followed by several paragraphs that have nothing to do with the original question, but instead tell how to use (yawn) unit testing.

      Relevancy: None.

    2. Re:Break down the problem and test components. by GagnierA · · Score: 1

      yeah, I was kinda wondering when he was going to relate or get on track with his answer too...it's kinda funny though that he would waste his time typing that hoping people would think he was smart, but half of intelligence is relevance and there was no clear examples of either there...hmmm...

    3. Re:Break down the problem and test components. by dubl-u · · Score: 1

      [...]followed by several paragraphs that have nothing to do with the original question, but instead tell how to use (yawn) unit testing.

      He could have connected the dots a little better, but he described a perfectly good approach. The poster asked How do you perform complete system regression testing? and a good way to do that is to break the system down.

      Note that the post you're griping about talked not just about unit testing, but integration testing. Both are important to getting an app like this well tested. If you have a system A->B->C then unit testing A, B, and C and then testing integrations A->B and B->C gets rid of most of your bugs.

      You do this because full end-to-end testing is much more expensive for the same level of quality than lower-level testing. In my experience it's still worth doing some end-to-end testing on a system with full unit and integration tests, but those aren't so much about catching interesting bugs as avoiding stupid mistakes and about verifying that what you've built is what the product manager intended, rather than a plauible but wrong interpretation of their requests.

    4. Re:Break down the problem and test components. by foandd · · Score: 1
      Hmmm... the ask /. question first laid out the problem in some detail, noted specifically that the current process was time consuming and error prone, then asked three questions.

      noahbagels' post laid out the process which is followed where he works, which addressed both the time consuming and error prone aspects as well as answering the first question. In noting specifically how they've gone about doing it, he answered the second question.

      You, of course, have tried (unsuccessfully I might add) to show yourself to be clever by noting that he didn't answer the third question. Even if that were so (which it is not), noahbagels' post would be quite relevant to the overall question as posted. However, he addressed the third question as well. Here, let me help you:

      Question: How do you test features that are easy for humans to observe, but not as easy for software to detect (ie, the light came on, the GUI updated when I pressed the external input, etc)?

      Answer: Given the complex and ever changing data we operate with, we can't test everything that only a human eyeball can spot.

      It may not be the answer you wanted to hear, but it was most assuredly relevant. So while it does appear that there's an idiot among us, it isn't noahbagels.

  3. Is there an Open Src ver of MS Test? by ivi · · Score: 1


    A few years ago, MS had a testing aid called
    - creatively enough (for MS, at least...)
    Microsoft Test.

    As you didn't even get it with MSDN Universal
    I never got the chance to evaluate or use it,
    but perhaps there is a counterpart in OSS...?

    I'd love to find one... TIA

    1. Re:Is there an Open Src ver of MS Test? by Ender_Stonebender · · Score: 2, Informative

      I think you mean MS Visual Test, which was (surprisingly) sold off to Rational Software. Rational has, I believe, in turn been sold to IBM. (At least, the first result of a Google search for "Rational Visual Test" is now an ibm.com page.) It was a Rational product when I first encountered it; I thought that it was pretty decent. However, it was based on Visual Basic; I would have preferred something based on C or C++ as those were the languages I was used to writing in at the time. There was a 30-day demo available; I'm sure with some digging you could turn up a copy of it. (Provided, of course, that you've got a Windows box sitting around; which is never a safe assumption on slashdot.)

      --Ender

      --
      Loose things are easy to lose. You're getting your hair cut. They're going there to see their aunt.
    2. Re:Is there an Open Src ver of MS Test? by NitsujTPU · · Score: 1

      Ahh.

      Judging by your UID, you are not new here... but let me tell you how times have changed. Microsoft now advertises on Slashdot!

      Yes my friend. The distribution of Linux users on this site is swiftly declining. As Slashdot becomes more mainstream, you can expect to see this trend continue.

  4. Mercury? by spacecowboy420 · · Score: 3, Informative

    Mercury has a decent suite:
    They have Quality Center:
    Which will help you write manual test scripts and create new test cases and do defect tracking.
    Quick Test Pro:
    Semi decent intergration into quality center so you write your test cases, then bust it out to code. You CAN just simply record a test and run it, but the lifetime of your script may vary. You need to build in tons of logic just to ensure the script can handle what it thrown at it and handle various error conditions.

    Caveats:
    If you like open source, this is not for you.
    If you don't like vbscript, you may find a difficult path to using other languages. There are options, but if you have a large automation team, a common language is useful. Vbscipt is sometimes the common denominator.

    If you ARE familiar with Load Runner or their performance tools, you'll love it.
    Even if you don't mind vbscript you may find their tool unweildy in the way it picks up objects.

    All in all, probably the BEST commercial offering - simply because a monkey can use it. There are good ones in opensource if you can code, i.e. "Push To Test" if you like java.

    ps: sorry there are no links, I would just google them anyway

    --
    ymmv
    1. Re:Mercury? by DukeLinux · · Score: 1

      We WinRunner and TestDirector for this at work. We built a WinRunner framework so that business analysts can basically "script" the test using excel data sheets. We run at night using TestDirector and review the results in the morning. There is a lot of up front work but once you get everything working it is pretty easy to maintain and use with minimal resource expense. Of course any major change in the application will break things. It is no panacea but we are having decent results.

    2. Re:Mercury? by lowmagnet · · Score: 2, Informative

      I have built testing frameworks in both WinRunner and QuickTest Pro, and I have to say the QTP framework was easier to write. Using the XLS-as-database structure built in to everything COM, we were able to write tests in Excel files, and do it in about 1,500 lines of code.

      Anything is simple if you break down each step into a window, object, action, input, output format. One thing we found that makes our lives somewhat easier is to forego the entirety of QTP except as an execution engine. We don't use actions in the QTP script except to call our test case, and we write the whole framework in a normal text editor, except when testing the engine itself. Our repository is stored in a database using descriptive object naming and a versioning scheme I came up with. It sits on the same SQL Server as TestDirector, too.

      There are some things I'd like to do better, but the way I built the framework, it should be easy to add these features in. We have already added several functional extensions to the regular object classes, and will continue to do so in the future.

      The same could be done in WinRunner, but the app doesn't support ActiveX and COM directly in code, choosing instead to use intermediary layers. For this reason we didn't go that route with WinRunner tests (We use both in-house) instead opting for direct coding of the engine. Then again, WinRunner is far from OO which makes life hell especially when talking to a VB/.Net appliacation under test.

      --
      Heute die Welt, morgen das Sonnensystem!
    3. Re:Mercury? by oogoliegoogolie · · Score: 1

      Yeah, my opinion on Mercury is mixed. One on hand test script development is a time consuming process that can involve nearly as much work as developing the product you are intending to test, especially when u are starting from the ground up. This isn't Mercury's fault, it's just the way it is, but Mercury will have you believe that all you do is install Winrunner, click the Record button and it records your test script as use your product, save the result and your work is done-all you have to do from now on is load your test script and click Run to test it. Rubbish!

      Anytime the product changes you much change you test scripts accordingly, although if they are properly written it may be just a couple object names or the physical description of GUI objects that need modification.

      On the other hand, it does remove the repetitiveness and tediousness out of regression and functional testing.

      At a price tag of close to $10,000 per licence, Winrunner is not for everyone.

  5. monkey presses button by Anonymous Coward · · Score: 0

    I just hope that you're mounting a scratch monkey during testing.

  6. Separate IO from the guts of the system by pfdietz · · Score: 3, Insightful

    Can you cleanly separate the internals of the system from the external interfaces? That way, interaction with the internals can be scripted without having to watch GUIs or blinking lights. You do have to test the part that drives the GUI/etc., but that is presumably easier than having to do all the system tests through it.

  7. A stopwatch? by Eyeball97 · · Score: 1

    I've seen it tried with a stopwatch... You are feeling sleepy... you are feeling sleepy...

  8. AutoIt? by Futurepower(R) · · Score: 4, Interesting


    Looks like the free AutoIt would help.

    Also see AutoIt Script home page.

  9. Much more importantly... by Anonymous Coward · · Score: 0

    Why does anyone call it regression testing? What is regressing? What is retrograding? Is your application regressing and if so, what the hell does that mean? Regressing into what??

    What is the point of using terms with self-manufactured meanings for things that are already well defined? Does it not suffice to say testing? Regression testing is a ridiculous name. Equally ridiculous are the people who perpetuate this stupidity.

    1. Re:Much more importantly... by Ithika · · Score: 2, Informative

      Regression testing is testing for regression... as a subset of general unit testing. It's the way you make sure that bugs you fixed yesterday don't reappear as a result of today's bigfixes. To ensure that you're always moving forwards, that you don't regress.

      --
      the layman's guide to computer science
    2. Re:Much more importantly... by SirSlud · · Score: 1

      Regressive testing checks to make sure that while you're adding or fixing existing functionality, that you're not breaking other functionality in the system.

      This can happen when developer A fixes a bug, and developer B either overwrites the change or assumes developer A made a mistake turns it back to the way it was.

      Regressive bugs are often the product of a failure in team communication or a weak development process; idealy you'd never have any regressive bugs, but this ain't a perfect world, and I ain't a perfect coder. ;)

      --
      "Old man yells at systemd"
  10. Good luck... by bluGill · · Score: 4, Informative

    Good luck. I have found that regression tests often cost just as much as just paying someone to do the test. The only time they are an advantage is when you run them against the daily builds. The problem is when you do that every day one test will fail because of a new feature, and the test now needs to be changed, not a bug written up. (Of course you will catch bugs too) So you have to have a full time programmer who only job is to keep the regression tests up to date. This can be more expensive than just hiring testers. (Kids are cheap and most of them can operate a computer well enough to do your testing if you keep track of them)

    Don't read too much into that downside. The more expensive programmer will in the end catch more bugs with the regression test suite, so your code has less bugs when you ship. However it is expensive.

    Automatic regression tests are NOT a substitute for manual regression testing! You still need to plan on some time (though not the 3 weeks you did before) for regression testing, just to make sure the automatic tests didn't miss anything. Much of this will be done as part of your normal tests, but make sure you are watching for regressions when you test.

    1. Re:Good luck... by dubl-u · · Score: 2, Interesting

      The more expensive programmer will in the end catch more bugs with the regression test suite, so your code has less bugs when you ship. However it is expensive.

      I think this can be true if you write your regression tests afterward. But if you're doing test-driven development, so that your app is designed from the beginning for testability, then I think the costs are pretty reasonable. In my experience, the time testing is more than made up for by the time and stress saved on debugging.

      On a recent 9-month project, a team of four developers I know built a web application with no external testing, just self-written automated tests and product managers playing with the app. After six months in production, they've had a total of two reported bugs and no scheduled downtime.

    2. Re:Good luck... by speedplane · · Score: 1

      I agree with you that testing is extremly expensive. At Microsoft only about half of their full-time programmers are contributing new code to products. The rest (literally thousands of programmers) are writing automated test code to make sure everything is working properly.

      --
      Fast Federal Court and I.T.C. updates
  11. If you are thinking of automated testing... by Spoing · · Score: 2, Insightful

    Keep in mind that if the UI or backend changes on a regular basis, you will also be making substantial changes to the automated tests. Part of this can be delt with by a good tool automatically, though for most substantive changes or ones that change the workflow in even minor ways that will not be the case.

    Also, people tend to think that automated testing takes less time...it *CAN* though expect that on many projects it will take much more time as automated tests are detailed and implementation specific; you can't create tests at the spec level unless your specs are detailed design documents too and even then only in a limited way.

    The time savings kicks in when you want to frequently repeat the tests across the whole project when even minor changes are made to the code in one place. It also allows you to be somewhat certian that only the things you expect to change do indeed change.

    If you do not have the budget or time to do complete manual tests, forget about autmating it unless you are dealing with a very static project that requires excessively detailed testing.

    I expect people to disagree on much of what I wrote above...when they do, keep in mind that situations can differ. These are just general rules of thumb and worked for the vast majority of projects I've been on.

    --
    A firewall can not protect you from yourself. Turn off what you do not need. Do not use the firewall to do your work.
    1. Re:If you are thinking of automated testing... by k3str3l · · Score: 1
      Well put.

      I am a software test engineer, I lead a small software test team, and am a huge advocate for automation when it is the best solution to the problem.

      James Bach, a leading test automation consultant, wrote an excellent article[PDF] debunking some of the reckless assumtions folks make about test automation.

      Generally speaking (and this is VERY context-dependent) I would like to have both skilled manual regression testing and regression suites written by skilled automation engineers, but if forced to pick one or the other, I will advocate for manual testing almost every time. Test automation excels in concert with manual testing, not in place of it.

      --
      There is just no way that you are the pine-scented air.
  12. Testing physical devices by dubl-u · · Score: 5, Insightful

    How do you test features that are easy for humans to observe, but not as easy for software to detect (ie, the light came on, the GUI updated when I pressed the external input, etc)?"

    For the GUI, I recommend instrumenting your app so you can programatically tell what's going on. An API is one way, but a quick and dirty way is just to keep an internal event log and then probe that. Then for free, you get a detailed log you can dump if there's an error in production.

    For the physical hardware, consider building a simulator. You could do it partly in hardware, adding simulation logic to your hardware controller, but running disconnected from the machinery. Or you could build another board that connects to your production board's inputs and outputs and simulates the machinery at an electrical level. Another option is to simulate your production board entirely, leaving the embedded code out of the testing loop.

    The right choices depend a lot on where you can get the best bang for your test automation buck. Unfortunately, starting with a lot of untested legacy code means you have a long slog ahead of you. Start with the modules that generate the most bugs, or the bugs hardest to find during manual testing and automate your tests of those. That will teach you a lot about good ways to test in your environment.

    1. Re:Testing physical devices by sonamchauhan · · Score: 1

      Hmm, but that's not really testing the GUI directly. Sometime back I read about an IBM tool that actually does OCR on a predefined section of the screen to verify the functionality of the user interface.

      There are also tools that query the actual components rendered on a window - sort of like the Spy++ tool included in Visual C++. This is the approach taken in the book "Effective GUI Test Automation" (ISBN 0-7821-4351-2)

      With cheap webcams, open-source software for detecting/comparing changes in images (similar to that used in surveillance software), and open-source computer vison libraries like Intel OpenCV, it may be possible to develop something that actually verifies the entire system - software and hardware - that the Ask Slashdot question mentioned. i.e. something that actually checks if a light comes on, etc.

    2. Re:Testing physical devices by dubl-u · · Score: 1

      Hmm, but that's not really testing the GUI directly.

      This is true. If you're going for theoretical purity, that might be necessary. In a commercial effort, though, you're more interested in bang for the buck.

      If you instrument your controller code (which is often easiest) then you catch all bugs except for GUI toolkit ones and in connecting the GUI library to your app, which is, I understand, not a big source of bugs, as it's simple stuff that programmers tend to hand-test pretty well.

      If you go further and instrument the GUI components, then you catch everything except bugs in the GUI components themselves. If that's a big source of bugs, the first step is to unit-test your GUI components.

      To take it further still, you can do screen capture and analysis. Bugs from there would be in low-level drivers and in your display hardware. Only if you're getting bugs in that would you need to do automated web-cam stuff.

      Either screen-capture or artificial vision stuff would be a huge effort. To make useful tests, you'll need to come up with the same model as the GUI toolkit, but soley by looking at output. I'd estimate this as 2-5x the work of writing those GUI components from scratch. If you GUI library really sucks that much, automated vision isn't the solution; you should just spend a year or two writing a reliable one.

      And even if you do build the automated vision stuff (which should include faking the mouse and keyboard, natch), it will test functionality, not aesthetics. And it will only test the things you specify; it won't notice other things wrong. For truly complete end-to-end tests, you need an artificial human, with skills equivalent to your graphic designer, your product manager, your programmers, and your QA people. Those are really expensive to build.

      Which is why it's the advice of people who do a lot of test-driven development with GUI apps to do your end-to-end tests at a level just below the GUI layer, with a very thin layer on top. That way you test 99% of your code for a lot less effort then testing the whole thing. Then you catch the rest of the bugs through casual use and either frequent releases or manual QA.

      It may be possible to develop something that actually verifies the entire system - software and hardware - that the Ask Slashdot question mentioned.

      Sure! And if we're talking over beers, I'm happy to fill a whole stack of napkins with sketches for this sort of thing. But the guy has a practical problem in a commercial environment, so I pointed him at the practical, cost-effective approaches for bug reduction.

    3. Re:Testing physical devices by sonamchauhan · · Score: 1

      Yes, I agree that testing is a 'maximum bang for buck' engineering discipline.

      I suggested the webcam for the physical blinking lights stuff the poster mentioned - this should be possible even today with open-source surveillance software I mentioned in passing (set zone, and the software continually analyse for image differences). For the GUI, screen capture would be more efficient.

      True, creating a testing tool that did screen capture would be a hassle. However, IBM already Also, most informed people also suggest what you did about the GUI testing (which the poster may know about already).

      However, IBM already has a product that does OCR on screen captures of mainframe applications.

  13. Mercury Interactive Toolsets by a9db0 · · Score: 1

    If you're not averse to proprietary software...

    Mercury Interactive makes a great set of tools for regression and load testing. If the app is Windows based, WinRunner or QuickTest will allow you to script end-user actions for repetitive testing scenarios. LoadRunner is a great way to test a back end system, not just for load but also regression. And, if you're not useing a bug-tracking mechanism already, TestDirector is a nice web based tool for bug tracking that integrates well with the other tools.

    I used these extensively for several years at a major entertainment company in Orlando to do testing, and they just flat work.

    --
    -- "Never underestimate the power of human stupidity." - R.A.H.
  14. For Java Swing, try QFTest by Webcommando · · Score: 2, Informative

    I've used a product called QFTest http://www.qfs.de/en/qftestJUI/ for performing long-term stress testing and functional testing on Java Swing applications.

    It is similar to many of the other products that do automated GUI testing, but will run on Linux and Windows (others are very Windows centric). It is also is scriptable using Jython.

    Doesn't relate directly to your needs but I thought others working with Java might find this information useful.

    --
    I love the sound of distortion in the morning -- webcommando
  15. JUnit by cuteseal · · Score: 1

    My previous project used JUnit very successfully, and had it integrated into their nightly build process. At the end of each day, all the developers checked in their modules and found out via the regression test reports the next morning, whether their changes caused someone else's module to blow up! :D

    --

    The friendliest digital photography forums on the net!

    1. Re:JUnit by k3str3l · · Score: 1
      I am a fan of JUnit (and the xUnit family) as well. It's a fabulous toolset for unit testing...which complements but is not the same thing as regression testing.

      A good unit test suite will catch many errors which might break existing functionality. Regression testing on the other hand tests changes after integration, tends to be more GUI-driven and user-oriented, is often best done manually, and is likely to find a somewhat different set of bugs than unit testing will.

      --
      There is just no way that you are the pine-scented air.
  16. GUI, hunh? by Intron · · Score: 2, Insightful

    Reminds me of HP storage. They removed a scriptable, serial port CLI in favor of an IE-only interface. Guess which is harder to use and I imagine is a bitch to test. Layering the GUI on top of the CLI would have been my choice, as either a user or a tester.

    --
    Intron: the portion of DNA which expresses nothing useful.
  17. Rational TestStudio by m1cajah · · Score: 1

    I've used Rational TestStudio and EnterpriseStudio for doing just this sort of thing.

    Other than having to give up your left testicle to afford it (I volunteered my boss for it), it proved itself quite capable of handling unit, integration, system, regression, and full-bore load testing. If you're also planning to do any load testing the "virtual users" will cost you the other testicle (I'd suggest you use multiple levels of management's PHBs - they get skittish after a couple of roosters become hens).

    I'd recommend "floating licenses" over "node-locked" licenses if your testers are geographically dispersed.

    On the upside; it's extremely easy to use, no problem integrating manual tests into the automated test suites, no problems integrating datapools from Excel spreadsheets (well, any CSV/TSV spreadsheet really). Our only problem arose with Java testing...none of us had any experience with Java other than simple javascripts for menus, so I can't honestly provide you with a review on its performance there.

    I was part of the team that made the selection of the test suite for our company and customer.

    We evaluated Rational, Mercury, and a few others I can't recall. We found Rational was easier to implement and integrate with our requirements management tool (which just happened to be Rational's RequisitePro). None of us - testers, tech writers, analysts, and management found the Mercury interface as easy to work with. I've seen Mercury put to use in quite a few places and all seemed to get the job done. I don't think you'll be poorly served by either.

    YMMV

    So, in the end (1) find one that can handle the specifics of your project (Mercury handles ActiveX better), (2) fits your budget, and (3) one who's interface doesn't make your work more difficult because you're having to constantly find things or look up how to perform them.

    Now, if load testing HTML only is a desire, you may want to evaluate OpenSTA (www.opensta.org). It's free (in both respects) and a breeze to implement and use.

    Hope this helps.

  18. Different kind of regressions by gourkha · · Score: 1

    Concerning tools I'd only used compuware QA suite a lot of time ago. Was not so bad even if things that it's exactly cheap. In the company where I work today regression testing are performed using internal script (shell or other funky languages). I think each case should be considered differently... My understanding is that none can test WEB application using same methodology used to test matlab like application.

    I problem we have today it's a conflict between R&D testsuites (corner cases, basic functionalities) and validation testsuites.
    Sure the validation testsuites simulate the real world behaviour but we can't afford all this wasting of time every time we have to checkin.
    At the end regression tests are there to improve productivity.

    --
    gourkha