Software Carpentry QMTest Testing Tool Released

soundsop writes: "The first tool resulting from the winners of a design competition by the Software Carpentry project has been released. The QMTest tool is a testing tool to replace software such as XUnit, Expect and DejaGnu. An issue tracking tool, called QMTrack (a la Bugzilla) is forthcoming. It looks like the winning design proposals for a config tool (autoconf replacement) and a build tool (make replacement) are not being implemented."

Comments

A Windows version, too!

[Tsar]

I checked the downloads page and found a Windows download for QMTest 1.0. Can't wait to try it out.

Re:What exactly does this do?

[Harumuka]

SC Test aims to produce regression testing software. This basically means that when software has new features added, regression testing tests to make sure it hasn't taken a step backwards (regressed). In plain english, regression testing tries to prevent new features from introducing bugs. A good introduction to regression testing is at AutomatedQA, although the software there is commerical their few words on regression testing are well worth reading.

This is what Webopedia has to say on the subject:

The selective retesting of a software system that has been modified to ensure that any bugs have been fixed and that no other previously-working functions have failed as a result of the reparations and that newly added features have not created problems with previous versions of the software. Also referred to as verification testing, regression testing is initiated after a programmer has attempted to fix a recognized problem or has added source code to a program that may have inadvertently introduced errors. It is a quality control measure to ensure that the newly-modified code still complies with its specified requirements and that unmodified code has not been affected by the maintenance activity.

Replacing Expect? It's not a testing tool.

[jdavidb]

Kudos to the QMTest folks; I'm looking forward
to the fruits of the software carpentry project. But, as Expect is an
automation tool, not simply a testing tool, I don't think it'll exactly be "replaced."
The well-known DejaGNU suite, written in Expect, might be a candidate for replacement, though.

I use Expect all the time, but have never used it for testing. I tried DejaGNU a time or two,
but never could figure it out. If you like Expect and like Perl, you might check out the
Expect.pm module project; it's really come to fruition recently, and
I've finally started doing that kind of thing in Perl instead of TCL/Expect.

XUnit still is more flexible

[prototype]

I think this is a nice product, but XUnit is still more flexible in regression and repetition testing. First off, this requires Python and an interface to Python from your application. That's not something I want to add to my system just to do testing. XUnit has basically been ported to every known language out there and is integrated into the languge. Also the way that you specify tests is maintained in a database. In a team development environment, you now have to have the db engine installed in everyone's workspace. While the db and python approach might offer a few more options maintenance wise, it makes things more complicated than they have to. XUnit has proven itself to be an excellent means of testing so if this is something new to you, try it out first before leaping into other technologies. No db or external language needed. Just my 2c worth on a quiet Christmas eve.

liB

an HTML user interface

[markj02]

I have written a lot of test cases in my time. I can't quite figure out why I would want to use QMTest.

First, I like writing test cases in a text editor, programmatically. It's tedious enough writing them in the first place, at least I can cut-and-paste and modify them quickly in an editor. Going through a web GUI does not seem like it's very efficient. Also, I don't particularly like using anything other than the implementation language and shell scripts for test cases; otherwise, people receiving the source code need to install additional tools. I also don't see anything in the white paper about support for the hard parts of testing, like configuration and compilation management for lots of extra C/C++ code, GUI testing, or web site testing (the latter usually require recording and playback).

Altogether, I'm not sure I ever felt I needed something like what QMTest seems to be doing. And the things that are actually difficult to test, it doesn't seem to provide useful support for. Can someone explain what I'm missing?

big design up front

[sohp]

So they've gathered a lot of requirements and done a lot design, but, as the lady says, "Where's the beef?" Why do all this work to replace open source tools that already exist? Why not take those tools and contribute back to them, or if the project owners don't like the contributions, take the things they like and build from there?

Perhaps this from their web page: "The Advanced Computing Laboratory at Los Alamos National Laboratory is providing $860,000 of funding for Software Carpentry, which is being administered by Code Sourcery, LLC." And this: CodeSourcery also provides training and strategic consulting services for companies considering the adoption of free or open source software.

Why use the MIT License? Why develop in Python? Why require that the submissions to the design competition not contain any source code? They require a language but not any source? Am I being too cynical in seeing how, after all this contributed design, coding, testing, etc is done, at taxpayer expense, Code Sourcery is now in a tremendous position as the sole-source solution for support and training to the shops that choose to use these tools? And to notice that choosing the MIT license allows them to take and wrap up all the source code into their products and not give back anything? These are questions that either are in the FAQ but not clearly answered, or not spoken about. Even the SC site itself has been retired and archived.

Thanks, I'll stick with XUnit, Bugzilla, cvs (and subversion when it's ready. For build config tools, well, if you do cross-platform C and C++, then autoconf or its successor, but that's just for one language and set of development requirements.

I'll be interested to see if anything widely used comes out of this exercise. So far, of the all the tools implementations promised for "Summer of 2001", we have QMTest 1.0. The rest? Late and unfinished.

Here's why

[devphil]

So they've gathered a lot of requirements and done a lot design, but, as the lady says, "Where's the beef?" Why do all this work to replace open source tools that already exist? Why not take those tools and contribute back to them, or if the project owners don't like the contributions, take the things they like and build from there?

The original Software Carpentry site had an excellent page explaining the reason for the whole project, but I can't find it. So I'll summarize and paraphrase from memory:

Because the original tools suck ass, that's why.

There was a great quote on the SC page, something to the effect that "when experienced *nix developers say that tools are easy to use, they really just mean that they've grown enough scar tissue that using those tools is no longer painful." I'm a long-time Old School *nix user, and I agree. Try teaching these "easy to use" tools to beginners and watch their faces contort.

Take make for example: we start with a syntax that seems custom-designed to fuck with the mind (so, rules have to be indented by one tab (why? don't ask why, they just have to be; probably because parsing was too slow back then without a leading marker character), so if you use eight spaces instead you'll get some useless error message, even though there's nothing visually apparently wrong.

Each command gets executed in a separate shell. That's always fun for beginners, and trips up experts too, until you build up enough scar tissue that ending all the commands with ";\" seems perfectly natural.

The POSIX specification for make requirements are unfortunately too weak to accomplish anything. An implementation of make must have some extensions in order to be useful. So we have a way to include other makefiles, but it's spelled differently for different versions. GNU make has some funky builtin pattern expanders, BSD make has some funky looping constructs -- and neither camp is willing to merge.

The SC project was going to replace all that crap with a tool of equal power that actually makes sense to a beginner, rather than looking like the dog's breakfast of features that make currently is.

(I, too, long for Subversions and the other Tigris projects to come to fruition.)