Slashdot Mirror
Software Carpentry QMTest Testing Tool Released
soundsop writes: "The first tool resulting from the winners of a design competition by the Software Carpentry project has been released. The QMTest tool is a testing tool to replace software such as XUnit, Expect and DejaGnu. An issue tracking tool, called QMTrack (a la Bugzilla) is forthcoming. It looks like the winning design proposals for a config tool (autoconf replacement) and a build tool (make replacement) are not being implemented."
I'm not exactly a developer, so could someone please explain to me what this software is meant to do? I take it that it is meant to help work out bugs... but how?
Remember, there were no nuclear weapons before women were allowed to vote.
I checked the downloads page and found a Windows download for QMTest 1.0. Can't wait to try it out.
Glad to see SC is making progress. Another one of my favorite SC spin offs is Quilt, a replacement for the dated make. Additionally, SCons is a similar program which won the SC build competition in August 2000. Although I personally haven't used Quilt nor SCons, they appear to be well-designed pieces of software. Hopefully Software Carpentry will act as a catalyst allowing the creative juices of progammers around the world to create the most well-designed software possible. So far, I'd say it succeeded.
What do you think of MusicCity now?
Merry Xmas to everyone. (offtopic ? who cares :)
...but a tool called SCons has just been released. It is based on the ScCons tool that was in the Software Carpentry Contest, and is written by the same folk.
Stand Fast,
tjg.
You sick wierdos who engage in this
extremely dubious practice of defaming
cmdrTACO must be al-qaeda sympathizers.
It's easy to be armchair anything.
It's easy to lament that something is not implemented.
But do you know that to really implement something _properly_ it takes real commitment - and not only sacrifies in money, time and effort ?
That is why I have utmost respect for those who participated in open-source projects.
Muchas Gracias, Señor Edward Snowden !
Kudos to the QMTest folks; I'm looking forward
to the fruits of the software carpentry project. But, as Expect is an
automation tool, not simply a testing tool, I don't think it'll exactly be "replaced."
The well-known DejaGNU suite, written in Expect, might be a candidate for replacement, though.
I use Expect all the time, but have never used it for testing. I tried DejaGNU a time or two,
but never could figure it out. If you like Expect and like Perl, you might check out the
Expect.pm module project; it's really come to fruition recently, and
I've finally started doing that kind of thing in Perl instead of TCL/Expect.
Secession is the right of all sentient beings.
I think this is a nice product, but XUnit is still more flexible in regression and repetition testing. First off, this requires Python and an interface to Python from your application. That's not something I want to add to my system just to do testing. XUnit has basically been ported to every known language out there and is integrated into the languge. Also the way that you specify tests is maintained in a database. In a team development environment, you now have to have the db engine installed in everyone's workspace. While the db and python approach might offer a few more options maintenance wise, it makes things more complicated than they have to. XUnit has proven itself to be an excellent means of testing so if this is something new to you, try it out first before leaping into other technologies. No db or external language needed. Just my 2c worth on a quiet Christmas eve.
liB
lol!
I'm not really familiar with software test suites, but the examples on the QMTest page is kind of like testing hardware; an input is given and an output is expected and thus tested. But is this enough for software development?
Is there any way to test other software scenarioes (ie GUI applications like page rengering, user interface, program response, program flow logic)? It seems that the most bugs often arise from an error in the program logic. Does the "real world" test programs other than having many people use the program and filling bug reports?
I have already downloaded both the linux and windows versions of the software. Not bad for an initial release. Although, this version does not seem to support Python 2.2. Thought this might be useful. May save some folks some time anyway. After taking a brief look at the tool. It shows some definite promise. The documentation is both clear and useful. I was up and running in under 10 minutes after I figured out the Python 2.2 incompatability issue. Any word on planned features for the next release?
I recently tried JUnit which is pretty simple. JUnit uses a java gui for running tests although the tests must be hard coded into test classes (its actually pretty easy). Looking at QMTest, it seems that the test scenarios can be modified using the web gui. Cool!
First, I like writing test cases in a text editor, programmatically. It's tedious enough writing them in the first place, at least I can cut-and-paste and modify them quickly in an editor. Going through a web GUI does not seem like it's very efficient. Also, I don't particularly like using anything other than the implementation language and shell scripts for test cases; otherwise, people receiving the source code need to install additional tools. I also don't see anything in the white paper about support for the hard parts of testing, like configuration and compilation management for lots of extra C/C++ code, GUI testing, or web site testing (the latter usually require recording and playback).
Altogether, I'm not sure I ever felt I needed something like what QMTest seems to be doing. And the things that are actually difficult to test, it doesn't seem to provide useful support for. Can someone explain what I'm missing?
Haha. When you categorized the trolls, you basically categorized every single message that is ever posted on /.
As an XP developer I take a serious interest in unit testing, so I browsed the site - claims claims claims, but no details... I perused their PDF white paper - more claims and absolutely no specifics.
If they can't explain exactly what you get with QMTest in a few paragraphs, then it is probably not worth my time. Thankfully I'm already quite pleased with JUnit, RubyUnit and CppUnit.
However, if other slashdot readers have more patience with the project - I encourage them to write a summary of their findings here!
- expensive, and
- invaluable for really testing a GUI and keeping it tested over the long term. It's even worth getting only a couple of licenses for a group of developers rather than go entirely without.
I've seen a lot of other automated approaches consed up for testing various GUIs, but nothing replaces a tool that can actually go out and interact with the GUI directly through the windowing system. The simplest tools will let you record mouse and keyboard events; the usable ones have enough understanding of the underlying interface primitives to reasonably abstract your actions. E.g., figure out and record that you're pressing button "foo" in box "bar" rather than clicking at 80,130, or that you're editing in a text entry box rather than just getting a bunch of keyboard events, so that when you reformat a dialog box the test case doesn't break. That encourages developers to do the right thing and build up and maintain *lots* of test cases; there are a lot more stupid corners for testing in a GUI app than the command line equivalent.This was also one of my concerns. Though I haven't yet successfully downloaded the source from this rather slow site, I do note that it also supports a command line interface.
I'm currently wrestling with dejagnu. Although the documentation of this tool seems to assume a familiarity with autoconf that I do not yet have, it is simple enough to use once you have set up the project test suite. I'd say it's probably pretty hard to beat. Expect is a pretty good tool upon which to base a test suite.
Well then you're not looking hard enough. Try http://scons.sourceforge.net/ for an implementation of winning build tool in alpha stage.
Speaking of build tools, there is a really nifty one called CONS It does automatic dependecy management, interfaces to repositories, and is written in Perl! I messed around with makefiles for a long time before I found this, and it made a fairly messed up kernel tree a cinch to build.
A deep unwavering belief is a sure sign you're missing something...
So they've gathered a lot of requirements and done a lot design, but, as the lady says, "Where's the beef?" Why do all this work to replace open source tools that already exist? Why not take those tools and contribute back to them, or if the project owners don't like the contributions, take the things they like and build from there?
Perhaps this from their web page: "The Advanced Computing Laboratory at Los Alamos National Laboratory is providing $860,000 of funding for Software Carpentry, which is being administered by Code Sourcery, LLC." And this: CodeSourcery also provides training and strategic consulting services for companies considering the adoption of free or open source software.
Why use the MIT License? Why develop in Python? Why require that the submissions to the design competition not contain any source code? They require a language but not any source? Am I being too cynical in seeing how, after all this contributed design, coding, testing, etc is done, at taxpayer expense, Code Sourcery is now in a tremendous position as the sole-source solution for support and training to the shops that choose to use these tools? And to notice that choosing the MIT license allows them to take and wrap up all the source code into their products and not give back anything? These are questions that either are in the FAQ but not clearly answered, or not spoken about. Even the SC site itself has been retired and archived.
Thanks, I'll stick with XUnit, Bugzilla, cvs (and subversion when it's ready. For build config tools, well, if you do cross-platform C and C++, then autoconf or its successor, but that's just for one language and set of development requirements.
I'll be interested to see if anything widely used comes out of this exercise. So far, of the all the tools implementations promised for "Summer of 2001", we have QMTest 1.0. The rest? Late and unfinished.
I haven't looked at QMTest, so I'm going by what $parent says. I have written a whole lotta tests, as well as designed and coded testing frameworks.
In my opinion, the number one criteria for judging a testing system is: it must be dead simple to write and maintain test cases. Because writing tests (for your obviously immaculate code) is annoying enough, and if the tool gets in your way, forget it--it just won't happen. This is probably especially true for volunteer projects, although it applies in no small measure to closed shops.
So, you can invent a fancy aparatus and even a "theory of testing". But your system had better not require learning much aparatus, or any theory. Keep in mind when tempted to create or use a fancy system, that most of the value of a test run is in the first bit of information: did it work, or not? Anything after that is gravy.
So if QMTest requires a lot of infrastructure and tools and web UI's, I'd guess it's probably too heavy. Correctness testing is something that should stay out of your way. Assuming you keep your codebase working--right? :-)
The evaluation of an action as 'practical' . . . depends on what it is that one wishes to practice.
Another developer has implemented SCCons.
Both are still in development, but roundup is being used by several organisations already. We hope to have a new release out next week that will fix some problems with 0.3.0 and implement some nice functionality too.
Interesting that not one of the competition winners seems to be involved in the QM effort - and it does not look to me as if QMTest uses many of the ideas from the original testing entries (Disclaimer: I did read them all, but it was a long time ago).
Projects like this need a good communicative leader and as far as one can tell from the (old) web site things changed a great deal when Greg Wilson had to step down.
It does look as if CodeSourcery are keen to make a buck out of their involvement; perhaps it should be regarded as a win for open source software that some projects have been spawned away from the Software Carpentry umbrella. There's an article here: Software Carpentry Spin Offs
The tool is called Scons, and it is much nicer to use than make. Instead of using timestamps to determine when rebuilding is necessary (which is very error prone in networked environments like NFS), it uses MD5 checksums. I encourage everyone to take a look at it.
Scons itself is based upon Cons, a build tool based on Perl.
I'm a leaf on the wind. Watch how I soar.
Expect.pm is great because you can use all those excellent perl modules (and you don't have to use tcl.)
One use I found for it is to put bash-like command-line editing on tools with lack command-line editing like Oracle's sqlplus. Just use it in conjunction with Term::ReadLine::Gnu and have ReadLine send commands to an Expect-controlled process. Sqlplus is finally usable!
-Bruce
sqlplus w/ command-line editing? You are the man!
Secession is the right of all sentient beings.
The original Software Carpentry site had an excellent page explaining the reason for the whole project, but I can't find it. So I'll summarize and paraphrase from memory:
Because the original tools suck ass, that's why.
There was a great quote on the SC page, something to the effect that "when experienced *nix developers say that tools are easy to use, they really just mean that they've grown enough scar tissue that using those tools is no longer painful." I'm a long-time Old School *nix user, and I agree. Try teaching these "easy to use" tools to beginners and watch their faces contort.
Take make for example: we start with a syntax that seems custom-designed to fuck with the mind (so, rules have to be indented by one tab (why? don't ask why, they just have to be; probably because parsing was too slow back then without a leading marker character), so if you use eight spaces instead you'll get some useless error message, even though there's nothing visually apparently wrong.
Each command gets executed in a separate shell. That's always fun for beginners, and trips up experts too, until you build up enough scar tissue that ending all the commands with ";\" seems perfectly natural.
The POSIX specification for make requirements are unfortunately too weak to accomplish anything. An implementation of make must have some extensions in order to be useful. So we have a way to include other makefiles, but it's spelled differently for different versions. GNU make has some funky builtin pattern expanders, BSD make has some funky looping constructs -- and neither camp is willing to merge.
The SC project was going to replace all that crap with a tool of equal power that actually makes sense to a beginner, rather than looking like the dog's breakfast of features that make currently is.
(I, too, long for Subversions and the other Tigris projects to come to fruition.)
You cannot apply a technological solution to a sociological problem. (Edwards' Law)