Slashdot Mirror
Open Source Experiment Management Software?
Alea asks: "I do a lot of empirical computer science, running new algorithms on hundreds of datasets, trying many combinations of parameters, and with several versions of many pieces of software. Keeping track of these experiments is turning into a nightmare and I spend an unreasonable amount of time writing code to smooth the way. Rather than investing this effort over and over again, I have been toying with writing a framework to manage everything, but don't want to reinvent the wheel. I can find commercial solutions (often specific to a particular domain) but does anyone know of an open source effort? Failing that, does anyone have any thoughts on such a beast?"
"The features I would want would be:
- management of all details of an experiment, including parameter sets, datasets, and the resulting data
- ability to "execute" experiments and report their status
- an API for obtaining parameter values and writing out results (available to multiple languages)
- additionally (alternately?) a standard format for transferring data (XDF might be good)
- ability to extract selected results from experimental data
- ability to add notes
- ability to differentiate versions of software
- automatically run experiments over several parameters values
- distribute jobs and data over a cluster
- output to various formats (spreadsheets, Matlab, LaTeX tables, etc.)
- provide a fancy front-end (that can be done separately - I'm thinking mainly in terms of libraries)
- visualize data
- statistical analysis (although some basic stats would be handy)
I also did lots of comp sci empirical experiments. My experience is that the tools used for experimenting itself is very ad-hoc and not easily scriptable. Most of the times we are required to tend the hour-long experiments to see what happened on the output and decide what to do next. And... the decision is often times not clear cut. Some sort of heuristic is needed. Not to mention about the frustations when the errors occur (especially when the tool is buggy, which is very often in research settings). So, considering this, what I would do is to construct a script and do the experiments in phases. Run it and see the result several days after.
I also noticed that from one experiment to another is sometimes radically different that I would doubt it is easily manageable.
--
Error 500: Internal sig error
Take a look at the object modeling system. It is currently being developed by Agricultural Research Service but many other agencies are cooperating.
http://oms.ars.usda.gov/
1. You cannot (?) afford commercial software.
2. It is impractical for you to continue writing your own software.
3. You cannot find open source software.
-------
Conclusion: Steal commercial software! -)
Did you consider R, a Splus clone? For Scientific Statistics a very flexible solution. http://www.r-project.org
What I would end up doing is setup an Ant build file for each experiment, under each algorithm.
And then you can update property files, using a quick shell script, or something along those lines at the end of the data set, as well as having build/run times that Ant can retrieve for you. Good solution, and you aren't reinventing the wheel.
Requires Java, which depending upon your ideology is either a good thing or a curse.
Dacels Jewelers can't be trusted.
I don't mean to sound cynical, but this seems to come across to me as a very nicely written:
Ne3D H3lp WIt M4H H4x0RiN!!!!!
I mean, let's face it, much of what modern hacking closed-sourced software consists of is throwing a variety of shit against a variety of programs in a variety of configurations and seeing what breaks and then following up to make an exploit out of it.
While this probably isn't the case here, it's very hard to read that note and not snicker just a tiny, tiny bit . . .
But what you are looking for, sir, is the cheap labor commonly known as a Graduate Student
- Many of these "grads" [as they are commonly known] have INDEED been able to " 'execute' experiments and report their status", as well as "writing out results (available in multiple languages)".
- The Graduate Student is often known for their abilities to create and distribute notes in lieu of bringing that onerous burden upon more high-ranking academic officials
- ...you don't even have to dream about doing "clustered work" or "outputing results to spreadsheets, Matlab, LaTeX tables, etc....". These fancy machines can definately do that...
- Of course, there are several "graduate students" that provide a fancy front end (and rear end, for that matter). I think that I would agree with your assesment that they do not need to have that feature, although it might make your days a bit more... ermm... *pleasant*
:-)
- As well, most graduate students have the capability of performing "basic stats", although most don't have an extensive faculty for performing such calculations...
- And don't you even worry about the price -- you'll see that they're quite affordable.
To conclude, you say that "There's no reason such software should be limited to computer science (nothing I'm contemplating is very domain specific). I can imagine many disciplines that would benefit". I would wholeheartedly have to agree with you: just about every discipline can do more and see farther by standing on the backs of their graduate students.In fact, I'm afraid to report that you are a bit behind the times in this department as these "Graduate Student" devices are quite common at universities and research labs.
We experimental high-energy physics folk have been using it (and PAW) for some time. It offers scripting and histogramming and analysis and a bunch of other features. And it's open source. Check it out.
I've been very happy using jdb (see below) to handle individual experiments, and directories and shell scripts to handle sets of experiments.
JDB is a package of commands for manipulating flat-ASCII databases from shell scripts. JDB is useful to process medium amounts of data (with very little data you'd do it by hand, with megabytes you might want a real database). JDB is very good at doing things like:
For more details, see http://www.isi.edu/~johnh/SOFTWARE/JDB/.
Just admit it. Perl slept with your wife. That's what this is really about, isn't it?
What you describe does indeed sound like High Energy Physics.
And the "middleware" you need are the GNU tools gluing together the specialized programs that do the specific things you want.
We have been using unix for a long time, and many of us prefer the combination of small targeted tools philosophy rather than a single monolithic package.
I will repeat, and you can stop reading now if you want. The GNU tools, unix, and specialized scriptable programs are already the "middleware" you seek.
If you are just missing some of the tools in the middle, here are the ones used in HEP. You might find more appropriate ones closer to whatever discipline you work in.
All the basic unix text processing tools and shells.
bash. csh. Perl. grep. sed. and so on.
Filename schemes ranging from appropriate to clever to bizarre.
(See other posts here)
Make it so that all the inputs you want to change can be done on the command line or with an input steering text file.
Same tools combined with some simple c-code to produce formats for spreadsheets or PAW or ROOT or whatever visualization or post-processing thing you need done. Has ntuple and histogram support automatically, which might be all you need.
Almost always I choose space delimited text for simple output to push into PAW, ROOT, or spreadsheets. I keep a directory of templates to help me out here.
Some people use full blown databases to manage output. For a long time there have been databases specific to the HEP needs. I recently have started using XML-style data formats to encapsulate such things in text files if the resulting output is more complicated than a single line. You mention XDF, sure, that sounds like the same idea.
CONDOR (U Wisconsin) has worked nicely for me for clustering and batch job submission when I need to tool through 100 data files or 100 diffrent parameter lists on tens of computers. The standard unix "at" is good enough in a pinch if you play on only 5 computers or so.
HEP folks use things like PAW and ROOT (find them at CERN) which contain many statistical analysis things and monstrous computation algorithsm. Or at least ntuples, histograms, averages, and standard deviations. You could go commercial or the gsl here if you prefer such things.
CVS or similar to take care of code versions.
Don't forget to comment your code.
We write our own code and compile from fortran or c or c++ for most everything else.
Output all plots to postscript or eps.
LaTeX is scriptable.
And use shells, grep, perl to glue it all together. Did I mention those already?
I get a good night's sleep more often than not.
And decide what to do next after coffee the following morning.
This is where you put your brain, and if you have done the above well enough, this is where you spend most of your time.
The answer I get each morning (as another post suggests) is always so suprising that I need to start from scratch anyway.
I bet that is what you are doing already. Probably no monolithic software will be as efficient as that in a dynamic research environment.
What did I miss from your question?
Oh, yes. Get a ten-pack of computation notebook with 11 3/4 x 9 1/4 inch pages (if you print things with standard US letter paper). And lots of pens. And scotch tape to tape plots into that notebook. Laser printer and photocopier. Post-it notes to remind yourself what you wanted to do next (or e-mail memos to yourself). Maybe I should have listed this first.
Good luck.
I ran into this problem when I was in graduate school, too. What I eventually did was to abandon make because of the limitations you are running into, and construct a special-purpose experiment running utility that would know about all the predecessors, etc. It turned out not to be too hard, actually. However, if you don't know perl or another language that gives you good pattern matching and substring extraction capability, then this will be very hard to do.
.. Check other patterns. ...
I just wrote two functions. (I wrote them in the shell, but if I were doing it again, I'd probably do it in perl.) construct() simply makes a file if it is out of date (see example below). Construct() is where are of your rules go: it knows how to transform a target filename into a list of dependencies and a command.
It uses a function called up_to_date() which simply calls construct() for each dependency, then returns false if the target is not up to date with respect to each dependency. If you don't do anything very sophisticated here, up_to_date will only be a few lines of code.
"construct" will basically replace your makefile. For example, if you did it in perl, you could write it something like this:
sub construct {
local $_ = $_[0]; # Access the argument.
if (/^base\.model(.)\.fold(\d+)\.test(.).run$/) {
@dependencies = ("base.fold$2.test$3.test",
"base.fold$2.train");
if (!up_to_date($_, # Output file.
@dependencies, # Input files.
"model$1")) { # Rerun if prog changed, too.
system("model$1 @dependencies > $_");
}
}
elsif (/^....$/) { #
}
}
What you've gained from this is a much, much more powerful way of constructing the rule and the dependencies from the target filename. Of course, your file will be a little harder to read than a Makefile--that's what you pay for the extra power. But instead of having many duplicate rules in a makefile, you can use regular expressions or whatever kind of pattern matching capability you want to construct the rules.
Distribution of jobs, running things with multiple parameter values, etc., all can be handed smoothly from the shell. This is really the sort of thing that UNIX was designed for, and the entire UNIX environment is your "experiment management software".
Funding agencies in the USA (NSF, NIH) and Europe have recently decided to target the construction of such software, and many competing projects have been given grants, most of which involve the production of open source software.
Relevant keywords are "eScience", "Experimental Data Management", "Experimental Metadata", and to some extent "Grid Computing".
Here is a paper which lays out the program of research.
I work for one such NSF & NIH funded project at Dartmouth College. We're developing such a tool : Java-based, completely open, available at sourceforge, currently in alpha, to be released for fMRI use in July, but designed from the start to be generalizable for all of experimental science. This is built on top of a pre-existing framework for semantic data management and modeling from Stanford.
I'll try to list some of the features relevant to your needs:
Finally, I would like to stress that our project is one of many, and that if it doesn't meet your needs, within a year there will be many competing "eScience" toolkits.
You may contact me for more information by reversing the following string: "ude.htuomtrad@exj".