Slashdot Mirror
Open Source Experiment Management Software?
Alea asks: "I do a lot of empirical computer science, running new algorithms on hundreds of datasets, trying many combinations of parameters, and with several versions of many pieces of software. Keeping track of these experiments is turning into a nightmare and I spend an unreasonable amount of time writing code to smooth the way. Rather than investing this effort over and over again, I have been toying with writing a framework to manage everything, but don't want to reinvent the wheel. I can find commercial solutions (often specific to a particular domain) but does anyone know of an open source effort? Failing that, does anyone have any thoughts on such a beast?"
"The features I would want would be:
- management of all details of an experiment, including parameter sets, datasets, and the resulting data
- ability to "execute" experiments and report their status
- an API for obtaining parameter values and writing out results (available to multiple languages)
- additionally (alternately?) a standard format for transferring data (XDF might be good)
- ability to extract selected results from experimental data
- ability to add notes
- ability to differentiate versions of software
- automatically run experiments over several parameters values
- distribute jobs and data over a cluster
- output to various formats (spreadsheets, Matlab, LaTeX tables, etc.)
- provide a fancy front-end (that can be done separately - I'm thinking mainly in terms of libraries)
- visualize data
- statistical analysis (although some basic stats would be handy)
Take a look at the object modeling system. It is currently being developed by Agricultural Research Service but many other agencies are cooperating.
http://oms.ars.usda.gov/
Did you consider R, a Splus clone? For Scientific Statistics a very flexible solution. http://www.r-project.org
Something like the AppLeS Parameter Sweep Template software might suit your needs. I've never used it myself, but it looks like it might be close to what you're looking for.
See here for other projects from the GRAIL lab at SDSC and UCSD.
We experimental high-energy physics folk have been using it (and PAW) for some time. It offers scripting and histogramming and analysis and a bunch of other features. And it's open source. Check it out.
I've been very happy using jdb (see below) to handle individual experiments, and directories and shell scripts to handle sets of experiments.
JDB is a package of commands for manipulating flat-ASCII databases from shell scripts. JDB is useful to process medium amounts of data (with very little data you'd do it by hand, with megabytes you might want a real database). JDB is very good at doing things like:
For more details, see http://www.isi.edu/~johnh/SOFTWARE/JDB/.
http://sourceforge.net/projects/pythonlabtools/
It might not satisfy all your requirements out of the box, but could you put something together with tcltest?
What you describe does indeed sound like High Energy Physics.
And the "middleware" you need are the GNU tools gluing together the specialized programs that do the specific things you want.
We have been using unix for a long time, and many of us prefer the combination of small targeted tools philosophy rather than a single monolithic package.
I will repeat, and you can stop reading now if you want. The GNU tools, unix, and specialized scriptable programs are already the "middleware" you seek.
If you are just missing some of the tools in the middle, here are the ones used in HEP. You might find more appropriate ones closer to whatever discipline you work in.
All the basic unix text processing tools and shells.
bash. csh. Perl. grep. sed. and so on.
Filename schemes ranging from appropriate to clever to bizarre.
(See other posts here)
Make it so that all the inputs you want to change can be done on the command line or with an input steering text file.
Same tools combined with some simple c-code to produce formats for spreadsheets or PAW or ROOT or whatever visualization or post-processing thing you need done. Has ntuple and histogram support automatically, which might be all you need.
Almost always I choose space delimited text for simple output to push into PAW, ROOT, or spreadsheets. I keep a directory of templates to help me out here.
Some people use full blown databases to manage output. For a long time there have been databases specific to the HEP needs. I recently have started using XML-style data formats to encapsulate such things in text files if the resulting output is more complicated than a single line. You mention XDF, sure, that sounds like the same idea.
CONDOR (U Wisconsin) has worked nicely for me for clustering and batch job submission when I need to tool through 100 data files or 100 diffrent parameter lists on tens of computers. The standard unix "at" is good enough in a pinch if you play on only 5 computers or so.
HEP folks use things like PAW and ROOT (find them at CERN) which contain many statistical analysis things and monstrous computation algorithsm. Or at least ntuples, histograms, averages, and standard deviations. You could go commercial or the gsl here if you prefer such things.
CVS or similar to take care of code versions.
Don't forget to comment your code.
We write our own code and compile from fortran or c or c++ for most everything else.
Output all plots to postscript or eps.
LaTeX is scriptable.
And use shells, grep, perl to glue it all together. Did I mention those already?
I get a good night's sleep more often than not.
And decide what to do next after coffee the following morning.
This is where you put your brain, and if you have done the above well enough, this is where you spend most of your time.
The answer I get each morning (as another post suggests) is always so suprising that I need to start from scratch anyway.
I bet that is what you are doing already. Probably no monolithic software will be as efficient as that in a dynamic research environment.
What did I miss from your question?
Oh, yes. Get a ten-pack of computation notebook with 11 3/4 x 9 1/4 inch pages (if you print things with standard US letter paper). And lots of pens. And scotch tape to tape plots into that notebook. Laser printer and photocopier. Post-it notes to remind yourself what you wanted to do next (or e-mail memos to yourself). Maybe I should have listed this first.
Good luck.
Draft relational schema:
// date-time stamp // description
// auto-num // foreign key to experiments table
// auto-nump ; // "True" if from experiment // ASCII check-sum to make sure not changed
// foreign key to experiments table // foreign key to dataSet table
s ion
// foreign key to softwareVersion // foreign key to experiments table
Table: experiments
----
exprmntID
exprmntWhen
exprmntDescr
outcome
Table: params
----
paramID
exprmntRef
paramName
paramValue
Table: dataSet
----
dataSetID
filePath
datasetDescr
isGenerated&nbs
CRC
Table: dataSetUsed
----
exprmntRef
dataSetRef
Table: softwareVersion
----
svID
softwareTitle
svVer
Table: softwareVersionUsed
----
svRef
exprmntRef
Just use something like MySQL or MS-Access, and perhaps some kind of CRUD[1] tool to create front ends. You can expand from there based on new needs you encounter.
[1] CRUD = typical Create, Read (list), Update, Delete screens.
(Note: slashdot's filter scrambles certain variable names.)
Table-ized A.I.
The features I would want would be:
management of all details of an experiment, including parameter sets, datasets, and the resulting data
This can be handled by an ad-hoc database, a flat file in most cases. If you were a Windows power user, you'd spend an hour or two putting together something in Access for it.
ability to "execute" experiments and report their status
make with a little scripting, or whatever you use as a build system.
an API for obtaining parameter values and writing out results (available to multiple languages)
additionally (alternately?) a standard format for transferring data (XDF might be good)
ability to extract selected results from experimental data
ability to add notes
Again, an ad-hoc database would be your friend.
ability to differentiate versions of software
This is conventionally handled with a configuration management system like CVS, Sourcesafe, or Clearcase.
I hate reinventing the wheel, too, and I'd love to see a good book on using standard free Unix tools like make, CVS, Postgres, perl or some other common scripting language, TeX, etc for cleanly and efficiently
automating complex computing processes and producing nice reports from them.
PAW and ROOT look interesting though they look like overkill for many apps.
Also, get a copy of Writing The Laboratory Notebook, some hardbound buffered laboratory notebooks, and Sakura 05 Pigma Micron archival pigment pens to keep your paper records. You'll thank me.
Distribution of jobs, running things with multiple parameter values, etc., all can be handed smoothly from the shell. This is really the sort of thing that UNIX was designed for, and the entire UNIX environment is your "experiment management software".
Have you considering using python? researchers at DIKU have created a benchmark tool for . It seems usefull. Python is also excellent for gluing modules together. :)
Generally I would recommend against exporting directly to spreadsheet formats - I tend to export to flat files (with field separators). Postgres' COPY command is extremely handy from scripts. I also find gnu-plot handy. Remember .. these tools might seem un-intuitive at first .. but when you have used them for earlier experiments you can re-use code between experiements :)
I hope this helps
A wellthought example on how to setup your code for experimental work is the lemur toolkit from CMU. This toolkit has a concept of "parameter" files that is very handy
I'm one of the principal designers of a system called SMIRP.
/.'ers, but we got seriously screwed when the prototype we did in Cold Fusion became production code and we realised that Allaire (and later Macromedia) would not computer redistribution for less than 10,000 units. I could try to get it running on another CF implementation (I think there's some Blue Dragon or something) but honestly, I'd rather rewrite the whole thing.
It started out as a very simple system that didn't act as much more than a set of tables with some simple linking structures. On top of that is an alerting system, (so you can track new experiments being done) a full text index, bots for automating certain procedures, and a system for transferring data to Excel.
What's surprising is that for the most part, the underlying structure stayed exactly the same even though we've been running all the operations in an inorganic chemistry lab on for, oh, four years now. I've been chewing over ways of rewriting it because, honestly, it's still the same prototype. I'd love to go with an all Perl solution... but the damned thing just works and I have other stuff to do.
Some lessons I've learned, problems I've run into:
A general interface. You really need a flexible structure because scientists never know what parameters they're going to use until they do the experiment. Our big success has been such a simple structure that people can throw a SMIRPSpace together in minutes.
Browser based interface. It's great because it's ubiquitous, but it's painful because of the inflexibility of forms. One big win with it is that you can get a horde of workstudies to form a pipeline. For example, a grad student might put a request in the system for an article, a workstudy recieves a notification of the change and hits the web to fill in details, another then gets notified and sends a request to the library, another gets notified and scans the result and finally the grad student sees a scanned copy of the article.
Excel based interface. It's great because people can play with data, but it's Excel...
XML is garbage. There's nothing you can do in XML that you can't do better with a flat file + regexes, or a SQL DBMS. XML is utterly, completely worthless.
Proprietary products. This won't be a huge surprise to
Reporting. This is *hard* to do. We still don't have any serious system for handling reports beyond "import the data to Excel and do it manually."
Funding agencies in the USA (NSF, NIH) and Europe have recently decided to target the construction of such software, and many competing projects have been given grants, most of which involve the production of open source software.
Relevant keywords are "eScience", "Experimental Data Management", "Experimental Metadata", and to some extent "Grid Computing".
Here is a paper which lays out the program of research.
I work for one such NSF & NIH funded project at Dartmouth College. We're developing such a tool : Java-based, completely open, available at sourceforge, currently in alpha, to be released for fMRI use in July, but designed from the start to be generalizable for all of experimental science. This is built on top of a pre-existing framework for semantic data management and modeling from Stanford.
I'll try to list some of the features relevant to your needs:
Finally, I would like to stress that our project is one of many, and that if it doesn't meet your needs, within a year there will be many competing "eScience" toolkits.
You may contact me for more information by reversing the following string: "ude.htuomtrad@exj".
The Computer Aided Engineering (CAE) world has much the same problem you do.
They model their products with several different analysis codes, each with its own input and output format. This generates a gob of data, and is currently managed in ad hoc ways, is not easy to integrate with other results and wastes the time of lots of engineers.
The product we've come up with to manage both the models, the process for executing the models, and the data generated by running the models is a software framework called CoMeT (Computational Modeling Toolkit).
We are also capable of managing different versions of the model, parameter studies, and some basic data mining. The whole thing is scriptable with Scheme.
Unfortunately, we are a commercial software company, and the software is still under development, although everything I mentioned above can currently be done. We are mostly working on a front end now, although we still need to make a few improvements to the framework and add support for many analysis codes.
The reason I'm replying to this is that your list of requirements is a perfect subset of ours. We are aiming our product at CAE in the mechanical and electrical domains (Mechatronics).
I know, it's not free, but we feel we've done some very innovative things and it has taken several people many years of low pay to get this far. We really want to make some money off it eventually....
If you want more information check out the web-site or email me here. We're in need of proving this technology in a production environment so maybe we can work something out.
-Craig.
http://roofit.sourceforge.net/
I'm in precisely the same situation as Alea, so I read the suggestions here with considerable interest.
I'd like to mention ExpLab.
Though I haven't used ExpLab yet, these folks have been associated with other very high quality work (CGAL) so I expect good things. Here are three goals they list for the project: