Slashdot Mirror

← Back to Stories (view on slashdot.org)

Open Source Experiment Management Software?

Posted by Cliff on from the open-source-for-science dept.

Alea asks: "I do a lot of empirical computer science, running new algorithms on hundreds of datasets, trying many combinations of parameters, and with several versions of many pieces of software. Keeping track of these experiments is turning into a nightmare and I spend an unreasonable amount of time writing code to smooth the way. Rather than investing this effort over and over again, I have been toying with writing a framework to manage everything, but don't want to reinvent the wheel. I can find commercial solutions (often specific to a particular domain) but does anyone know of an open source effort? Failing that, does anyone have any thoughts on such a beast?"

"The features I would want would be:

  • management of all details of an experiment, including parameter sets, datasets, and the resulting data
  • ability to "execute" experiments and report their status
  • an API for obtaining parameter values and writing out results (available to multiple languages)
  • additionally (alternately?) a standard format for transferring data (XDF might be good)
  • ability to extract selected results from experimental data
  • ability to add notes
  • ability to differentiate versions of software
In my dreamworld, it would also (via plugin architecture?) provide these:
  • automatically run experiments over several parameters values
  • distribute jobs and data over a cluster
  • output to various formats (spreadsheets, Matlab, LaTeX tables, etc.)
Things I don't think it needs to do:
  • provide a fancy front-end (that can be done separately - I'm thinking mainly in terms of libraries)
  • visualize data
  • statistical analysis (although some basic stats would be handy)
The amount of output data I'm dealing with doesn't necessitate database software (some sort of structured markup is ok for me), but some people would probably like more powerful storage backends. I can see it as experiment management 'middleware'. There's no reason such software should be limited to computer science (nothing I'm contemplating is very domain specific). I can imagine many disciplines that would benefit."

3 of 122 comments (clear)

  1. Object Modeling System by Anonymous Coward · · Score: 5, Informative

    Take a look at the object modeling system. It is currently being developed by Agricultural Research Service but many other agencies are cooperating.

    http://oms.ars.usda.gov/

  2. Oh that's easy.... by Anonymous Coward · · Score: 5, Funny
    The Academic Community, especially those strange AI people, have long sought complicated programs and machinery that could automate all of their work and projects, keep track of complicated "parameter sets, datasets, etc....".

    But what you are looking for, sir, is the cheap labor commonly known as a Graduate Student
    • Many of these "grads" [as they are commonly known] have INDEED been able to " 'execute' experiments and report their status", as well as "writing out results (available in multiple languages)".
    • The Graduate Student is often known for their abilities to create and distribute notes in lieu of bringing that onerous burden upon more high-ranking academic officials
    • ...you don't even have to dream about doing "clustered work" or "outputing results to spreadsheets, Matlab, LaTeX tables, etc....". These fancy machines can definately do that...
    • Of course, there are several "graduate students" that provide a fancy front end (and rear end, for that matter). I think that I would agree with your assesment that they do not need to have that feature, although it might make your days a bit more... ermm... *pleasant* :-)
    • As well, most graduate students have the capability of performing "basic stats", although most don't have an extensive faculty for performing such calculations...
    • And don't you even worry about the price -- you'll see that they're quite affordable.
    To conclude, you say that "There's no reason such software should be limited to computer science (nothing I'm contemplating is very domain specific). I can imagine many disciplines that would benefit". I would wholeheartedly have to agree with you: just about every discipline can do more and see farther by standing on the backs of their graduate students.
    In fact, I'm afraid to report that you are a bit behind the times in this department as these "Graduate Student" devices are quite common at universities and research labs.
  3. Re:Perl is only useful for maintaining your job by asciirock · · Score: 5, Funny

    Just admit it. Perl slept with your wife. That's what this is really about, isn't it?