Open Source Software For Experimental Physics?

jmizrahi writes "I've recently started working in experimental physics. Quite a few programs are used in the lab for assorted purposes — Labview, Igor, Inventor, Eagle, to name just a few. They are all proprietary. This seems to be standard practice, which surprised me. Does anybody know of any open source software intended for scientific research? Does anybody work in a lab that makes an effort to use open source software?"

Comments

Your Reqs Are Too Specific, Try R or Octave

[eldavojohn]

Does anybody know of any open source software intended for scientific research? Does anybody work in a lab that makes an effort to use open source software?

We discussed R which describes itself as:

R is a language and environment for statistical computing and graphics. It is a GNU project which is similar to the S language and environment which was developed at Bell Laboratories (formerly AT&T, now Lucent Technologies) by John Chambers and colleagues. R can be considered as a different implementation of S. There are some important differences, but much code written for S runs unaltered under R.

R provides a wide variety of statistical (linear and nonlinear modeling, classical statistical tests, time-series analysis, classification, clustering, ...) and graphical techniques, and is highly extensible. The S language is often the vehicle of choice for research in statistical methodology, and R provides an Open Source route to participation in that activity.

One of R's strengths is the ease with which well-designed publication-quality plots can be produced, including mathematical symbols and formulae where needed. Great care has been taken over the defaults for the minor design choices in graphics, but the user retains full control.

While it's not geared specifically towards experimental physics, that's probably going to be your most fruitful endeavor.

Then there's the Matlab knock-off of Octave which describes itself as:

GNU Octave is a high-level language, primarily intended for numerical computations. It provides a convenient command line interface for solving linear and nonlinear problems numerically, and for performing other numerical experiments using a language that is mostly compatible with Matlab. It may also be used as a batch-oriented language.

Octave has extensive tools for solving common numerical linear algebra problems, finding the roots of nonlinear equations, integrating ordinary functions, manipulating polynomials, and integrating ordinary differential and differential-algebraic equations. It is easily extensible and customizable via user-defined functions written in Octave's own language, or using dynamically loaded modules written in C++, C, Fortran, or other languages.

I'm surprised you're surprised that you only find proprietary software in the highly specialized realm of "experimental physics." I mean, you have to be like a PhD in physics with a good deal of programming knowledge to make something accurate & useful (and there's probably gotta be like 50 failed projects before you get a good successful one).

You're probably wondering why there's not a project of Firefox or OO.o quality for experimental physics but I'll tell you why: it's too specialized and your user base is ridiculously small. You're not going to find a company that is going to benefit greatly (or at all maybe) by releasing their product into the wild for a community to grow. There's probably not a community for it to grow in.

You should tell us what specifically you are looking for something to do ... I have no idea what Labview, Igor, Inventor, or Eagle do. Ask yourself why these programs are standards and then maybe add to Octave's wish list or contribute to it even! Unfortunately, this isn't easy--I myself started to implement proper handling of sparse matrices in Octave but gave up as I was trying to form low level requirements ... You probably already know though that this is going to have to be done in C or another very low level, very quick language.

If you're looking for something specific or outline some high level

Re:Your Reqs Are Too Specific, Try R or Octave

[Anonymous Coward]

It also depends on what is meant by "experimental physics". CERN has released one hell of a lot of physics software over time. That which is maintained and comes in a tarball is largely listed on Freshmeat, but there's also plenty of unmaintained but useful code, along with code only released via CVS. The same is true of many US experimental physics labs and NASA.

Your question simply isn't specific enough for me to identify whether you want OpenAtom, MultiBody Dynamics, Geant3, Geant4, Electron Gamma Shower, MIT Photonic Bands, a statistical language like R, visualizing software like DX4, the Interactive Spectral Interpretation System or GGobi, plotting software like Octave, general acquisition/control software such as COMEDI, data acquisition software over VMS such as MIDAS, or just a driver like the Cogent CIF Driver for Fieldbus. Maybe those aren't your cup of tea and you're more into CFD, in which case OpenFoam and NASA's FUN3D might be more what you want, along perhaps with the CFD General Notation System. Or maybe something in geophysics will help, like Seismic Unix.

Or maybe you've jury-rigged something Professor Heinz Wolff-style out of spare parts from an HVAC and need the BACNet package. Of course, data manipulation/storage then becomes useful and you'd need something like PETSc, the Vector Signal Imaging Processing Library, the Stanford Exploration Project Library or HDF5. Then, perhaps, the system you're controlling uses a modern high-speed, low-latency network, in which case perhaps DAPL would be of use. You could even use SensorWeb for general-purpose sensor monitoring.

Yes, this post is saturated with names. Deliberately so. (It could hardly be accidental.) This is not to overwhelm you, as much as it is to emphasize that Open Source science is very well established - so much so that a highly general question cannot be meaningfully answered. Maybe what you want is in the list above, maybe it isn't, but Slashdot would run out of disk space before I ran out of packages out there. Since nobody wants that, it is far more practical to give you the general idea and some pointers on where to look.

Qtiplot & root

[denominateur]

For simple tasks, even with big data sets, I've had good results with qtiplot http://soft.proindependent.com/qtiplot.html. For really complex stuff, there's root from CERN http://root.cern.ch/

Re:Fermilab...

[MoellerPlesset2]

Actually Fermi's Linux is their build of Scientific Linux, which is a distro they have in collaboration with CERN and others.

It's originally based off Redhat/Fedora IIRC. Or department uses it.

EPICS

[ilbrec]

Many larger places in the world use EPICS (Experimental Physics and Industrial Control System). An experiment I am a (small) part of use EPICS for control.
It is an open source control systems frequently used for particle accelerator control and observatory telescope control. We use it slightly differently, but for what we need to do, it works very well. It is maintained primary by Advanced Photon Source at Argonne National Laboratory. You can read more at the following URL:
http://www.aps.anl.gov/epics/
In case you are wondering, no, they don't use EPICS for LHC. They use a commercial SCADA program called PVSS (for the most part anyway).

He means I think experimental control

[goombah99]

The programs he names:
labview, igor, matlab, are not simply data analysis tools but also have hardware control modules. You can run your data acquisition from Igor, and matlab and of course Lab view.

things like R or octave or scipy dont' have control and acquisition modules that I know of.

That said, it seems to me that since scipy is in python that any python DAQ system would fit the bill of a scientific software system.

So the real question is, what DAQ's are available for python.

and the answer is going to be, that daq softwares tend to get written to drive specific hardware systems. e.g. labview makes a whole suite of DAQ boards. there's oscilloscopes and so forth.

As a scientist you don't want to get bogged down building a custom daq. SO the real bottom line is what commercial DAQs are open in design.

The only system I know that might possibly be in the open is the OASIS daq's developed for flow cytometry. these are mass produced and were developed at the National Labs. But I don't know how it is lic.

of course, one can always use an oscilloscope or DAQ made to be controlled by GPIB or simmilar common language. In that case it's pretty easy to write your own as long as you can find a suitable open source GPIB output driver.

EPICS and RTEMS

[joelsherrill]

Since you said experimental physics... :)

The Experimental Physics and Industrial Control System (EPICS) is a set of Open Source software tools, libraries and applications developed collaboratively and used worldwide to create distributed soft real-time control systems for scientific instruments such as a particle accelerators, telescopes and other large scientific experiments.

EPICS is often used with the free real-time operating system RTEMS to build custom control systems.

Users of EPICS+RTEMS include Stanford Linear Accelerator Center (SLAC), Argonne National Labs, Brookhaven National Labs, and Canadian Light Source.

Re:Stay away from Labview

[goombah99]

Your knowledge of Labview and Igor seems limited and out of date. Labview is far more sophisticated now that the primitive description you use. (for example, you can bundle all the wires into a single "object" wire) so there's no "endless" drawing of connections like you describe. It becomes more like drawing UML, except when you are done making the vi, it actually runs.

Igor is a very nice fast system. the command line syntax is a little archane but not terrible hard. But it has complete gui controls so you can use it before you memorize all the commands. It's not worse than GNU plot or matlab or perl data language or scipy or R in terms of having syntax oddities. it seems like all grpahics packages try to compromise between powerful short commands and completeness. (sort of the perl ethos out of neccessity to make it interactive rather than a structured program)

but it's not as good a general programming language as say Scipy. But it is more user freindly.

Labview can be frustrating because sometimes it is faster to write code. But here is one big advantage labview has over everything I have used for data acquisition. It's the only code I'm willing to modify during an experiment. it's not possible to make a syntax error, it is very hard to make big errors when simply moving or split, and you don't have to worry about the thread g wires in an existing program, and most importantly for DAQ since you dont' explicitly program the thread syncronization you cant screw it up with changes. So it's not too scarey to change a working program in the lab to do something a little extra right in the middle of some expensive instrument time.

Labview is a extremely crappy language for data analysis. It's designed for live instruments and real time work not off line anaylis.

right now I'm a big fan of scipy but it too has it's frustrations. it's wildley incomplete, undocumented, and they just changed all the 3d graphics to a much better but way too complicated system and the old programs can't be maintained with the new releases.

Sage

[hoytak]

http://www.sagemath.org/. Sage rocks. It's python based, brings together many of the useful libraries already mentioned (numpy, scipy, matplotlib, etc.), and has a very active mailing list. Can't recommend it enough.

f2py

[goombah99]

Python has mad fortran all the mor valuable. the combination of fortran and python is a match made in heaven.

fortran is a wonderfully good algorithmic language but it is terrible at everything else. python is great at everything else, but slow as a dog.

one of fortrans strengths when combined with python is oddly enough it's simplicity and lack of sophistication. unlike c++ It compiles so fast that you can feasibly have python write fortran on the fly optimized for the algorithm and compile it, all during run time.

Fortran is good for non-computer scientist in part because it is quite hard for a typo to be a logic error. (e.g. = instead of ==) as a result it's comparatively easy to debug compared to C.

but it can't do all the fun memory management tricks, introspection, or objects that advanced languages do.

But with f2py you can push all of that into python and then just have it call short simple fortran routines for speed.