Open Source Software For Experimental Physics?

jmizrahi writes "I've recently started working in experimental physics. Quite a few programs are used in the lab for assorted purposes — Labview, Igor, Inventor, Eagle, to name just a few. They are all proprietary. This seems to be standard practice, which surprised me. Does anybody know of any open source software intended for scientific research? Does anybody work in a lab that makes an effort to use open source software?"

Comments

Fermilab...

[samriel]

has its own Linux distribution. Let's hope to god that it's stable enough to not crash the LHC.

https://fermilinux.fnal.gov/

Re:Fermilab...

[MoellerPlesset2]

Actually Fermi's Linux is their build of Scientific Linux, which is a distro they have in collaboration with CERN and others.

It's originally based off Redhat/Fedora IIRC. Or department uses it.

Re:Fermilab...

[Roger W Moore]

FermiLinux was what they used before switching to Scientific Linux. The original FermiLinux was about 1-2 years out of date because they took about a year to certify the corresponding version of RedHat (pre-Fedora). We never used it on the Desktop Linux cluster I designed and used to manage because of that although now they use Scientific Linux things are a lot better.

Your Reqs Are Too Specific, Try R or Octave

[eldavojohn]

Does anybody know of any open source software intended for scientific research? Does anybody work in a lab that makes an effort to use open source software?

We discussed R which describes itself as:

R is a language and environment for statistical computing and graphics. It is a GNU project which is similar to the S language and environment which was developed at Bell Laboratories (formerly AT&T, now Lucent Technologies) by John Chambers and colleagues. R can be considered as a different implementation of S. There are some important differences, but much code written for S runs unaltered under R.

R provides a wide variety of statistical (linear and nonlinear modeling, classical statistical tests, time-series analysis, classification, clustering, ...) and graphical techniques, and is highly extensible. The S language is often the vehicle of choice for research in statistical methodology, and R provides an Open Source route to participation in that activity.

One of R's strengths is the ease with which well-designed publication-quality plots can be produced, including mathematical symbols and formulae where needed. Great care has been taken over the defaults for the minor design choices in graphics, but the user retains full control.

While it's not geared specifically towards experimental physics, that's probably going to be your most fruitful endeavor.

Then there's the Matlab knock-off of Octave which describes itself as:

GNU Octave is a high-level language, primarily intended for numerical computations. It provides a convenient command line interface for solving linear and nonlinear problems numerically, and for performing other numerical experiments using a language that is mostly compatible with Matlab. It may also be used as a batch-oriented language.

Octave has extensive tools for solving common numerical linear algebra problems, finding the roots of nonlinear equations, integrating ordinary functions, manipulating polynomials, and integrating ordinary differential and differential-algebraic equations. It is easily extensible and customizable via user-defined functions written in Octave's own language, or using dynamically loaded modules written in C++, C, Fortran, or other languages.

I'm surprised you're surprised that you only find proprietary software in the highly specialized realm of "experimental physics." I mean, you have to be like a PhD in physics with a good deal of programming knowledge to make something accurate & useful (and there's probably gotta be like 50 failed projects before you get a good successful one).

You're probably wondering why there's not a project of Firefox or OO.o quality for experimental physics but I'll tell you why: it's too specialized and your user base is ridiculously small. You're not going to find a company that is going to benefit greatly (or at all maybe) by releasing their product into the wild for a community to grow. There's probably not a community for it to grow in.

You should tell us what specifically you are looking for something to do ... I have no idea what Labview, Igor, Inventor, or Eagle do. Ask yourself why these programs are standards and then maybe add to Octave's wish list or contribute to it even! Unfortunately, this isn't easy--I myself started to implement proper handling of sparse matrices in Octave but gave up as I was trying to form low level requirements ... You probably already know though that this is going to have to be done in C or another very low level, very quick language.

If you're looking for something specific or outline some high level

Re:Your Reqs Are Too Specific, Try R or Octave

[morgan_greywolf]

Agreed. I've know people who work in experimental physics labs. Some of them code. And the use of open source software is quite a bit more than one might think. Check out this guy's blog. He regularly hacks together interesting scripts in Python, for instance.

Re:Your Reqs Are Too Specific, Try R or Octave

[Hatta]

I'm surprised you're surprised that you only find proprietary software in the highly specialized realm of "experimental physics." I mean, you have to be like a PhD in physics with a good deal of programming knowledge to make something accurate & useful (and there's probably gotta be like 50 failed projects before you get a good successful one).

To me, that makes it all the more important that ones work is distributed as widely as possible so that others may build upon it.

Re:Your Reqs Are Too Specific, Try R or Octave

[Anonymous Coward]

It also depends on what is meant by "experimental physics". CERN has released one hell of a lot of physics software over time. That which is maintained and comes in a tarball is largely listed on Freshmeat, but there's also plenty of unmaintained but useful code, along with code only released via CVS. The same is true of many US experimental physics labs and NASA.

Your question simply isn't specific enough for me to identify whether you want OpenAtom, MultiBody Dynamics, Geant3, Geant4, Electron Gamma Shower, MIT Photonic Bands, a statistical language like R, visualizing software like DX4, the Interactive Spectral Interpretation System or GGobi, plotting software like Octave, general acquisition/control software such as COMEDI, data acquisition software over VMS such as MIDAS, or just a driver like the Cogent CIF Driver for Fieldbus. Maybe those aren't your cup of tea and you're more into CFD, in which case OpenFoam and NASA's FUN3D might be more what you want, along perhaps with the CFD General Notation System. Or maybe something in geophysics will help, like Seismic Unix.

Or maybe you've jury-rigged something Professor Heinz Wolff-style out of spare parts from an HVAC and need the BACNet package. Of course, data manipulation/storage then becomes useful and you'd need something like PETSc, the Vector Signal Imaging Processing Library, the Stanford Exploration Project Library or HDF5. Then, perhaps, the system you're controlling uses a modern high-speed, low-latency network, in which case perhaps DAPL would be of use. You could even use SensorWeb for general-purpose sensor monitoring.

Yes, this post is saturated with names. Deliberately so. (It could hardly be accidental.) This is not to overwhelm you, as much as it is to emphasize that Open Source science is very well established - so much so that a highly general question cannot be meaningfully answered. Maybe what you want is in the list above, maybe it isn't, but Slashdot would run out of disk space before I ran out of packages out there. Since nobody wants that, it is far more practical to give you the general idea and some pointers on where to look.

Re:Your Reqs Are Too Specific, Try R or Octave

[gmueckl]

Well, the average scientific experiment really is buggy, feature-incomplete and crash-prone. And this doesn't even include the software for controlling them. All that *really* matters is getting solid data.

Scilab is the GPL alternative to Matlab

[swaltman]

.. but the specialized libraries aren't as mature. http://www.scilab.org/

Python, SciPy, VPython, etc...

[Anonymous Coward]

There are tons of opensource software tools used in scientific research. One widely used language for building and extending these tools is Python.

Try a google search for +Python +physics to get started, but also try +Python Labview, +Python Igor, and so on. You will find that lots of people have built Python tools to extend the proprietary ones, or do preprocessing or postprovcessing of data.

Re:Python, SciPy, VPython, etc...

[bzipitidoo]

There's Sage, for advanced mathematics. Built with Python.

Stay away from Labview

[hankwang]

I have some experience with Labview and Igor.

Labview has the advantage of being easy to learn for non-computer-savvy people that want to control and read special hardware (electrical monitoring equipment, servo motors, etc.). Unfortunately, it rather terrible for complex programs that do not naturally fit in LabVIEW's paradigm of deterministic data flow paths from user input to screen or file output. For example, storing things in persistent variables that are not visible to the end user are a horrible kludge. Reusing VIs (program modules) in other projects require you to endlessly draw wires on your screen rather than simply copy/paste something in an editor. Data processing that involves anything else than applying a standard library function (such as searching arrays for special conditions) that would have been 20 lines of straightforward C code will take you half an hour in LabVIEW, even if you use the "C formula node" that has no debugging facilities whatsoever. You will find yourself spending most of your time moving lines on your screen because there is no free space left on the flow diagram for that extra feature that you need to implement. Stay away from Labview if you can. Even Labview representatives will tell you that more experienced programmers tend to not like Labview so much.

I haven't used Igor myself, but I have watched over other people's shoulders. You can do quite a bit with it, but the syntax of the language is quite inconsistent; the central paradigm is that variables are either scalar or "wave", which is an array that implicitly represents a row of (x,y) values with fixed steps in x. Wave variables always have global scope, which makes functions that deal with wave variables unpleasant, especially if there is recursion.

Myself, I have been a happy Gnuplot (FOSS) user since 1996. Gnuplot is quite limited in terms of data analysis and dealing with higher-dimensional datasets, and also has a very inconsistent syntax due to historical reasons, but the latter doesn't bother since I have used it as it evolved. For serious (CPU-intensive) data analysis, I have always written C/C++ programs (with gcc, of course), using Gnuplot only for plotting.

I have played a bit with R (FOSS) for data analysis and visualisation, which I liked, but which I have never used apart from making a few drawings. Octave can be useful if you need to collaborate with Matlab people.

For hardware control, I also prefer C/C++. Unfortunately you will likely have to do that under Windows unless you want to write your own lowlevel drivers for Linux (I tried once and gave up when I had to read a 200 page document describing the PCI bridge controller to figure out how to do a DMA transfer).

(I'm out of the academic world now, working at a high-tech company where I have to work with Microsoft Office most of the time and god I do hate that...)

Re:Stay away from Labview

[goombah99]

Your knowledge of Labview and Igor seems limited and out of date. Labview is far more sophisticated now that the primitive description you use. (for example, you can bundle all the wires into a single "object" wire) so there's no "endless" drawing of connections like you describe. It becomes more like drawing UML, except when you are done making the vi, it actually runs.

Igor is a very nice fast system. the command line syntax is a little archane but not terrible hard. But it has complete gui controls so you can use it before you memorize all the commands. It's not worse than GNU plot or matlab or perl data language or scipy or R in terms of having syntax oddities. it seems like all grpahics packages try to compromise between powerful short commands and completeness. (sort of the perl ethos out of neccessity to make it interactive rather than a structured program)

but it's not as good a general programming language as say Scipy. But it is more user freindly.

Labview can be frustrating because sometimes it is faster to write code. But here is one big advantage labview has over everything I have used for data acquisition. It's the only code I'm willing to modify during an experiment. it's not possible to make a syntax error, it is very hard to make big errors when simply moving or split, and you don't have to worry about the thread g wires in an existing program, and most importantly for DAQ since you dont' explicitly program the thread syncronization you cant screw it up with changes. So it's not too scarey to change a working program in the lab to do something a little extra right in the middle of some expensive instrument time.

Labview is a extremely crappy language for data analysis. It's designed for live instruments and real time work not off line anaylis.

right now I'm a big fan of scipy but it too has it's frustrations. it's wildley incomplete, undocumented, and they just changed all the 3d graphics to a much better but way too complicated system and the old programs can't be maintained with the new releases.

Re:Stay away from Labview

[novakyu]

for example, you can bundle all the wires into a single "object" wire

Oh, yes you can bundle them, but that just means you have fewer visible wires; the underlying scheme hasn't changed at all.

For example, suppose you write a sub-VI and later on, you want to modify it to take more inputs. In real programming languages like Python, that would be as simple as dealing with keyword argument lists. You don't have to mess with the function definition; you just pass the information to the function and write something within the function to do something with that input (i.e. you don't have to think about the whole pipeline, just the end-points).

Try doing that in LabVIEW. When the smallest property of those bundles change (i.e. more number of wires in the bundle, or a different data type for one wire), you have to change a number of things to make the things connecting to that bundle work again.

I've worked with LabVIEW for some time, and I've modified and re-written moderately complicated program, and at the end of the day, if I can imagine the program growing in its functions and me maintaining that program 1, 2 years from now, I would stay very far, far away from LabVIEW, regardless of how easy initial UI design is (it's not like designing UIs with toolkits like Qt is that difficult).

Let me just put it this way: modifying a LabVIEW program is like trying to modify an electronic circuit---except that, very often, you won't have a circuit diagram in your hand, and you won't have any explanations as to why certain things were wired in a certain way.

Re:Stay away from Labview

[goombah99]

Try doing that in LabVIEW. When the smallest property of those bundles change (i.e. more number of wires in the bundle, or a different data type for one wire), you have to change a number of things to make the things connecting to that bundle work again.

Sounds like someone has not learned you can index bundles by name, not just by position in the struct. You are treating them like an array but you can also treat them like hash's.

But agreed it's not as simple in some cases as writing code.

On the other hand if you were writing the syncronization of multiple threads by hand your head can explode but in labview you don't even think about it most of the time.

it's a terrible generla purpose programming language. it's a great data aquiristion language and RAD gui builder.

plus it's the only language I know where it's safe to let the enduser mess with the gui.

that makes it perfect for experimental laboratory environements.

Re:Stay away from Labview

[ultralame]

Try doing that in LabVIEW. When the smallest property of those bundles change (i.e. more number of wires in the bundle, or a different data type for one wire), you have to change a number of things to make the things connecting to that bundle work again.

If you have been working with LV for a while, you should know about Type Definitions and possibly LVOOP/LV Class architecture. In short, you can EASILY do what you are talking about, as long as you have some idea how to use good practices in labview (and no, it doesn't take significantly more time, any more than avoiding GOTO statements).

In short, there are very simple "BKM"s in LV as there are in any language that will allow you to deal with exactly that (and other) situations.

Let me just put it this way: modifying a LabVIEW program is like trying to modify an electronic circuit---except that, very often, you won't have a circuit diagram in your hand, and you won't have any explanations as to why certain things were wired in a certain way.

And this is different from a poorly written text program how? I write my LV code in such a way that you can understand it by looking at it, amendable and commented.

Labview's biggest weakness is that it IS so easy to get a working program up that people who have never seen it before and get things done- sometimes rather horribly. But compare that to someone trying to write a deterministic piece of code in VB who has never seen a line of code before. They just can't (in a short period of time). But like any language, there are good and bad practices.

Qtiplot & root

[denominateur]

For simple tasks, even with big data sets, I've had good results with qtiplot http://soft.proindependent.com/qtiplot.html. For really complex stuff, there's root from CERN http://root.cern.ch/

Geist3D

[dedazo]

I know of this one, if only because it's written in Python and I ran into it looking for some unrelated 3D tools. It does do physics simulations, AFAIK.

EPICS

[ilbrec]

Many larger places in the world use EPICS (Experimental Physics and Industrial Control System). An experiment I am a (small) part of use EPICS for control.
It is an open source control systems frequently used for particle accelerator control and observatory telescope control. We use it slightly differently, but for what we need to do, it works very well. It is maintained primary by Advanced Photon Source at Argonne National Laboratory. You can read more at the following URL:
http://www.aps.anl.gov/epics/
In case you are wondering, no, they don't use EPICS for LHC. They use a commercial SCADA program called PVSS (for the most part anyway).

He means I think experimental control

[goombah99]

The programs he names:
labview, igor, matlab, are not simply data analysis tools but also have hardware control modules. You can run your data acquisition from Igor, and matlab and of course Lab view.

things like R or octave or scipy dont' have control and acquisition modules that I know of.

That said, it seems to me that since scipy is in python that any python DAQ system would fit the bill of a scientific software system.

So the real question is, what DAQ's are available for python.

and the answer is going to be, that daq softwares tend to get written to drive specific hardware systems. e.g. labview makes a whole suite of DAQ boards. there's oscilloscopes and so forth.

As a scientist you don't want to get bogged down building a custom daq. SO the real bottom line is what commercial DAQs are open in design.

The only system I know that might possibly be in the open is the OASIS daq's developed for flow cytometry. these are mass produced and were developed at the National Labs. But I don't know how it is lic.

of course, one can always use an oscilloscope or DAQ made to be controlled by GPIB or simmilar common language. In that case it's pretty easy to write your own as long as you can find a suitable open source GPIB output driver.

Re:He means I think experimental control

[PitaBred]

For those of us (like me) who were confused when the DAQ acronym started being peppered in the parent post, it means "Data AcQuisition".

Re:He means I think experimental control

[Chris Burke]

For those of us (like me) who were confused when the DAQ acronym started being peppered in the parent post, it means "Data AcQuisition".

Oh man, that makes way more sense than what I was thinking, "Dark Age of Quamelot".

Re:He means I think experimental control

[civilizedINTENSITY]

Every lab I've worked in used GPIB controlled instruments. Python "import gpib" makes this trivial.
If you want an environment like MatLab or LabView, there is Scilab:

The scilab serial port interface provides direct access to peripheral devices such as modems, printers, and scientific instruments that you connect to your computer's serial port. This interface is established through a serial port object.
If you want to communicate with PC-compatible data acquisition hardware such as multifunction I/O boards, you need the Data Acquisition Toolbox. If you want to communicate with GPIB- or VISA-compatible instruments, you need the Instrument Control Toolbox. Note that this toolbox also includes additional serial I/O utility functions that facilitate object creation and configuration, instrument communication, and so on.

There are many such toolboxes: DATA ACQUISITION AND REAL-TIME

EPICS and RTEMS

[joelsherrill]

Since you said experimental physics... :)

The Experimental Physics and Industrial Control System (EPICS) is a set of Open Source software tools, libraries and applications developed collaboratively and used worldwide to create distributed soft real-time control systems for scientific instruments such as a particle accelerators, telescopes and other large scientific experiments.

EPICS is often used with the free real-time operating system RTEMS to build custom control systems.

Users of EPICS+RTEMS include Stanford Linear Accelerator Center (SLAC), Argonne National Labs, Brookhaven National Labs, and Canadian Light Source.

RHIC and LHC

[Rubedo]

Here at the Relativistic Heavy Ion Collider at Brookhaven Lab, for large-scale data analysis at plotting we use the open-source C++ framework ROOT (root.cern.ch). ROOT is also used at the LHC. For hardware related tasks the needs of experimental physicists are very specific to the task at hand, and we must hand-write most of the code we need. Only for very small-scale projects can software like LabView (yuck) be useful, and there just isn't any open-source equivalent of Labview that I'm aware of.

Labview

[Brit_in_the_USA]

I find labview very very good for experimental work. There are educational licenses and site licenses, and with the compiler you can distribute your programs for free.

At the end of the day I could always justify the expense of the software when the equipment it is controlling is orders of magnitude more expensive and some of the automation I was able to roll out saved weeks if not months of time.

The is also a great community of labview users who freely share (source) code they have developed - many under specific open source licenses.

At the end of the day I wouldn't get to hung up about open source when you are dealing with equipment and budgets where the software is 1% of the cost - just use the best tool to meet the requirement or risk project overrun and funding issues.

Re:Labview

[j_cavera]

I've been a LabVIEW developer for 20 years now (since v2 came out in '89) and a C-coder for about almost as long and I can say this with about 99% certainty: LabVIEW is not for everything, but what it is good at, there is no good replacement (open source or otherwise). LabVIEW is second to none for data acquisition, control, (some) analysis, (some) simulation and (some) SCADA. On the flip side, unless you've a lot of experience with LabVIEW and/or a lot of time to kill, don't try anything with recursion, distributed computing or high-end visualization. I guess I'm not really sure what the problem is here: For less than $2k, you can pick up a copy of LabVIEW and save your boss hundreds of hours of your time. For about $5k, you can get the whole dev package and compile things to .exe's for deployment all over the company.

Just my $0.02...

Re:Labview

[JustinOpinion]

At the end of the day I could always justify the expense of the software when the equipment it is controlling is orders of magnitude more expensive and some of the automation I was able to roll out saved weeks if not months of time.

Depends very much on the application. I once had a LabVIEW system that was getting camera input. I needed to do some simple image processing (and I mean really simple), but the only way I could do that in LabVIEW was to buy a gigantic add-on package that was very expensive. So in the end I coded it myself, but what should have been a 5-minute coding exercise (just modify one of the LabVIEW VIs to export some data) ended up being quite a bit more complicated because LabVIEW is closed-source (so I couldn't just go into a VI and make the simple change I needed).

This is just one example where LabVIEW's closed-source nature slowed me down. On the other hand, the community is quite good and there are many useful VIs being freely offered on the forums.

At the end of the day I wouldn't get to hung up about open source when you are dealing with equipment and budgets where the software is 1% of the cost - just use the best tool to meet the requirement or risk project overrun and funding issues.

Open-source isn't just about money. The ability to modify the code can frequently be very helpful, especially when you're trying to do something non-traditional (which seems like all the time in experimental physics!). In science there is also the issue of "knowing what's going on" inside the software if you're ever going to trust the data that comes out of it. And we shouldn't ignore the educational benefit (and all-around fun) of being able to get into code and modify it. Science is about learning (in particular for students; but tinkering is also crucial for established scientists!).

Re:Labview

[diddleydoo]

You know that you can bind external DLLs relatively easily in LabVIEW don't you?

There's nothing stopping you from using a DLL from an open-source program for the image processing.

Shane.

Re:Labview

[The_Wilschon]

when you're trying to do something non-traditional (which should be all the time in experimental physics!).

There fixed it for you.

My experience with LabView (used it once for a little undergrad acoustics experiment) was that it didn't do things quite the way I expected from its presentation. The whole block diagram data flow model thing made me think that everything was going to run in parallel. In reality, it did not, so real-time processing just wasn't quite the same thing as building a circuit that did what it looked like LabView was going to do. Of course, this isn't a very strong complaint, and somebody who used it often would quickly learn exactly what it did and did not do and how.

My own offering: In high energy experiment, we use primarily ROOT (root.cern.ch), which is a horrible, bloated, feeping creature, but gets the job done. Replacing or rewriting it would be too much effort, so everyone just limps along and pretends there isn't an elephant in the room. I can't say I want to recommend using it, but I do recommend using it anyway. At least it is FOSS, so you can in principle fix it if you want.

astronomy

[interglossa]

A lot of astronomers use IDL but NASA projects in general must put their software in the public domain, so it is not surprising to find packages like the Astronomical Image Processing System (AIPS), the C Flexible Image Transport System (FITS) I/O (CFITSIO) libraries and many other packages and interfaces in SciPy.

Replacement for Eagle

[grimsnaggle]

It's a tad less polished and maybe a bit more buggy than Eagle, but FreePCB ( here ) is FOSS Windows software. It works sufficiently well for me to be very productive. I've used it to make all of the circuitry for the Stanford Solar Car Project ( here ).

It's much easier to use than Eagle and does make you go through as much bigma to get a board made.

Re:apt-cache search

[habig]

The vast majority of this list are packages in the old, fortran-based CERNLIB. Still around bug in legacy code mode. The modern replacement is Root.

Aimed squarely at high energy experimentalists, and used for data analysis and simulation. In that field, DAQ code is nearly universally highly custom since the electronics it is interfacing with is also custom. Usually built with FOSS tools, though, such as gcc.

GnuRadio as an Labview alternative

[Falstius]

I wrote the DAQ I use for my research using the GnuRadio framework (which is written in python and c++). It is relatively easy to write custom C++ modules for high speed computation. There is also a graphical programming utility called GRC for simple LabView style "programming". Unfortunately, the code still isn't very mature, but it is under very active development. It is difficult to argue with the simplicity of labview for a quick interface to NI hardware, but once you leave the academic world where licenses and hardware cost real money, the appeal is quickly lost.

When I get time, I'll play with comedi and the comedi module for gnuradio for working with NI DAQ cards.

GRACE

[digitalderbs]

I use all of the tools listed by other commentators. But one of the most useful tools not yet listed is GRACE (or xmgrace). It produces publication quality figures, includes many useful features like linear and non-linear regression, statistical analysis, convolutions, interpolations, Fourier transforms and it supports complicated multi-graph overlays well.

Re:GRACE

[16384]

xmgrace graphics aren't very pretty, at least when compared with matplotlib (http://matplotlib.sourceforge.net/gallery.html. I still use it for quick and dirty graphics, otherwise I use matplotlib.

All things considered, I recommend IGOR

[BishopBerkeley]

In the interest of full disclosure, I've been an IGOR user for about a decade, and I have worked for the company as a consultant.

IGOR is a fantastic tool. Will it do everything? Just about. Its programmability makes it nearly infinitely flexible.

But, for me, the two features that make me shudder at the thought of dispensing with IGOR are the support and OS interoperability.

IGOR Pro is perhaps the best supported and best documented piece of software on the planet. (Yes, I have contributed a minor amount to said documentation.) The company stands behind the program completely, and the user community usually offers solutions to anyone's problem within minutes. And, the documentation is simply phenomenal: the comprehensive and highly linked pdf version of the manual ships with the software is also installed as a searchable, highly organized database that can be navigated many ways.

Then, of course, is the bonus that everyone gets with their purchase of IGOR: you get Mac and Windows versions as part of your purchase. (I've been told the Windows version runs well on Linux under Crossover.)

And, oh yeah, the cost is a fraction of Matlab's or LabView's.

Scientific Linux Is for You!

[b4upoo]

Scientific Linux OS is written by the scientists and engineers at CERN!
      Its base is from Red Hat.
      It is a jewel of an OS and is exactly what you need.
      Distrowatch.com probably has it listed and a path to downloading it. It is free.
      Good luck and do something good that helps people!

What are you asking?

[DoctorNathaniel]

Professional experimental physicist here.

There are two main types of software physicists have to deal with: hardware interface, and data analysis.

Hardware interface is often the the tougher one: slow controls, data aquistion, environmental monitoring - these all need to interface to hardware through various drivers. LabView is an obvious candidate for table-top experiments, since it is possible to set up working control and readout systems more or less out of the box. There is really no good open-source solution for this for the same reason that open-source drivers took a long time coming to Linux: the user base is just too small to write the code.

My own experience is that it's far better to write your own code, using whatever drivers you can scrounge - it's far more efficient at getting what YOU want done as quickly as possible once it's running. However, the time to write and debug this code is extensive. It's particularly bad since often students will write this code and then disappear, leaving you with badly-documented half-working code.

However, this is basically true of many LabView installations as well.

On the data analysis side, there are many good packages that serve as starting points. ROOT (http://root.cern.ch) is an excellent package for doing event-based data analysis in nuclear and high-energy physics, including efficient ntuple storage and histogramming. It's really a toolkit, not a program, so it allows you to do your own analysis by writing your own code.

I'm not familiar with other big packages, but I know that I frequently use raw C, C++, gnuplot, perl, and python to do little jobs.

There are other tasks as well. Blogging software can be good for logbooks. Wikis are good for in-house documentation.

It really depends on specifics. But basically it depends on where your project falls on the quick-and-dirty vs long-life vs high-performance judgements.

Sage

[hoytak]

http://www.sagemath.org/. Sage rocks. It's python based, brings together many of the useful libraries already mentioned (numpy, scipy, matplotlib, etc.), and has a very active mailing list. Can't recommend it enough.

Perl Data Language

[Dr. Zowie]

Perl Data Language is quite nice -- it has all the blended hackish glue-code goodness of Perl (which could be a plus or a minus depending on your personal style). CPAN (the big Perl repository) has a lot of free stuff for access to serial, parallel, and USB ports so you can control your equipment and acquire data. PerlXS (built into Perl) gives you nice access to C libraries, and PP (a meta-language that comes with PDL) helps you sidestep a lot of cruft if you have to use a C module to import large volumes of data.

PDL has built-in commands for plotting 2-D stuff via PGPLOT or PLPLOT, and 3-D stuff via GL. I use it for all my publications.

Some prefer NumPy or SciPy, the Python equivalents of PDL, but (IMAO) Python isn't as expressive as Perl, and the external libraries, while extensive, cannot compete with CPAN for completeness or hugeness.

f2py

[goombah99]

Python has mad fortran all the mor valuable. the combination of fortran and python is a match made in heaven.

fortran is a wonderfully good algorithmic language but it is terrible at everything else. python is great at everything else, but slow as a dog.

one of fortrans strengths when combined with python is oddly enough it's simplicity and lack of sophistication. unlike c++ It compiles so fast that you can feasibly have python write fortran on the fly optimized for the algorithm and compile it, all during run time.

Fortran is good for non-computer scientist in part because it is quite hard for a typo to be a logic error. (e.g. = instead of ==) as a result it's comparatively easy to debug compared to C.

but it can't do all the fun memory management tricks, introspection, or objects that advanced languages do.

But with f2py you can push all of that into python and then just have it call short simple fortran routines for speed.

Re:f2py

[Bill_the_Engineer]

Agree. However, Fortran still evolves and Fortran 2003 (and 2008) is suppose to add some OO capabilities beyond encapsulation in modules.

I will point out that I've seen more Perl than Python in scientific computing. But most of my scientist coworkers tend to be older..

Things i have tried.

[drolli]

My different generations of meas and evaluation software stacks

Measurement software:

0) C
1) C/perl/java evaluation
2) C/java/jython evaluation:matlab
3) matlab/labview (connected by http+xml+xslt)
4) matlab with DAQ+instrument control toolbox
5) C/tcl evaluation:matlab; real time plotting: gnuplot

C was only used to bind non-preexisting daq functions. I dont write GUIs in C. For GUIs i prefer java, and the best free combinations i used were 2) and 5) would i be free to choose i would use 2), however in the environment i am currently working some co-workers write in tcl, so i found in more productive to learn tcl. (Not because of my productivity, but learn fast. However, one more conflict in the group could have killed the group. This selection was purely driven by social purpose - i gave to say: sucessfully.).

Otherwise the general idea would be that to use java and implement the GUI in java and the scripting languange in one of the many languages existing on the top of java. If you use a standalon scriptiong language for performance reasosn (i tested that once, but i could affot to go without it): python is a great choice for semi-numerical (and numerical) programs.

For evaluation i used and

1) perl
2) python
3) octave
4) matlab

1-3) each plotting with gnuplot. I know there are other solutions around, but when i tested them, gnuplot was always the rock-stable, feature-rich workhorse (think: LaTeX friendly output: epslatex - great in combination wit a makefile for you figures in a thesis!), with the only disadvantage of some things beeing historically grown.

4) is great, but do yourself the favor and alway try to keep the core functionality working in octave. I mean matlab is a great product, but you may prefer not to tie yourself to a package which costs than much money (think about what will happen after you leave your current univerity? do you still want to access you old data?). My current strategy is to keep the functions loading experimental data/evaluating it working in octave, and gui functions in matlab (only exception: i use the statistical toolbox moderatly. I could rewrite the function i use, but it would take a week. I am working in a reasearch company, and here it is more reasonable to buy the toolbox.

Oh. and by the way from time to time i am misfortunate enough to work in labview. it is a PITA. My personal experience say: rewriting the program in another language usually saved time.

Maybe it's Alzheimer's but...

[Bentronathon]

Does anyone remember that category where you could submit questions to /. and then get responses from the community? I wonder what happened to that...

Python(x,y)

[artor3]

I'm sure this will be lost among the sea of comments, but I'll throw it out there anyway. As a former LabVIEW/Matlab user, I have had tremendous success with the Python(x,y) package.

All the development is done in Eclipse. You have a WYSIWYG editor built into Eclipse, with which to design Qt GUIs. The signal/slot methodology in Qt is similar to LabVIEW, but easier to work with once you've learned it, since you can make connections in text or the drag and drop interface, whichever is easier.

You can use the generic Qt widgets to display Matplotlib graphs, which use similar syntax to Matlab.

You have numpy/scipy, which are fantastic for processing data.

You have pyvisa/serial/parallel, which will control any instrument you have, or even ones you make/program yourself, such as FPGAs or MCUs.

You can use Python's pickling to store your data in rapidly-accessible modes (I've opened hundred million datapoint sets in seconds), or CSV/Excelerator to store smaller chunks of data in more human-readable modes. Python also provides a fairly simple database system if you need one, and you can always use Zope or Django if you want a web interface (though that will be harder to learn).

On top of all that, there are many more field-specific modules available in Python than in LabVIEW.

I've been using it for months now (I should emphasize that I had never even heard of Qt or numpy or any of those things before that), and I cannot wait to drop LabVIEW. If you think Perl is bad, try to debug a 5 year old LabVIEW program, developed by ten different people, each of whom was using LabVIEW because they don't know how to program properly. One of our old VIs, which only sends a string of commands over a serial port and then reads a DMM, weighed in at over 350 MB. No one can even figure out how it works any more. The same system, done with Python(x,y), came out to under 200 lines of code.