Slashdot Mirror
Running a Research Lab on Free Software?
"[Hardware Manufacturers] seem to get very upset when somebody asks them what the register-level interface to their card is. Who could blame them? Their Windows DLL is the perfect solution under [most] circumstances.
I'm not the only one around here getting frustrated, but all before me have been defeated. It seems I am to be as well, for today I have started to learn Visual Basic.
Has anyone had any *positive* experiences trying to move a lab from proprietary to free software? Surely the government-funded researchers of the world have a responsibility to ensure that their work is free, as in freedom. However, I have found out the hard way that it's usually just not worth the effort, following such ideals. You just get frustrated by apathetic colleagues, useless product support, and the conventional wisdom that it's OK to ignore your ideals, so long as you get the experiment working. Additionally, my ordeals convince my peers that free software isn't worth the trouble."
Funny thing is, the two and a half weeks that the "all BSD" lab lasted, we had no work orders concerning crashes. I learned BSD only because the LAN manager in question made me learn it to move a email server over to it (Novell GroupWise used to be VERY expensive, and this square-headed manager didn't want to pay a bunch of cash to run one secure and isolated mail server on Novell stuff).
Business is business I guess.
Actually, I sometimes feel that this is the number one symptom of the obstacles that OSS/free software face. Multiple times, I have been labeled a nut for wanting to us Linux/OSS. I had a roommate in the dorms once who insisted that I would be "happier" using Windows.
As far as experience moving labs from Windows to OSS, I have never moved an entire "lab" so-to-speak to Linux, but it is my experience that in the past three or so years since I was introduced to Linux, it has made big inroads into becoming easier to install and use. The hardware support is definitely better. I remember when I first tried Linux, support for USB mice was still experimental.
These days, migrations from Windows to Linux have been relatively smoothe. I've had great success moving a fileserver from a Windows NT workstation to a less-used Linux box. Usually, it will eventually go down because of a power outage - so I recently purchased a UPS.
The bottom line is, I think that the problem of colleages who drag their feet on using Linux/OSS will be reduced as major distros become increasingly easy to setup and operate and hardware support improves.
Wh47 d1d j00 541, 31337 15n't t3h r0xor5 ne m0r3???
Forget the old stuff, concentrate on new projects, number crunching and data servers. Use the strengths of free software first, then move into areas you can. Old stuff is working, as well as any old M$ junk works, and you will only be frustrated working with equipment no one cares about. "Rebuild" it till it smokes. You are familiar with the superiority of free software available for networking, number crunching and programming. Take old computers no one wants and make them useful. Something as simple as databases, web servers and email servers are helpful. Take the hardware advice you get here and put it into your next proposal. The bottom line will speak for itself and you will have proven the dependability of free software on comodity hardware already.
Friends don't help friends install M$ junk.
I'm doing my Ph.D. at a mobile robotics lab. Most (all?) of the robots we bought, (either from iRobot or ActivMedia) came with Linux pre-installed. I don't think we could even have Windows if we wanted it. I think this is going to be much more frequent in the future. For things that don't have a display anyway (no we don't have 17" monitors on our robots!) what's the point of running Windows?
Opus: the Swiss army knife of audio codec
Here's the problem, though (I've faced nearly exactly the same issue as the original poster).
1. You have programming skills, no one else in the lab does (they're all scientists with pure science backgrounds, you're the only one with a mixed engineering/science background), except the staff programmer. He is good, but also thinks there's nothing wrong with VB. Fine. Your code needs to be read, supported, and developed once you're gone.
2. You need a little experimental control program with a GUI that is reasonably easy to debug and modify, and you need it TWO MONTHS AGO.
Deploying something that meets both of these criteria is relatively straightforward if you swallow your pride and write something in VB. Hell, all of the interface card and hardware vendors provide VB example code. Doing it any other way (remember, GUI, easily changed, *yesterday*, etc.) is a heck of a lot harder. Either that, or I'm way behind in the quality of Open Source RAD tools.
Good luck to the original poster. Like many of his predecesors, I gave up and wrote the experimental control in VB. But my data analysis and primary desktop environment is still Linux.
Put my fist through my alarm clock with its ding-dong death inside my ear. - The Blackjacks.
who run the labs. In our lab we wouldn't make the purchase when the products could't work on more than one open platforms. It's not like we hate Windows and proprietary products.
A researcher with certain year of experience would have thru the nightmare of getting obsoleted hardware to work - it's not that the hardware itself is getting out-dated, it's about its interface i.e. the device driver. E.g. the VPL optical gloves are still usable today as we've the driver source came with them, even when the company behind were out of contact for years; while we have to kiss those 3d controllers goodbye when their Windows device driver no longer functional in modern Windows system.
Really, we had no problem when we had to code in VB, as long as it works effectively, and gives results for our papers. We just don't want our money wasted on endless upgrade-cycle(if upgradable), we are not running a corporation afterall.
Well it's so simple you're gonna laugh, I'm controlling a two-axis pulsed-laser spectrometer. The problem is that I've got a National Instruments PCI-7324 stepper motor card to work with and even Comedi (which looks kewl BTW) doesn't support it. I can't blame them, there's just no information out there to start with. NI won't even reply to my emails.
:(
I thought it would be clever to implement a GUI using QT and then simply compiling on Windows with the stubs filled in to use the DLL. How wrong I was. The GUI looks great, but Trolltech's official position is that the Non-Commercial version of QT for Windows isn't supported any more, so you have to use QT 2.3 when 3.1 is out for Linux. I would have even ported my code back to 2.3 but then I would still need MS Visual C++ or Borland C++ Builder 5 (yes, I have version 6 and that's not binary compatible with the QT distribution). In case you're wondering, I did try MinGW but in the time I had available I didn't get anywhere because qmake still looks for a C++ Builder 5 DLL which I don't have. I could download the MS VC++ version of QT 2.3 but I expect to have the same problems all over again. And really if you've paid good money for a development environment like Borland C++ Builder 6, you might as well just use it instead. Oh and I did try to get the NI software running under Wine, I think I might have had some luck that way but I couldn't see how to get the Windows DLL talking to the PCI card in the time I had.
Then came a flurry of ideas for changing the stepper card to something else, say a timer/counter card, and use Comedi. But I quickly came to the conclusion that I've wasted enough time, I have a working stepper card already, and my supervisor really wanted a product using VB anyway because it's the easiest way to get physicists with zero coding experience to use and modify the application to fit their needs. It's logic I can't really beat right now, I wish it weren't so, I *so* wish I could hand him a free software solution with the same benefits to him as a VB solution, but I just can't.
Oh and if you were still wondering, I have to use Win95. That or replace it with the free (as in beer) OS of my choice, which would be Mandrake 9.1. Maybe I'll get an upgrade to Win2K but it bothers me to pay good money for a piece of software I wouldn't need if a) the original Win95 worked without blue-screening randomly, or b) NI would have some Linux support.
There's a guy upstairs who used to have an entire lab running GNU/Linux until he battled Agilent's and NI's lousy Linux support. He finally gave up and converted his lab from Linux to Windows (at great financial cost) so he could keep working on his experiments. Along came the customary driver conflicts, forced expensive updates, etc. etc. It's a sad tale, and it's what I was told when I began asking around in desperation. Now I know he wasn't kidding around
I work with a research professor at the University of Washington. I'm hard pressed to find a windows machine on my floor (I think there are two, and they suck). Everything is Solaris. The main server room has a huge Sun Enterprise Server and a 19 computer linux cluster. My group just got its own 2TB storage server, it's linux based... The whole dept. seems to moving from Solaris => Linux if anything... but no where near Windows.
Consider whether you really believe that ALL software NEEDS to be open source, I mean, as long as the OS and the core tools are, is there really a problem with someone writing a fairly esoteric piece of software that few people will ever use and charging money for it?
I suppose VB isn't really that esoteric, but as others in this thread have pointed out, open source isn't free-as-in-free-beer, because it takes someone time to write it, and if that someone is you, then your time costs money. Bite the bullet, spend the time researching if the system works.
I've been told that our lab distrusts LabView because of previous irritations. I don't have any experience with it personally. But as for the Linux version, there still aren't any drivers for my stepper board (NI PCI-7324).
Your time is your most important resource. Don't waste it recoding.
That's some good advice, I agree. But still my ideals die hard.
Comedi is awsome.
I second that sentiment!
"Alessandro Rubini's excellent book Linux Device Drivers (another fine O'Reilly publication)" Ahh, knowledge, what could be finer. Free software, free info. Go get it and become the research tech God you want to be.
Well I've read Linux Device Drivers, nearly cover to cover, and I've implemented Linux drivers for the chip I designed during my Engineering Honours project. So I'm not scared of doing it although I know of how much effort it takes. The main thing is you need to have specs for the card. Such things don't seem to exist outside of NI's vaults.
We have a lightning research group (previously mentioned on Slashdot here and here). We use LabView 6.1, on RedHat (7?), to communicate with PIC microcontroller-based instrument control boxes out in the field, to continously monitor the local electric field (which is a good indicator of the favorability of lightning initiation, either natural or triggered), and to arm, calibrate, and disarm all of the oscilloscopes in the various experiments. We leave the system unattended during the winter months, just in case a frontal storm comes through and we get a strike within a half kilometer or so...
.VI files are available to access the Comedi drivers from LabVIEW, and our fields monitoring program appears to have been running, unattended, without restart, for over 100 days on our dual Athlon MP 1900+ box.
We use the Comedi drivers to interface with our National Instruments DAQ card.
And if you can't program in LabVIEW, you might oughta practice asking if I want fries with that burger.
Ce n'est pas un vrai mouvement de robot!
There's two basic tricks I've discovered over the last couple of years of slowly incresing the use of open source platforms and tools in our research (ok, we do behavioral science stuff, but the politics of IT change are the same).
First, re convincing colleagues that open source / free software has a role in your work: do something they envy. Produce a tool they want to use, or find some existing software that does something useful and cool, or even just do the great unix thing of tying a bunch of small programs that do one or two things well together to do something that no existing monolithic package really offers. Then point out that it either can't be done on the current platform of choice, or, while it can be done, it requires spending $$$ on some proprietary solution. Doing something like this tends to legitimize the use of the toolset you'd like to use, and gives you a good foot in the door for more abitious moves later.
Second, re working with third party suppliers who don't currently produce software or drivers or whatever that work with non-MS platforms. If there's more than one vendor who supplies something that does what you want, pick the smallest one. They're more likely to be interested in finding niche markets, less likely to be bogged down by bureacracy when it comes to doing something new or different. And a three-person company is more likely to have two of the three who've recently been working in your field & remember what it's like trying to do the usual research thing of trying to get an existing tool to do something that no-one's done before - hence more likely to give you access to the kind of more detailed information you might need, even if they can't really expend the effort themselves right now.
Anyway, that's my take on 'what worked' after a couple of years of win-some, lose-some politics around research and IT.
I work for a Navy research lab and we use Linux, Perl, C, Apache, Mozilla, Gnome, Perl/Tk, GnuPlot, etc almost exclusively for our control and monitoring systems. We have had no difficulty controlling Opto22 devices, GPIB, RS422, R485, and one-wire devices. The only windows we use is on the desktop which is unfortunately mandated (see NMCI).
If you need real-time look into QNX.
BTW we operate what is probably the deepest running webserver. We have a vehicle monitoring system that is controlled via a Linux/Apache machine setting at depth reporting topside via fiber connection. This system controls power reports temperatures, pressures, inclination, depth, smoke, leak, etc.
I've been working with one of the Physics profs at my school (U of Rochester) for the past year, helping him update the software they use for cosmic particle experiments. We use a data aquisition board and particle detectors designed by FermiLab. The software runs on Linux, and accesses the DAQ board through the serial port. My job has mostly been adding a GUI to the program, so that the students running the experiments can concentrate more on getting results than understanding the weird command line interface for the program. For more info on the project, see the FermiLab page for the QuarkNet project, and the PARTICLE project page at the university.
We set up a small undergraduate research lab at the University of Manitoba's EE Dept. For the summer we are doing research in networking and telecommunications. All of our workstations are running slackware linux and we find most of what we need in this distro. OK, so this is slightly contrived (we're doing networking after all) but we rely big time upon iptables, tcpdump, iptraf, Sun's java, and lotsa unix utilities. I think people underestimate how many useful tools are in linux.
You dared me did ya?
Well according to your next post you think I'm working on NMR quantum computing. I'm not. It's an optical scheme involving coherent transients in rare-earth doped inorganic crystals. So we aren't using a bought system, it's a system that's been put together over a decade, out of all sorts of discrete pieces of equipment, many of which were not bought by us for this purpose, all of which must be usable under a given OS for the whole experiment to use that OS. The only software that really can't be replaced is the hardware drivers, and the whole issue is that they can't be replaced without information from the company.
Whilst you are correct that much of the hardware we use, such as DSOs, waveform generators, etc. are OS-independent (via GPIB busses most of the time), we have some things that we simply can't replace and yet they don't work without proprietary drivers. Such as the USB->ISA converter in the pulse sequencer. We do fortunately know everything we need to know about the ISA card, but ARS Technologies manufactured the USB->ISA converter and it uses a magical black-box driver. No help with the specs when I asked them. So we would have to pull it all apart and use the ISA card directly from the PC (which I'm open to do BTW). Also the NI DAQPads are USB devices with equally mysterious USB converters, once again requiring black-box drivers under Windows.
I'm sure you realise that I am one of the people who *wants* to dump Windows in our lab. But I have to start by proving it's both possible and simple, on my own specific project, which for the present month is to retrofit an old optical spectrometer to be controlled from a PC. And before I got here, we had a PC running Win95, and using a broken VB app that talks to an NI PCI-7324 stepper card.
Now that's the thing that consumed going on two weeks of my time now. Instead of implementing a solution using VB, I felt adventurous. I tried GNU/Linux, but found that there's most likely no way without some hardcore reverse-engineering, to get the card going under Linux. Not all of that time was wasted on the card, some of it went into experiments with QT as a cross-platform development environment. Still, I've given up (at least for now) as I still have to use Windows.
Ah, that's what my knee jerk reaction was all about. Discresion is the better part of valor. Use hardware that's got drivers already for new projects and use the old junk till it blows up. I presume someone will have sucess stories that I was only able to dream of four years ago.
I get ill thinking of VB. Learning C++ and the win95 API was easier for me. One week of happy brainwash VB training tape scarred me for life, "methods" twitch. I saved myself from that hell with a nice little "windows XX API how-to" book with examples and a watcom compiler. It made sense and offered greater control. MFC was required to talk to devices and it was a step in the wrong direction but control, display and communication modules were still seperate Others were lazy or stupid and spageti code VB was used on many other projects with horrid USB interfaces.
Good luck, you suffer a legacy of bad choices and are going to be forced onto VB. So you enter the downward spiral of the M$ maze, chasing mindless changes, befudled by tools that don't work the way they should and mysterious crashes, delays and poor data rates. You shall suffer nights of rebuilding win3.1 machines to take care of those old DA boards that don't have win2k much less XP drivers. If you can even read the poorly commented and ill disciplied spagetti code you have, you will suffer the pain of "porting" VB 4, 5, 6 to whatever is the current version, which might require complete re-writes to save time. Read letters to the editor in VBmagazine if you don't believe me. The more you learn, the worse it looks. Don't gripe too hard, the boss might have written some of that crap.
When it's all said and done, using a seperate machine listen to the device and learn how to talk to it might save you time. Selecting reasonable hardware for future project surely will save you time. Once you get aquisition working once you will be able to replace legacy stuff that breaks. The VB/disposable hardware route can hardly be called a success story.
Friends don't help friends install M$ junk.
In some cases, esp. where federal monies are concerned, law requires that if a commercial solution exists already that they be used over development of similar/same alternatives. So is the environment is primarily windows (or some versions of linux) then development with free software, where similar commercial products are available, can be illegal.
I've been using Delphi for over 2 years now and I can say that Delphi is on par with C++. In Delphi you're not limited to the handful of functions that VB gives you and you can choose to go deeper if you want to, straight down to the assembly level. /. knocks Delphi down as another "Quick 'n dirty" development environment but it most certainly isn't.
Everybody here on
Yuioup
You're REALLY not gonna like this. Not long ago NI came and gave a talk at my school (University of Texas) and basically said "the future is .NET." When asked about Linux, his response was basically "Why would anyone want to use Linux? Windows works." This angered a lot of the crowd (mostly CS students) and then he added that any job applicants better learn .NET or they had no chance. They really don't seem interested in Linux at all.
The basic fact of the matter is that there are industries where free software is not an option. Sometimes you have to abandon your ideals to get things done (as it seems those who came before you did.) Why do you think researchers are always begging for money? They have to pay for expensive, shitty software and there's no other alternative to do the things they want to do. It sucks, but Linux isn't the best solution to every problem (it probably would be in this case, but it's not an option because your hardware doesn't work.) Sometimes that's the way it goes.
My point was that Delphi is the best of both worlds: it was equally powerful whether you needed it for a quick`n`dirty GUI app or an extremely complex project.
Perl - $Just @when->$you ${thought} s/yn/tax/ &couldn\'t %get $worse;
I'm a Physics UG at the University of Oxford, and have done the odd part-time job writing code for some of the groups (I worked for CMP last year and will start in the Teaching Course next month). All of the code that I have written was released as OSS, because other research groups can then modify it and use it in their experiments, without affecting the originality of our work.
The systems I worked on in CMP were almost entirely Red Hat, and some of the code I had to work with was being ported to C/Linux from Delphi/Windows it was stupid to ask people to pay for Delphi licenses just to tweak a few lines of code in a detector controller. Some of the older computers were running Win95 with X11 and SSH clients and were just used as terminals for the beefier Linux boxen.
The computer I'll be using in my next project is a little meatier, a four CPU Sparc box. There are a couple of these in the department, each with a few dozen SunRays attached. The code developed on here will still be OSS, and the box has a fair amount of the GNU userland sitting side-by-side with the Solaris rubbish.
Other than that, my knowledge is a bit sketchy and is inferred through what I hear or see around the department. There are a number of Windows workies around, because software like SPSS or Origin or Minitab only exists on Windows, and there either don't exist Unix or Linux alternatives, or they aren't yet mature enough to want to switch onto. Many of these workstations also have Cygwin or at least DJGPP.
But the main point to be answered is whether or not a research group should be OSS. My opinion is yes as far as in-house code is concerned, because this facilitates collaboration between groups using similar code, hence quickly smoothing bugs. OSS also neatly fits the philosophy of shared information many scientists have.
Should the computers all use Open Source operating systems? I just don't think this should be a requirement in most cases. Yes Linux and the BSDs are stable and mature enough to be used in a research environment, but then so are Slowlaris and Mac OS X, and you can develop and build your code on both of these systems too. The only situation OTTOMH in which having an OSS OS would be directly beneficial is when custom-built embedded devices are required, for instance in HEP detectors or beam controllers. In these circumstances the ability to modify the OS kernel would be useful, but if you're just developing your C code in a Unix environment to be used on bog-standard Unix or Linux machines, then a good set of manpages and a functional cc is really all that's needed.
Great for prototyping. Horrible for large projects. Try debugging nested loops. What a friggin mess. Anyone who suggests LabVIEW for anything other than a one-day project should be shot. I spent (wasted) 1 1/2 years doing a commercial project with it. Seems nice at first, but slowly you very quickly get diminishing returns from it. I'd rather do stuff in assembler. At least I can see what the fuck's going on. Optimization sucks because who the hell knows what NI does under the hood. Go with VB or C/C++ or ANYTHING before LabVIEW. Please.
Recently this went so far that I had a very candid talk with one of their sales people. I made it clear I would move to a different manufacturer if they would not provide drivers or some means for me to write them. He would ask within their organisation. Several days later I received an email titled "Solution to your GPIB driver problem". To my astonishment, it gave contact details for several other manufacturers of GPIB cards!
So, isn't this weird? They'd rather lose our custom than provide us with a driver, or sufficient details to write our own! And given the amount of stuff we are buying from them that's a pretty big decision.
So now we are talking with National Instruments, and they are very pleased to have us as customers. Moreover, they have Linux drivers for their GPIB cards. I haven't seen these drivers yet, and I sure hope they will be of acceptable quality.
I'm interested in hearing what experiences other people here have had with GPIB cards under Linux - especially those of NI, which we are about to buy. When you mention lousy Linux support, does that include their GPIB cards?
I was in a similar position...I was using Linux to run a laser tweezer setup in our small research group. Linux was such a pain - inconsistent libraries, poor support, flaky drivers unsupported by vendors, etc.
I switched to Windows 2000 Pro for the vastly superior support for data acquisition hardware, framegrabbers, and so on. Along the way I switched from doing GUI stuff in wxWindows/C++ to Matlab. The GUI stuff in Matlab 6.x is pretty good. It's got a bigger learning curve than LabView but it can also be used to implement much of your data analysis as well, with its vast libraries of mathematical functions and decent performance.
With the new hardware coming out, our lab now has a standing policy to only purchase items with USB or FireWire interfaces whenever possible, so the PCs don't even need to be opened up anymore. These drivers are rarely if ever available for Linux.
Use whatever makes your lab the most productive. Standard lab software like LabView, Matlab, and Mathematica are a safe way to implement software since they're so popular and they're more efficient and productive than C/C++ or VB. If you need high performance computing, then go consider F90, C++, etc., but instrumentation should never be controlled like that. You need real time? Buy a real time board.
The problem is, even in the 20K-200K range of hardware, the software usually sucks. Or the install isn't stable.
One example, (albiet cheaper one) I've been dealing with is a simple scale that sends data over a serial port to a PC. Using VB, you can occasionally get reliable results while taking data into Excel. Even through the software to do this cost as much as the scale.
On the other side of my fence, I've been using Linux for about a year now to do video capture and acquire temperatures from a networked DAQ unit. It's effortless after installed.
I'm looking into similar linux based solutions to running two old units, a DDA-06 and a DAS-16 to record thermocuple data realtime. The linux projects for these are actually really good, far exceding the windows based controls. The problem is the age of the boards (ISA, mid 80's) which are actually still sold today by several vendors including Omega. Unfortunatly, you need to be a god of electrical juices inorder to calibrate the darn things.
-=fshalor
I'm an astrophysicist in a non-profit research institute. We have very few 'special' hardware requirements like the poster, and so are not as limited by what OS hardware vendors are willing to support, but I've come across similar situations.
When it comes to proprietary hardware with proprietary drivers on a limited number of (non-free) OSes, you're stuck. Everytime you talk to the vendor, or the support staff, you need to ask them about Linux (or BSD, or whatever) support. Be the bee in their bonnet that gets them thinking about supporting other operating systems. They don't necessarily have to GPL their software and drivers (I know, sacrilege!), but the ability to use your hardware on Linux means your one step closer to moving the lab over to Linux. Also, even if the data-acquisition is on, for example, Windows, that doesn't mean the data-analysis has to be on the same OS.... (Unless the data format is proprietary =8-( )
However, being a researcher, you should be used to the concept of peer-reviewed publications: nothing is published in established journals without having being scrutinized by other researchers in the same field. The same concept applies to OSS: open source software, at sometime in its public life, is viewed by enough people that bugs, cheats, etc., will probably be caught (things slip through, as they do in the scientific peer-review process, but the idea is sound). If you're doing research in the uncharted regions of physical science, you can't expect that someone would have written all of the software you need to get there and understand what you discover. This means that you, the researcher, are obligated to write the software. This software should be open sourced and peer-reviewed, saving your collegues and other researchers the same headache. What is done in scientific research is often governed by the 'publish or perish' doctrine. But keep in mind that what you publish might not have to be scientific papers: if everyone in quantum computing labs around the world knows of or makes use of your software, you will have more exposure than publishing a few obscure papers.
Also, despite the fact that you might be analysing unique data, I would hazard to guess that large amounts of the mathematics and statistics you would be using are not unique. Save yourself time when writing your software and don't re-invent the wheel: use publicly available mathematical and statistical packages. There are enough out there that I'm not going to bother giving URLs...
I look forward to seeing your project (not in VB or C#) on sourceforge =;-)
(Oh, and when your Windows-bound collegues ask about using your software on their OS, you can say "Sorry, it's only supported on Linux." That'll make you feel better, trust me =8-)
#include "cunning_plan.h"