Slashdot Mirror

There are some packages on CRAN that claim to implement parallel processing for R -- go to http://cran.r-project.org/web/packages/ and search for the text "parallel" to find several examples. I haven't tried any of them out yet, but sooner or later I'm going to have to.

And actually, I think that "scripting" languages in general will have a very bright future in the parallel processing world. If memory management and garbage collection are implemented invisibly (and well!) in the core language, then the programmer can concentrate on the application logic and not have to worry about the kind of allocation headaches discussed in TFA. Python and R, where I spend most of my coding time these days, both offer very nicely implemented versions of function mapping, which I see as the key to making multiple processors useful for a wide variety of tasks. And no, the memory management and GC aren't quite there yet in either language, but they will be.

Hmm, reminds me a bit of R, the plotting part of it, at least. There are a few examples of the kind of plotting you can achieve in R here.

Don't bother saying anything about KOffice or any other Office product becoming popular until it can be installed on Windows with a setup.exe or an MSI.

- First as you said yourself in your follow-up : KOffice is part of the KDE software that can be installed on Windows with their package manager.

- OpenOffice.org
Installs on Windows with a very standard installer.
The only minor problem in my opinion is getting the plugins. It uses the kind of plugin manager as the older versions of FireFox (you can't directly search and browse the installable plugins from there, you have to go to a website first). Also the plugin manager doesn't help you to restart the "quicklaunch" if a restart is needed.
It cool be great if I could install LanguageTool with a simple click from within the manager, the same way as AdBlock+ in recent versions of Firefox. But I'm nit-picking. Back to the subject.

- Gnome Office :
It's not an actual suite, its a lose collection of separate software that cover the needs of an Office suit. All use the same library underneath (GTK+) which has been ported to windows since ages (back at the begining of the GIMP on Windows port). As such you can find installers for :
- AbiWord (word processing)
- Gnumeric (spreadsheet whose accurate statistic formula are done in collaboration with R projet)
(and probably other GTK stuff if you need them).
In fact, as they are small separate software with a very small footprint (compared to behemoths like OO.o), they are quite popular and often recommended for people wanting to build for free small lightweight Windows installation on underpowered hardware.

- For the VI vs. Emacs flamewar combatant out there (the kind who'll immediately scream that they don't need an actual office suite as every needed function and even more is available in some Emacs mode/Vim plugin), both softwares are also available for Windows, if that's your kick. (And yes, I'm not sarcastic. I'm definitely sure that here on /. you'll find at least a dozen of people who can be more productive with a complex emacs-based stack).

So as we can see, the three major players of Linux/BSD's office suites (and the two editors behind most holy wars) are installable on Windows (and on Mac OS X for that matters too).
Yes they are indeed cross-platform.

KOffice was more of a problem until recently the whole KDE switched to Qt4 during is 4.x branch and took opportunity of the major overhaul to be rebuilt with cross-platfrom portability in mind.

I work in a laboratory - a wind tunnel and I don't think the problem is too much specialization. I don't know what the guy's setup is but from my point of view the problem is hardware support. Most hardware manufacturers simply do not acknowledge that there is an OS different from windows and for these guys open source / free software is a completely alien concept.

So basically, if you want to use free software you are going to have to do it all on your own. From low level hardware drivers (with no support or help whatsoever) up to libraries and high level environments.

But do you know what is interesting? At our lab we have some old equipment (30-40 years old) and they all have service manuals that usually includes electronic circuits diagrams. Basically you had all the information to fix anything. But now the only option available is to send the instrument to the manufacturer and hope they are in the mood of fixing it.

Software? It is a disgrace usually. You by a $200.000 equipment and you get a software that handles the instrument pretty well but does nothing else (how do you interface with anothe equipment???) and the damn software includes a hardware key!!! I ask myself who would "pirate" this software? It is useful only with the equipment the company sold for tons of money and nothing else. I mean you already payed a small fortune for it!!! Often they do not provide an API so that you can try to interface the software with another environment. This drives me insane!

Why do we still by equipment from this company? The first reason is that I will usually have the same problem with other options. Another reason is that most of these equipments are expensive so you have to make sure it will work and the best way to ensure this is to get the same stuff some other lab you know uses.

All this could change is there was an awesome open source environment. But here is the catch: most experimenters I know are very poor programmers. But there is light in the end of the tunnel there are projects like http://www.comedi.org/ that provides drivers for DAQ cards. National Instruments has binary only linux drivers for some of their boards.

Just yesterday my boss agreed to release as open source some code I'm developing. It is basically interfaces to the R (http://www.r-project.org/) environment. Much of it is very similar to matlab's daq toolbox. For now I will mostly be using the code on windows (since many instruments we have do not offer any other option) but I will try to slowly migrate to Linux or some other free environment.

Does anybody know of any open source software intended for scientific research? Does anybody work in a lab that makes an effort to use open source software?

We discussed R which describes itself as:

R is a language and environment for statistical computing and graphics. It is a GNU project which is similar to the S language and environment which was developed at Bell Laboratories (formerly AT&T, now Lucent Technologies) by John Chambers and colleagues. R can be considered as a different implementation of S. There are some important differences, but much code written for S runs unaltered under R.

R provides a wide variety of statistical (linear and nonlinear modeling, classical statistical tests, time-series analysis, classification, clustering, ...) and graphical techniques, and is highly extensible. The S language is often the vehicle of choice for research in statistical methodology, and R provides an Open Source route to participation in that activity.

One of R's strengths is the ease with which well-designed publication-quality plots can be produced, including mathematical symbols and formulae where needed. Great care has been taken over the defaults for the minor design choices in graphics, but the user retains full control.

While it's not geared specifically towards experimental physics, that's probably going to be your most fruitful endeavor.

Then there's the Matlab knock-off of Octave which describes itself as:

GNU Octave is a high-level language, primarily intended for numerical computations. It provides a convenient command line interface for solving linear and nonlinear problems numerically, and for performing other numerical experiments using a language that is mostly compatible with Matlab. It may also be used as a batch-oriented language.

Octave has extensive tools for solving common numerical linear algebra problems, finding the roots of nonlinear equations, integrating ordinary functions, manipulating polynomials, and integrating ordinary differential and differential-algebraic equations. It is easily extensible and customizable via user-defined functions written in Octave's own language, or using dynamically loaded modules written in C++, C, Fortran, or other languages.

I'm surprised you're surprised that you only find proprietary software in the highly specialized realm of "experimental physics." I mean, you have to be like a PhD in physics with a good deal of programming knowledge to make something accurate & useful (and there's probably gotta be like 50 failed projects before you get a good successful one).

You're probably wondering why there's not a project of Firefox or OO.o quality for experimental physics but I'll tell you why: it's too specialized and your user base is ridiculously small. You're not going to find a company that is going to benefit greatly (or at all maybe) by releasing their product into the wild for a community to grow. There's probably not a community for it to grow in.

You should tell us what specifically you are looking for something to do ... I have no idea what Labview, Igor, Inventor, or Eagle do. Ask yourself why these programs are standards and then maybe add to Octave's wish list or contribute to it even! Unfortunately, this isn't easy--I myself started to implement proper handling of sparse matrices in Octave but gave up as I was trying to form low level requirements ... You probably already know though that this is going to have to be done in C or another very low level, very quick language.

If you're looking for something specific or outline some high level

Coming from a data mining background, InforSense's platform allows you to not only manipulate your data, perform calculations (and data mining) but then visualize your results in an AJAX environment. You can call open source apps like R and Cytoscape directly from InforSense's workflow building application, so none of your pre-existing work gets discarded.

Have you looked at data mining solutions? Someone mentioned Pentaho already, but there's also:

Rapid Miner

Orange Data Miner

all of which are packed with enterprisey features. But you may have to learn some stats. Once you get past what you can do with the pre-packaged stats methods, then head for R, or write a RapidMiner plugin in python.

There was a thread about the R language a couple of weeks ago. Look it up and read it....

I have used both SAS and R for projects large and small. Each has its own use, but I strongly disagree with the SAS marketing rep's argument that "We have customers who build engines for aircraft. I am happy they are not using freeware when I get on a jet." Actually, I would find it terrifying to think about the engineers using canned packages that are subject to no external review whatsoever, rather than R and its routines, which are freely available for all to see.

When choosing between the two, there are two questions I would ask:
1) What format does my data come in?
2) Is my data small enough to fit in memory?
If the answer to 1 is "SAS dataset" or the answer to 2 is "NO" then I would use SAS. It keeps datasets on disk, so if you are handling TBs (or even GBs) of data, there is no better software package. A lot of big data dumps come in SAS, too, so while it is easy to dump to CSV, if you are repeatedly accessing the same data source, SAS makes sense. Also, if you already have significant infrastructure written in SAS (data processing, reporting, etc.) it may make sense to continue doing it that way.

In all other cases (and even in some of the above cases, when considering overall cost), R is better. Easier to program in; easier to write C/Fortran/Python/etc. extensions for; contains real functions (SAS just has macros); object oriented; and FREE (GPL and zero cost). That last bit is important because if for some reason the SAS Institute goes bust (stranger things have happened), and all of your code is in SAS, you are fucked. Whereas if you have been writing in R, you will always have your existing software and can, in principle, continue developing it yourself or hire someone to maintain/upgrade it for you.

R is great, even if it is a slightly awkward language. It easily blows Octave out of the water in almost every way, but is inferior (as a language) to python+scipy, although I believe that it has far more packages available for it. Just check out CRAN some day. It's amazing.

I'm an electrical engineer. Most folks in my field would use excel and then Matlab. I have no statistical background. I started using it because it has great graphics and plotting capabilities, much better than excel -- more precision, many more types of plots, no jaggies when imported into word, and more. See here for some great examples from Deepayan Sarkar's lattice system, an implementation of William Cleveland's trellis graphics:

http://lmdvr.r-forge.r-project.org/figures/figures.html

That's just one of several ways to create precise, good-looking graphics.

As I used it more, I started to appreciate R's language features, too. It's a great all-purpose data manipulator. It's great with 2-d tables, but handles other data nicely, too.

Along the way, I even picked up some statistics:)

Well, it is a programming language; it's Turing-complete. And there are a fair number of packages on CRAN to make general-purpose programming tasks easier. But yes, mainly it's good for math, especially statistics.

R?

That said, there are doctors with very good statistical skills. He is a molecular pathologist at Johns Hopkins University and one day he came up and asked me if I knew any open source statistical software.

I directed him to R http://www.r-project.org/, where he wrote and used an R implementation of the Kolmogorov-Smirnov test in his next paper. His BA is in chemistry, but it's essentially required that all physicians doing research (anything you ever see published in a journal) have a good knowledge of statistics and the math behind it.

I have found one problem with open source toolchans - producing good quality graphics. At the end of the day you have to present the data, and gnuplot just isn't cutting it anymore.

Try R instead. It's FOSS and the graphics are fantastic. (I haven't used Mathematica, but the 2D plots are better than Matlab's. You can also easily link it up to ggobi or Mondrian for visualising multidimensional data.)

1)First, ESS, Emacs speaks statistics, found at http://ess.r-project.org/ . This lets you interface interactively with R, SAS, Stata, etc., all from the common Emacs interface. As a statistician, it's the one piece of software I could not do very well without!

2) The 'ido' package, with flex matching, in my .emacs,

(require 'ido)
(ido-mode t)
(setq ido-enable-flex-matching t)

This lets you open files and switch buffers with fuzzy matching, really nice when you have lots of things open.

See: http://www.emacsblog.org/2008/05/19/giving-ido-mode-a-second-chance/

3) Make the mouse jump away when you type over it.
(mouse-avoidance-mode 'cat-and-mouse)

4) Open two windows side-by-side (C-x 3) one with LaTeX code, one with a pdf, then use this in your .emacs, (add-hook 'doc-view-mode-hook 'auto-revert-mode), when you compile the .tex file into PDF, the PDF automatically updates in Emacs, I used that a lot while working on my CV.

5) The thunderbird extension that lets me compose replies in Emacs using emacsclient.

6) org-mode http://www.org-mode.org/

7) preview-latex, now part of AUCTeX, this lets you see preview versions of formulae and graphics inline in your .text file, *while you edit*. Your formula is replaced by what it will look like when compiled.

8) EmacsWiki: http://www.emacswiki.org/

All you need is R

I've seen a lecture by the main developer of SAGE. It seems to be more a tool for doing mathematics research. I've heard of scientists using S-PLUS and R (the open source alternative to S) for their research. In any case, any of these tools is probably better than a spreadsheet for serious scientific research.

Particularly when R is free and, being specifically a statistics package, will probably do a better job than a general-purpose mathematics package.

There is already a system for enforcing citation.
It's called referees and editors. It works a lot
better than licenses and lawyers ever will.

I currently have 8 contributed packages on
CRAN.
All have conventional open source licenses.
None say anything about citation.

However R does have a "citation" function that
tells users how to cite R and how to cite R
packages.

Citations are cheap. Scientists will cite if
you just tell them how. It helps if you are
part of a well understood system, so citation
is simple.

One language that is being used in the sceintific community right now is CUDA - which runs on a GPU and is C based.

In addition, Fortran to C tools have been around for some years. To say that Fortran is the only scientific language is BS. R, S Plus, Octave, matlab, perl and CUDA to name a few. Taking R as an example - it provides an code interface that allows you to write optimised C/C++ routines and utilise those in the language itself.

If it were my call, scientists would be required to take a statistics nee optimization course, whose course required the use of the R programming language: http://en.wikipedia.org/wiki/R_programming_language. R combines research grade tools with a language much like Scheme. It has interesting concepts for creating data sets and manipulating them. Finally, some of the algorithms available in this environment simply couldn't be written by a clever student with a semester of comp sci and a copy of "Numerical Recipes".

More info here:

http://www.r-project.org/

My girlfriend is finishing her M.Sc thesis in economics (finance). As one might guess, they use Excel a lot, but at least in her school more sophisticated statistics is done almost exclusively with R. If you haven't checked out R yet, it's a really fantastic tool.

Personally, I'm in computational physics. I mainly use Fortran, C, matlab and python.

How about R?

• Many people are much more proficient in Excel than in any other software, due to familiarity. The cost of moving the data over to other software and then figuring out how to do what you want to do is often not worth the time required.
  
• Spreadsheets are inherently more flexible than a database. This is both a strength and weakness, but there are plenty of times when you need to do some sort of funky calculation to the data that is much, much easier to implement in a spreadsheet than via SQL (calculating a moving average of X records comes to mind).
  
• I would argue that a DB is often not the best choice, given the difficulty of implementing certain calculations in SQL (yes, I know about Codd and relational theory, but just because something can be done in an RDBMS does not mean it can be done easily). Often, something like S+/R, Matlab/Octave, SPSS, Tableau, etc., is the better choice- they generally allow you to perform manipulations as flexibly as in Excel, with better performance and less likelihood of errors, but more easily than doing everything in SQL. But many people aren't familiar with these. That doesn't mean they shouldn't be, but they aren't.
  

Perhaps you already know about GNU octave http://www.gnu.org/software/octave/, which is a free MATLAB replacement. It uses the same language. Alternatively, GNU R http://www.r-project.org/ has a different language, but similar functionanlity.

Ruby and Erlang are the two I've spent the most time with so far. I like Ruby enough so far, that I've decided to write the initial
batch of install scripts for OpenQabal in Ruby.

Outside of that wish-list, I also harbor some vague hope of one day finding time to dabble with Forth, Fortran, Perl, and maybe Dylan.

R.

http://www.r-project.org/

It can do basic stats easily, procedurally generate journal quality graphics, and do just about any advanced procedure you could dream up. R: it's science!

I am interested in the Standard Deviation, SEM, and a one- or two-tailed T-test. As a molecular immunologist, that's about all I need for 99% of my data analysis.

[...]
But if you can suggest a good data analysis application that runs on Linux, I will listen, and will surely try it.

Regarding interfacing with R note that R itself can do minimal symbolic
differentiation out-of-the-box as shown by this sample R session:

> deriv(expression(x^2))
2 * x

and has a partially developed interface with yacas via the addon package Ryacas.
After installing Ryacas and yacas this R code works:

> library(Ryacas)
> x = Sym("x")
> deriv(x^2)
expression(2 * x)

This Ryacas interface includes a partial recursive decent R parser that translates
R code to yacas code and an XML-based OpenMath connection in the other direction.
Communication is via sockets. See

http://ryacas.googlecode.com/

Unlike Sage, symbolic computation is not really the focus of R but R does have
1000+ free addon packages including interfaces to numerous other free and
commercial systems. The addon packages are listed in these repositories which
focus on general items, interfacing and biology, respectively:

http://cran.r-project.org/
http://www.omegahat.org/
http://www.bioconductor.org/

Also there is a graphics gallery with sample R graphics:

http://addictedtor.free.fr/graphiques/

The R home page can be found by entering the single letter R into google.

Are you aware of R it's intended as a OSS version of S-plus but I'd say it's better than matlab in many ways. You have to dig into the packages to get really interesting functionality, but out of the box I prefer it to matlab.

The only things I needed non FOSS tools for was the emulator, and SPSS for crunching the statistics, since I couldn't find an open source equivalent. (Believe me, I tried)

Have you never encountered R? It's been around at least 10 years. I first used it about 3 or 4 years ago in an intro to stats for engineers class, where the professor recommended it specifically because it was FOSS. My current boss does a ton of stats on a regular basis and loves R.

Not that I want to run them out of a patent (if they TRULY deserve it), but I think most decent programmers might be able to effect the same results with existing code and some head-banging on the wall.

Yes, I think so too. Get started :)

CC.

Well, http://www.murdoch-sutherland.com/Rtools/ provides the tools necessary to build http://www.r-project.org/ on windows. When compiling the latest version, R-2.6.0, just released it uses gcc 4.2.1 to compile.

I don't have windows to test the compiler out but I tried the binaries and they work fine. BTW, R is a fairly large project.

Why use a database or spreadsheet? Why not something like the R Project?

Actually I had the understanding that a lot of installer makers force you to have an EULA,

So why not use an open source installer? On Windows Inno Setup is very good; it doesn't force you to do this. (R offers the GPL in an information screen, with instructions saying "Please read", and "When you are ready to continue, click Next". I think that's about the right level: you want users to be aware of the GPL, but they don't need to accept it to do an installation.)

Definitely seconded, although the focus of those projects is not symbolic computation per-say. Both R and Octave are very good tools - R is an industrial strength statistical environment (it probably has the most active user base of any of these projects - certainly its contributed materials are formidable) and Octave tends more toward numerical computation.

R is located at http://www.r-project.org/

Octave is at http://www.gnu.org/software/octave/

Bar charts work just fine. For fancier plotting options your best bet is R.

We use R. I t produces visually appealing neat & clean graphics. My only bugbear is that the raster renders are external syscalls and often handle things like fonts and anti-aliasing poorly. We weer also looking at Rserv as a graphing service rather than invoking an R instance each time we want a graph. http://www.r-project.org/

Xix.

I think Acrobat Reader also leaks memory, at least when used as a plugin in a Gecko based browser, very badly, perhaps for the same reason.

I'm curious why so many people are concerned with the ability of calc to do statistics. Is this just a carryover from the MS Windows world where Excel seems to be used for all sorts of things it isn't well suited for? Why not do your stats in R, which is much more powerful than Calc or Excel?

you'd think they could afford statisticians. Survival analysis anyone? http://cran.r-project.org/

R is a system for statistical computation and graphics. It consists of a language plus a run-time environment with graphics, a debugger, access to certain system functions, and the ability to run programs stored in script files. I runs on a variety of CPU's. You may be able to get this to run on some PDA's like a zarus I would think. This would be your most powerful option. Check out R here: http://www.r-project.org/

Depending on your needs, the R project is an excellent choice too.

If you're a programmer, and even if you're not, use R to plot from spreadsheet programs, databases, and flat files! I highly recommend the book "R Graphics" by Paul Murrell if you're interested in not being constrained to what Excel and others limit you to! Murrell's grid package for R can have you building publication quality plots from scratch, it's very powerful.

Can you name me an open source spreadsheet-like program that is not an Excel clone?

What about R http://www.r-project.org/?

It provides far more powerful tools for the analysis and display of data than Excel. Granted, it is basically a programming language, so it is not as intuitive as a spreadsheet, but it can be used for everything from adding columns to completing cutting-edge statistical analysis, including publication-quality graphics. The spare nature of the interface would prevent any marketing goons or pointy-haired bosses from thinking they know enough to mess with it, while allowing anyone who spends a day or two getting up to speed to do just about anything you can imagine doing with numbers.

yp

Since this is science, that information *should* be publicly available somewhere.

Despite arguing (with tongue in cheek) that a click-through is a good idea, in the latest release of R I removed the requirement. The license is still displayed, but there's no requirement to "agree" to it to continue with installation.

For people who are handling the installers for other projects: this was a one line change to our Inno Setup script, from

LicenseFile=${SRCDIR}\\COPYING

to

InfoBeforeFile=${SRCDIR}\\COPYING

R is OSS and has the commands he wants (image, countour, and wireframe) it sounds like exactly what he wants: code that makes these plots that he can look at.

I'm out of mod points, but this is insightful. The R programming language is GPL'ed and works on lin/win/osx (packages for major distros). It is an interpreted language (except for a few internal commands), and so the source code for the several different 3D plotters is included with the program. Some you might have to install yourself, but this can be done by the install.packages command.

You might want to have a quick look at output from different 3D commands (persp, scatterplot3d and wireframe).

The introductory documentation might be a bit confusing - especially since it's often written for and by statisticians, but there's a mailinglist with a huge googleable archive and I've often found that "google:r-project search term" will get me what I need.