Use of Math Languages and Packages in Research?

CEHT asks: "As a research programmer at the university, I have encountered numerous times when I need to choose which language(s) or package(s) to use for different projects. Tradeoffs and performance issues have to be considered: results from one package may be more compatible with the data from other researchers, another package may find the solution faster and use less resources, and so forth. Maple, Matlab, Magma, and Mathematica are among the most well-known packages. Libraries such as IMSL is also popular. Of course, there are smaller (and mostly free) packages that tend to target specific types of problem, such as LiDIA, Singular, and LAPACK. The question is, how useful are these [and other] math packages? Do researchers use only one or two packages for most of their projects? Or do people like to mix things a little by pulling the strength of different packages together to solve a math problem? If not, do researchers write C/C++ programs and use GMP or Matpack to solve math problems?"

Comments

Fortran

[luzrek]

In Experimental Nuclear Physics (ENP) there is a healthy mix of Fortran , C, and C++ (and some others). There is a healthy schepticism of "black box" programs and libraries so programs like Mathematica and Mathlab are pretty much not used. Also, most of the problems are pretty specific (and time consumming to run) so everyone seems to run specialized code (Example: Radware is very popular in Nuclear Spectroscophy). Of course it helps that most ENP's are pretty competant with computers and electronics (amoung other things).

Re:Fortran

[wcspxyx]

Your points are well taken given the data you got from my first message. But you miss the mark. Allow me to explain.

I took a few classes from my Dad, so I do know how he tought, and I know what he expected of his students and I know how hard he worked with his students to help them succeed. So he may have been (or still is) an asshole, but he really cared about his students and that they succeed. He also felt that this was information that his students needed to learn and, since the CompSci dept. at that school wasn't interested, he did it himself.

And there is no nuke-navy thing going on here. My Dad never served in any military outside of ROTC Marching Band when he was an undergrad. It wasn't any macho attitude that made him make his classes difficult. It was respect for the material and respect for what his students were going to be asked to do with thier knowledge once they graduated. Nuclear Engineering is some serious shit. Not to be taken lightly. My Dad may have been an asshole. But students graduating from his department could run circles around graduates from other schools.

Now, as I said before, I went to the same school. The CompSci department was horrible. I know of very few CompScis from that school that learned anything in their classes. Most learned on their own or brought their knowledge with them. And I know even fewer of my friends from school that graduated with a CompSci degree that are still working as CompScis. The profs spent most of their time teaching how to format your code so that it prints nicely and how to use canned routines and other such sillyness.

To give you an example, there was a Numerical Analysis course that was required for most engineers. The did about 5 programs over the course, and it was required that the code be done in FORTRAN using LINPACK. It was trivial. I don't think any of the code that I wrote for that class was longer than 50 lines of FORTRAN. Didn't learn a thing. IMHO, there is no real benefit to doing something like this. It's especially not a semester long course. You just get your 'A' and go on.

And I'm sorry; Gear method is about a 3 day coding project, tops. The freaking pseudo-code is in the textbook fer cryin' out loud. It's not like you are inventing a wheel, just implementing one. If it were a graduate level project, I would assume that there was some research going on there, something interesting or new, not a simple implementation. I agree with you there. But that's not what was implied in the argument I had with the prof. He said that any implementation of the Gear method would be a semester long grad project. And to this day I disagree. Especially since a dozen or so undergrads a year did it in my Dad's course.

Re:Fortran

[dragonsister]

Wow. You come from a place full of TLAs, don't you? (Three Letter Acronyms.)

I handed in my PhD thesis in nuclear physics earlier this month ("The effect of ground-state spin on fission and quasi-fission anisotropies" - I'm in fusion-fission research.) I used basically nothing but FORTRAN all the way through. One of the reasons this was practical was that my work involved only a few new equations - not all that much symbolic manipulation to be done. Further, it was reasonable to tweak and optimize the FORTRAN to get faster calculation times ... when the newest Alpha in the department takes four days to crunch a single data point, believe me you devote a little time to optimizations and evaluating approximations!

For relatively pure and simple number-crunching like I was doing, FORTRAN is pretty well suited to the task. However, that isn't the real reason I used it. I used FORTRAN because everyone else in the department (and in most of the rest of the discipline, world wide) used FORTRAN - when I start my new job in may, the rest of my research group can look at my documentation and source code, and tweak and change it as required. And send it to other researchers who can do likewise. (A common type of change is to track and output a variable not previously considered important. This may be hard to introduce into Math-something calculations.)

(I write Perl for a website at home. Oh, the contrast!)

I've met Radware, but did most of my Nuclear Spectroscopy (during my honours year) using a solution peculiar to the ANU nuclear physics department; DCP. This in-house program slices, dices, drops the knife and needs to be poked or re-started occasionally ... Argh; it gets the job done, and I understand many of its important features are not available in other programs, but it would really benefit from a total rewrite from a highly trained, intelligent, and approachable programmer like my partner. (The author of DCP is a prickly old fellow with lots of experience and (to the best of my knowledge) no theory to speak of. DCP reflects its author.)

What is the best tool for the job? Depends on the job, and the skills of you and the people around you.

Rachel

age-old answer: it depends

[coult]

I'm a research mathematician, and I use lots of tools:

matlab for design prototypes of numerical algorithms and for visualizing data.

mathematica for doing messy algebra/calculus/differential equations.

my own c/c++ code, with a lapack backend, for doing large-scale computations (matlab and mathematica are too slow for big computations).
So, the answer is e) all of the above!

Perl Data Language

[sleepingsquirrel]

Let's not forget about PDL, the Perl Data Language. Think of Matlab combined with the goodness (i.e. CPAN packages) of perl.

Inexact floating point calculations...

[$$$$$exyGal]

A general question. With some programming languages... floating point is by its nature inexact. It is probably best if you imagined that after every floating point operation, a little demon came in and added or subtracted a tiny number to fuzz the low order bits of your result. (I quoted that from here).

Do any of the listed tools/languages take care of this problem for me? I understand the nature of the problem, but it is still very frustrating. What do the "pure" math programming languages do with this issue?

--sex

Re:Inexact floating point calculations...

[Exitthree]

That's true with all programming languages, not just some. Floating point accuracy is an inherited limitation of using a computer, you can partly work your way around it, but it never really goes away.

One way of getting around it is making your own method for storing a floating point number, not using a built-in type with so many bits (like using more than 32-bits). Now, you can keep adding "bits" in your interpretation of a floating point number until you reach the desired accuracy.

I can make a list of booleans to represent a floating point number and have the list longer or shorter depending on how precise I want the number to be. Then I use my own addition, multiplication, etc. algorithms on this number to get my result. This makes the process slower since you are basically rewriting the functionality of a chip to accomodate a higher number of bits than which it was designed, but it's possible.

As for mathematical tools dealing with the issue for you, I think you can specify the precision you would like, and it adjusts the answer accordingly. (At least I believe it is the case with Mathematica.)

Python and Numeric

[kognate]

I've used python and Numeric and coworkers of mine have used them in real physics type stuff and it is great (I think Fermi lab uses Numeric a great deal).

I'm surprised you also haven't mentioned R. It's a stats
package (gpl'd) modled after S. http://www.r-project.org
and it is very powerful with a great community behind it. It's an amazingly powerful tool for analysis.

Other trade offs

[briancnorton]

The trade offs are a lot more than you mentioned. If you are doing MATHEMATICAL research, Mathematica AFAIK has the most extensive capabilities for expansion and programmability. If you are doing somethign like environmental modelling or complex systems analysis, then something like Matlab may be more important.

I am not able to articulate this well, but the type of research you are doing is MUCH more important of a consideration than computation speed or resource consumption. If you need supercomputer time, then you had better ask the admin what you need to use. I know a bunch of people that do environmental modelling, and I have never seen or heard of anybody writing their own C++ to do it. Researchers GENERALLY have better things to do than re-invent wheels.

Re:Octave

[decoutt]

Octave is a nice MATLAB clone, developed from chemical engineers in the beginning, but now used extensively in virtually any area that math is usefull.

Many packages have their open source counterparts: Octave for MATLAB, R-system for SPLUS (statistics algebra system), and so forth. But IMHO you raise another issue: you can use each of these packages to do whatever calculations you want, since all of them are extended in the C/Fortran end, i.e. they can use programs written in these languages. Custom code is readily integrated. And above all, the GNU Scientific Library. If you don't like or you don't trust the numerical solvers integrated in MATLAB, you can investigate the source in the GSL.

And yes, you can use all of these together. So, what is the question again?

Mathematics past

[Shadow Wrought]

Just out of curiosity, anyone know what mathematicians, engineers, and phycicists would do in regards to these complex problems before there were these programs mentioned? What about before slide rules?

depends, or, if you have to ask slashdot...

[foog]

You're talking about two different classes of software: "numerical linear algebra packages" and "computer algebra systems". Maple and Mathematica are the latter, Matlab is the former. I don't know about Magma.

Hardcore numerical programmers use LINPACK/LAPACK with platform-optimized BLAS (this latter is often commercial, or at least proprietary to the platform vendor) directly from Fortran. They usually use modern commercial Fortran 90 or Fortran 95 compilers, too.

On numerical linear algebra stuff where you aren't going to recruit and pay a Fortran programmer with a PhD in applied mathematics, most sane people use Matlab or GNU Octave or one of the many other Matlab clones. A lot of people like Numerical Python, if I had a big new project to do, I'd seriously consider it.

Yes, crazy "researchers" who don't want to learn Fortran and think Matlab is too slow or too expensive will write numerical code in C++. Some of them do fine work, too.

Excel and other spreadsheets are fine for small bits of numerical analysis, too. Don't turn up your nose at 'em, you can email your boss your whole analysis and he doesn't have to learn Matlab to do anything with it. Excel is also slowly replacing Qbasic as the computing lingua franca of the Amateur Radio/hobbyist-electronics community.

The class of people who just doodle out the singular integral equations for the airfoil design they're brainstorming seem to like Mathematica a lot. I wish I were more like that. Maxima is seeing a renaissance now that its licensing and distribution issues are cleared up (it's GPL now). I should check it out. There's also GNU (Emacs) Calc, which I use regularly as an RPN desktop calculator. It is actually much more powerful than that and will do all kinds of HP-calculator-style graphing and computer algebra with a liberal sprinkling of Mathematica-style syntax, but I don't use those features much, because they're wicked slow.

emacs calc

[Anonymous Coward]

Dave Gillespie's excellent emacs package Calc ignores your floating-point hardware and instead uses lists of integers to represent floating-point numbers internally, IIRC. And you can specify how many decimal places to remember; you want to compute something to 1000 decimal places, you got it.

(I would guess some other programs do this, too, but I'm not as familiar with them, since emacs calc does pretty much everything.)

Arbitrary Precision Floating Point?

[jvkjvk]

I have a related issue, in this regard. Some of the problems I am working on require arbitrary precision floating point numbers. E.g., one number might be 3.2334, but it needs to be multiplied by 3.4568902349830983945873908730987578439345, and I need all the resultant digits.

The problem is that the output of one calculation is fed into the input stage of another, that output being the input of the first calculation, in a circular style, so that small rounding changes may have a large affect on the final outcome.

Now, at some points, the precision may be truncated (where the effect will be unnoticable to the equations), but at certain points I need the exact number.

I have heard that with Lisp you can have numbers as large as you like, but I don't know how hard it is to perform complex numerical tasks in Lisp. Also, speed is an issue (I want it to be as fast as possible).

Any suggestions as to how to accomplish this?

heres one data point...

[edhill]

I'm a post-doctoral researcher at a engineering college and I use Linux for all of my data acquisition and analysis. The following environments are used:

LabVIEW, PERL, shell scripts, and/or C for data acquisition

C++, MatLAB, and/or shell scripts for data analysis

and you can get some of my codes from Sourceforge:

http://sourceforge.net/projects/qaxa
http://sourceforge.net/projects/ssnooper

and others are available by sending me an email.

Ed
http://cesep.mines.edu/people/hill.htm

Black Box

[luzrek]

Matlab is a Black Box mostly because it has huge numbers of built in functions. In order to make sure that all of these functions work properly (you would be amazed by how many commercial functions don't actually do what is advertised) by building our own and then compairing the two.

The other major factor is that nuclear physics is perpetually underfunded and buying commercial software is ussually not nessasary (since we would have to make sure it worked properly anyway).

BTW we do use "building block" type programs and libraries for our interfaces. A good example is SpecTCL at the National Superconductiong Cyclotron Laboratory. I have used GTK/GDK in my applications, others have used Qt. However, the numbercrunching and datacrunching parts are nearly all custom. The data processing is simply too complex and too specialized to trust to prepackaged software. The numbercrunching applications are too time consuming to use a generalized program, everything has to be optimized.

Matlab primary; old languages also used

[jim_deane]

I was taught Matlab in my computational physics graduate class, which biases me toward Matlab in my own research. I also own Mathematica, but have not taken the time to master its language and command structure. Mathematica was an award at a conference where I presented a paper, but I purchased Matlab for myself.

There are two primary advantages which I see in Matlab. The first advantage to me is its abilities with matrices and arrays; it can do things in a couple of lines of code which can take some roundabout programming and subroutines in other more conventional languages.

The second is Matlab's graphical abilities. Display of data is very important, both in the final product (thesis, paper) and in the research process itself. After a brief introduction to graphing in Matlab, it becomes a trivial task to choose and use various display options for your data.

In physics, it seems that we stick with what works until something better is found. That applies to our theories and to our tools. It is not uncommon for us to use Fortran, Pascal, or even various types of Basic to perform simple calculations and experiments.

Much of what one uses may be determined partially by chance--what software package was available at your institution, what professor did you study under, did your undergraduate degree require a programming course? The work involved in switching from one major package to another, for instance from Matlab to Mathematica, simply seems like too much effort for very little sure return.

Jim Deane

Re:skeptical

[ihateashcroft]

The difference in speed between Matlab and C/C++ is roughly the difference between Java and C/C++. Unless you use a few certain functions, Matlab compiles your code into Bytecode-like instructions called M-code, which is then interpreted like in Java's VM. It also has a built in converter to change M-code into C++ code that can be compiled by an external C++ compiler.

Instead of C/C++

[bauernakke]

A highly unknown but very efficient (faster than Pascal w gc) and easy to program is ML (Meta Language) Seems to be perfect for Math computation.

SuperMongo...

[hak hak]

unfortunately. We were taught it in one of our astronomy classes to analyze and plot data. It has a very arcane syntax and doesn't have a lot of capabilities (no matrices; there is a very crude way to represent square matrices as sets of vectors). I now use Octave or Matlab as much as possible for numerical work. For symbolic math, I mostly use Maple. I also have a little experience with Mathematica, but I like Maple better (although Mathematica looks nice). As far as I know Maxima, it can't do as much as Maple or Mathematica, but it should be good enough for most symbolic computations.

Re:Matlab, C, VB, local scripting

[foog]

oof. here's the link:
Numerical Recipes -- not so good

Matlab is mega bucks

[Anonymous Coward]

Mathworks (the owners of Matlab) has been aggresssivley increasing prices on Matlab while reducing licensing flexibility. All this started after they bought Matrix-X (can you say monopoly... I thought you could :-).

They heavily discount for Universities and students, because that gets you hooked. Matlab scripts are so cryptic, that the idea of switching to another package causes great fear and trembling.

I've also got quite a few gripes about the language itself. Yes, matrix aware languages are great; but there are much better implementations out there. Try Mathematica or Python/Numeric.

Do yourself a favor and avoid Matlab.

Just my $20.

Re:Matlab, C, VB, local scripting

[the gnat]

Clearly this breaks down for certain applications, but most of the science currently being done (read: molecular biology, and no, not bioinformatics) is not algorithm-bound.

Most of the bioinformatics being done that I'm aware of is not algorithm-bound either.

People do tend to find a language and stick to it, though. Usually Perl. You get the occasional Python diehard as well, but my experience has been that while I'd far rather use Python for a large project, I'd rather use Perl for anything with significant amounts of text processing. There are times when weird kludges and shortcuts are actually a good thing. I know someone who programs in Lisp whenever possible. C is usually the last resort of people who think it'll be faster than Perl. Sometimes this is the case. Sometimes they simply can't program worth shit.

The real problem is that many bioinformaticists have no concept of software engineering. This applies on many levels. First, they can't write reusable, maintainable code. Second, they have no concept of algorithms or recursion. Third, they never get to the point where they can write software reflexively. The best code, in my experience, is the stuff that's pounded out in under an hour, but which has been thought about for days beforehand. I think everyone wanting to do bioinformatics should be forced to take an intermediate CS class before they're allowed to do research, rather than sitting down with an O'Reilly book and starting to write code. They'll waste less of their time and everyone else's this way.

Frankly, however, two-thirds of the time of any bioinformaticist is spent interpreting and reformatting the crap data that biologists give us.

LiveMath (aka Theorist) Rocks!

[OsamaBinLogin]

I was in the math software industry for well over a decade. My experience is that each manufacturer is irrationally optimistic about their own package, and irrationally pessimistic about competitors. Kindof like commercial software in general, but the venue is more fuzzy, lending to more irrationality.

- the way a user does this or that, varies widely. Often, methods are not obvious, so for one person, "mathblob does gizzyggy calculations" is true, for another, it's false. EG mathcad has (had?) a version of Maple inside, but it was awkward and hokey to use. BUT they could claim all of Maple's capabilities.

- The algorithms vary widely. If MathBlob has a specific optimization for gizzyggy calculations with flipex inversions, then you could say that MathBlob is great at gizzyggy calculations, and the others are useless. These optimizations can easily give you 1000x performance improvement or more (think n^n).

- The domain venues vary widely. Numerical programs typically do zero symbolic stuff, but do the numeric stuff lightyears faster than symbolic programs. A group theory program may have trouble adding floating point numbers. With hardware double floats, you can't get anything past 10^308 or so, which is a disaster to some but no big deal to others. When someone says MathBlob does EVERYTHING, really they mean MathBlob does Everything that I think is important.

So like, I once had Mathematica take like 30 seconds to add 2 + 2. (It had to load in a ton of packages.) Then it reported it was out of memory.

With that said, LiveMath is THE BEST math program in the universe, way better than any of the others here. :-) It's designed to be interactive, rather than a programming language. Check it out at livemath.com

(disclaimer: I'm the original author)

Re:Octave (useless bragging)

[theLOUDroom]

Wanna hear something cool?

I just got Octave & Gnuplot running on my Sharp Zaurus. I can do my DSP type calculations, anywhere!

Someone is currently porting gtktiemu, at which point I'll have a TI-89 emulator, which will let me handle just abount any engineering math type stuff I need to do with one pocket-sized deivce.

Now if my fold-up keyboard would just show up.....

For Statistics

[EnlightenedDuck]

As a statistician, I prefer R. Matlab's approach to statistics is to implement a bunch of formulas one could look up - R (or S-plus - I prefer the open source version) gives an interface that is closer to doing statistics. R has far more routines implemented than minitab (or Matlab, if one sticks to statistics). Additionally, most of the interesting applied statistical research that I've seen is implemented in R.

Re:Depends

[Raiford]

I have been using math packages such as Matlab, Mathematica, Maple, Macsyma since the late 80's. I have also done my share of FORTRAN and C++ programming. It is going to depend on what your need is. If you are developing something that will be used by a group then a stand-alone application written in FORTRAN or C++ is appropriate. If it is something that you need for yourself and your require rapid prototyping of a concept or numerical experiment then a math environment is appropriate. You are not going to be doing any "grand challenge" computing with a math environment. If you don't want to deal with a bunch of graphics coding then go with the math packages. Matlab (and the GNU semi-clone Octave) are good general purpose environments. Mathematica is a great computer algebra system (and much more) but the cost is prohibitive (and getting worse). So it is really dependent on what your needs are or the needs of your group.