PDL 2.4.0: Scientific Computing for the Masses

Dr. Zowie writes "Perl Data Language 2.4.0 was just released; get it here. This release includes even more powerful array slicing, a complete GIS cartography package, API access to the Gnu Scientific Library, and a host of other goodies. Between PDL and its less-mature siblings Numeric Python and Octave, the established commercial languages' days appear numbered."

Comments

Maple?

[infernalC]

Maple doesn't get an established commercial languages link?

BTW, Maxima, Macsyma, etc, is free and has been around for years.

Re:Maple?

[pr0ntab]

Well, the author of the article is an idiot anyway, as PDL is basically a Perl interface to LAPACK.
PDL is not a CAL, DE-solver, or anything like that.

So Maple's days are not numbered. Go figure! :-)

Re:Maple?

[Dr. Zowie]

Heh. Touche on the symbolic logic stuff, though calling PDL a Perl interface to LAPACK is like calling Maple a graphing calculator. Yes, it can do those things -- but that's a tiny part of the functionality.

Re:Maple?

[Anonymous Coward]

That's because maple sucks. Mathematica long ago won that fight.

Re:Maple?

[Anonymous Coward]

Q: 2.4 What is PDL good for ?

Just in case you do not yet know what the main features of PDL are and what one could do with them, here is a (necessarily selective) list of key features:

PDL is well suited for matrix computations, general handling of multidimensional data, image processing, general scientific computation, numerical applications. It supports I/O for many popular image and data formats, 1D (line plots), 2D (images) and 3D (volume visualisation, surface plots via OpenGL - for instance impelmented using Mesa), graphics display capabilities and implements lots of numerical and semi-numerical algorithms.

Through the powerful pre-processor it is also easy to interface Perl to your favourite C routines, more of that further below.

Re:I'm seeing red

[Anonymous Coward]

they've gone open source ....

Re:I'm seeing red

[Anonymous Coward]

Is that what that is? Thank god I don't have a subscription that red is nasty, worse than the Games section.

Where is the GIS?

[smishra]

I looked through the website but could not find and GIS related modules. I also did a Google search (gis site:pdl.perl.org)but that too came up blank.

Re:Where is the GIS?

[ChrisDolan]

Download it. In the tarball is
PDL-2.4.0/Demos/Cartography_demo.pm

Re:Where is the GIS?

[Dr. Zowie]

The GIS stuff is a collection of vector and pixel coordinate-transform methods that allow you to plot, overlay, and compare map data from all the conventional formats. There are some sample global-scale maps of Earth included in the distribution, but they're not especially high resolution since they're mainly intended to demonstrate the capability.

We designed the GIS code to be useful for planetary, astronomical, and solar work -- it's "just" a set of tools for dealing with sampled and vector data in a spherical system.

Re:I'm seeing red

[elmegil]

sshhhhhhh :-)

Re:I'm seeing red

[kworthington]

This might help the Editors: I've noticed that these glitches happen when Michael is the editor who posts. I have screenshots if needed. (not trolling, it's an observation. I have nothing against Michael)

R: Open-source statistical languate

[StatFiend]

Another open-source statistical language is R. Its commercial cousin is S-Plus.

Octave v. Matlab

[djdead]

I use Octave at home to test anything I'm doing for the "Matlab" sections of my homework. And while I think it's a great program and works well, for large computations Matlab is much much faster. There is one routine in particular that takes about 4 hours to run at home and only 15 minutes to run at school. And no, this isn't because my home machine is P-MMX 100 and school has has 3GHz P-4's. The machines are pretty closely matched.

Re:Octave v. Matlab

[jmv]

This may depend on the actual code you have. If your code is spending most of its time doing matrix manipulations (which should be the case), then Matlab and Octave should have almost the same speed. The reason is simply that both use the ATLAS and FFTW libraries for most of the stuff. (BTW, make sure Octave is indeed compiled with those libraries and not generic stuff)

Re:Octave v. Matlab

[djdead]

Here is the code to which I was refering. It's for creating 1d binary cellular automata. The main block is below:

%%this for loop is going to calculate the number for each cellar from a binary combination of the previous row
%%use its value and its neighbors from the previous generation and store it in 'n.' it then encodes the next slot
for i=2:x+runs
n = (2^2)*copyRedds(1,i-1) + (2^1)*copyRedds(1, i) + (2^0)*copyRedds(1, i+1);
nextRedds(1, i) = ruleArray(1, 9 - (n + 1));
end

%%now calc the first and last
n = (2^1)*copyRedds(1, 1) + (2^0)*copyRedds(1, 2);
nextRedds(1, 1) = ruleArray(1, 9 - (n + 1));
n = (2^2)*copyRedds(1, x+runs) + (2^1)*copyRedds(1, x+1+runs);
nextRedds(1, x+1+runs) = ruleArray(1, 9 - (n + 1));

I'm using gnu octave 2.1.36 on windows. I just installed the binaries that they distributed so I'm not really sure what it was compiled with.

Re:Octave v. Matlab

[t]

Ah you did post the code, I thought you were pulling a SCO for a minute there.

Looks to me like the problem is the line right above the part you posted:
nextRedds = zeros(1, x+2*runs);
The reason this is bad is because you reallocate it slightly bigger at the beginning of every run. This wastes a lot of time in malloc. You should allocate it before the loop at the biggest size necessary, and then initialize it as you already are.

This part
redds = [redds; nextRedds(1, a:b)];
is especially bad because you are forcing Octave to grow your matrix on every loop. You should also preallocate this before the loop. In general, allowing Matlab/Octave to automatically enlarge arrays as needed results in disasterous performance. This is in general bad style, especially if one day you decide to code this up in C or something for speed, you would have to recode all your allocation code.

Re:Octave v. Matlab

[djdead]

I can look into that. The reason I wrote it this was because if the whole matirx was allocated at the begining, then it would be ~20000x10000 entries. I believe that octave/matlab only store things as doubles so the matrix would take, if I've done my math correctly, is ~1.49GB. I don't have that much main memory so that's why I wrote it this way.

Re:Octave v. Matlab

[t]

Well your loop seems to have some flaws. You allocate
nextRedds = zeros(1, x+2*runs);
But you only use a range of [1:x+1+runs]. Is the rest meant to be zero?

Regardless, copyRedds for example will at its largest be 2*numIters + x+2*sum([1:numIters]). redds will add to its initial size x*sum([1:numIters]) elements. And nextRedds will at most be x+2*numIters. You should allocate all of these at the max sizes required, and use the necessary indicies within the loop instead. e.g.,
redds = [redds; nextRedds(1, a:b)];
would be
redds(length(redds)+ [1:x] = nextRedds(1, runs+[1:x]);

I have no idea where you got your 1.49GB figure from as `x' is unknown to me. sum([1:N]) = (N+1)*(N/2) is about N^2. For N=10k then that's about 3/4 GB of RAM. So I guess your number is plausible. If you don't have that much RAM then now you know why it runs so slow, you can't escape that fact that that is indeed how much RAM will be trying to use at the end of the loop. I'm not sure about Windows, but I think it will continually allocate swap on the local harddrives which would allow you to run this monstrosity without running out of memory. I'd be willing to bet that the computer at the school has several GBs of RAM, that would explain the enourmous speed difference.

Re:Octave v. Matlab

[djdead]

Hey thanks for you advice. I misread your first post (sorry it's exam time, I'm a bit distracted).

anyway i haven't implemented changes to the nextRedds allocation yet, but perhaps next week. I thought about it for a minute and decided that i should be able to use a full sized redds matrix which would be 10000x502 elements. While I'm sure you're not too surprised it did cut the time down from 4 hours to 2:45. Then I tried to run the code on Matlab on one of the math servers but it had a problem allocating that much memory to me. Oh well, there are a bunch of standalone windows boxes that should let me take all I want.

What I was thinking before was that I would have to allocate a matrix to take the final size of nextRedds at every row. I believe that the final size of nextRedds is the length of the orginial vector redds plus twice the number of iterations. that would mean a matrix that was 20502 elements long by 10000 elements high. I realized though that I could just use the truncated part of that so instead of 10000x20502 it would only have to be 10000x502.

I was gonna email you to thank you but your email address isn't on display, so this will have to suffice.

Re:I'm seeing red

[kworthington]

Excuse this reply to my own post, but there's now a post that is also red, and not submitted by Michael.

Since all scientists use Perl

[Anonymous Coward]

Good thing Perl is a required course in most degree programs for science, otherwise it might not have much of an impact.

scipy

[d-Orb]

Well, I don't know about how mature/not mature Scientific Python or Octave are with respect to PDL, but I like Python better and I was used to Matlab in the past anyway.

At present, I am using Scipy, a nice more complete version of Numerical Python. Together with IPython, I get a very nice numerical environment. Unfortunately, while Scipy is very nice, it is still a bit of a bleeding edge product. But it is **very** fast for large array computations. I also like the fact that you can link fortran routines easily (yes, people still use fortran, it's useful and easy).

I also use Octave because I miss the ease of generating plots in Matlab (yes, I could do this with scipy, but somehow, I resort to using Octave). It is a very complete program, with many toolboxes. Given that some of the Matlab toolboxes can also be incorporated, there is a vast array of functions for you to play around with.

On the other hand, I think that none of the "established languages" are a good comparison. IDL is extremely powerful for Remote Sensing/Image Processing tasks (my area of research). It is simple to use, and a bit of a standard in the field. From the PDL changelog, the cartographic features in PDL amount to no more than transformations... Mathematica is extremely powerful in symbolic Maths, which as far as I can tell, is not what pdl is about. And Matlab is turning into the VB of scientists (at least, it is multiplatform :D)

Oh well, I'll have to give it a go :-D

Re:scipy

[gnuadam]

Right about mathematica. Does anybody know if there is a free (or Free) alternative for symbolic mathematics?

Re:scipy

[t]

Just curious but are you generating plots just for looking or for eps? I've been looking for awhile for the best eps plot generator (for latex) and I'm currently using epstk with octave. Seems to be the best, most importantly it allows one to numerically specify dash length, thickness, and color of plot lines. Necessary for busy plots. It also has the ability to use arbitrary symbols for plotting.

Re:scipy

[pawsa]

Check out Maxima - while it may be not as complete as mathematica, it can do a lot of symbolic algebra, including derivatives, integration and expression simplification (optimize function is great, it can eg. factor out common subexpressions).

Re:scipy

[Dr. Zowie]

There are several device-independent graphics packages for PDL. The main one is PGPLOT, a venerable but powerful package written in FORTRAN in days of yore. PGPLOT has output modules for everything from a PASCAL turtle to (yes) eps. There are interactive devices (X windows and such) and hardcopy devices (PostScript, EPS, gif, jpeg, png, and such).

What impressed me most about PGPLOT when I started using it is the strong device-independence. For example, it's difficult to say "Give me a 600x400 pixel X window" since pixels aren't device independent. It's much easier to say "Give me a 6-inch by 4-inch X window". Takes some getting used to, but then when you go to stick your output into a publication you can generate the same plot and send it to (say) the eps device instead of the X-windows device.

The PDL front-end has a record-and-play feature, too, so you can define a PGPLOT X window, noodle around with it, and then say "Replay all that into this hard-copy device" and get exactly the same plot, rendered on the other device.

except of course...

[joto]

...that Mathematica is not a data language competing with the others at all.

Gis?

[Anonymous Coward]

I don't mind circle jerks, but i don't want to play with some other guy's gis.... Can't they get a better name, like 'muff', or 'vag'?

Comparisons...

[Dr. Zowie]

Yep, you're right that Mathematica is not a good comparison -- I stuck that in mainly as a reference to the numerical part of Mathematica, but the symbolic stuff is pretty much unmatched (though Maple fans might disagree).

Much of PDL's development has been motivated by a need for something "like IDL, but more powerful", and I think that's really where PDL shines best: in remote sensing and image processing tasks. It helps a lot that all of CPAN is already present, and that the file I/O and indexing have many fewer "gotchas" than those of IDL. The PGPLOT back-end is great, too, for actual device-independent plotting: how many hours have you spent tweaking your IDL plots to actually print right on the PostScript device?

It's (IMHO) a Good Thing that we have all three of numpy/scipy, Octave, and PDL: each has a different set of strengths. Ultimately, each group really should use the tool that suits them best (and it shouldn't cost more than the workstation it runs on...). The reason I've more-or-less committed to perl development rather than Python or Octave is that it has a nice "natural language", expressive feel to it: it's easy to build pipeline-style, imperative-style, or evaluated-style constructs, whichever is most convenient for the current application.

Of course, the open-source languages have the added benefit that results derived using them are actually reproducible, whereas closed-source languages might conceal irreproducible bugs (in the language or the reduction code) that other groups can't identify.

Re:Comparisons...

[Sara Chan]

Yep, you're right that Mathematica is not a good comparison -- I stuck that in mainly as a reference to the numerical part of Mathematica, but the symbolic stuff is pretty much unmatched (though Maple fans might disagree).

Might disagree? No "might" about it! Mathematica is better than Maple for numerical data analysis. Considering symbolics, though, Maple's ability to solve ODEs, for example, is much better than Mathematica's. Also, generally, Maple is supposed to be more reliable than Mathematica (though I've not done a full detailed comparison myself).

Re:I'm seeing red

[Anonymous Coward]

I've noticed it happens because Slash is written in Perl, and as everyone knows, Perl sucks.

PDL is cool

[Anonymous Coward]

because it gets you a nice Gimp filter named "Warp Sharp".

Maxima

[Anonymous Coward]

Maxima is what you are looking for.

It actually has an interesting history, because in many ways it was the first symbolic math program (many argue that other programs, such as Maple and Mathematica, are clones of Macsyma). It went out of use for awhile, then was open sourced, and now is experiencing somewhat of a renaissance.

So there's some irony, in that the commercial programs are actually "alternatives" to the now-free program.

This is quite true.

[pr0ntab]

More importantly recent versions of MATLAB JIT-compiles all the functions you run into a VM-bytecode-like thing, whereas Octave is a straight interpreter (AFAIK), so if you use a lot of recursive function calls or iterations and stuff.... heheh, you'll notice the difference right there.

Re:I'm seeing red

[Anonymous Coward]

Red? That one's blue on my screen. Almost the same shade as the games section. Seems like a bad day for Mr Daddypants.

Macsyma and Maxima

[screwthemoderators]

The sourceforge Maxima is a decendant of DOE (Department of Energy)Maxima. It was one of three licensed versions of the program. The commercial "Macsyma" is dead, but still property of Symbolics Technologies, I believe. The fact that any of the code is available and publicly licenced at all is a minor miracle.

A lack of Parallelism

[ChaoticCoyote]

All of these tools address different aspects of numerical computing. A mixture of languages and tools will generally produce the best results.

I've been experimenting with a number of scientific programming packages, ranging from traditional languages like Fortran 95 to new developments like SciPy. Of the "new" approaches, I like SciPy the best, given its support for MPI and ease of linking to traditional languages.

Support for NUMA and SMP architectures is severely lacking in most "free" packages. This may, in some respects, be due to the lack of parallel support on gcc (although there is an effort underway (gomp) to add OpenMP support to gcc).

Parallelism is important to any large-scale numerical application -- and PDL, as yet, does not appear to support SMP, NUMA, or cluster architectures. I know there are attempts at adding parallel support to Perl, but haven't seen much activity with them.

GSL does not implement any parallel algorithms; according to this post by Brian Gough (), GSL is not designed to support parallelism.