Slashdot Mirror

I wrote some parellel code using MPI in university. It takes a lot of work to get the hang of at first, and many people who I know that were good at programming had lots of trouble in this course, because programming for parallelism is very different than programming for a single processor. On the other hand, you can get much better performance from parallel algorithms. However, I think that we could do just as well sticking with the regular algorithms, and having a lot of threads each running on a different core. If you look at an RDBMS, it would be nice if you could sort in less than n log(n) time, but it's even better if you just sort in n log (n), but can run 128 sorts simultaneously. I seem to remember some news about Intel saying they would have 128 core chips available in the near future.

I'm not sure how this got modded up, just a quick Google search reveals that FreeBSd clustering is very doable.

Check out LAM/MPI or see pages by people who've done it

I haven't used Open MP, but I took a class in parallel processing, and we used LAM-MPI. If they are anything alike, then anything you program takes 10 times as long, plus you have to explicity tell it how to split up and collect the data in an efficient manner. Which is often the hardest part. Anyway, I think that this kind of stuff is only necessary for applications which are required to be highly parallel. Otherwise, it would probably be easy just to add a couple threads to you application, and let the OS figure out how to schedule them properly. Oh, and the other thing, most parallel algoriths only work well on a large number of processors/processes. For instance, you can sort N items in O(1) time, but you need to run N processes. Most of the time you don't have anywhere near N processors, so you are running more than 1 process per processor. You don't end up getting extra performance once you factor in the overhead.

At my work, we develop for a smattering of platforms, ranging from Linux, MacOS X and Cygwin to Solaris, IRIX, and a plethora of custom-built supercomputers. It's all done using C++ and MPI. MPI is a standard that specifies how nodes communicate with each other, and what methods/functions are called to do this. MPICH is an implementation put out by Arlington National Labs. LAM is another implementation, put out by Indiana University. LAM is a much nicer implementation IMO, but it's not available on quite as many platforms.

I would recommend these two books: How to Build a Beowulf and High-Performance Computing.

Bottom line: If you have computationally intensive calculations to do, beowulf clusters are a cheap alternative to pricey supercomputers. But if you've just heard they're cool but don't have anything to do with one, it's probably not worth it, as applications have to be specifically (re)written to take advantage of a cluster, they don't get automagically faster.

Take a look at OSCAR. We built a nine node cluster out of IBM e-servers using it. It was really quite straightforward.

As far as languages go, you'll need an MPI library (like MPICH, or LAM/MPI (which is also a runtime environment), but the actual code used is usually C, C++, or Fortran. BTW, OSCAR comes with MPICH and LAM/MPI.

Mosix is nice, because it treats the cluster like a single, large, multi-cpu box by simply allocating threads to different boxes. The nice thing about this is that any multi-threaded program can take advantage (as stated in the parent post).

However, this also can cause problems. Most threaded programs are written assuming that all the threads have high speed (i.e. system bus / cpu cache) access to shared information. When we introduce the latency incurred by a network, this can cause programs to run alot slower then they would if they simply had all the threads on a single box. Obviously, it all depends on how the program was written, and what it does.

If you are writting a program specifically for a cluster, I would suggest instead looking at something like LAM-MPI. This allows for a much more controlling approach to be taken. It is more work (you have to decide how the work will be split) but it allows for much better control of where and what is being done and how to optimize it.

I work with and help maintain a small mosix (now openMosix) cluster at a university. While mosix only really shines when you are running many long, computation-intensive jobs which are not I/O bound, there is no reason you can't mix mosix with other clustering implementations.

For example, we run a LAM MPI implementation on our cluster which allows us to carefully arrange which parts of each job go on each CPU and maximize efficiency. What's more, if we miscalculate, and one job on a heavily loaded machine tears off on a long cpu-burst, mosix will step in and migrate it over to a less loaded system for the duration of the burst.

None-the-less, it helps to inform your users not to start up 50 I/O bound jobs on one node and expect them to migrate. You end up having to give users access to multiple nodes to help balance load and this reduces security.

All in all, I've found mosix is very useful if your users know how to code for it. Standard software will typically not benefit too much. That said, if you have a couple of cd-rom drives in your machine, grip performs quite nicely: the ripping takes place wherever the drives are, but the mp3 encoding tends to migrate across the cluster beautifully :).