Slashdot Mirror
High-Performance Linux Clustering
An anonymous reader writes "High Performance Computing (HPC) has become easier, and two reasons are the adoption of open source software concepts and the introduction and refinement of clustering technology. This first of two articles discusses the types of clusters available, uses for those clusters, reasons clusters have become popular for HPC, some fundamentals of HPC, and the role of Linux in HPC."
For some very good information on F/OSS based clustering, check out aggregate.org. They have really neat ideas, that are reasonably well doccumented and freely implementable/usable. I built a little cluster (AFAPI on a WAPERS switch) with them for my highschool senior project, and it was a great experence.
Though mpp's are kind of like clusters, and the boundary between the two is vague, I think there's definately a distinction. In many MPPs, nodes share access to memory, just at a performance penalty. Often the scientific binary is written using a message-passing tool like MPI, but the OS is often run with direct memory access. Definately from a systems-administration point of view, an mpp is different from a cluster. In an MPP you don't have 4000 root hard drives and 4000 power supplies to replace when they break. An mpp may be like a (fast) cluster from the programmer's point of view, but they are a lot simpler to deploy and manage. (Blue Gene, xt3, altix)
I also contest some of the distinctions drawn about vector processor systems. The two vector systems currently on the market, the cray X1 and the NEC SX-8 are clusters. Each node just happens to be a vector-smp. The earth simulator is a 640 node cluster of 8-way SMP boxes, where each of the processors in the smp is a vector cpu. However, the predominant programming method even on these boxes is with explicite message passing like MPI. Co-array fortran and Unified Parallel C are faster, but slow to catch on.
Good summary of the common case though.
Rocks has a great system for making high-performance clusters from similar machines. A Rocks cluster consists of a front-end ("master") node and a bunch of compute nodes (and I think special-purpose nodes).
The master gets a full Linux (RedHat-based) install. It's a NFS/DHCP/Kickstart server for the compute nodes, and runs whatever other services you want the compute nodes to use. The master has two network cards and acts as a firewall (NAT optional).
The compute nodes boot via DHCP and Kickstart, downloading their kernel and whatever other OS files you want to their local disk. You decide how much NFS or local disk to use.
Job queueing is handled by, e.g., Sun Grid Engine (an Open Source queueing package) or some other queueing software.
Here's the neat thing: to make a change to a compute node setup, you change the Kickstart config and reboot all the compute nodes (as they finish whatever queued work they're doing, or immediately if you want). That makes the sysadmin's life easy, while still maintaining the speed of having the OS on the local disk.
Raise your children as if you were teaching them to raise your grandchildren, because you are.
Do you watch DVDs? Do you dream of squeezing all your DVDs onto a harddrive and streaming them to a media PC attached to your TV?
;)
You could copy the DVDs at ~8GB each to some large harddrives or you could transcode them to much smaller formats with all the garbage removed and go from ~8GB/movie to less than 4GB/movie. But to do this you need lots of processing power. A cluster works very good for this and the software is already there for you:
http://www.exit1.org/dvdrip/doc/cluster.cipp
For the cost of some overpriced Dell crap video editing PC you could build a decent diskless cluster. Who needs harddrives, monitors, video cards, keyboards, mice, etc. At least more than one set.
burnin