Slashdot Mirror
Building Your First Cluster?
An anonymous reader asks: "I'm interested in building a DIY cluster using Linux and will be using conventional Linux software. However, the number of possible ways to do this is huge. Aside from Beowulf, there's Mosix, OpenMosix, Kerrighed, Score, OpenSSI and countless others. Therein lies the problem. There are so many ways of clustering, development seems to be in fits and starts, most won't work on recent Linux kernels and there's no obvious way to mix-and-match. What have other people used? How good are the solutions out there?"
Rocks is a great tool to build a cluster. It includes lots of monitoring tools and such, so you can see the status of nodes, etc. However, I'm not quite sure how large you're planning on going... May be overkill for a 4-noder. =)
Sounds like you're trying to solve a problem that you don't have.
Cluster need special software to take advantage of the disturbed computing. They are built with a specific task in mind. Or do you already have a need and just failed to tell us?
For me, I run my network with distcc (http://distcc.samba.org/) So all of my Gentoo boxes can compile using shared computing power. It cut a typical 33Min app down to less then 2 mins doing this. And works wonders for my slower laptop.
With distcc, all you need to do is have the same tool chains. (glibc, gcc, coreutils, etc) You can even specify how many threads per box you want running to fine tune your network.
On the other hand, if you just want to learn, then you should try them all. The all suit different needs.
While a few early parallel computers used hypercube based interconnects (eg, CM-1), there hasn't been a lot of interest in hypercubes since then. The advantage of hypercubes is that their diameter only increases logarithmically with the number of nodes in the network. Their disadvantage is that the node degree increases with the dimension, meaning that each node must be configured with sufficient ports to support the maximum dimensionality of the network -- making hypercube-based networks either expensive or non growable. (This can be solved to an extent by using cube-connected cycles.)
No, seriously, if you're setting up a cluster where your work can be batch-queued, or intend to run MPI, then Rocks http://www.rocksclusters.org/ is the way to go. It also comes with tools such as SGE (Sun Grid Engine) or OpenPBS pre-configured, Intel compilers and libraries ready for you to drop a license onto (but of course the entire GNU suite is there as well, including Ada), has more monitoring tools (plus some nice web-based interfaces) than you can shake a stick at, and runs on IA-32/AMD-64/IA-64 (Itanium). It also has a Roll to help build a tiled display wall, which would be a really cool use of a small cluster.
They're also really great guys.
On the other hand, Oscar is supposed to be good, and if you're not into the whole batch-mode thing, you can get OpenMosix up and running using http://clusterknoppix.sw.be// ClusterKnoppix, and just fire jobs off into space and let them find their own unburdened node.
But still, Rocks is really an elegant and clean way to go, plus it will scale up in case you're going to deploy a huge one of these for real after you get your feet wet.
the more accurate the calculations became, the more the concepts tended to vanish into thin air. R. S. Mulliken