What's The Best Linux Distribution For Clustering?

syn1 asks: "There has been a proliferation of Linux distros over the last couple years. Many are specialized for specific tasks or needs. In terms of Beowulf Clusters, there are a growing number of distros specialized for these clusters. Although the old favorite among specialized Beowulf distros is Extreme Linux, other distros such as Syclid Linux and Scali Linux are catching up in terms of user share. Additionally, more people are using conventional distros (Red Hat, Debian, Mandrake, SuSE, etc..) and adding Beowulf support. I am just wondering what fellow Slashdotters think about these various distros when it comes to Beowulf Clusters and which ones they think are best."

Comments

the best Beowulf distro? Scyld, of course.

[eleitl]

It's a second generation Beowulf, with some
very interesting features (see below). You can
download it for free or purchase it for cheap
(see link at http://www.scyld.com/ )

http://www.scyld.com/clustering_overview.html

[...]
Scyld Beowulf installation is easy. It's like loading Linux onto a single PC.

The Scyld Beowulf software provides the capability to start, observe, and control
processes on cluster nodes from the cluster's front-end computer.

Scyld Beowulf's cluster process control, BProc, decreases time to start processes
remotely. With process migration times of ten milliseconds, BProc provides an
order of magnitude improvement over other job spawning methods. Additionally
BProc provides insight into job and cluster performance.

Scyld Beowulf features Large File Summit (LFS) support via Scyld's Linux kernel
updates and GNU C library which support 64 bit file access on the ext2 filesystem.
Scyld Beowulf also includes utilities modified to take advantage this. (Basic text
utilities, scp, ftp client and server).

Scyld Beowulf includes GUI-based cluster node configuration, control and status
tools.

Scyld Beowulf ships with a customized version of the popular MPICH message
passing library. This version is modified to take advantage of the unique process
creation and management facilities provided by BProc which makes running MPI
applications easier than before.

Scyld Beowulf includes MPI-enabled linear algebra libraries and Beowulf
application examples.

Re:the best Beowulf distro? Scyld, of course.

[Coolfish]

Wow, imagine a beofwulf cluster of these...

oh wait.

Clustering is way custom

[Crutcher]

There are twqo basic types of clustering:

1) Process clustering - This beowulf, it is designed to rip every last shred of CPU time out of boxen. It is a VERY custom, machine dependant thing. A good B-cluster will be so hand tweaked as to be almost unrecognizable as what ever distro.

2) Server clustering - this is failover stuff, and distros can do this much better. Most people call it something like High Availability. But you are still likely to teak it up.

This is not a very good question, because clusters tend to be so custom. Its like asking: "Whats the best frame to base a kit car on?" There /is/ a valid answer, but it simplifies more than it educates.

-- Crutcher --
#include <disclaimer.h>

Extreme Linux is a little out of date

[GC]

I think the last version of Extreme Linux was (searches for his Extreme Linux CD) is based on RedHat Linux 5.0 - it's a little out of date now - code has moved on considerably.

For you I would like to recommend some reading:

Building Linux Clusters by David HM Spector published by O'Reilly, (hmmm site seems to be down, come back later, or check Google cached version)

This book comes with a CD together with clustering software. It also comes with step-by-step instructions. I believe, however, that there are some errata, which means that some hacking will need to be done to get your cluster online.
It also goes through some aspects of choosing hardware etc...

A more in-depth resource, without step-by-step instructions, but with in-depth discussions on granularity of Beowulf systems and whether they are actually good for the tasks you have in hand is:
How to Build a Beowulf, A guide to the implementation and application of PC Clusters by the MIT Press

Also check the The Beowulf Project Site and the The Beowulf Underground Site

Have fun!
---

Moderation on story submissions?

[yerricde]

I really wish Slashdot would start moderation on articles too

You really wish you were looking at Kuro5hin. All logged-in users are always moderators at all times, and all logged-in users can vote +1 or -1 (remind you of [e2]?) on story submissions in the public queue.