Slashdot Mirror

Beowulf clusters can be made of any platform Linux (or other flavors of unix) runs on. The software needed to run a Beowulf cluster, PVM or MPI, are open source. Compaq used Alpha because it's there flaqship CPU and it's the highest performance CPU available. For more info, see the Beowulf Project.

We build turn-key Beowulf systems for a living. You may want to consult:

Beowulf.org

Beowulf FAQ

Beowulf Underground

Beowulf Quick Start

In addition, you may want to contact us directly about helping you convince your management that turn-key and supportable Linux Beowulf Clusters are available and do useful things. Take a look at our success stories.

Doug Eadline,
deadline@plogic.com
Paralogic, Inc.

Since they weren't mentioned, I thought I'd mention them here: Mosix, Beowulf, and I couldn't find SCI.

We are a small Internet development shop, running a few servers and a mixed bag of development stations. Currently, there are three Linux boxen on our network, running the latest RedHat releases. We are looking to put in three more systems, for a total of 5 running some Linux/UNIX like OS.

• A MySQL server box
• A Firewall
• A Mail/Fileserver
• 2 Webservers running Apache / PHP(Zend)

When we perform this upgrade, we are willing to change operating systems if there is a demonstrable benefit. Due to recent slashdot postings we have started looking at OpenBSD as our server OS. Now, we do understand that RedHat is not the only Linux distribution available, but we don't really want to get into a Linux/Linux war here. We don;t mind changing if we should for technical reasons - but the Linux world seems more hip and vibrant, and we really like the penguin T-shirts we have... so if we can stay on Linux then we want to.

So far, we like what we hear about OpenBSD - but we don't know if the things we like are inherent in the relative designs of OpenBSD or if they are results of policy choices by the OpenBSD team. If they are the results of policy decisions, then with any luck a Linux distribution could be found that exhibited the same characteristics?

Features we like about OpenBSD:

• It seems like the release/testing cycle is extremely carefully controlled. While a freewheeling machine with lots of OpenSource code on the desktop is a good thing, for a server it seems that a smaller group exercising testing/release control is a more controlled system.
• The integrated crypto looks great, the one time use passwords look like a big winner here.
• There are a lot of references to OpenBSD's security and stability - but none with any specific examples or technical backing.
• The file layout on OpenBSD seems like a winner, it looks like things live in a well thought out and logical set up - not in a mishmash like RedHat.

Assumptions:

These systems will be running the server software they need, and X11 + (Gnome||KDE) for administration and so on. They will not be running the latest stuff from Linuxberg or a bunch of things that would be on a desktop OS. So we are going to try very hard not to introduce any instabilities. We aren't going to be compiling running games, sound drivers and the like that integrate directly into the kernel.

The questions are:

1. Is OpenBSD more secure in some fundamental way that a well maintained Linux distribution?
2. Is OpenBSD more stable than a well maintained Linux distribution?
3. Will the OpenSource software we normally need (firewall, Apache, PHP4, Perl, Python) and so on probably compile on OpenBSD?
4. Does OpenBSD have something like clustering support (Beowulf) and failover?
5. Is the performance of a well maintained OpenBSD system better than a well maintained Linux distribution?
6. Does Linux have anything like the one time use password system?
7. Does OpenBSD support multiple CPU's better then Linux?

Thanks for taking the time, and hopefully we can keep the flames down to nothing and talk about technical issues this time.

Linux has proven to be an essential component in building clusters of PCs (pioneered by Beowulf), and its popularity is increasing in the world of scientific and high-performance computing. With a modular design and free source code that has been ported to several CPUs, the Linux kernel is also ideally suited for computer science research. Several companies have introduced Linux products to support powerful desktops and high-performance computing clusters. Dozens of universities and laboratories are using Linux for scientific computation and research. Companies are beginning to market preconfigured Linux clusters using the latest Intel or DEC Alpha CPU. The marketplace is evolving. The Extreme Linux community wants to help.

http://www.beowulf.org/listarchives/beowulf/1999 /03/0027.html


http://www.eet.com/story/OEG19990302S0026

Plans call for the system to scale to peak performances of multiple tens of teraflops, many times faster than any supercomputer in existence today. One teraflop is equivalent to a trillion calculations per second.

"Cluster", "Linux" and "Beowulf" are popular buzzwords at the moment, and I can see a bandwagon developing. However, it's a bandwagon which isn't going anywhere yet, because it doesn't have direction.

Beowulf is an interesting technology, but I don't think it has all that many applications outside of scientific research. For Linux clustering to achieve credibility as a viable means of replacing mainframes and high-end servers, a more balanced architecture, providing for high availability and ease of administration needs to be developed.

Luckily, I have the answer, am currently preparing a business plan and intend to begin seeking out interested parties in a month or two. If there are any venture capitalists out there who are interested in investing in a venture with more than hype and PR behind it, let me know. :-)

Meanwhile, some guy at Dell UK is inviting people to participate in building a large Beowulf at Dell's "Proof of Concept Lab" in Limerick, Ireland, at the beginning of September.

By an amazing coincidence, I'd already booked those two weeks off to go home and catch up with the family. A trip down to Limerick is on the cards, methinks...

D.
..is for Dastardly.

What does the fact that an operating system not being open source have to do with the ability to write applications for it?

Because with something like this, you are likely to want to make modifications to the operating system. For the most obvious example, Be's proprietary status will make it difficult to strip out that pretty GUI, which you don't particularly want to be running on every node in your cluster.

I was under the impression that these days network shared memory is generally considered a bad idea for performance reasons (by abstracting away the distinction between local and nonlocal data you abstract away much of your ability to optimise for performance), but there are a number of other kernel modifications the beowulf people have made.

Looking at www.beowulf.org, I see that they do use special software to achieve high performance with multiple NICs. In particular, Beowulf Ethernet Channel Bonding.

Has anyone tried this in a web server environment? Maybe this isn't the best long-term solution, but for now, it should kick ass! Perhaps c't could be convinced to try a retest?

Looking at www.beowulf.org, I see that they do use special software to achieve high performance with multiple NICs. In particular, Beowulf Ethernet Channel Bonding.

Has anyone tried this in a web server environment? Maybe this isn't the best long-term solution, but for now, it should kick ass! Perhaps c't could be convinced to try a retest?

As many people have already noted, using the Promise Ultra33 is an excellent way to approach Software RAID with IDE drives. There is some very useful information at Erik Hendriks' website at NASA Goddard:

http://www.beowulf.org/bds/disks.html

They found that most of the "dual" channel IDE ports built into motherboards are not truly independent because of a shared buffer in the controller. This is a "feature" of the IDE controllers used and effectively limits the collective performance of the two IDE channels to roughly that of a single channel. The IDE channels on the Promise board are truly independent though. As their benchmarks show, placing one drive on one of the motherboard controllers and one on each of the Promise controllers yielded nearly three times the disk bandwidth of a single channel. Of course, this is for data striping, not RAID5, but the principle is the same.

For those interested in building a RAID5 server, this configuration makes a lot of sense. Use two disks on each channel of the Promise and two disk on the motherboard controller...five data and one parity and roughly 3x the bandwidth of a single drive.

i quote from the beowulf mailing list faq v2 at: http://www-unix.mcs.anl.gov/~utha yopa/bwfaq2.html

2. Where can I get the Beowulf software? [1999-05-13]

There isn't a software package called "Beowulf". There are, however, several pieces of software many people have found useful for building Beowulfs. None of them are essential. They include MPICH, LAM, PVM, the Linux kernel, the channel-bonding patch to the Linux kernel (which lets you 'bond' multiple Ethernet interfaces into a faster 'virtual' Ethernet interface) and the global pid space patch for the Linux kernel (which, as I understand it, lets you see all the processes on your Beowulf with ps, and maybe kill etc. them), DIPC (which lets you use sysv shared memory and semaphores and message queues transparently across a cluster).

As for the licensing info, presumably the patches to the Kernel will be GPL'ed. Check out: http://www.beowulf.org/software/softw are.html. I couldn't find any specific information on liscensing, although they do refer to the software necessary to implement Beowulf as:

implemented as an add-on to commercially available, royalty-free base Linux distributions

hope this helps
alex

Residents of upscale neighborhoods on the outskirts of Tokyo (Japan) have been harried lately by large packs of electronic dogs. The dogs patrol the area at night causing a wide variety of damage ranging from spreading trash about the lawn, digging up flower beds, and destroying any metal objects they can reach (including mailboxes). Speculation by local authorities includes the possibility that these "dogs" are networked together by some sort of wireles LAN as a large "Beowulf cluster." (See related information on the Beowulf Project at CESDIS.

Information compiled from the few eyewitness accounts available indicates that the dogs lounge in well lit areas by day, recharging energy cells via photo voltaic panels, and roam the neighborhood at night. Their "mission" appears to be the gathering of raw materials for the construction of more dogs.

Watch-out Tabby! Fido has an attitude!

D. Keith Higgs
CWRU. Kelvin Smith Library

The beowulf newsgroup had a couple short threads a couple months ago about consistently abyssmal performance on redhat 5.2 SMP machines running 2.x with > 512 MB of RAM. The two threads [ one, two] deal with users who had horrendous performance problems with their new machines (both running 2.2.2, the same kernel as in the report) when they used more than 512 MB of ram, but the performance jumped right back up when they used 512 or less. Check out the articles to see how bad the performance was; it's pretty surprising, and presents an interesting opportunity for detractors of linux:

Linux definitely has some hardware/kernel combinations that would seem OK by design on paper, but exhibit peculiar behavior in practice, especially with SMP. I wouldn't rule out the possibility of the testers (or financial backers) hand-picking kernels/hardware configurations that could affect results while seeming perfectly viable to the layman.

It seems very likely to me that if Microsoft did not outwardly donate the hardware to the testing company, they at least made suggestions on its configuration. The open nature of linux development and bug disclosure could easily be used by companies wishing to stage biased demonstrations; Microsoft almost certainly does a thorough job tracking linux kernel development and bug reports.

-- Scott

The beowulf newsgroup had a couple short threads a couple months ago about consistently abyssmal performance on redhat 5.2 SMP machines running 2.x with > 512 MB of RAM. The two threads [ one, two] deal with users who had horrendous performance problems with their new machines (both running 2.2.2, the same kernel as in the report) when they used more than 512 MB of ram, but the performance jumped right back up when they used 512 or less. Check out the articles to see how bad the performance was; it's pretty surprising, and presents an interesting opportunity for detractors of linux:

Linux definitely has some hardware/kernel combinations that would seem OK by design on paper, but exhibit peculiar behavior in practice, especially with SMP. I wouldn't rule out the possibility of the testers (or financial backers) hand-picking kernels/hardware configurations that could affect results while seeming perfectly viable to the layman.

It seems very likely to me that if Microsoft did not outwardly donate the hardware to the testing company, they at least made suggestions on its configuration. The open nature of linux development and bug disclosure could easily be used by companies wishing to stage biased demonstrations; Microsoft almost certainly does a thorough job tracking linux kernel development and bug reports.

-- Scott

• I'm involved in admin'ing a few of these sleek, slick, mean monsters. To sum it up: (8x)~300mhz + 8gb-ram = $250k. Unless you have very specific needs, you don't need one.
• For less than $80k you can fill a rack with 20 600mhz Alphas (512mb-ram each, dual ether, 2U cases) and get a few decent switches. That's a total of 12000mhz of Alpha and 10gb-ram for less than 1/3 the price. Even after you factor out ovehead, the o2k cost:performance falls way short. On the other hand, "Origin" is a cool name and they come in a very shiny contoured plastic box.
• Nevertheless, having NUMA-kernel support can't be a bad thing. Who knows where it might come in handy?
• Offtopic:On another note, as one who has a (small) cluster, I've finally gotten sick of the word beowulf being tossed around. Does anyone have even a faint idea what a beowulf is? All beowulfs are clusters, but the obverse is not true. For an explanation, visit http://www.beowulf.org and actually read their docs.