Wanted: Turn-Key 10-Node Beowulf Cluster

forgotten password writes: "I'd just started working on my morning M&Ms, when I was asked where my group can buy a good turn-key ~2CPUx10-node Beowulf cluster in two hours. I suspect the time frame is longer than that, although the window-of-opportunity for the money is apparently on the order of days, and a quote before the procurement meeting would help. Any ideas? Who's good? What it should cost? Thanks!" If you're quick, maybe you can become the world's newest manufacturer of custom beowulf clusters.

Comments

Atipa

[p14-lda]

Clemson University purchased a setup w/ 512 nodes from Atipa, they delivered it onsite. Can't beat that. Call 888-222-7822, and ask for Bret, tell him the PARL sent you

Re:Atipa

[donglekey]

I went to a party at Comdex Chicago that was for Atipa employees, and listend to a talk from the CEO and they are ass-kickers. They have built clusters for Motorola (I think) and NASA. they certainly wouldn't be a bad choice.

Re:Atipa

[Dyelar]

Be warned, Atipa was recently bought by MicroTech computers, and they are sometimes known for not being very good about support, or providing decent equipment. I know that Microtech fired a whole bunch of the Atipa employees though. Before this though, Atipa was looking very good.

If you are interested, SGI can sell you turn key beowulf solutions also. You can also go to http://www.beowulf.org and they have a list of commercial companies that provide beowulf clustering solutions.

Re:Atipa

[Andrewkov]

I tried calling, I think his phone number is slashdotted..

Re:Atipa

[TgrMan]

I don't know what you may have experienced with Atipa in the past but I can definitely assure you that things are on the right track now. With a team of engineers, of which I am one, who actually have engineering degrees from accredited schools, many things are in the works behind the scenes. Keep an eye on us over the next few months and I think you'll be pleasantly surprised. I am from one of the schools on the cutting edge of Beowulf which worked with Becker when he was at NASA's Goddard Space Flight Center and more recently all of the guys at Scyld, so I can assure you that Atipa and its partners are working harder than ever to become the premiere Beowulf provider.

Re:Atipa

[jmauro]

By far and away MicroTech is the sleaziest of all possible computer vendors. They only really have money, because for a while the state of Kansas dictacted that computers must be bought in state. Their PC's break after about 1.5 years and they try to wiggle out of complete support contracts, which would have 1.5 years left on them by saying stupid shit like memory is too expensive so we're not going to replace it (even though they are contractually obligated to do so). And they're laptop support is utter crap. You'll call about a failed laptop. Trace the problem down to bad memory (himem.sys won't even start because it can't access sections of memory), request to have it replaced, but they'll send it back saying Norton anti-virus fixed the problem. But the problem pops up in the returned box. Oh yea, and you can't find drivers for anything. Sorry to be so mean, but I've really had more than I can take of Microtech. Stay away from them like the plague.

Re:Atipa

[xmedar]

How about 10 nodes in 3U for Linux?

here

And why didI get the "Your comment violated the postercomment compression filter. " ???

planning

[bwhalen]

So it would appear that out of nowhere a multithreaded application appeared that needed this kind of horsepower??

Re:planning

[cweber]

Probably not. In science there's sometimes money that needs to be spent by a certain deadline. More often than not the principal investigator will then suddenly remember that proposal by some group member to spend big bucks on some obscure (for him/her anyway) technical thing and get on the bandwagon. At that point things need to move fast...

Been there, done that. Planning has not much to do with it, but getting a busy and distracted person's ear does.

Remember: Computing per se is not the primary focus of most compute power consumers.

Re:planning

[caseydk]

no, what it ACTUALLY is is a federal group of some sort...

the fiscal year ends in 10 days and any money not spent goes back to the treasury and scrutinized...

one of the contracts that i'm on right now is freezing as of 09/30 unless our client gets at least some of their funding soon...

the House seems to be busy doing other things right now instead of approving the budget... understandable though...

Western Scientific ...

[Bob(TM)]

... will sell you one.

Price depends on bells and whistles, but the 8 node, dual processor P-III system we got with SCI cards ran around $35K.

http://www.wsm.com

x4? xMore?

[DestroyahX]

I wonder what the limits and power of a 4-way quad Xeon Beowulf cluster would be like.

With RAM so cheap nowadays, one could have at least a capable cluster without monster chips. Coupled with Xeons, I fail to imagine how great the price:performance ratio is compared to Big Iron like IBM's Power3 and Power4 systems...

*drool*

Re:x4? xMore?

[DestroyahX]

Commodity parts. Big Iron peice are more expensive than PC pieces.

I still think that the $:performance ratio is very exciting on the PC side.

Don't forget Linux, which is free/

Re:x4? xMore?

[markhahn]

they suck, basically. a 4-way Xeon is sitting
on a miserable 800 MB/s (450 sustained) bus,
so unless your code is totally cache-friendly,
the CPUs will starve. and if your code is so
cache-friendly, you should probably be looking
at different hardware in the first place.

Re:x4? xMore?

[DestroyahX]

they dont make 400mhz mobos for xeons? if nto now, they will for intel's upcoming 2.2ghz xeons.

Re:x4? xMore?

[choprboy]

As someone who actually owns a quad Xeon (Intel Sitka 4x400MHz 1M cache) and is building 2 more for a cluster... the answer, as always, is "depends". It all depends on what the intended use is. For a embarrassingly simple parallel processing job (aka running 20 seti@home jobs) the price/performance ratio can be quite poor. Prices are definately down, but the motherboards and RAM are still fairly expensive (typically you need EDO ECC DIMMs with high end server boards, not the cheap SDRAM). You can pick up several 1GHz barebones Athlons for the same price and run the data serially thru each at a faster pace.

On the other hand, if you have a true multi-threaded, highly integrated task that requires high inter-process communication, separate boxes are a poor choice. Something like a large relational database or multi-dimensional vibration calculation wherein each calculation requires knowledge of it's neighbors motions, is far superior on a multi-CPU box. Unless you implement an expensive Dolphin/Myrinet network, the process communication alone, be it over ether, SCSI, or FC, kills a multi-box solution. Not to mention the fact quad Xeon boxes typically take 4-8GB of RAM so everything is always local.

So they want a cluster in 2 hours to do .... ?

[Rosco P. Coltrane]

You can't buy (or make) a x node cluster and expect to run Quake x times faster : parallel processing machines require parallel programming. This is usually means that you want a cluster to solve a particular problem, that the problem is specified, the parallel application is properly designed, architectured and implemented.

I have the feeling that friend forgotten_password's group have no clue about Beowulf clusters.

Re:So they want a cluster in 2 hours to do .... ?

[bbay]

Surely we can give the fellow the benefit of the doubt, and assume that he knows at least this much.

Re:So they want a cluster in 2 hours to do .... ?

[Diclophis]

Not together true... http://www.mosix.org/ is a kernel patch that will allow seamless clustering...

Re:So they want a cluster in 2 hours to do .... ?

[bort13]

so you're saying that just because they might not need it is a good reason for not buying such a cool toy?

for shame!

Re:So they want a cluster in 2 hours to do .... ?

[xcjohn]

*or* they know exactly what they're doing and you're bein an idiot and assuming the guys a fsckin moron. People dont go out and buy 10 node clusters of 2 way machines on a whim.

Re:So they want a cluster in 2 hours to do .... ?

[carlos_benj]

remove your left eye to email me.

Gee, I know he was asking for a quote, but if the price is that steep just for the RFP I'm afraid we'll have to pass on the consulting fees....

(I normally don't respond to .sigs, but given the topic....)

I dont think you can ...

[Tensor]

I never saw a place that sold ready made beowulfs ... i think you need to "assemble them" yourself. And if u can manage that in two hours you should come work for us ! Check out http://www.beowulf.org/

If I remember my Beowulf correctly...

[Slashdolt]

It should cost an arm, but not necessarily a leg.

Re:If I remember my Beowulf correctly...

[nobody69]

Bad Slashdolt, bad!

Beowulf Site (www.beowulf-underground.org)

[p14-lda]

Check out www.beowulf-underground.org That is the place for everything beowulf. It is run by the guys in the Parallel Architecture Research Lab at Clemson University.

this month's Linux Journal...

[Kraken137]

on page 7, has an ad for Aspen Systems, advertising out-of-the-box beowulf clusters. I've never used them myself, but they have an impressive list of clients in the ad, including Los Alamos, MIT, NIST, NOAA, and Sandia.

www.aspsys.com

Scratch That

[Tensor]

These two places make and sell Clusters

http://www.par-tec.com/

http://www.aspsys.com/

I have no knowledge about how good these are.

BTW, you DO know that you need a special type of paralell programming to really make use of all the added processors rite ? its not like your regular apps will run faster.

Good luck
T

From Linux Journal

[kperrier]

A quick search of the ads in my linux journal:

• http://www.microway.com/
• http://www.aspsys.com/
• http://www.swt.com/

I hope this helps!

Kent

Vendors

[PenguinX]

Penguin Computing ships beowulf clusters

IBMdoes a lot of linux stuff, they even have beowulf traning classes - I imagine that they have some turnkey solution.

Compaq sells 'em. too.

In other words, almost any company that sells Linux servers sells beowulf clusters o' servers as well. And if you want training, quite a few of them out there have classes for it too :)

Re:Vendors

[shogun]

Penguin Computing [penguincomputing.com] ships beowulf clusters

This ask slashdot might not of been neccesary if the original questioner wasn't filtering slashdot's banner ads. :)

Maybe something here ...

[Midnight Thunder]

I have never set one up, but maybe visiting the Beowulf might be a good starting point. Other links include the Beowulf Clusters page at Yahoo and the Oak Ridge Extreme Linux Page.

If you don't find any answers to your quest then you could always buy 10 dual-processor machines, configure one and then copy its HD image to the other 9 ( I have never tried this ).

OUCH

[Tensor]

1. Microway's Dual 1GHz Pentium III Beowulf Cluster
Package Pricing (Including Server):
8 Processors: $ 8,625
16 Processors: $16,325
32 Processors: $31,725
64 Processors: $62,525

dammm this is like a 1k per proc ! i am sure you can build it cheaper

Re:OUCH

[BrentN]

Not if the price includes the networking costs. If you're using "real" interconnects (i.e. not 100BT), the networking costs can be as much as a quarter of the total cost. Also, if you buy rackmount enclosures, they are typically more expensive than if you use minitower cases. But even so, when we built our 168 node Pentium cluster, we paid $900/node exclusive of networking costs.

Linux Networx

[Snodgrass]

We build beowulf clusters and have (if I do say so myself) the most advanced cluster management utilities in the industry.

Check us out.

I'm not a salesman, but we've got some people that would be more than happy to talk to you. :)

Re:Linux Networx

[Snodgrass]

I am a man and will to refer to myself as such.

However, if you inquire at the web site a salesperson will contact you. :)

Re:Linux Networx

[Snodgrass]

3 months?? Hmmm...somebody's slacking then. I don't know what else to tell you except you can try the 800 number...I'll kick up some dust and see where the ball was dropped.

By the tone of your question

[alen]

I bet you either work for the government or do IT consulting for them. Is it for some kind of scientific work or to help kill of the terrorists?

Re:By the tone of your question

[psychalgia]

interesting thought, im right behind you. Perhaps an encryption cracking machine? hmmm....

Re:By the tone of your question

[alen]

Actually at my last posting in SETAF the planning started 10 months out. Every one put their dream sheet in for unfunded projects months out and hoped for the best. That's how they found the money to dig up the streets and lay fiber.

Re:By the tone of your question

[psychalgia]

heh, well, its that way EVERYwhere else too, not just gvmt.

Don't forget Scyld.

[pi_rules]

Check out Scyld [scyld.com]. If I'm not mistaken Donald Becker (one of the founders of Beowulf) is the head of the company ... or at least has something to do with it.

Re:Don't forget Scyld.

[becker]

The rtl8139 driver code keeps churning exactly because I don't have control over the changes. Look at who is making the modifications -- it's not me.

Many of the changes do come from my driver releases, but are incompletely or incorrectly copied over the driver in the kernel.

The was all part of the kernel development fall-out about two years ago. I only wanted to release working drivers into the kernel, while Linus wanted all development moved into the regular kernel releases. Linus just started accepting patches from anyone, which resulted in much lower quality drivers.

Re:Don't forget Scyld.

[jfunk]

Hmmm, I have 2 8139s and a Rhine and I have never had a single problem with them other than the fact that I get about twice the performance out of the Rhine...

I also upgrade my kernel regularly.

Are you sure that your cards or MB chipset aren't bad or anything?

Also remember that DB wrote many rock-solid drivers for high-end cards like 3Com, etc. I have some of those, too.

Besides, when you buy cheap-ass $5 NICs and they don't work properly, you should blame the cards first.

Re:Don't forget Scyld.

[evil_one]

Donald isn't the maintainer anymore. Starting with (I think?) 2.2.19 it has a different maintainer, and although the drivers rock at 100mbps, there are issues with 10 mbps. Consider replacing them with Becker's drivers, which you can find at scylid.

Applications?

[jnik]

As other have pointed out, getting the cluster and getting it running isn't the hard part. Parallelizing your application, choosing a parallel library/utility, and coding to it will take a lot of time. A beowulf cluster is basically just a big bag of Linux boxen until you code specifically to it.

Of course, if the issue is just spending the funding fast, go right ahead. Don't expect results for a long time, though.

Just wondering...

[teaserX]

Would 2 Quad boxes be better than 4 duals? I don't think they're *much* more expensive (per slot)and administering two boxes might be more practical than four. The performance gained by eliminating communications overhead may make for a desirable price/performance. I don't know; I was just thinkin'...

Re:Just wondering...

[zaius]

Two quad boxes would theoretically outperform 4 duals, because you have more bandwidth between the processors. Of course, if it's not a parallel task in the first place, extra processors won't give you anything.

You're wrong about the price though. Two quad boxes would be _much_ more expensive than four duals, because you can no longer use your traditional PIII/Athlon processors. You have to up it to an Alpha or Xeon. Both those processors and their respective motherboards cost 1.5-3x the price of PIII's, for the equivalent number of processors.

The point of Beowulf clusters is to be able to put bunches of computers that are not very powerful in and of themselves together to make something really powerful, so using quad processor boxes kinda defeats the purpose. The ideal price/performance balance is two processors per box, and this is easy to see by looking at the commercially available solutions.

Re:Just wondering...

[RollingThunder]

Probably depends where most of your collisions take place, but I do know that when you have multiple Pentiums (not sure on Xeon based), that when one CPU is accessing the memory or I/O, the other has to sit and wait. That's one of the big benefits of a multiple Athlon system - both CPU's can access memory simultaneously.

We bought one of the APPRO 1U dual TBirds, and this thing -screams-. It also howls, but that's the four big blower fans. :)

APPRO is at http://www.appro.com, and Anandtech had a writeup on the server.

Re:Just wondering...

[j.e.hahn]

(Note: all of the below refers to servers from the Dell product line. It's what I use, because it's what my company buys. Other vendors are just as suitable, for the most part)

Hmm... I'd say no. We have a cluster of 8 quads, plus a bunch of smaller, older duals. They're beautiful clusters, but I'd rather have dual processor machines. Here's why:

1) A quad processor box is 4U of rackspace. I've never seen one smaller. They may be larger still.
2) I can buy a 1U dual processor box. This means 2x the CPU per Unit of rack. Hence greater compute density.
3) A quad processor box has greater bus contention than a dual.
4) If you lose a node, you lose a larger chunk of processing power, and it costs more to replace.

In my mind the ideal node size, at least with commodity hardware, is dual. (If you start talking really custom gear, then it's another story entirely...) However, YMMV.

Re:10,000 G

[bteeter]

Errr.

No, I don't think so. The article you are refering to was actually about building a server with 1 TB of storage for under $10,000. I don't think it had anything to do with clustering. See it here:

http://slashdot.org/article.pl?sid=01/07/19/155421 6&mode=thread

That was over 2 months ago, so the price is probably even less than $10K now...

Take care,

Brian
--
AssortedInteret.com
Featuring 100% Linux Based Web Hosting
Try us Risk Free with our 30-Day Money Back Guarantee
http://www.assortedinternet.com/hosting/

Linux Labs

[jjo]

Linux Labs will happily sell you clusters, either in standard or custom configurations.

My rough estimate from reading their website is that a 10-node 1.33-GHz Athlon cluster from them would price out at something like $16-17k.

Missing Critical Linux

[Outland Traveller]

I've run into guys from an company called mission-critical linux www.missioncriticallinux.com at the local LUG meetings. I know that they do custom clusters. Perhaps they can help?

See Commercial sites at bottom of www.beowulf.org

[bwhalen]

As stated previously look there...

What do you run on the darn thing ?

[beanerspace]

I'm glad someone is asking about the cost. But I think I've got the hardware thing worked out.

I've got a bunch of old Pentium I's and II's in a basement of a charity I do some volunteer work for. Can get a hold of a router. I've got all the nerd-boys a bit pumped to beowulf a cluster this winter when it's too cold to paintball.

So my question is, what to run on the blasted thing once you get it up ? Is there anything open source out there worth looking into ? Or am I just going to have to buy an application. If so, which one, how much ?

Re:What do you run on the darn thing ?

[psychalgia]

wow, you sound like my brother... are you adopted?! Beowulfs in lieu of paintball, and in charge of pumping up the nerds...like looking into a nerd-mirror.

Re:What do you run on the darn thing ?

[weakethics]

What a question!!
Hell, I don't need beer, but what's life without some fun?

Re:What do you run on the darn thing ?

[cr0sh]

Most of the apps for Beowulf clusters tend to be scientifically oriented apps. It really comes down to there being only certain kinds of problems that can be broken down into parallel processes that can be executed concurently, then the answers "recombined" at the end to get the result.

IIRC, Mosix allows for using the machine as one large machine, essentially allowing each process (or groups of processes) a single CPU, but funneling the result back to the main controller, so to the user, it looks like one large machine. This is different from a standard parallel processing Beowulf cluster, which behave like classical parallel processing machines.

For that domain (parallel processing), aside from coding your own stuff (hard to due, even if you are a master coder, from what I understand - due to having to understand what classes of problems can be broken down into parallel tasks, and then actually applying that knowledge to real tasks), there is one application that would prove to be "fun" - Ray Tracing.

Fortunately, POVRay has a paralleled version available, for Beowulf clusters. I don't have a link, but I know it exists (heck, you can probably get it at the POVRay site).

One other disclaimer: Everything I have said should be interpreted as "coming out my ass", simply because I have no experience at building or using Beowulf clusters or any other parallel processing architectures. Most of what I know about them have come from /. discussions, various web sites of Beowulf cluster projects, books, and magazine articles (as well as one funky book I picked up at the library detailing programming in Fortran for a Connection Machine - eeepp!). The POVRay stuff is true, though...

Re:What do you run on the darn thing ?

[tolldog]

As much fun as rendering seems over a large network, if you are really doing any major rendering and you don't have Cray-linked SGI's .. you are better off treating it as a NOW (network of workstations).
The POVRay rendering dropped off at 8 procs or so, if i can remember correctly.

We render a hand full of frames per box and do this on several boxes. This keeps the network overhead between machines low.

Sometimes when people talk about beowulf, what they really mean or really want is just a bunch of managed computers running tasks. If the task can be broken down at submission time, like most renders, then take advantage of a NOW.

Re:What do you run on the darn thing ?

[asland]

search for mpipov. Installed it on the sun cluster at my school the other day.

Re:What do you run on the darn thing ?

[ryanvm]

One other disclaimer: Everything I have said should be interpreted as "coming out my ass"...

Heh - I think with Slashdot, this is usually an assumed competence. But we appreciate you candidness anyway.

Re:What do you run on the darn thing ?

[duffbeer703]

No shit sherlock.

The guy knows what a Beowulf does, he needs somebody to BUILD one for him.

Goddamn karma whore.

Re:What do you run on the darn thing ?

[cr0sh]

Good points.

In a way, from what I understand, this is kinda what Mosix does for you, automatically, and and thus different from a true Beowulfed parallel processing app.

I guess it also depends on if you want to learn parallel processing techniques, or if you are trying to get a job done.

The drop off I would imagine is due to slow interconnects among the nodes - I am sure there would be better gains the higher speed your interconnects were.

Anybody know how the POVRay for Beowulf machines really works? Does it break the viewport into n sections, handing each section to a different node, or does it process multiple rays (ie, a node per ray), or what? Anybody got a link on this?

Re:What do you run on the darn thing ?

[SubtleNuance]

If you just want to tinker, and ball-busting computing power is unnecessary, you really shouldnt be building a Beowulf cluster.
Have a look at the info contained at MOSIX.org they provide kernel patches && apps to build clusters also, but these clusters allow -depending on your configuration (how/who/what of each computer in the cluster) - processes to migrate from 'more idle nodes'.

From what i understand, you could setup your 'head or workstation' machine to let procs dribble off onto other nodes.

have a look - much more usefull to geek exploration than beowulf... even if beowulf gets 'all the /. mentions'

Re:What do you run on the darn thing ?

[beanerspace]

..., then why do you need one?

Because we can !

Aspen Systems

[iso]

There's an ad in the latest Embedded Linux Journal for a company called Aspen Systems (1-800-992-9242) advertising Beowulf clusters. Thtat's about the sum total of all I know about them ;).

- j

Cheaper solution

[NumberSyx]

Better and cheaper idea, do it yourself. Instead of buying 10 dual proccessor systems at $2000+ each, goto Fryes and get 30 Emachines or whatever they are selling for $299, add a couple hundred dollars for 30 decent network cards, one monitor for the control node (barrow a couple more to do the installs) a few hours to install and configure RedHat 7.1, which comes with the clustering software and you'd be done by morning. You will probably get better performance at half to two thirds the cost. This is what clustering is all about, turning cheap off the shelf systems into a super computer.

Re:Cheaper solution

[dsb3]

> This is what clustering is all about, turning cheap off the shelf systems into a super computer.

No, this is not what clustering is about. Beowulf, specifically, is about gaining raw performance without concern for stability or fault tolerance/redundancy.

There are many other uses for and of clustering; not all of them are concerned with speed (your super computer quotient, if you will), less of them are concerned with saving money.

Granted, the original post specifically mentioned Beowulf, but from that one should not assume that clustering == Beowulf.

Oh, and why on earth do you need a monitor to do the install??

Re:Cheaper solution

[markhahn]

beowulf isn't just clustering, it's compute-intensive clustering. that means that run-of-the-mill 100bT cards will likley not be enough - think Myri2000, SCI, Quadrics, gigabit.
whitebox machines also tend to be truely horribly configured - you probably don't want UMA video stealing dram bandwidth, for instance. and even for a smallish cluster, it's smart to get better power supplies and fans, simply because the noname stuff has short expected lifespans.

Re:Cheaper solution

[huckda]

Oh, and why on earth do you need a monitor to do the install??

I'll tell ya why you need a monitor to do the install...

so that when the first machine is up and running you can download ROBOCODE and do some javabot simulations on your new cluster and see how many battles you can simulate in 1 hr. =)

JOY JOY HAPPY HAPPY JOY JOY!!!

Re:Cheaper solution

[i-sob]

The $299 is more than likely the price AFTER a few hundred dollar rebate that requires a long term contract with an ISP (CompuServe, MSN, or AOL most likey) ...now what would this company that obviously has serious work to do on a Beowulf cluster want with 30 four-year subscriptions to MSN? :)

Re:Cheaper solution

[Jeff Knox]

Not true. You can get eMachines at fry's for aroud 299 bucks (or the 3 I got for 375, with a 75 dollar rebate, NO ISP contract, plain old rebate) without any form of ISP contract. Add a 80 buck 17inch monitor and you have a nice system. I bought one for my girlfriend. And then a few more.

Re:Cheaper solution

[milkmandan9]

Forget the eMachines.

I've just finished building a (really small) 4-node cluster for some research I'm doing.

Each node is a Duron 750 w/256MB RAM and a 100Base net card, power supply and floppy drive. They're vertically stacked on some 2' threaded rod from McMaster Carr.

But get this: I built the whole thing for $170 per node . That's everything, shipped to my door.

They load the kernel off the floppy and grab the NFS root from a fileserver I've got set up, so no need for hard drives. Granted, there's a bit more network and memory overhead (both /var and /tmp sit in memory), but I can buy more nodes from the money that I've saved on hard drives.

Re:Cheaper solution

[i-sob]

I'm impressed, and I stand corrected.

Re:Cheaper solution

[Cytotoxic]

milkmandan9's $170 per node cluster is worthy of a feature article all to itself. Somebody run over to his place and make him write it up!

Re:Cheaper solution

[tmuka]

I too would appreciate pictures of this creation. I am intrigued by your approach.

Re:Cheaper solution

[ErikZ]

I don't understand how you stacked motherboards on a rod. Got any pics?

fast and cheap

[AeiwiMaster]

How to make a Beowulf cluster in 2 hours for free.

1) Find a single machine to use as cluster server.
2) Install some linux (I prefers debian.)
3) Set it up as a diskless client server.
Debian have some diskless packages, but I prefers
to use cluster-nfs form clusternfs.sourgeforge.net
it makes it easier to maintain the system.

you can find a howto for setting up a mosix cluster
at the cluster-nfs site dropping the mosix stuff and installing
mpi and pvm should do it.

4) To use your groups computers as diskless nodes
go to http://www.rom-o-magic.net make some boot floppies
and use them to boot against your server.

5) Have fun with your new Beowulf cluster.

Whenever you need a Beowulf cluster just boot up
via the floppys.
In this setup it is only one the server you need to maintain.

Knud

Re:fast and cheap

[AeiwiMaster]

Sorry
The links is Cluster-nfs and rom-o-matic

Knud

Re:fast and cheap

[chriss]

it's http://http://www.rom-o-matic.net/

chriss

We've bought from two places

[BrentN]

We bought a 168-node Pentium cluster from Atipa, and we're negotiating for a 1024-node (yes, that's right) Athlon cluster from Linux Networx.

Linux Journal Ads for Clusters

[Duke]

RTFLJ :)
From the Oct. 2001 Linux Journal
http://www.microway.com - Alpha, Athlon & Pentium
http://www.aspsys.com - Athlon and Pentium
These ads mention clusters specifically.

Some vendors..

[pdiaz]

..listed here:

http://planetcluster.org/links.php?op=viewslink& si d=20

Hope that helps

APPRO AMD Servers

[Krieger]

You could purchase 10 APPRO AMD MP servers. These machines are based off of the Tyan boards so have all the built-in high end features you could wish for, while still being relatively cheap and easily networkable given the builtin NICs. Plus SCSI and potential for RAID if you feel you need it. and all in 1U.

http://www.appro.com/

or for the AMD systems specifically http://www.appro.com/1124.html

Should give you enough information to put together a 10U Beowulf cluster...

Re:APPRO AMD Servers

[tomblackwell]

We had bad luck with Appro servers in the past (non-Beowulf, though). They specified high-end processor cards and shipped very low-end cards. The Appro chassis became something of a joke around our office.

Who is this for?

[PhipleTroenix]

On Friday the government authorizes $20,000,000,000.00 for defense. The following Thursday a request goes out to procure Beowulf cluster in two hours. Let me guess this is to heat your dorm room right?

Your tax dollars at work.

Re:Who is this for?

[alen]

I totally agree. I spent 8 years in the army and the last four years at this link . October 1 is the beggining of the fiscal year for the US Government. That means every last penny from FY 2001 funds has to be spent or "lost".

I remember some cool shopping trips with my US Government VISA card in September. Basically we throw stuff in the cart just to spend the money. Whether you need it or not important.

Re:Who is this for?

[demigod]

You don't have to heat the dorm room to much.

http://www.terrasoftsolutions.com/products/briQ/po wer.shtml

Don't know that it's a beowulf cluster though.

Re:Who is this for?

[alen]

Scanner. Some software. Don't really remember. It's the way things work. If you don't spend your budget, it gets cut next year. And nobody wants their budget cut. So you find a way to spend the money. I didn't like it, but I didn't contrlo the money.

My first job out of the army was consultant for a gov't agency. They didn't have the same mentality. Maybe that's why we had a surplus.

RTS+FPS Sim

[Da VinMan]

Not encryption... modeling of the ground actions to be take in Afghanistan or other areas. Probably need to know just how hard it's going to be. Couple the micro-management concerns of a first person shooter with the strategic elements of a real-time strategy with the parallel concerns of mutiple agents in the field and you could probably simulate a proper battlefield in short order.

Hmm...

My $0.02 worth of a guess. :)

Scyld

[4of12]

Haven't dealt with them directly, but I believe they have Don Becker, one of the Beowulf pioneers.

I think they have developed a system to help provide a single system image, along the lines of MOSIX, but not MOSIX, IIRC. This can help managing such a cluster which could otherwise be like managing 10 separate machines - a hassle.

RLX Technologies

[buffy]

Check out http://www.rlxtechnologies.com/. We've got Scyld stuff running very well. Processors are not very beefy, but we can get 24 of 'em in three rack units (or 336 in a 42U rack), and have some very nice tools to manage the whole shebang. Very little power required, easy to wire.

Just to say it out loud...I work there, so it's a blatant self-promotion, I guess. But it is a bad ass little product that packs a punch.

Wow!

[quintessent]

Just imagine...

a Beow...

oh wait. Never mind.

Re:Could you imagine...

[Lxy]

Hey moderators: I know you guys look for these posts just to mark them -1, but occasionally these are on-topic posts. This is one of those times. "Imagine... a beowulf cluster of these" is a staple here on /. and needs to be recognized every once in awhile!

What is turn-key?

[Spiff28]

Honest to goodness curious question that I'm sure the slashdot crowd could answer: what exactly does "turn-key" mean here?

Re:What is turn-key?

[Juju]

That all you have to do, is turn the key so that it starts. Think like a car not like a PC where you need to install stuff.

In other words, no installation, everything is configured to work out of the box...
Must be a very big box though... ;o)

Re:What is turn-key?

[Weasel Boy]

Their tech steps away from the keyboard, and you log in.

Re:What is turn-key?

[bn557]

I think it'd take 2 boxes, cuz you'd need to keep the power cables and ethernet cables separate(and those DAMMED user manuals)

...now turn to page 557 of the idiots guide to configuring a beowulf cluster...

Buy it in two days

[tie_guy_matt]

Ah yes, you must work for the government.
Yes with the fiscal year ending we all have lots
of money to spend this year or we'll loose it
next year. Don't you love working for the government?
Sometimes I think the only job in America that
is better than working for the government is being a domestic cat.

LOL [NT]

[Juju]

I said [NO TEXT] ;o)

Western Scientific

[dieman]

Has nice setups. www.wsm.com

More like 3-4 grand for 1TB fileserver

[Jeff Knox]

You can easily build a 1TB fileserver (P3 1GHZ, or Athlon, 1gig of ram, etc etc) for 3-4 grand, using quailty IBM drives. None of that Maxtor garbage. Its probably closer to 4 grand with the ibm drivers, and closer to 3 with Maxtor drives.

Other vendors

[Kismet]

Try Aspen Systems or Einux Network Solutions

Not sure how "out of the box" the Einux ones are, but you can get a 10 node dual-Athlon setup for around $25k.

Smaller companies

[Felidae]

You might want to check around to see what companies locally might make them. IBM clusters are avaliable through a number of smaller companies, most will ship them just about anywhere in the country, but you can save a ton on freight be finding one local. There is one in Houston, TX that I know of if you are in the Texas area, that is PCPC, Inc. (www.pcpcdirect.com) that custom configures and builds IBM clusters

YASOB: Yet Another Supplier Of Beowolves

[nels_tomlinson]

Microway also sells these toys. If you want Alpha, look here. If your budget runs more to Athlons, look here. Unfortunately, you will have to choose between 8 and 16 CPUs on the low end here; they don't have a 10 CPU Athlon cluster.

You should think a bit about whether the extra abilities of the Alpha boxes are worth the extra bucks for your application. One thing which I think that I remember about the Alphas is that they use a crossbar switch to link the several processors on a motherboard to memory, et cetera. This should give better throughput. They also have huge caches which should help with big matrices. I think that if you have lots of little problems which should be run in parallel, more nodes with lower price and capability per node might be the way to go.

I remember back in the days of the XT, Microway used to sell math coprocessor and video boards for PCs which cost more than the box you hooked them to, along with high-grade compilers which would put that hardware to work. They were once the place to get hardware and software for doing seroius number-crunching on a PC.

Re:YASOB: Yet Another Supplier Of Beowolves

[jaxon6]

i used to work there, so i know a bit about these guys. they don't have limits on what you get. they don't seel '8-node clusters' or '16-node clusters' only. if you want 10, you get 10. wanna add a cd-rw? sound card? kick-ass geforce3(they were fun to setup) i don't remember the days of the xt, so i can't comment on that, but they do a good job of linux clusters.

QLITech Linux Computers

[James Hetfield]

Looks like QLITech, (Made famous by buying out Tuxtops this spring) is offering clustering boxes these days as well.
1U/2U AMD/Intel CPUS, and Gigabit or Dolphin Interconnects. Cool Stuff.

I've dealt with these guys for servers and a Laptop, pretty good pricing, and damn good service.

http://www.qlitech.net

YellowDogLinux G4 Cluster

[Magus]

b riQ/</A>

This is really cool, 8 node cluster size of a regular Tower. Each node is the size of a CD-rom drive. And I believe that they will soon be putting out Black Lab linux which looks to be a Cluster software package. If these wern't so expensive, I would get one now.

Re:YellowDogLinux G4 Cluster

[Knobby]

You can buy the BriQ's from Total Impact, but they're not cheap as you pointed out.

I'm glad to see that TerraSoft is working with TotalImpact. They were selling rack mounted clusters comprised of g3's ripped from iMacs.. This is definitely a step in the right direction!

Cray sells Beowulf clusters

[totallygeek]

Here is an article in which Cray builds clusters. Cray is most known for supercomputing.

off topic: instability, windows and science

[nels_tomlinson]

Your sig line mentioned that someone should do a paper on windows instability. Here is one of them. It's the third in a series.

One portion which shocked me was:

Our final piece of analysis concerns operating system crashes. Occasionally, during our UNIX study, tests resulted in OS crashes. During this Windows NT study, the operating system remained solid and did not crash as a result of testing. We should note, however, that an early version of the fuzz tool for Windows NT did result in occasional OS crashes. The tool contained a bug that generated mouse events only in the top left corner of the screen. For some reason, these events would occasionally crash Windows NT 4.0, although not in a repeatable fashion.

They crashed a unix os? Wow! That doesn't match up with my limited experience. The only way I've ever done that was by trying to do stupid things as root, like running mindi with a buggy kernel. I wouldn't have thought that this would be a problem for a normal user.

Here is something which didn't surprise me at all:

Our 1995 study found that applications based on open source had better reliability than those of the commercial vendors. Following that study, we noted a subsequent overall improvement in software reliability (by our measure). But, as long as vendors and, more importantly, purchasers value features over reliability, our hope for more reliable applications remains muted.

Mine, too.

a bit of overkill perhaps...

[neurovish]

...but, as one other post pointed out (only one?) Cray has mentioned plans of selling beowulf clusters utilizing alpha processors. If I had the money to throw out my car window, I'd go for one of these.

From the Scyld website

[evil_one]

Scyld Vendors Scyld Beowulf Professional Product Scyld has partnered with industry leaders for them to provide Enterprise Level systems which consist of pre-integrated, supported hardware systems loaded with the Professional Scyld Beowulf:

• Compaq
• PSSC Labs
• Penguin Computing
• eLinux
• Aspen Systems
• Custom Fit, Inc.
• Atipa

A suggestion to save cash on power consumption...

[SpamapS]

If this is going to be a sustained computing effort, you can save a lot of cash on power consumption by going with the RLX servers. RLX Servers They use Transmeta's Crusoe processor, so they actually can pack 336 of these in a single rack! Definitely worth a look.

get it all from the web...

[RadioheadKid]

First you could got here
And then here

How hard can it be, it's not like you're launching rockets...Wait a second, 2 days to buy, quick access to cash, damn he probably is launching rockets!

www,openclustergroup.org

[Glasswire]

download, install.

re: turn-key beowulf cluster...

[christopherwg]

any particular reason to specify a beowulf cluster?
and such a specific request for number of nodes too...
i didn't have the time(or patience) to read _all_
the replies, but i didn't see any mention of
project appleseed anywhere. goto:

http://exodus.physics.ucla.edu/appleseed/applese ed .html

and add this cluster configuration to your comparison matrix.

Re:Cheaper solution: WOW!

[milkmandan9]

Pics of the $170/node cluster threaded together.

FSB is 133, if I'm not mistaken. Cache is ... whatever is on the Durons. Sorry I don't know off the top of my head.

Been There Done That

[elinenbe]

This summer I was employed at a finite element analysis company in Philadelphia, and I designed, budgeted, and built a 10 processor AMD System (Octavian) based on gigabit ethernet, 1.2 Gig MP AMD chips on Tyan mobo's with half a gig of RAM per node, and it did not cost more then $13000 plus some setup time. (This was with a gigabit switch, etc AND the current cost with dropping prices would be less then $10000) The computer was designed to run LS-DYNA (a Livermore software finite analysis program) and it has not let them down.

Here are some benchmarks:

Octavian Benchmark runs
as of 8/17/01

Problem description:
Acetabular cup with a spherical metal ball compressing the liner into the
shell.
The effect of holes used for screw fixation to the bone is included
Total Mesh Statistics
45814 Nodes
37696 8-noded solid elements
2 contact pairs
Dynamic Relaxation Solution:

Execution Statistics
2 Processors : 62 Minutes
6 Processors: 34 minutes
10 Processors: 24 minutes

Analysis of the execution throughput indicicates a linearly increasing speed
(1/wall clock time)

And here is a review of the results:

the cluster is performing BEAUTIFULLY, and we have been crunching problems on it pretty much nonstop since it was brought online. It has really saved our butts, as we could never have met some key project deadlines without the speed. I've included some bechmarking stats below FYI for a contact problem that took only 24 minutes to run on octavian using 10 cpus. The comparable time on our $50K dual CPU octane workstation is 3 hours and 26 minutes, which translates into a speed up of 8.6 times for about 1/5 the cost.

So, What I am trying to say is build the thing yourself. You will know much more about the system, and you will be able to install any software you want to without having to deal with "customer service". Also, it will save you a bundle as a turnkey solution is nearly 3 times the price. (Even if this cluster was built with Myrinet it would still be far less then any of the pre-built solutions) Lastly, design the cluster for what you need. If your problems involves lots of RAM, then spen money there. If CPU is the bottleneck spent money there. If communication is the bottleneck....

Best of Luck,
Eric

Terra Soft-pre configured Black Lab Linux clusters

[Douglas]

Complete 8 CPU cluster, $17319.00 + Shipping

THE briQ Cluster

8x 5.25" slit case
8x PowerPC 7400 (G4) 500 MHz
8x 1MB L2 Cache at 250 MHz
8x 512 MB SDRAM, 3.3V (up to 8 GB per cluster)
8x 2.5" IDE 40 GB, internal to each briQ
4x 3.5" Ultra100 IDE 130GB, in alternate slots
with 4 briQ CPUs
300 Watt power supply
Internal 8 port 10/100 Ethernet switch with
Gigabit Ethernet upliink.

Installed Software:
YDL 2.0 or Black Lab Linux

PowerPC 7400 (G4) 500 MHz
1 MB L2 Cache per CPU at 1/2 processor speed

Northbridge - IBM CPC710-100+
100 MHz 64-bit 60x System Bus
Southbridge - Winbond W83C553

2x168 Pin DIMMs, up to 1Gb of SDRAM
Open Firmware Boot ROM/BIOS, supports remote boot
up to 40GB 2.5" IDE internal to the briQ case
up to 2 3.5" IDE external to the briQ case
16550C UART, DB-9 connector
10/100Base TX Ethernet, Dual 10/100 optional
Power Consumption: 20 to 40 Watts per CPU node
64-bit, 66mhz PCI, Custom 66MHz PCI connector
Programmable Vacuum Florescent Display (VFD)
2 lines X 20 characters Two push button switches
bi-color LED

Sure, I can sell you one in less than an hour, but

[(H)elix1]

but it may take a while before it gets shipped (grin)

rm -rf /bin/ladin

Hello? (Totally OT - Mod me down, I need the fun!)

[cr0sh]

Hello?

I wasn't replying to the article, but to an individual who was asking what he can use a Beowulf cluster for, as he obviously didn't know.

I am the last person needing to whore, since I am capped, and been that way for months...

Re:your .sig

[choprboy]

No, you didn't miss anything... Apparently I'm missing my dictionary though. In case anyone wondered about the origin, I used to play way to much xEvil.

Re:Cheaper solution - add shared storage

[paled]

does anyone know of an affordable shared storage solution for this?
I know that SCSI can operate on a shared bus for 2 nodes, but I'd really like to test out Oracle 9i Real Application Cluster (RAC) (formally Parallel Server) on a 4 node setup on Linux - but would need a shared storage device for log files, data files, control files.

FC would be a little pricey.
NetApp filers are outta my league.
thanks.

Re:Spelling police!

[BardicStorm]

One entry found for inaccurate.

Main Entry: inaccurate
Pronunciation: -'a-ky&-r&t, -k(&-)r&t
Function: adjective
Date: 1738
: not accurate : FAULTY
- inaccurately /-ky&-r&t-lE, -k(&-)r&t-, -ky&rt-/ adverb

www.m-w.com is your friend. :)

Well..

[mindstrm]

Buy ten machines, and build it yourslef? I mean.. that's all a beowulf is... it was a project to work on using off the shelf hardware for parallel processing.

DO you have some app that needs it? I mean, you can't just run anything....
Did some department just come up with some app already designed with the PVM libraries or something?

Hmmmph..

[Da VinMan]

Yeah, you're most likely right of course. But, my idea is much more fun.

Think about it though, how likely is the military to be in possession of simulation software like I mentioned with full-out graphics capabilities that actually runs on those spiffy supercomputers they own? Maybe very likely. If not, then a Beowulf cluster is the perfect starting point along with existing code perhaps from the likes of Loki.

Hmm....

;+)

What's it for?

[Mister Transistor]

If you are looking for a huge webserver or FTP server or such, then maybe a Content-addresable switch is the thing for you, and you really don't NEED a "beowulf" cluster to achieve your goal.

I recently set up a "web farm" of 10 Linux boxes, using off the shelf Compaq DL580 servers (running SuSE linux, pretty much straight out of the box). The magic component that made it a "cluster" was the Content-addressable switch which sent incoming web traffic in a load-balanced fashion to the 10 stock web servers.

Finally, the serial consoles of all 10 were run to a terminal server hooked to a stock DL360 running Red Hat for access to the consoles of the 10 servers, if necessary.

I can have that up for you in 3 days from receipt of the hardware :)

/. helps war effort!

[gsfprez]

or was it so obvoius that he's asking this for those reasons?

I'm not complaming - i'm just finally glad to see /. as something USEFUL.

fight on!

Scyld Computing Corporation

[danFL-NERaves]

They gots the mad scylds.. Scyld is a highly regarded vendor in the Beowulf field. They host the beowulf maillist (beowulf.org), offer technical certifications and work with the open source community. They are all that and a can of Moxy.

Scyld is a software company so they don't sell directly. To get a turnkey system they have partnered with hardware vendors who will sell a pre-integrated cluster along with hardware support and Scyld support for the integrated Beowulf Professional Edition software. Visit their site at:

Scyld Computing Corporation

Or go directly to their partners:

Compaq

PSSC Labs

Penguin Computing

eLinux

Aspen Systems

Custom Fit, Inc.

Atipa

Dan

Re:Spelling police!

[bn557]

copy and paste... what I did for my sig. I don't fix other peoples typoEs. I'm knot a secratarie

+2 Funny on the MQR standard

[MarkusQ]

Who modded this down?!?

It's a joke for gosh sake; he was playing off the first person's Grendel reference by alluding to The Three Billy Goats Gruff!

-- MarkusQ

Turn key is great but,

[levinas]

What I would rather see is a software rather then a hardware solution to beowulf. Right now to make use of a cluster one needs to have the know how to compile software to work in that environment this limits the usage of clusters to people in the know for example universitys. It would be better if more work was done in the creation of a linux distro and some generic applications(no doubt one already exists).

btw, just IMHO but isn't the whole point of creating a cluster to make use of your old hardware that is gathering dust rather then having to buy a mainframe. Why buy a dozen ghz pentiums when you can get a thousand 486's from a scrap dealer?

My $0.02

[bogdant]

I do think that getting a Beowulf in two hours is very wise. This is not like getting a new PC. You gotta consider the application(s) that you want to run on that cluster, and what kind of network infrastructure you need, and a zillion other things.

Anyways, AFAIK GSH sells PC & SUN clusters. You might wanna check them out.

You can also visit http://www.parallelcrunchers.net for more infos related to Beowulfs.

This is technically off-topic

[biglig2]

But am I the only person here to think "working on my morning M&Ms?"

why not a Mac?

[mr.ska]

I've seen a few stories about researchers looking for a cheap, quick, and powerful cluster for their research purposes who decided to go with Apple hardware. One group got an 8-node iMac cluster up and running in less than a day (which I can't find the link for.. sorry).

A quick search of Apple's site actually mentions clusters [apple.com]. Perhaps 10 dual G4's would suit your needs?

I want more pics! And a write up!!!!

[ShaneTheGeologist]

Please!

It's gotta be said.

[Penguinoflight]

Imagine a Beowolf Cluster of these!!

Beowulf

[PegQuin]

Call Eric over at eLinux.