Wanted: Turn-Key 10-Node Beowulf Cluster

forgotten password writes: "I'd just started working on my morning M&Ms, when I was asked where my group can buy a good turn-key ~2CPUx10-node Beowulf cluster in two hours. I suspect the time frame is longer than that, although the window-of-opportunity for the money is apparently on the order of days, and a quote before the procurement meeting would help. Any ideas? Who's good? What it should cost? Thanks!" If you're quick, maybe you can become the world's newest manufacturer of custom beowulf clusters.

Comments

Atipa

[p14-lda]

Clemson University purchased a setup w/ 512 nodes from Atipa, they delivered it onsite. Can't beat that. Call 888-222-7822, and ask for Bret, tell him the PARL sent you

Re:Atipa

[donglekey]

I went to a party at Comdex Chicago that was for Atipa employees, and listend to a talk from the CEO and they are ass-kickers. They have built clusters for Motorola (I think) and NASA. they certainly wouldn't be a bad choice.

Re:Atipa

[Dyelar]

Be warned, Atipa was recently bought by MicroTech computers, and they are sometimes known for not being very good about support, or providing decent equipment. I know that Microtech fired a whole bunch of the Atipa employees though. Before this though, Atipa was looking very good.

If you are interested, SGI can sell you turn key beowulf solutions also. You can also go to http://www.beowulf.org and they have a list of commercial companies that provide beowulf clustering solutions.

Re:Atipa

[Andrewkov]

I tried calling, I think his phone number is slashdotted..

Re:Atipa

[jmauro]

By far and away MicroTech is the sleaziest of all possible computer vendors. They only really have money, because for a while the state of Kansas dictacted that computers must be bought in state. Their PC's break after about 1.5 years and they try to wiggle out of complete support contracts, which would have 1.5 years left on them by saying stupid shit like memory is too expensive so we're not going to replace it (even though they are contractually obligated to do so). And they're laptop support is utter crap. You'll call about a failed laptop. Trace the problem down to bad memory (himem.sys won't even start because it can't access sections of memory), request to have it replaced, but they'll send it back saying Norton anti-virus fixed the problem. But the problem pops up in the returned box. Oh yea, and you can't find drivers for anything. Sorry to be so mean, but I've really had more than I can take of Microtech. Stay away from them like the plague.

Western Scientific ...

[Bob(TM)]

... will sell you one.

Price depends on bells and whistles, but the 8 node, dual processor P-III system we got with SCI cards ran around $35K.

http://www.wsm.com

If I remember my Beowulf correctly...

[Slashdolt]

It should cost an arm, but not necessarily a leg.

Beowulf Site (www.beowulf-underground.org)

[p14-lda]

Check out www.beowulf-underground.org That is the place for everything beowulf. It is run by the guys in the Parallel Architecture Research Lab at Clemson University.

From Linux Journal

[kperrier]

A quick search of the ads in my linux journal:

• http://www.microway.com/
• http://www.aspsys.com/
• http://www.swt.com/

I hope this helps!

Kent

Vendors

[PenguinX]

Penguin Computing ships beowulf clusters

IBMdoes a lot of linux stuff, they even have beowulf traning classes - I imagine that they have some turnkey solution.

Compaq sells 'em. too.

In other words, almost any company that sells Linux servers sells beowulf clusters o' servers as well. And if you want training, quite a few of them out there have classes for it too :)

Maybe something here ...

[Midnight Thunder]

I have never set one up, but maybe visiting the Beowulf might be a good starting point. Other links include the Beowulf Clusters page at Yahoo and the Oak Ridge Extreme Linux Page.

If you don't find any answers to your quest then you could always buy 10 dual-processor machines, configure one and then copy its HD image to the other 9 ( I have never tried this ).

OUCH

[Tensor]

1. Microway's Dual 1GHz Pentium III Beowulf Cluster
Package Pricing (Including Server):
8 Processors: $ 8,625
16 Processors: $16,325
32 Processors: $31,725
64 Processors: $62,525

dammm this is like a 1k per proc ! i am sure you can build it cheaper

Don't forget Scyld.

[pi_rules]

Check out Scyld [scyld.com]. If I'm not mistaken Donald Becker (one of the founders of Beowulf) is the head of the company ... or at least has something to do with it.

Re:Don't forget Scyld.

[jfunk]

Hmmm, I have 2 8139s and a Rhine and I have never had a single problem with them other than the fact that I get about twice the performance out of the Rhine...

I also upgrade my kernel regularly.

Are you sure that your cards or MB chipset aren't bad or anything?

Also remember that DB wrote many rock-solid drivers for high-end cards like 3Com, etc. I have some of those, too.

Besides, when you buy cheap-ass $5 NICs and they don't work properly, you should blame the cards first.

Just wondering...

[teaserX]

Would 2 Quad boxes be better than 4 duals? I don't think they're *much* more expensive (per slot)and administering two boxes might be more practical than four. The performance gained by eliminating communications overhead may make for a desirable price/performance. I don't know; I was just thinkin'...

Re:Just wondering...

[zaius]

Two quad boxes would theoretically outperform 4 duals, because you have more bandwidth between the processors. Of course, if it's not a parallel task in the first place, extra processors won't give you anything.

You're wrong about the price though. Two quad boxes would be _much_ more expensive than four duals, because you can no longer use your traditional PIII/Athlon processors. You have to up it to an Alpha or Xeon. Both those processors and their respective motherboards cost 1.5-3x the price of PIII's, for the equivalent number of processors.

The point of Beowulf clusters is to be able to put bunches of computers that are not very powerful in and of themselves together to make something really powerful, so using quad processor boxes kinda defeats the purpose. The ideal price/performance balance is two processors per box, and this is easy to see by looking at the commercially available solutions.

Re:Just wondering...

[RollingThunder]

Probably depends where most of your collisions take place, but I do know that when you have multiple Pentiums (not sure on Xeon based), that when one CPU is accessing the memory or I/O, the other has to sit and wait. That's one of the big benefits of a multiple Athlon system - both CPU's can access memory simultaneously.

We bought one of the APPRO 1U dual TBirds, and this thing -screams-. It also howls, but that's the four big blower fans. :)

APPRO is at http://www.appro.com, and Anandtech had a writeup on the server.

Linux Labs

[jjo]

Linux Labs will happily sell you clusters, either in standard or custom configurations.

My rough estimate from reading their website is that a 10-node 1.33-GHz Athlon cluster from them would price out at something like $16-17k.

Missing Critical Linux

[Outland Traveller]

I've run into guys from an company called mission-critical linux www.missioncriticallinux.com at the local LUG meetings. I know that they do custom clusters. Perhaps they can help?

What do you run on the darn thing ?

[beanerspace]

I'm glad someone is asking about the cost. But I think I've got the hardware thing worked out.

I've got a bunch of old Pentium I's and II's in a basement of a charity I do some volunteer work for. Can get a hold of a router. I've got all the nerd-boys a bit pumped to beowulf a cluster this winter when it's too cold to paintball.

So my question is, what to run on the blasted thing once you get it up ? Is there anything open source out there worth looking into ? Or am I just going to have to buy an application. If so, which one, how much ?

Re:What do you run on the darn thing ?

[cr0sh]

Most of the apps for Beowulf clusters tend to be scientifically oriented apps. It really comes down to there being only certain kinds of problems that can be broken down into parallel processes that can be executed concurently, then the answers "recombined" at the end to get the result.

IIRC, Mosix allows for using the machine as one large machine, essentially allowing each process (or groups of processes) a single CPU, but funneling the result back to the main controller, so to the user, it looks like one large machine. This is different from a standard parallel processing Beowulf cluster, which behave like classical parallel processing machines.

For that domain (parallel processing), aside from coding your own stuff (hard to due, even if you are a master coder, from what I understand - due to having to understand what classes of problems can be broken down into parallel tasks, and then actually applying that knowledge to real tasks), there is one application that would prove to be "fun" - Ray Tracing.

Fortunately, POVRay has a paralleled version available, for Beowulf clusters. I don't have a link, but I know it exists (heck, you can probably get it at the POVRay site).

One other disclaimer: Everything I have said should be interpreted as "coming out my ass", simply because I have no experience at building or using Beowulf clusters or any other parallel processing architectures. Most of what I know about them have come from /. discussions, various web sites of Beowulf cluster projects, books, and magazine articles (as well as one funky book I picked up at the library detailing programming in Fortran for a Connection Machine - eeepp!). The POVRay stuff is true, though...

Re:What do you run on the darn thing ?

[tolldog]

As much fun as rendering seems over a large network, if you are really doing any major rendering and you don't have Cray-linked SGI's .. you are better off treating it as a NOW (network of workstations).
The POVRay rendering dropped off at 8 procs or so, if i can remember correctly.

We render a hand full of frames per box and do this on several boxes. This keeps the network overhead between machines low.

Sometimes when people talk about beowulf, what they really mean or really want is just a bunch of managed computers running tasks. If the task can be broken down at submission time, like most renders, then take advantage of a NOW.

Re:What do you run on the darn thing ?

[ryanvm]

One other disclaimer: Everything I have said should be interpreted as "coming out my ass"...

Heh - I think with Slashdot, this is usually an assumed competence. But we appreciate you candidness anyway.

Re:What do you run on the darn thing ?

[duffbeer703]

No shit sherlock.

The guy knows what a Beowulf does, he needs somebody to BUILD one for him.

Goddamn karma whore.

Re:What do you run on the darn thing ?

[cr0sh]

Good points.

In a way, from what I understand, this is kinda what Mosix does for you, automatically, and and thus different from a true Beowulfed parallel processing app.

I guess it also depends on if you want to learn parallel processing techniques, or if you are trying to get a job done.

The drop off I would imagine is due to slow interconnects among the nodes - I am sure there would be better gains the higher speed your interconnects were.

Anybody know how the POVRay for Beowulf machines really works? Does it break the viewport into n sections, handing each section to a different node, or does it process multiple rays (ie, a node per ray), or what? Anybody got a link on this?

Re:What do you run on the darn thing ?

[SubtleNuance]

If you just want to tinker, and ball-busting computing power is unnecessary, you really shouldnt be building a Beowulf cluster.
Have a look at the info contained at MOSIX.org they provide kernel patches && apps to build clusters also, but these clusters allow -depending on your configuration (how/who/what of each computer in the cluster) - processes to migrate from 'more idle nodes'.

From what i understand, you could setup your 'head or workstation' machine to let procs dribble off onto other nodes.

have a look - much more usefull to geek exploration than beowulf... even if beowulf gets 'all the /. mentions'

Re:What do you run on the darn thing ?

[beanerspace]

..., then why do you need one?

Because we can !

Aspen Systems

[iso]

There's an ad in the latest Embedded Linux Journal for a company called Aspen Systems (1-800-992-9242) advertising Beowulf clusters. Thtat's about the sum total of all I know about them ;).

- j

Cheaper solution

[NumberSyx]

Better and cheaper idea, do it yourself. Instead of buying 10 dual proccessor systems at $2000+ each, goto Fryes and get 30 Emachines or whatever they are selling for $299, add a couple hundred dollars for 30 decent network cards, one monitor for the control node (barrow a couple more to do the installs) a few hours to install and configure RedHat 7.1, which comes with the clustering software and you'd be done by morning. You will probably get better performance at half to two thirds the cost. This is what clustering is all about, turning cheap off the shelf systems into a super computer.

Re:Cheaper solution

[milkmandan9]

Forget the eMachines.

I've just finished building a (really small) 4-node cluster for some research I'm doing.

Each node is a Duron 750 w/256MB RAM and a 100Base net card, power supply and floppy drive. They're vertically stacked on some 2' threaded rod from McMaster Carr.

But get this: I built the whole thing for $170 per node . That's everything, shipped to my door.

They load the kernel off the floppy and grab the NFS root from a fileserver I've got set up, so no need for hard drives. Granted, there's a bit more network and memory overhead (both /var and /tmp sit in memory), but I can buy more nodes from the money that I've saved on hard drives.

We've bought from two places

[BrentN]

We bought a 168-node Pentium cluster from Atipa, and we're negotiating for a 1024-node (yes, that's right) Athlon cluster from Linux Networx.

RTS+FPS Sim

[Da VinMan]

Not encryption... modeling of the ground actions to be take in Afghanistan or other areas. Probably need to know just how hard it's going to be. Couple the micro-management concerns of a first person shooter with the strategic elements of a real-time strategy with the parallel concerns of mutiple agents in the field and you could probably simulate a proper battlefield in short order.

Hmm...

My $0.02 worth of a guess. :)

Scyld

[4of12]

Haven't dealt with them directly, but I believe they have Don Becker, one of the Beowulf pioneers.

I think they have developed a system to help provide a single system image, along the lines of MOSIX, but not MOSIX, IIRC. This can help managing such a cluster which could otherwise be like managing 10 separate machines - a hassle.

Wow!

[quintessent]

Just imagine...

a Beow...

oh wait. Never mind.

Re:Could you imagine...

[Lxy]

Hey moderators: I know you guys look for these posts just to mark them -1, but occasionally these are on-topic posts. This is one of those times. "Imagine... a beowulf cluster of these" is a staple here on /. and needs to be recognized every once in awhile!

What is turn-key?

[Spiff28]

Honest to goodness curious question that I'm sure the slashdot crowd could answer: what exactly does "turn-key" mean here?

YASOB: Yet Another Supplier Of Beowolves

[nels_tomlinson]

Microway also sells these toys. If you want Alpha, look here. If your budget runs more to Athlons, look here. Unfortunately, you will have to choose between 8 and 16 CPUs on the low end here; they don't have a 10 CPU Athlon cluster.

You should think a bit about whether the extra abilities of the Alpha boxes are worth the extra bucks for your application. One thing which I think that I remember about the Alphas is that they use a crossbar switch to link the several processors on a motherboard to memory, et cetera. This should give better throughput. They also have huge caches which should help with big matrices. I think that if you have lots of little problems which should be run in parallel, more nodes with lower price and capability per node might be the way to go.

I remember back in the days of the XT, Microway used to sell math coprocessor and video boards for PCs which cost more than the box you hooked them to, along with high-grade compilers which would put that hardware to work. They were once the place to get hardware and software for doing seroius number-crunching on a PC.

Re:So they want a cluster in 2 hours to do .... ?

[carlos_benj]

remove your left eye to email me.

Gee, I know he was asking for a quote, but if the price is that steep just for the RFP I'm afraid we'll have to pass on the consulting fees....

(I normally don't respond to .sigs, but given the topic....)

Re:x4? xMore?

[choprboy]

As someone who actually owns a quad Xeon (Intel Sitka 4x400MHz 1M cache) and is building 2 more for a cluster... the answer, as always, is "depends". It all depends on what the intended use is. For a embarrassingly simple parallel processing job (aka running 20 seti@home jobs) the price/performance ratio can be quite poor. Prices are definately down, but the motherboards and RAM are still fairly expensive (typically you need EDO ECC DIMMs with high end server boards, not the cheap SDRAM). You can pick up several 1GHz barebones Athlons for the same price and run the data serially thru each at a faster pace.

On the other hand, if you have a true multi-threaded, highly integrated task that requires high inter-process communication, separate boxes are a poor choice. Something like a large relational database or multi-dimensional vibration calculation wherein each calculation requires knowledge of it's neighbors motions, is far superior on a multi-CPU box. Unless you implement an expensive Dolphin/Myrinet network, the process communication alone, be it over ether, SCSI, or FC, kills a multi-box solution. Not to mention the fact quad Xeon boxes typically take 4-8GB of RAM so everything is always local.

off topic: instability, windows and science

[nels_tomlinson]

Your sig line mentioned that someone should do a paper on windows instability. Here is one of them. It's the third in a series.

One portion which shocked me was:

Our final piece of analysis concerns operating system crashes. Occasionally, during our UNIX study, tests resulted in OS crashes. During this Windows NT study, the operating system remained solid and did not crash as a result of testing. We should note, however, that an early version of the fuzz tool for Windows NT did result in occasional OS crashes. The tool contained a bug that generated mouse events only in the top left corner of the screen. For some reason, these events would occasionally crash Windows NT 4.0, although not in a repeatable fashion.

They crashed a unix os? Wow! That doesn't match up with my limited experience. The only way I've ever done that was by trying to do stupid things as root, like running mindi with a buggy kernel. I wouldn't have thought that this would be a problem for a normal user.

Here is something which didn't surprise me at all:

Our 1995 study found that applications based on open source had better reliability than those of the commercial vendors. Following that study, we noted a subsequent overall improvement in software reliability (by our measure). But, as long as vendors and, more importantly, purchasers value features over reliability, our hope for more reliable applications remains muted.

Mine, too.

From the Scyld website

[evil_one]

Scyld Vendors Scyld Beowulf Professional Product Scyld has partnered with industry leaders for them to provide Enterprise Level systems which consist of pre-integrated, supported hardware systems loaded with the Professional Scyld Beowulf:

• Compaq
• PSSC Labs
• Penguin Computing
• eLinux
• Aspen Systems
• Custom Fit, Inc.
• Atipa

Been There Done That

[elinenbe]

This summer I was employed at a finite element analysis company in Philadelphia, and I designed, budgeted, and built a 10 processor AMD System (Octavian) based on gigabit ethernet, 1.2 Gig MP AMD chips on Tyan mobo's with half a gig of RAM per node, and it did not cost more then $13000 plus some setup time. (This was with a gigabit switch, etc AND the current cost with dropping prices would be less then $10000) The computer was designed to run LS-DYNA (a Livermore software finite analysis program) and it has not let them down.

Here are some benchmarks:

Octavian Benchmark runs
as of 8/17/01

Problem description:
Acetabular cup with a spherical metal ball compressing the liner into the
shell.
The effect of holes used for screw fixation to the bone is included
Total Mesh Statistics
45814 Nodes
37696 8-noded solid elements
2 contact pairs
Dynamic Relaxation Solution:

Execution Statistics
2 Processors : 62 Minutes
6 Processors: 34 minutes
10 Processors: 24 minutes

Analysis of the execution throughput indicicates a linearly increasing speed
(1/wall clock time)

And here is a review of the results:

the cluster is performing BEAUTIFULLY, and we have been crunching problems on it pretty much nonstop since it was brought online. It has really saved our butts, as we could never have met some key project deadlines without the speed. I've included some bechmarking stats below FYI for a contact problem that took only 24 minutes to run on octavian using 10 cpus. The comparable time on our $50K dual CPU octane workstation is 3 hours and 26 minutes, which translates into a speed up of 8.6 times for about 1/5 the cost.

So, What I am trying to say is build the thing yourself. You will know much more about the system, and you will be able to install any software you want to without having to deal with "customer service". Also, it will save you a bundle as a turnkey solution is nearly 3 times the price. (Even if this cluster was built with Myrinet it would still be far less then any of the pre-built solutions) Lastly, design the cluster for what you need. If your problems involves lots of RAM, then spen money there. If CPU is the bottleneck spent money there. If communication is the bottleneck....

Best of Luck,
Eric

Re:Linux Networx

[Snodgrass]

I am a man and will to refer to myself as such.

However, if you inquire at the web site a salesperson will contact you. :)

Sure, I can sell you one in less than an hour, but

[(H)elix1]

but it may take a while before it gets shipped (grin)

rm -rf /bin/ladin

Hello? (Totally OT - Mod me down, I need the fun!)

[cr0sh]

Hello?

I wasn't replying to the article, but to an individual who was asking what he can use a Beowulf cluster for, as he obviously didn't know.

I am the last person needing to whore, since I am capped, and been that way for months...

Re:Cheaper solution - add shared storage

[paled]

does anyone know of an affordable shared storage solution for this?
I know that SCSI can operate on a shared bus for 2 nodes, but I'd really like to test out Oracle 9i Real Application Cluster (RAC) (formally Parallel Server) on a 4 node setup on Linux - but would need a shared storage device for log files, data files, control files.

FC would be a little pricey.
NetApp filers are outta my league.
thanks.

Well..

[mindstrm]

Buy ten machines, and build it yourslef? I mean.. that's all a beowulf is... it was a project to work on using off the shelf hardware for parallel processing.

DO you have some app that needs it? I mean, you can't just run anything....
Did some department just come up with some app already designed with the PVM libraries or something?

Hmmmph..

[Da VinMan]

Yeah, you're most likely right of course. But, my idea is much more fun.

Think about it though, how likely is the military to be in possession of simulation software like I mentioned with full-out graphics capabilities that actually runs on those spiffy supercomputers they own? Maybe very likely. If not, then a Beowulf cluster is the perfect starting point along with existing code perhaps from the likes of Loki.

Hmm....

;+)

/. helps war effort!

[gsfprez]

or was it so obvoius that he's asking this for those reasons?

I'm not complaming - i'm just finally glad to see /. as something USEFUL.

fight on!

+2 Funny on the MQR standard

[MarkusQ]

Who modded this down?!?

It's a joke for gosh sake; he was playing off the first person's Grendel reference by alluding to The Three Billy Goats Gruff!

-- MarkusQ

This is technically off-topic

[biglig2]

But am I the only person here to think "working on my morning M&Ms?"

why not a Mac?

[mr.ska]

I've seen a few stories about researchers looking for a cheap, quick, and powerful cluster for their research purposes who decided to go with Apple hardware. One group got an 8-node iMac cluster up and running in less than a day (which I can't find the link for.. sorry).

A quick search of Apple's site actually mentions clusters [apple.com]. Perhaps 10 dual G4's would suit your needs?