Where to Spend $1M on a Cluster?

Natchswing asks: "My university has been given a $757,825 NSF grant to build, 'A 256 node (128 pair) Beowulf parallel computing cluster ... to improve the realism of gravity-wave modeling by permitting treatment of the three dimensional problem and multiple wave interactions.' They want to pay a company to just show up and drop off a functional cluster rather than build it themselves. Since word has leaked out regarding the purchase intent, every computer manufacturer under the sun (including Apollo himself) has called up trying to sell their cluster. Since I'm no cluster expert, I'm writing Slashdot. If you had $0.7 mil to buy a pre-built cluster who would you go with and why?"

Comments

Competitive Bidding

[duffbeer703]

Have the companies submit bids... then compare them and make a decision.

This isn't rocket science.

Me!

[Jahf]

I'll do it ... send via paypal to my /. username @ yahoo.com ...

Penguin Computing

[retostamm]

Penguin Computing does this kind of stuff for a living. I think they are an all open source shop, too... There may be others, too.

Re:Penguin Computing

[DA-MAN]

Penguin Computing does this kind of stuff for a living. I think they are an all open source shop, too... There may be others, too.

As a Systems Engineer who has worked with a number of vendors, I would say that Penguin is the bottom of the barrel in service and quality control.

We have five clusters at our facility, the slowest of which is on the top500 in the 150 range. We've tried big and small vendors.

Penguin is the absolute worst. No two scsi hard disks had the same firmware version, the raid controller was DOA, etc. We buy/borrow a node from each vendor and evaluate them before buying clusters, and out of all the vendors the Penguin is the one that would crash or hang all the time. After months of trying, they were never able to get this going properly. Regardless of the fact that we shipped it back twice and were told each time that we'd get back a whole new machine (it wasn't).

I would personally recommend Appro, IBM or Western Scientific in that order. Service and quality hardware are their game.

Only on ask slashdot...

[Txiasaeia]

could you say, "Imagine a beowulf cluster of those!" and actually be asking a legitimate question!

You got a grant with NO PLAN?

[Anonymous Coward]

Shit, my department needs to take lessons from you guys. We need to specify the budget down to the last rubber foot and network cable just to have them review the application, and you got a grant without even having an idea what you were going to spend it on?

Re:Do it with an apple

[coldcup]

A cluster of storage? Perhaps you mean the Xserve itself.

They even have a page on clusters.

RFP is the answer

[hectorh]

I think that you should look at your intended application.

- How much disk space are you going to need in total?
- How much disk space are you going to need per node?
- How much RAM is each node going to need?
- Is your application going to benefit from a low-latency or a high-bandwith connection between nodes?
- What about cpu? which cpu family will provide the best bang/$ for your calculations? PPC or X86? x86-64 maybe?

Once you know what you need, put it together in an RFP and send it out to every company that shows up under a google search for "beowulf cluster"

Review the responses and pick the best.

Since you are asking this question here, I'm going to refrain from suggesting the better option which is to build your own.

Hector

Microway

[brsmith4]

I run a 48 Node Microway beowulf and I must say that it is the most stable system available. Everything came assembled and ready to go (of course, I built the enclosure and did the networking, but they will do that for you if you'd like). If you're not very knowledgeable about beowulfs, how do you know you'll need so much power? Do you know how well the software you will be using will scale? Is it close to embarassingly parallel or does it lose efficiency over X nuber of nodes? What type of resources and consumption does the program use? Is it extremely processor hungry, or does it deal with dense matrices and require low-memory latency and high bandwidth or both? Do you know if you will need the power of Myranet or will you be able to get by on GigE?

These are important questions you must ask your researchers and yourself before you purchase this cluster. But, to answer your question, I believe Microway is the best choice and I plan on having them build our next cluster in the next fiscal year.

-brian

Negotiate a success story

[elliotj]

Whoever you chose to go with (I'm partial to Apple, but that's just me - and just because they have sexy hardware), see if you can get them to give you either more for your money, or free implementation/consulting help, or something like that in exchange for using your implementation as a success story. I think Virginia Tech got a bunch of free stuff from Apple when they decided to build their supercomputer.

All these vendors want to be able to talk about their work. Letting them use you for marketing may help you get more for your money.

Not Angstrom

[Anonymous Coward]

I currently maintain some Opteron based Angstrom Microsystems Linux clusters. We've had them for less than a year, and already 30% of our nodes have had to be replaced. Support has been a nightmare.

Sadly, I was not around when the proposal was made, otherwise I would have rejected this cluster outright. There is no way to hook external storage up to this beast. There is no USB, Firewire, SCSI, external SATA, or fibre channel options. You can't even run an ATA cable out of the thing without drilling holes into the blade walls.

Personally? I'm looking at an XServe or an IBM Bladecenter.. but maybe it's just because I'd like some real support.

cluster experience

[Robbat2]

First of all, you really should put out an RFP for your cluster.

We've got a 128 node (1 cpu per node) cluster from Atipa http://www.atipa.com/ that cost CDN$ 0.25M.
128 P4 Xeon, 1GB RAM, 120Gb IDE, Gigabit Ethernet.
I'd expect you to get a lot more for your USD$ .75M, like maybe doubling your size and getting AMD64 nodes. Look at your primary problemset first, see if it's IO-intensive or CPU-intensive to figure out what you want in the way of disk/networking.

The only thing I don't like about it is Atipa's configuration of Redhat8 (they didn't offer anything newer at the time). Look for something newer there.

Atipa is one of the suppliers for SGI-branded clusters as well.

I'd really like a cluster from http://adelielinux.com/en/, but I wasn't aware of them at the time we did our RFP and cluster purchase.

ring..

[ivano]

...Apple
..Dell
...IBM
and *talk* to a sales rep. I know how hard this is (not!) but asking Slashdot is kinda silly. Sure you might want some impartial advice but /. might not be the right place :) Ring these people and decide for yourself (you're a smart man, no?). From the media Apple is getting for its "out-of-the-box" clusters I would seriously put them as an option.

..and good luck ! it sounds like a good project

ciao