Ask Slashdot: Clusters On the Cheap?

First time accepted submitter serviscope_minor writes "A friend of mine has recently started a research group. As usual with these things, she is on a shoestring budget and has computational demands. The computational task is very parallel (but implementing it on GPUs is an open research problem and not the topic of research), and very CPU bound. Can slashdotters advise on a practical way of getting really high bang for buck? The budget is about £4000 (excluding VAT/sales tax), though it is likely that the system will be expanded later. The computers will probably end up running a boring Linux distro and Sun GridEngine to manage batch processing (with home directories shared over NFS)."

Comments

Re:Uhm AWS EC2 Cluster Compute

[jpedlow]

AWS EC2 was my response aswell. :)

for raw horsepower on the short - medium term, use AWS http://aws.amazon.com/ec2/

ec2 should do well for this, imho :)

trade-off

[TheSHAD0W]

Actually, that's a good question... Assuming no time constraints, at what point does it make sense to buy hardware rather than use the cloud? Take that budget above (roughly US$6K) and the best hardware you can get for that price: How many months would you need to run it, flat out, to equal the number of floating-point ops EC2 would give you for that cost?

Re:trade-off

[subreality]

Sometimes, never. Don't forget to add up power, cooling, sysadmin time... And that's before getting to intangibles like being able to spin up 400 cores for an hour and getting your result same-day instead of only owning 40 cores and having to wait until tomorrow.

Cloud computing really cleans up for batch computing jobs like this.

Has she investigated existing clusters?

[Goonie]

Many universities/consortia have supercomputers available on which researchers can apply for (or buy) time. For example, my university is a member of VPAC, which has a big-arse cluster shared between a number of institutions. She might get much better bang for buck if she uses the money for that, rather than splashing out for dedicated hardware.

buy one Opteron 6100-based box

[Chalex]

You can get a SuperMicro reseller to sell you one workstation with 4 sockets of CPUs and a bunch of RAM. UK£ 4000 = 6 299.2 U.S. dollars

That buys you a box with 4 x Opteron 6134 (32 cores) and 128GB RAM (32 x 4GB sticks). And some hard disks.

Re:buy one Opteron 6100-based box

[toruonu]

Yes, my recommendation would be also, we do loads of LHC data analysis and simulations and have found that for real science real cores outweigh hyberthreaded ones so we run Opteron 6172 x2 in supermicro chassis that fits 4 servers into 2RU. The cost of such a box of course is ca 11keur, but it gives 96 cores 192GB ram. Now she can get for half the money that she has about half of that so 48 cores 96GB ram should be doable using SM boxes and you can scale up/down with CPU frequency to adjust the cost and maybe adjust total RAM alongside to fit in the budget. If she plans to later expand she may actually want to spend the money to get the 2U chassis with only 2 of the 4 machines present and later add one/two more by just buying the board with cpu/ram.

Re:Uhm AWS EC2 Cluster Compute

[subreality]

+1. It is very nice to be able to spin up 50 instances, run the hell out of your job, then delete them. It gets done faster, and you don't have to deal with maintenance, upgrades, and obsolescence. Realized you need more RAM? Just adjust it! And so on. It'll likely come out cheaper than owning your own after you add up all the hidden costs (power, cooling, space, time, etc).

The only downside is there are no GPUs. But that's not really a downside: if you do end up developing a GPU version, your cluster configuration would completely change (1x2 cores per box, 3-4U boxes with many PCI-E slots, instead of 2x8 cores or however many you can economically cram into a 1-2U pizza box), so the investment you'd make now would be completely wrong for that future development. With cloud servers you minimize sunk costs.

I use Rackspace Cloud and it performs as promised. It's definitely worth a look.

Amazon AWS.

[Haven]

$1.60 / hour for the largest non-GPU cluster instance. This also provides you with rather fast interconnects and scalability with multiple instances.

Only £4,000 in hardware would be a waste of money. You wouldn't have all that much computing power, and it would be obsolete immediately.

BOINC Project?

[bradley13]

She could also consider creating a BOINC project. She could then do some publicity locally and on forums, to get people to choose her project. I've never tried creating a BOINC project, so I don't know how hard this is. However, I do run the client as a background task, and I imagine many other people do as well.

Re:EC2 is expensive

[TheRaven64]

The problem is the constraints. The cheap cluster in my old department cost £100k. £4k does not buy you a lot of hardware. You will probably find a lot more lying around in the undergrad labs. For some of my work as a PhD student, that's exactly what I used - each lab had 40 machines on a GigE network and closed overnight, and for work that wasn't that latency sensitive, I could distribute it across the machines there and run it at night without anyone minding.

If you're serious about needing a cluster, then you need to spend a lot more than £4K. If you only need a cluster for a short time, then £4K can buy you a chunk of time on someone else's hardware. Since this is the UK, they should contact the Manchester Supercomputing Centre, which provides this kind of service to UK universities at quite a reasonable price (and will also lend you people who are good at optimising code for their systems). If the university doesn't already have some clusters lying around, then you should get in contact with a few other research groups. £4K won't go very far, but if half a dozen research groups each put in £4K then that gives you enough for a reasonable cluster to share between the various users.