How To Build a Homebrew PS3 Cluster Supercomputer

eldavojohn writes "UMass Dartmouth Physics Professor Gaurav Khanna and UMass Dartmouth Principal Investigator Chris Poulin have created a step-by-step guide designed to show you how to build your own supercomputer for about $4,000. They are also hoping that by publishing this guide they will bring about a new kind of software development targeting this architecture & grid (I know a few failed NLP projects of my own that could use some new hardware). If this catches on for research institutions it may increase Sony's sales, but they might not be seeing the corresponding sale of games spike (where they make the most profit)."

Comments

"super" computer:

[Ethanol-fueled]

I get a kick out of these types of articles.

Last year, Khanna's construction of a small supercomputer using eight Sony-donated Playstation 3 gaming consoles made headlines nationwide in the scientific community.

I'm not trying to be a smartass, but why did he mention in TFA that his supercomputer cost $4000 if the 8 consoles were "Sony-donated"? ALso, like the iPod example at the top of the post, most research use of the technology won't come from actual iPods or consoles but from customized versions of the underlying technology such as the Opteron-Cell hybrid Roadrunner supercomputer. If one wanted to build their own home "super" computer then why not just use CUDA and a few Nvidia cards?

Re:"super" computer:

[eltaco]

see afidel's posting.
basically it comes down to the costs of having your own personal power station in the TCO to run a cluster.
this started (well, really hit it off) a few years back, when the pentium M and centrino tech became widespread. basically, to my knowledge, it was the first time you could actually have more processors with less jiggahertz, that consumed less power in total and still had more flops than the others. it swayed everyone from "more powerful cpus plz" train of thought to the "more cpus, less power-consumption". (also cpus/chips in general will eventually hit an upper barrier, making parallel computing a necessity)
I haven't checked the facts on the ps3, but seeing how much nether-region sucking is going on, ps3s probably fit into this scheme.

Re:"super" computer:

[J05H]

Answer: PS3s were used because of the vector processors - they are significantly faster than general purpose CPUs for some of Dr. Khanna's needs and the general vision of the project. These are chips designed for raytracing which makes them perfect for some forms of scientific processing.

Also a rack unit full of PS3s looks way cooler than some crufty old PCs pulled from a dumpster.

Josh - PS3Cluster tester

ibm

[sreid]

why would ibm be involved in this if it means they will sell less servers?

Why PS3s?

[whyloginwhysubscribe]

I don't understand why this isn't possible with normal PC hardware - what is special about the PS3 - or is it just because it is better value for money?

Re:Why use PS3s?

[ookabooka]

If I recall correctly, Sony sells hardware either at-cost or at a slight loss because they make their money on the games. I know this was true for the original xbox as modded xbox clusters were demoed as extremely cost efficient compared to making the computers yourself. I used a moded xbox as an early TiVO as it was way cheaper than making a similar setup myself.

Re:Pretty much useless

[blind biker]

How is it useless, when the guy who built it, used it already for a month? And it has replaced 200 supercomputer nodes, for his purpose? I'd say that's very fucking useful.

But you know what, maybe you should send him an e-mail and try to convince him how his cluster is useless. Make it a nice, insightful and intelligent e-mail, like your post.

Re:Why use PS3s?

[afidel]

The best MIPS/watt for CUDA is probably either the 9600 GSO or the GTX280 depending on whether you're memory or processor constrained. The 9600 can be had for about $75 for 768MB variety (forget the 512/1024 parts they much lower performing) and has 96 stream processors running at up to 650Mhz. The GTX280 costs about $400 and has 240 650Mhz stream processors (though I believe they might be slightly more advanced then the ones on the 9600 I'm not sure how much of that is exposed by CUDA). Power usage is 46W peak for the 9600 and 180W for the GTX280.

Power and maintenance?

[lemaymd]

Researchers pay not only for the initial capital outlay required to install a supercomputer, but also for its power, cooling, the building it resides in, and its maintenance. This PS3 cluster might be cheap from the researchers' standpoint if they don't pay for any of these things directly, but I imagine their departments won't be real thrilled if a bunch of researchers start building their own individual "cheap" supercomputers! Those issues aside, it sounds like they're doing pretty cool stuff with those machines, so maybe more supercomputers should be cell-based!

Re:Power and maintenance?

[Anonymous Coward]

Researchers pay not only for the initial capital outlay required to install a supercomputer, but also for its power, cooling, the building it resides in, and its maintenance.

Someone noted that "AFRL did it already". While doing it, they noted and were able to take advantage of:
1) low power consumption compared to a typical 1U unit (or desktop PC) used in clusters. Very low draw when in idle mode.
2) low heat output, and engineered to be tolerant of warm/hot environments found in typical users' homes.
3) low maintenance, with a consumer system designed for low MTBF, and easily/cheaply swapped out if a unit is a lemon.

The AFRL lead investigator also noted that a PS3 ain't the end all/be all. Given the lack of ram, "his" cluster was set to spawn off specific types of processing that a Cell CPU with middling ram could do a good job on, to get a good FLOP/$ ratio. The AFRL cluster mixes in a number of x86 1U systems as job masters.

diy distributed computing

[doronbc]

I've thought about Folding @Home and I've always wondered why can't there be a diy distributed computing server that could be setup. Something like this PS3 cluster but could be replicated with any home pc.

My setup for PS3Cluster

[J05H]

I helped Chris with the documentation, testing and image capture on this project. I see some "it doesn't do this!" comments above - please remember this is a young project that started out of one researcher's need to solve a specific type of problem. If you want to see this advance, it's all open source so start hacking.

So my setup:
1 40Gb Playstation3 w/ HDMI cable out and keyboard
Hauppauge HDPVR digitizer
PC running Windoze and Photoshop
TV hanging off the HDPVR for reference

Software as described on PS3Cluster.org including Geoff's Cell libraries, boot image on USB and Fedora 8 for PPC.

Plugged everything together, installed Fedora 6 the first time around since we knew that worked, then redid it with Fedora 8. Added the MPI libraries and ran the little Pi test code. Digitized the whole install as video, proofed out the process in terms of instructions. Did frame grabs from the video, cropped etc in Photoshop. Lots of work, totally worth it seeing the project posted here.

Oh, and it runs X - kinda cool having Firefox running on a game deck.

Enjoy,
Josh

Re:Limited use

[MikeBabcock]

I tell you what: you go ahead and buy $4000 of those Dual core kits, and we'll compare your output from a well-written algorithm versus the Cell system designed by this team.

Some interesting code examples for using the Cell have been demonstrated and it has immense processing power that most people don't recognize immediately. Check out this Dr Dobb's Journal article for an example.

Re:Why doesn't Sony make it ?

[Sarusa]

Well the funny thing is they're still losing money on every PS3 sold. At least from the last cost analysis I saw, which was back in July. They are counting on you watching Blu-Ray disks or buying games, and any PS3 in a computing scenario won't be doing any of that (barring someone 'misallocating resources' *koff koff*).

So your idea makes sense. They are partnered with Toshiba to produce the low(er) cost Cell add-ons outside of a PS3. And you should see http://en.wikipedia.org/wiki/Cell_%28microprocessor%29#Possible_applications for some others.

The sad thing is that a Cell is far more suited for supercomputing than it is for writing games on, but Sony seems insistent that the primary purpose is making game devs's lives miserable while IBM and Toshiba seem more focused on using it where it's actually useful.

This is probably a silly question

[Chrisq]

This is probably a silly question but why/how are they running PPC Linux (which is presumably for the Power PC) on PS/3s which have cell processors?

I guess that either the PS3 has a PPC chip as well, or it runs some sort of emulation mode. I can't find either documented.