How To Build a Homebrew PS3 Cluster Supercomputer

eldavojohn writes "UMass Dartmouth Physics Professor Gaurav Khanna and UMass Dartmouth Principal Investigator Chris Poulin have created a step-by-step guide designed to show you how to build your own supercomputer for about $4,000. They are also hoping that by publishing this guide they will bring about a new kind of software development targeting this architecture & grid (I know a few failed NLP projects of my own that could use some new hardware). If this catches on for research institutions it may increase Sony's sales, but they might not be seeing the corresponding sale of games spike (where they make the most profit)."

Comments

Oblig. OMG...

[Anonymous Coward]

something to finally run Vista?

ibm

[sreid]

why would ibm be involved in this if it means they will sell less servers?

Re:ibm

[oodaloop]

FEWER servers! FEWER! Aauughhhh!

Re:ibm

[Anonymous Coward]

This isn't really the place to start criticizing grammar and spelling, unless you REALLY want to live a life full of frustration and torment....?

(Though it could be worse, I suppose - you could go to digg etc. intsead...:p)

Re:ibm

[fuzzyfuzzyfungus]

IBM is probably banking on the existence of people who want Cell processors in systems with more than 256megs of RAM. Other IBM value-adds would presumably include rack mountability, support for netbooting and other convenient management stuff, and so forth.

If your application leans almost entirely on the CPU with very little need for RAM, and you have an army of screwdriver monkeys(or grad students) to do all the legwork, the PS3 is an excellent deal. If you need something with RAM capacity that wasn't a joke in 2001, and/or management features that won't have you tearing your eyes out when you have 10,000 of them, then IBM smells opportunity.

Re:ibm

[dbIII]

If you wait out the two weeks of used car salesman tactics from a place like Mercury that can sell you cell processors in systems with more than 256megs of RAM you'll find out that unless you have an endless budget you are probably better off with the ten quad core Xeon systems you could get for the same price.

That is why a system made of game consoles makes a lot more sense than very similar hardware in a rackmount case. Other cell hardware has been priced into complete irrelevance by salesfolk having too much control over the process.

On the other hand there is the nvidia CUDA solutions, hardware doing things a slightly different way but proudly printing their prices on the net instead of two weeks of mindless chatty emails before you get the price.

Why use PS3s?

[JanusFury]

Why would you want to use PS3s for a homebrew supercomputing cluster if it means you have to write and optimize code for the SPEs to get benefit out of it? The PS3's linux environment doesn't let you utilize the GPU or all of the built-in SPEs and it doesn't have a lot of RAM available either. It seems like it would be cheaper to build a cluster out of commodity PC parts, and maybe use GPUs+CUDA to get more muscle without having to completely hand-roll your own accelerated computation code (since CUDA is roughly C). I can't imagine that the PS3 would end up cheaper for these purposes, considering it includes a Blu-Ray player along with a bunch of other things you're not going to be using.

Re:Why use PS3s?

[drinkypoo]

Why would you want to use PS3s for a homebrew supercomputing cluster if it means you have to write and optimize code for the SPEs to get benefit out of it? The PS3's linux environment doesn't let you utilize the GPU or all of the built-in SPEs and it doesn't have a lot of RAM available either.

Well, I'll bite; if the cell is the fastest processor for your workload, the PS3 is the cheapest way to get one, even at only six usable SPEs and no GPU. Doesn't the PS3 have GigE? That's plenty fast enough to shovel data in and out of the system.

Re:Why use PS3s?

[ookabooka]

If I recall correctly, Sony sells hardware either at-cost or at a slight loss because they make their money on the games. I know this was true for the original xbox as modded xbox clusters were demoed as extremely cost efficient compared to making the computers yourself. I used a moded xbox as an early TiVO as it was way cheaper than making a similar setup myself.

Re:Why use PS3s?

[ASBands]

since CUDA is roughly C

Not quite. CUDA looks a lot like C in that it has C-family syntax but the biggest limitation it has is that there is no application stack - which means no recursion. CUDA also lacks the idea of a pointer, although you can bypass this by doing number to address translation (as in, the number 78 means look up tex2D(tex, 0.7, 0.8)). The GPU also has other shortcomings, in that most architectures like to have all their shaders running the same instruction at the same time. For this code

if (pixel.r < pixel.g){
//do stuff A
}else if (pixel.g < pixel.b){
//do stuff B
}else{
//do stuff C
}

The GPU will slow down a ton if the pixel color causes different pixels to branch in different directions. Basically, the three sets of shaders following different branches of that code will be inactive 2/3 of the time.

In the Cell, you really do just program in C with a number of extensions added onto it like the SPE SIMD intrinsics and the DMA transfer commands (check it out). The Cell really is 9 (10 logical) processors all working together in a single chip (except in PS3, where there are only 7 working SPEs). Furthermore, your 8 SPEs can be running completely different programs -- they're just little processors. Granted, you have to be smart when you program them to deal with race conditions and all the other crap you have to deal with for multithreaded programming. The Cell takes about 14 times longer to calculate a double precision floating point than a single (and there aren't SPE commands to do four at once like you can with singles).

So which is more powerful? It really depends what you're doing. If your task is ridiculously parallellizable and doesn't require the use of recursion, pointers or multiple branches, the GPU is most likely your best bet. If your program falls into any of those categories, use a Cell.

Re:Why use PS3s?

[afidel]

The best MIPS/watt for CUDA is probably either the 9600 GSO or the GTX280 depending on whether you're memory or processor constrained. The 9600 can be had for about $75 for 768MB variety (forget the 512/1024 parts they much lower performing) and has 96 stream processors running at up to 650Mhz. The GTX280 costs about $400 and has 240 650Mhz stream processors (though I believe they might be slightly more advanced then the ones on the 9600 I'm not sure how much of that is exposed by CUDA). Power usage is 46W peak for the 9600 and 180W for the GTX280.

Why PS3s?

[whyloginwhysubscribe]

I don't understand why this isn't possible with normal PC hardware - what is special about the PS3 - or is it just because it is better value for money?

Re:Why PS3s?

[Darkness404]

A) Although the cell is a pain to code for, it is much better than whatever PC you can get for ~$400 which will probably contain a mid-to-low-range dual core x86 CPU, whereas the PS3 gives you a Cell CPU which is much, much, faster than the x86 CPU.

B) PS3s are uniform. Other than HD differences, a PS3 built in 2008 will be the same PS3 built in 2012 (assuming the PS3 lasts that long) this allows for a uniform cluster without worrying about differing parts (for example, the Core i7 built in 2008 will not be the same as the Core i7 built in 2012 and getting a 2008 Core i7 is going to be a pain)

C) PS3s are the new fad. It isn't going to be hard to set up a supercomputer cluster with PS3s compared to using a mismatch of older computers because again, the PS3 is uniform.

Re:Subsidized Supercomputers

[makapuf]

AGAIN, revenue of console sales is not N*const (positive or negative), but const1+N*const2 where const2 is negative (it's a gain per console) but upfront costs=const1(R&D, licences ...) are big. So the fact that the total is negative implies const is negative, but in fact it's mostly that N*const2 is still less than const1. (I hope this makes sense to some at least)

Limited use

[idiot900]

Couple issues with this as an alternative to the garden-variety x86 cluster connected with InfiniBand:

Slow network interconnect. For problems that are not trivially parallel, network latency is usually a big deal. Ethernet doesn't cut it.
Lack of RAM. 'Nuff said.
Have to care about Cell and PS3 architecture. The codes ("codes" has a slightly different meaning in the context of supercomputing) have to be modified to take advantage of this very specific architecture. Software always outlives hardware, so in the long run the effort may not be worth it.

That said, it's really cheap. If your application isn't held back too much by these issues then enjoy your insanely cheap cluster!

Re:Limited use

[raftpeople]

This is why:
8 PS3's = 8 cells
8 cells X 7 available SPE's per cell = 56 SPE's
56 SPE's X 4 simultaneous FP calcs = 224 FP calcs per cycle

You would need to get quite a few of those x86 dual core kits to match that performance

Re:Limited use

[MikeBabcock]

I tell you what: you go ahead and buy $4000 of those Dual core kits, and we'll compare your output from a well-written algorithm versus the Cell system designed by this team.

Some interesting code examples for using the Cell have been demonstrated and it has immense processing power that most people don't recognize immediately. Check out this Dr Dobb's Journal article for an example.

Re:Message from Government Man

[johanatan]

Or maybe he did catch it and thinks that sort of thing belongs on Reddit?

Imagine a beowulf cluster..

[Anonymous Coward]

Imagine a beowulf cluster of those!

Re:"super" computer:

[steveha]

I'm not trying to be a smartass, but why did he mention in TFA that his supercomputer cost $4000 if the 8 consoles were "Sony-donated"?

Oh come on, you are being pedantic. Clearly what he meant was "$4000 worth of consoles", never mind that they were donated. $X worth of consoles is a useful number if someone is considering buying PS3s and setting up a supercomputer; it's also a fun number to compare to the cost of renting time on some large supercomputer.

The original Wired article is informative:

http://www.wired.com/techbiz/it/news/2007/10/ps3_supercomputer

He asked for Sony to donate the PS3s because he didn't think the NSF would give him grant money to buy video game systems. Now that he has actually built the supercomputer and it does everything he hoped it would do, perhaps other researchers will be able to justify the money to set up their own clusters (without donations from Sony).

The numbers are a no-brainer: he used to spend $5000 to do a single simulation run using rented supercomputer time. For less than the cost of a single simulation run, you can set up your own supercomputer and make simulation runs whenever you feel like it.

ALso, like the iPod example at the top of the post, most research use of the technology won't come from actual iPods or consoles

Um, he is using actual PS3 consoles to do actual research.

If one wanted to build their own home "super" computer then why not just use CUDA and a few Nvidia cards?

If you think that is a good way to make a super computer, why don't you go ahead and do it, and make a web site explaining how it is done?

Meanwhile, he thought he had a good way to go with the PS3, and it did in fact work as he expected, so what's the problem?

Anyway, here's why he thought it was a good idea. From the above linked Wired article:

According to Rimon, the Cell processor was designed as a parallel processing device, so he's not all that surprised the research community has embraced it. "It has a general purpose processor, as well as eight additional processing cores, each of which has two processing pipelines and can process multiple numbers, all at the same time," Rimon says.

Khanna says that his gravity grid has been up and running for a little over a month now and that, crudely speaking, his eight consoles are equal to about 200 of the supercomputing nodes he used to rely on.

steveha

Re:Oblig. OMG...

[windsurfer619]

At first I laughed... But then I realized that, no, Vista won't be able to run on this.

Vista doesn't support the PowerPC architecture.

Re:"super" computer:

[afidel]

Old CPU's have a much lower MIPS/Watt and a lower MIPS/interconnect so they have a higher cost. Many organizations have found it's cheaper to retire an old supercomputer and add a few nodes to the new one even if it is more capital outlay to get the same performance. Basically a Cell does many times as much useful work than a P4 at a fraction of the power budget.

Re:Message from Government Man

[Anonymous Coward]

Luke: Your overconfidence is your weakness.

Emperor: Your faith in your moderators is yours!

Re:Subsidized Supercomputers

[timeOday]

The number of PS3's sold will never be enough to hurt Sony's bottom line, but will boost the image of the console. Having credible scientists call your product a "supercomputer" is worth something. Does Ferrari's Forumla 1 racing team pay for itself? Nah, it's an investment to promote an image.

Invader Zim is non-free

[tepples]

It's not my fault you didn't catch the Invader Zim reference.

Invader Zim is non-free. It's easier to catch pop culture references if they are pre-1923 or otherwise free.

Re:Pretty much useless

[blind biker]

How is it useless, when the guy who built it, used it already for a month? And it has replaced 200 supercomputer nodes, for his purpose? I'd say that's very fucking useful.

But you know what, maybe you should send him an e-mail and try to convince him how his cluster is useless. Make it a nice, insightful and intelligent e-mail, like your post.

Re:Invader Zim is non-free 23-skidoo!

[hguorbray]

sorry, but that's stupid -how many pop culture references from 1923 are relevant to TODAY's pop culture:

seeya snookums, me and the squeeze are the bees knees in our raccoon coats, we're gonna get jazzed up in our hupmobile on hootch and go check out Mary Astor's horse after we hit the blind pig.

I agree its unfortunate that this stuff is non free, but pre 1923 means that most talkies would be out of bounds as well -including stuff you can see on tv all the time.

I'm just sayin'

Power and maintenance?

[lemaymd]

Researchers pay not only for the initial capital outlay required to install a supercomputer, but also for its power, cooling, the building it resides in, and its maintenance. This PS3 cluster might be cheap from the researchers' standpoint if they don't pay for any of these things directly, but I imagine their departments won't be real thrilled if a bunch of researchers start building their own individual "cheap" supercomputers! Those issues aside, it sounds like they're doing pretty cool stuff with those machines, so maybe more supercomputers should be cell-based!

Re:Invader Zim is non-free 23-skidoo!

[mollymoo]

sorry, but that's stupid -how many pop culture references from 1923 are relevant to TODAY's pop culture:

A perfect illustration of the fact that copyright terms are way too long.

I wish I had one

[uassholes]

For the dick licks that say it's useless, I guess you missed all the previous articles about scientists who have been doing the same thing:

http://www.physorg.com/news92674403.html

http://dgl.com/itinfo/2003/it030528.html

http://www.lbl.gov/Science-Articles/Archive/sabl/2006/Jul/06.html

http://folding.stanford.edu/English/FAQ-PS3

What a ripoff!

[Anonymous Coward]

What a complete farce! Here I was all excited to go see this PS3 cluster "guide". From TFA:

"Found at www.ps3cluster.org, the resource fully illustrates how to create a fully functioning and high performance supercomputer with the Sony Playstation 3."

And what is actually *on* the site?? How to install Linux on a PS3 (as if there weren't any guides for that out there already). Then, they show the magical touch where they download the stock Fedora Open MPI implementation, and configure it using all *TWO THREADS* of the Power PC unit.

No mention that Open MPI doesn't even utilize the synergistic processors on the Cell. No benchmarks. Nada. They can boot Linux, and run a networked application that has absolutely NOTHING to do with the CELL architecture itself.

From the site: "One of the authors (Khanna) estimates that his MPI computations run much faster than on desktop workstation chipsets, and that his original 8 PS3 (i.e. 64 core) Cell cluster had comparable if not better performance to a 200 Node IBM Blue Gene system."

B.S. (And I am being generous.) Their MPI isn't using any 64 processors (when there are actually only 56 available cores for use on the PS3). His data sets may run about as fast as they would on 8 older Apple laptops, but there is no way they're anywhere near a Blue Gene. My tax dollars had better not have been used to fund this "research"....

Re:What a ripoff!

[enslaved_robot_boy]

If you clicked on some of the links you would find some quantitative data hotshot.

Re:"super" computer:

[eltaco]

see afidel's posting.
basically it comes down to the costs of having your own personal power station in the TCO to run a cluster.
this started (well, really hit it off) a few years back, when the pentium M and centrino tech became widespread. basically, to my knowledge, it was the first time you could actually have more processors with less jiggahertz, that consumed less power in total and still had more flops than the others. it swayed everyone from "more powerful cpus plz" train of thought to the "more cpus, less power-consumption". (also cpus/chips in general will eventually hit an upper barrier, making parallel computing a necessity)
I haven't checked the facts on the ps3, but seeing how much nether-region sucking is going on, ps3s probably fit into this scheme.

Re:Why doesn't Sony make it ?

[Malekin]

I betcha there's enough talent out there to make a small desktop unit. An affordable, out of box, mass produced Sony supercomputer./p>

I think Sony feels that's exactly what the PS3 is.

Re:"super" computer:

[lysergic.acid]

it's not that simple. sure you can make up for a lack of per-CPU processing power through cluster computing, but at some point it becomes more practical or even cheaper to go with a smaller cluster using a better processor architecture.

you could use hundreds of P3s or even P4s and still not achieve the same real-world performance as a couple dozen cell processors or modern GPGPU stream processors. that's because P3s & P4s are general-purpose CPUs designed for SISD/scalar processing. they're great for the bulk of general-purpose commodity computing applications like running an OS, web browser, word processor, etc., but high-performance computing problems typically involve processing very large data sets that greatly benefit from data parallelism. so if you had two processors, one scalar and one vector, each with the same power consumption and clock rate, the vector processor would be an order of magnitude faster at performing HPC tasks than the processor with the scalar architecture.

and the combined use of parallelization at multiple levels will always be more efficient than relying solely on a single form of parallelism. blindly adding more cheap 32-bit scalar CPUs won't get you as good of results as building a smaller cluster comprised of 64-bit fully-pipelined stream processors with multithreaded superscalar cores that support VLIW. in the former case, you're only employing task-level parallelism, whereas in the later case you're taking advantage of bit-level, instruction-level (pipelining + superscalar + VLIW), data, and task-level (multiprocessing + multithreading) parallelism. you'd not only save power by using fewer (more power-efficient) processors, but you'd also reduce memory coherence & bandwidth problems, not to mention the space savings.

Re:"super" computer:

[J05H]

Answer: PS3s were used because of the vector processors - they are significantly faster than general purpose CPUs for some of Dr. Khanna's needs and the general vision of the project. These are chips designed for raytracing which makes them perfect for some forms of scientific processing.

Also a rack unit full of PS3s looks way cooler than some crufty old PCs pulled from a dumpster.

Josh - PS3Cluster tester

Games

[daybot]

If this catches on for research institutions it may increase Sony's sales, but they might not be seeing the corresponding sale of games spike

Come on - that's the whole point. This is what you'll need to run the PS3 version of Crysis!

Re:Subsidized Supercomputers

[CronoCloud]

Meaning the 80GB systems are NOT 100% backwards-compatible.

Not 100% but very still very high. Out of my collection of 64 PS2 games, only 2 have enough problems that I consider them unplayable on the PS3: Tekken Tag Tournament (doesn't run full speed) and Fallout Brotherhood of Steel (with very pronounced texture glitching)

But now, the PS3 ships without hardware or software emulation to play PS2/PS1 games.

Sigh, why do people keep getting this wrong. Although the latest release PS3 consoles can't play PS2 games, ALL PS3's can play PS1 games since that's entirely software emulation.

diy distributed computing

[doronbc]

I've thought about Folding @Home and I've always wondered why can't there be a diy distributed computing server that could be setup. Something like this PS3 cluster but could be replicated with any home pc.

My setup for PS3Cluster

[J05H]

I helped Chris with the documentation, testing and image capture on this project. I see some "it doesn't do this!" comments above - please remember this is a young project that started out of one researcher's need to solve a specific type of problem. If you want to see this advance, it's all open source so start hacking.

So my setup:
1 40Gb Playstation3 w/ HDMI cable out and keyboard
Hauppauge HDPVR digitizer
PC running Windoze and Photoshop
TV hanging off the HDPVR for reference

Software as described on PS3Cluster.org including Geoff's Cell libraries, boot image on USB and Fedora 8 for PPC.

Plugged everything together, installed Fedora 6 the first time around since we knew that worked, then redid it with Fedora 8. Added the MPI libraries and ran the little Pi test code. Digitized the whole install as video, proofed out the process in terms of instructions. Did frame grabs from the video, cropped etc in Photoshop. Lots of work, totally worth it seeing the project posted here.

Oh, and it runs X - kinda cool having Firefox running on a game deck.

Enjoy,
Josh

Re:Why doesn't Sony make it ?

[Sarusa]

Well the funny thing is they're still losing money on every PS3 sold. At least from the last cost analysis I saw, which was back in July. They are counting on you watching Blu-Ray disks or buying games, and any PS3 in a computing scenario won't be doing any of that (barring someone 'misallocating resources' *koff koff*).

So your idea makes sense. They are partnered with Toshiba to produce the low(er) cost Cell add-ons outside of a PS3. And you should see http://en.wikipedia.org/wiki/Cell_%28microprocessor%29#Possible_applications for some others.

The sad thing is that a Cell is far more suited for supercomputing than it is for writing games on, but Sony seems insistent that the primary purpose is making game devs's lives miserable while IBM and Toshiba seem more focused on using it where it's actually useful.

Re:"super" computer:

[Chrisq]

or bank reclaimed assets from a sunken business?

What type of processor do Woolworth's POS tills use?

This is probably a silly question

[Chrisq]

This is probably a silly question but why/how are they running PPC Linux (which is presumably for the Power PC) on PS/3s which have cell processors?

I guess that either the PS3 has a PPC chip as well, or it runs some sort of emulation mode. I can't find either documented.