How To Build a Homebrew PS3 Cluster Supercomputer

eldavojohn writes "UMass Dartmouth Physics Professor Gaurav Khanna and UMass Dartmouth Principal Investigator Chris Poulin have created a step-by-step guide designed to show you how to build your own supercomputer for about $4,000. They are also hoping that by publishing this guide they will bring about a new kind of software development targeting this architecture & grid (I know a few failed NLP projects of my own that could use some new hardware). If this catches on for research institutions it may increase Sony's sales, but they might not be seeing the corresponding sale of games spike (where they make the most profit)."

Comments

"super" computer:

[Ethanol-fueled]

I get a kick out of these types of articles.

Last year, Khanna's construction of a small supercomputer using eight Sony-donated Playstation 3 gaming consoles made headlines nationwide in the scientific community.

I'm not trying to be a smartass, but why did he mention in TFA that his supercomputer cost $4000 if the 8 consoles were "Sony-donated"? ALso, like the iPod example at the top of the post, most research use of the technology won't come from actual iPods or consoles but from customized versions of the underlying technology such as the Opteron-Cell hybrid Roadrunner supercomputer. If one wanted to build their own home "super" computer then why not just use CUDA and a few Nvidia cards?

Re:"super" computer:

[CannonballHead]

The cost for his initial Playstation grid was $4,000.

It does say that the cost was that for his grid, so that's badly written if they were donated; however, it doesn't actually say that HE paid it :)

Re:"super" computer:

[SIR_Taco]

If one wanted to build their own home "super" computer then why not just use CUDA and a few Nvidia cards?

Not only that, but why wouldn't you just buy a shit-load of old computers from an auction or bank reclaimed assets from a sunken business? $4000 could buy a huge number of old Pentium 3s or even 4s if you know where to look.
One of the main points of a cluster is the fact that you can make up for brute per-processor speed by just having more hardware.

Re:"super" computer:

[steveha]

I'm not trying to be a smartass, but why did he mention in TFA that his supercomputer cost $4000 if the 8 consoles were "Sony-donated"?

Oh come on, you are being pedantic. Clearly what he meant was "$4000 worth of consoles", never mind that they were donated. $X worth of consoles is a useful number if someone is considering buying PS3s and setting up a supercomputer; it's also a fun number to compare to the cost of renting time on some large supercomputer.

The original Wired article is informative:

http://www.wired.com/techbiz/it/news/2007/10/ps3_supercomputer

He asked for Sony to donate the PS3s because he didn't think the NSF would give him grant money to buy video game systems. Now that he has actually built the supercomputer and it does everything he hoped it would do, perhaps other researchers will be able to justify the money to set up their own clusters (without donations from Sony).

The numbers are a no-brainer: he used to spend $5000 to do a single simulation run using rented supercomputer time. For less than the cost of a single simulation run, you can set up your own supercomputer and make simulation runs whenever you feel like it.

ALso, like the iPod example at the top of the post, most research use of the technology won't come from actual iPods or consoles

Um, he is using actual PS3 consoles to do actual research.

If one wanted to build their own home "super" computer then why not just use CUDA and a few Nvidia cards?

If you think that is a good way to make a super computer, why don't you go ahead and do it, and make a web site explaining how it is done?

Meanwhile, he thought he had a good way to go with the PS3, and it did in fact work as he expected, so what's the problem?

Anyway, here's why he thought it was a good idea. From the above linked Wired article:

According to Rimon, the Cell processor was designed as a parallel processing device, so he's not all that surprised the research community has embraced it. "It has a general purpose processor, as well as eight additional processing cores, each of which has two processing pipelines and can process multiple numbers, all at the same time," Rimon says.

Khanna says that his gravity grid has been up and running for a little over a month now and that, crudely speaking, his eight consoles are equal to about 200 of the supercomputing nodes he used to rely on.

steveha

Re:"super" computer:

[afidel]

Old CPU's have a much lower MIPS/Watt and a lower MIPS/interconnect so they have a higher cost. Many organizations have found it's cheaper to retire an old supercomputer and add a few nodes to the new one even if it is more capital outlay to get the same performance. Basically a Cell does many times as much useful work than a P4 at a fraction of the power budget.

Re:"super" computer:

[TheKidWho]

Seriously, you're better off getting a few NVIDIA Tesla cards and using CUDA then trying to get code running on the PS3...
Nevermind the fact that you can't use the GPU at all in the PS3 Linux Environment.
$4000 gets you 2 Tesla cards + 1 GPU which should push about ~2 Teraflops for optimized code.

Re:"super" computer:

[ozphx]

So by "sony-donated" its "sony-still-pushing-teh-cell-is-teh-awesomes" bandwagon?

The "real" cell chip is used in "real" supercomputers - not the stripped down one with only 7 SPUs in the PS3. We're talking like 32 of the babys.

Thats how you make an efficient supercomputer - not by wiring together a bunch of toys (even if they do run *nix).

Re:"super" computer:

[Anonymous Coward]

You are so right, he should have stopped everything and just started using your method. Don't worry about the time he spent setting it up and writing about it. And if something better comes along while delevoping that he should just stop and listen to all the people saying PS3s are not good.
DAMN they should just fire him and get a bunch of slashdotters to do it better. ;)
I'll be impressed when all the complainers build their supercomputers.

Re:"super" computer:

[eltaco]

see afidel's posting.
basically it comes down to the costs of having your own personal power station in the TCO to run a cluster.
this started (well, really hit it off) a few years back, when the pentium M and centrino tech became widespread. basically, to my knowledge, it was the first time you could actually have more processors with less jiggahertz, that consumed less power in total and still had more flops than the others. it swayed everyone from "more powerful cpus plz" train of thought to the "more cpus, less power-consumption". (also cpus/chips in general will eventually hit an upper barrier, making parallel computing a necessity)
I haven't checked the facts on the ps3, but seeing how much nether-region sucking is going on, ps3s probably fit into this scheme.

Re:"super" computer:

[Anonymous Coward]

You sound like a real fun guy.

Re:"super" computer:

[lysergic.acid]

it's not that simple. sure you can make up for a lack of per-CPU processing power through cluster computing, but at some point it becomes more practical or even cheaper to go with a smaller cluster using a better processor architecture.

you could use hundreds of P3s or even P4s and still not achieve the same real-world performance as a couple dozen cell processors or modern GPGPU stream processors. that's because P3s & P4s are general-purpose CPUs designed for SISD/scalar processing. they're great for the bulk of general-purpose commodity computing applications like running an OS, web browser, word processor, etc., but high-performance computing problems typically involve processing very large data sets that greatly benefit from data parallelism. so if you had two processors, one scalar and one vector, each with the same power consumption and clock rate, the vector processor would be an order of magnitude faster at performing HPC tasks than the processor with the scalar architecture.

and the combined use of parallelization at multiple levels will always be more efficient than relying solely on a single form of parallelism. blindly adding more cheap 32-bit scalar CPUs won't get you as good of results as building a smaller cluster comprised of 64-bit fully-pipelined stream processors with multithreaded superscalar cores that support VLIW. in the former case, you're only employing task-level parallelism, whereas in the later case you're taking advantage of bit-level, instruction-level (pipelining + superscalar + VLIW), data, and task-level (multiprocessing + multithreading) parallelism. you'd not only save power by using fewer (more power-efficient) processors, but you'd also reduce memory coherence & bandwidth problems, not to mention the space savings.

Re:"super" computer:

[J05H]

Answer: PS3s were used because of the vector processors - they are significantly faster than general purpose CPUs for some of Dr. Khanna's needs and the general vision of the project. These are chips designed for raytracing which makes them perfect for some forms of scientific processing.

Also a rack unit full of PS3s looks way cooler than some crufty old PCs pulled from a dumpster.

Josh - PS3Cluster tester

Re:"super" computer:

[Anonymous Coward]

Now that he has actually built the supercomputer and it does everything he hoped it would do, perhaps other researchers will be able to justify the money to set up their own clusters (without donations from Sony).

So what you're saying is that, thanks to this researcher's creativity and effort, computer researchers in the future may get free playstations? Sweet!

Re:"super" computer:

[Anonymous Coward]

Seriously, you're better off getting a few NVIDIA Tesla cards and using CUDA then trying to get code running on the PS3... Nevermind the fact that you can't use the GPU at all in the PS3 Linux Environment. $4000 gets you 2 Tesla cards + 1 GPU which should push about ~2 Teraflops for optimized code.

So, where's the link to the supercomputer that you built with that hardware?

Yeah, that's what I thought.

Re:"super" computer:

[andy_t_roo]

"If one wanted to build their own home "super" computer then why not just use CUDA and a few Nvidia cards?"

its been done - http://www.nvidia.com/object/personal_computing.html
$10k for ~250x the processing power of a desktop

Re:"super" computer:

[TrancePhreak]

The Cell in the PS3 has 7 cores, 1 is reserved for the hypervisor. You get 6 cores.

Re:"super" computer:

[andy_t_roo]

tesla cards are basically 8800's with double the memory - for personal computing power, a straight 8800 (or 2) is probably just as useful, and quite a bit cheaper.

Re:"super" computer:

[Anonymous Coward]

The "real" Cell has 8 SPE's rather than 7. And will cost you much more than a PS3.

Re:"super" computer:

[TheKidWho]

Not true. The Tesla C1060 is about the same as a Geforce GTX 280 in raw processing power and # of stream processors.

Re:"super" computer:

[TheKidWho]

It's the planned project for the summer. Putting together a GPU based SuperComputer with ~3-4TFlops of Performance for under $6,000. I plan on running CFD code on it.

Re:"super" computer:

[Chrisq]

or bank reclaimed assets from a sunken business?

What type of processor do Woolworth's POS tills use?

Re:"super" computer:

[bigblackcar]

It's just one of the best excuses to put 10 ps3s in the office and start playing massive group games.

"What? You have 10 ps3s in the office?"

"It's for our cluster computer, dear."

Re:"super" computer:

[tolan-b]

Seems to me that you'd be better off waiting for Cell on a PCI card type solutions like the SpursEngine from Leadtek that's quoted as launching at around $285. There's plenty of the PS3 going to waste if you're just using the Cell for processing and ignoring the GPU, BluRay drive and so on.

Re:"super" computer:

[TheKidWho]

I should add, the C1060 supports Double Precision Floating Point numbers which is necessary for accurate simulations.

Re:"super" computer:

[Anonymous Coward]

The dude had his supercomputer up and running and doing useful work BEFORE the SpursEngine was available. Also, the SpursEngine will need a PC to plug into, while a PS3 is a ready-to-go standalone box, so even with SpursEngine available it MIGHT make sense to keep using the PS3.

Re:"super" computer:

[J05H]

I'm pretty sure that PS3Cluster uses the GPU - but am not 100% because I only tested and did image-capture for the project. Whatever it is when you gang a bunch of them together it pumps out teh complex maths just fine.

Cell processors on a PCI card or in blades would be nice but neither of those are real "commodity" hardware solutions. One of the nice things with PS3Cluster is that the Playstation consoles can be scaled to arbitrarily large clusters.

Re:"super" computer:

[spazdor]

I also don't think Sony pays retail prices for Sony hardware. ;)

Re:"super" computer:

[Anonymous Coward]

Sheep piece of shit 11?

 

Re:"super" computer:

[Anonymous Coward]

You jest, but most likely P3 processors. A lot of POS systems are based on WinNT 4 or Win2k.

Re:"super" computer:

[whohou]

It's just one of the best excuses to put 10 ps3s in the office and start playing massive group games.

"What? You have 10 ps3s in the office?"

"It's for our cluster computer, dear."

"Cluster computer?"

"*sigh*, all right, they're FOR PLAYING GAMES AND GOOFING OFF, ok!?!"

oh wait..

Re:"super" computer:

[Herby Sagues]

They fake their performance numbers, I don't know what they might do with their pricing. A single Cell processor tops at less than 10GFlops double precision. A low end supercomputer runs at 100TFlops. So you need at least ten thousand PS3 to have something worthy of the name of a supercomputer under theoretical conditions. In real conditions, with the best interconnect available, you probably need a few hundred thousand consoles. Yet, these folks are claimingfor comparison that the BlueGene supercomputer (from 2005, by the way) tops at 280GFlops, when in reality it runs at more than 280TFlops. I have a difficult time believing Sony didn't notice the math error.

Re:"super" computer:

[Michael Hunt]

So does the D10U arch (GTX260/GTX280). Unless you need the ability to say 'Nvidia says it should have worked' when it doesn't work, you're wasting your money buying a Tesla card with a D10U chip on it when you can get the exact same thing (exactness may not include domain validation and/or burn-in) for half the price on a consumer-level VGA card.

Re:"super" computer:

[TheKidWho]

Right, but the parent was claiming the Tesla card was just an 8series which is wrong depending on which version one is referring to.

And yes I understand the GTX and the C1060 are for all intents and purposes the same, the main difference being RAM and graphics output. My new system is going to start out with 2 dual GTX280's Instead of the more expensive Tesla board.

Oblig. OMG...

[Anonymous Coward]

something to finally run Vista?

ibm

[sreid]

why would ibm be involved in this if it means they will sell less servers?

Re:ibm

[oodaloop]

FEWER servers! FEWER! Aauughhhh!

Re:ibm

[Anonymous Coward]

This isn't really the place to start criticizing grammar and spelling, unless you REALLY want to live a life full of frustration and torment....?

(Though it could be worse, I suppose - you could go to digg etc. intsead...:p)

Re:ibm

[Anonymous Coward]

How?

Re:ibm

[rhsanborn]

It's showcasing the usefulness of IBM's cell processors for exactly this kind of thing. They have very good reason to be involved as it may mean that there is interest in using their processors for smaller computers at a higher volume to do modeling and research.

Re:ibm

[fuzzyfuzzyfungus]

IBM is probably banking on the existence of people who want Cell processors in systems with more than 256megs of RAM. Other IBM value-adds would presumably include rack mountability, support for netbooting and other convenient management stuff, and so forth.

If your application leans almost entirely on the CPU with very little need for RAM, and you have an army of screwdriver monkeys(or grad students) to do all the legwork, the PS3 is an excellent deal. If you need something with RAM capacity that wasn't a joke in 2001, and/or management features that won't have you tearing your eyes out when you have 10,000 of them, then IBM smells opportunity.

Re:ibm

[Enderandrew]

IBM is the first company to see the writing on the wall and invest in new markets, even if those markets invalidate their current holdings. They went from a typewriter company, to a mainframe company, and an operating system company, to a PC company, to a server company, to a virtualization company, and now a SaaS company.

Re:ibm

[Grimbleton]

Or YouTube

Re:ibm

[dbIII]

If you wait out the two weeks of used car salesman tactics from a place like Mercury that can sell you cell processors in systems with more than 256megs of RAM you'll find out that unless you have an endless budget you are probably better off with the ten quad core Xeon systems you could get for the same price.

That is why a system made of game consoles makes a lot more sense than very similar hardware in a rackmount case. Other cell hardware has been priced into complete irrelevance by salesfolk having too much control over the process.

On the other hand there is the nvidia CUDA solutions, hardware doing things a slightly different way but proudly printing their prices on the net instead of two weeks of mindless chatty emails before you get the price.

Re:ibm

[Skjellifetti]

They had to do that because there is a world market for maybe five computers so none of those markets were going to last very long.

Re:ibm

[Anonymous Coward]

IBM is using this to their advantage to sell more servers. The Worlds fastest SuperComputer is a special hybrid of Opteron and Cell Processors. The more software that is developed for the Cell the higher volume of Custom Supercomputing solutions they can sell.

Worlds Fastest Supercomputer - Roadrunner - Sustained 1.026 Petaflop

http://www.top500.org/system/9707

Re:ibm

[adisakp]

IBM is probably banking on the existence of people who want Cell processors in systems with more than 256megs of RAM. Other IBM value-adds would presumably include rack mountability, support for netbooting and other convenient management stuff, and so forth.

The newest Cell processors from IBM have two major features that customers wanted:

1) Support for normal DDR2 RAM. The original CELL (PS3 version) only supports RAMBUS XDR Memory. There is support for 16GB of DDR vs 256MB of XDR (PS3).
2) Support for pipelined double-precision operations on the SPU's. The original CELL (PS3 version) stalls the SPU's during all double-precision floating point operations and has no support for vectorized double-precision. The pipelining can increase the throughput of common double-precision operations by a factor of 5-8. There is still no support for vectorized double-precision floats.

Subsidized Supercomputers

[MeanMF]

Does Sony make any money on PS3 hardware sales? Last I heard they were selling them at around $100 under the cost of production.

Re:Subsidized Supercomputers

[Shados]

I dont know about the newer models without the backward compatibility and stuff, but previously, they were definately net loss. The -last- thing Sony wants is to sell a million PS3 with 0 attach rate. Of course, those numbers would still count to impress developers, and may be a catalyst, but....

Re:Subsidized Supercomputers

[makapuf]

AGAIN, revenue of console sales is not N*const (positive or negative), but const1+N*const2 where const2 is negative (it's a gain per console) but upfront costs=const1(R&D, licences ...) are big. So the fact that the total is negative implies const is negative, but in fact it's mostly that N*const2 is still less than const1. (I hope this makes sense to some at least)

Re:Subsidized Supercomputers

[Spatial]

The Wii is the only current generation console sold at a profit.

Re:Subsidized Supercomputers

[timeOday]

The number of PS3's sold will never be enough to hurt Sony's bottom line, but will boost the image of the console. Having credible scientists call your product a "supercomputer" is worth something. Does Ferrari's Forumla 1 racing team pay for itself? Nah, it's an investment to promote an image.

Re:Subsidized Supercomputers

[Bearhouse]

Does Ferrari's Forumla 1 racing team pay for itself? Nah, it's an investment to promote an image.

Most of the budget is paid for by sponsors.

Re:Subsidized Supercomputers

[Grimbleton]

Who sponsor them based... on an image that's promoted ;)

Re:Subsidized Supercomputers

[VinylRecords]

Some of first generation of PS3s, the 60GB, had the PS2 Emotion Engine in them to provide perfect hardware emulation for playing PS2/PS1 games making the PS3 100% backward-compatible.

The 80GB version of the PS3 saved SONY the cost of having to manufacture a PS2 essentially inside of a PS3 by replacing the hardware emulation with software emulation. The upside was cost, the downside was that many games cannot be played on a PS3 without the hardware Emotion Engine. Meaning the 80GB systems are NOT 100% backwards-compatible.

And many of the components have become cheaper to manufacture now or have been created out of more efficient and less costly parts since the PS3 first launched. But now, the PS3 ships without hardware or software emulation to play PS2/PS1 games.

The reason for this is not only to save money on initial production costs of the PS3, but it also extends the life of the PS2 greatly for those who were not able or willing to buy the backwards-compatible PS3s, but it also allows SONY and third-party developers to offer old PS1 (and eventually PS2) games for sale on the Playstation Network.

Is SONY making money on the PS3s being sold now? No. But compared to the negative cost of manufacturing a PS3 when it was first in production SONY is in much better territory but they still aren't in the black.

Re:Subsidized Supercomputers

[CronoCloud]

Meaning the 80GB systems are NOT 100% backwards-compatible.

Not 100% but very still very high. Out of my collection of 64 PS2 games, only 2 have enough problems that I consider them unplayable on the PS3: Tekken Tag Tournament (doesn't run full speed) and Fallout Brotherhood of Steel (with very pronounced texture glitching)

But now, the PS3 ships without hardware or software emulation to play PS2/PS1 games.

Sigh, why do people keep getting this wrong. Although the latest release PS3 consoles can't play PS2 games, ALL PS3's can play PS1 games since that's entirely software emulation.

Re:Subsidized Supercomputers

[Anonymous Coward]

Hang on, I need to feed the info into the supercomputer to process this.

Re:Subsidized Supercomputers

[BenFenner]

Does Ferrari's Forumla 1 racing team pay for itself? Nah, it's an investment to promote an image.

Incorrect. At the end of the day Ferrari is interested in racing and racing only. No, it's F1 team doesn't pay for itself, but unlike every other car manufacturer, they don't use racing to sell their cars. They sell their cars so they can go racing. They started out racing and when sponsorship money wasn't enough they sold cars to the public at an abnormally high price. Buyers understood, and still understand today that they are funding the racing team with a donation and getting a decent car out of the transaction.

Re:Subsidized Supercomputers

[nasch]

When someone mentions a product as losing money, they're usually talking about negative marginal revenue. After all, a brand-new fantastically profitable product will often be "losing money" at the beginning if you count fixed costs, but nobody would describe it so.

Re:Subsidized Supercomputers

[Anonymous Coward]

Actually, according to the founder of Ferrari, the racing team is the company. The street cars that are produced and sold (based in part on the 'image') are to fund the racing effort. In other words, and I wish I could recall who said this first, "Some teams race so they can sell cars, one team (Ferrari) sells cars so they can race."

Re:Subsidized Supercomputers

[powerlord]

People keep "getting it wrong", because a lot of owners/potential owners have a huge PS2 library but relatively few PS1 disks.

Speaking as a 60GB owner, I think the lack of any PS2 compatibility in the current models on sale is sad.

Re:Subsidized Supercomputers

[piltdownman84]

Does Ferrari's Forumla 1 racing team pay for itself? Nah, it's an investment to promote an image.

I couldn't find Ferrari's numbers but McLaren made £3.2M in 2007. They did get £41M in 'free' parts from Mercedes though, so you do have a bit of a point.

Re:Subsidized Supercomputers

[Spatial]

Haha, I didn't know that. Man, six dollars and the hardware is so much worse than the others. Sony and MS must be taking heavy losses.

Awesome power!

[iSzabo]

This is just one of those steps towards homebrew genetic algorithms. It won't be long until someone goes "The Internet, pfft - supercomputing clusters are for porn".

but seriously....

[powerspike]

Most of my friends purchased a ps3 because it's the cheapest profile 2(is that what it's called?) blueray player on the market, they have 1-2 movies each, and just use it as a showcase, sony will never make cash out of them....

All this sounds like is the same thing.

Everytime i see an article like this, i feel even more sorry for the 12 people that purchased the console to play games!

Re:but seriously....

[hchaudh1]

I have bought about 14-15 disk PS3 games to date. And spent about $300 on the PSN. Someone I know (the crazy gamer kind) has spent between $4K-5K on games alone. I don't know what your friends are trying to showcase, but I would say if they don't have a use for the PS3, they were idiots to buy it in the first place.

The PS3 has sold consoles in the millions (the last number I heard somewhere was between 18-21 million). And I, neither those other millions I am sure, have any plans of throwing it out for the next 4-5 years at least. Guess what we will be doing with the PS3 then. Interesting statistic btw, the PS3 is selling more than the 360 did at the same point in its lifetime.

Saddam Hussein

[Anonymous Coward]

Saddam Hussein was going to do this with PS2s.

Re:Saddam Hussein

[mowall]

Saddam Hussein was going to do this with PS2s.

Sadly he didn't have time to do the write-up.

Re:Message from Government Man

[blhack]

Pay no attention to the man being arrested behind me for disclosing how to build a supercomputer using gaming consoles and beaten by the fine men and women of the LAPD

Please....please go back to reddit.

Why use PS3s?

[JanusFury]

Why would you want to use PS3s for a homebrew supercomputing cluster if it means you have to write and optimize code for the SPEs to get benefit out of it? The PS3's linux environment doesn't let you utilize the GPU or all of the built-in SPEs and it doesn't have a lot of RAM available either. It seems like it would be cheaper to build a cluster out of commodity PC parts, and maybe use GPUs+CUDA to get more muscle without having to completely hand-roll your own accelerated computation code (since CUDA is roughly C). I can't imagine that the PS3 would end up cheaper for these purposes, considering it includes a Blu-Ray player along with a bunch of other things you're not going to be using.

Re:Why use PS3s?

[Darkness404]

....About the cheapest computer you can build/buy that would be of any use as a supercomputer would be $200, add in a $150 GPU, and thats $350, about the price of a used PS3. Most supercomputers need fast CPUs, not a ton of RAM (though, the more RAM the better), and so it becomes that a PS3 is about the same as doing it with commodity computers only the PS3 has a much faster CPU.

Re:Why use PS3s?

[drinkypoo]

Why would you want to use PS3s for a homebrew supercomputing cluster if it means you have to write and optimize code for the SPEs to get benefit out of it? The PS3's linux environment doesn't let you utilize the GPU or all of the built-in SPEs and it doesn't have a lot of RAM available either.

Well, I'll bite; if the cell is the fastest processor for your workload, the PS3 is the cheapest way to get one, even at only six usable SPEs and no GPU. Doesn't the PS3 have GigE? That's plenty fast enough to shovel data in and out of the system.

Re:Why use PS3s?

[ookabooka]

If I recall correctly, Sony sells hardware either at-cost or at a slight loss because they make their money on the games. I know this was true for the original xbox as modded xbox clusters were demoed as extremely cost efficient compared to making the computers yourself. I used a moded xbox as an early TiVO as it was way cheaper than making a similar setup myself.

Re:Why use PS3s?

[Anonymous Coward]

Wrong. Ram to processing power is really application dependent, and they tend to scale linearly together for most applications. It's a big part of designing supercomputer specs.

Re:Why use PS3s?

[Jeff DeMaagd]

The Blu-Ray drive and the controller are the only things in the system you won't really use, at least much. The rest of the system is a computer, even if it is an unusual architecture. I don't know if the system can install an OS over a USB drive or CF card vs. optical disc, I've never tried to install Linux.

I'd say it's a very powerful computer for $400, assuming you can program for it.

Re:Why use PS3s?

[ASBands]

since CUDA is roughly C

Not quite. CUDA looks a lot like C in that it has C-family syntax but the biggest limitation it has is that there is no application stack - which means no recursion. CUDA also lacks the idea of a pointer, although you can bypass this by doing number to address translation (as in, the number 78 means look up tex2D(tex, 0.7, 0.8)). The GPU also has other shortcomings, in that most architectures like to have all their shaders running the same instruction at the same time. For this code

if (pixel.r < pixel.g){
//do stuff A
}else if (pixel.g < pixel.b){
//do stuff B
}else{
//do stuff C
}

The GPU will slow down a ton if the pixel color causes different pixels to branch in different directions. Basically, the three sets of shaders following different branches of that code will be inactive 2/3 of the time.

In the Cell, you really do just program in C with a number of extensions added onto it like the SPE SIMD intrinsics and the DMA transfer commands (check it out). The Cell really is 9 (10 logical) processors all working together in a single chip (except in PS3, where there are only 7 working SPEs). Furthermore, your 8 SPEs can be running completely different programs -- they're just little processors. Granted, you have to be smart when you program them to deal with race conditions and all the other crap you have to deal with for multithreaded programming. The Cell takes about 14 times longer to calculate a double precision floating point than a single (and there aren't SPE commands to do four at once like you can with singles).

So which is more powerful? It really depends what you're doing. If your task is ridiculously parallellizable and doesn't require the use of recursion, pointers or multiple branches, the GPU is most likely your best bet. If your program falls into any of those categories, use a Cell.

Re:Why use PS3s?

[JDevers]

Well, that is all true except most simulations eat a ton of RAM (we aren't comparing 20 GB to 25GB here, the PS3 has 256MB of memory) and the PS3 does NOT have a faster CPU. If you think it does, look at the folding@home stats for a PS3 versus a mid/high end GPU. The GPUs are really what are interesting here, the main CPUs of both systems are slow enough to make them of no interest when only talking 8-20 units...

From the folding@home Wiki:
as of August 24, 2008, GPU clients accounted for the majority of entire project's throughputâ"over 1.8 petaFLOPs of computational powerâ"at an approximate ratio of 9 clients per teraFLOP

On April 26, 2007, Sony released a new version of Folding@home which improved folding performance drastically, such that the updated PS3 clients produced 1500 teraFLOPS with 52,000 clients versus the previous 400 teraFLOPS by around 24,000 clients.[22] Lately, the console accounts for around 40% of all teraFLOPS at an approximate ratio of 35½ PS3 clients per teraFLOPS.

So using those numbers, the PS3 is about a fourth as powerful as the average GPU running folding@home, and of course we know that the average GPU isn't nearly as fast as the fastest available.

Re:Why use PS3s?

[Anonymous Coward]

Second...RAM is usually the limiting factor when it comes to fast simulations and calculations.

Re:Why use PS3s?

[afidel]

The best MIPS/watt for CUDA is probably either the 9600 GSO or the GTX280 depending on whether you're memory or processor constrained. The 9600 can be had for about $75 for 768MB variety (forget the 512/1024 parts they much lower performing) and has 96 stream processors running at up to 650Mhz. The GTX280 costs about $400 and has 240 650Mhz stream processors (though I believe they might be slightly more advanced then the ones on the 9600 I'm not sure how much of that is exposed by CUDA). Power usage is 46W peak for the 9600 and 180W for the GTX280.

Re:Why use PS3s?

[kramulous]

Not that I am, but if I was some home/small business artist/modeller who needed some serious render time to generate the frames of a computer animated movie/demo, I'd be making one of these clusters ... It would be perfect for this kind of thing.

Re:Why use PS3s?

[afidel]

Nope, not nearly enough ram for even a moderately complex scene's geometry let alone the textures unless you want your output looking like a game (IE most graphic artists will want photorealistic output which is more than a game console is capable of).

Re:Why use PS3s?

[rm999]

Yeah, but a large part of the cost is the bluray player, which is useless in a supercomputer. I guess you could probably sell the drive/laser for 100 bucks to offset your costs.

Re:Why use PS3s?

[Chandon Seldon]

Well, I'll bite; if the cell is the fastest processor for your workload, the PS3 is the cheapest way to get one, even at only six usable SPEs and no GPU. Doesn't the PS3 have GigE? That's plenty fast enough to shovel data in and out of the system.

What workload is actually faster on a Cell than on a modern quad-core CPU or video card? I mean - it's possible that such a workload exists, but the niche between a general purpose CPU and the hundreds of FPUs in a video card has got to be pretty damn small.

Re:Why use PS3s?

[Bearhouse]

Or keep it in, and use your cluster to break the encryption, rip and recode the Blueray disk.

As per:

http://www.exit1.org/dvdrip/index.cipp

The PS3 cluster uses Fedora, so how about it someone (more talented than me)?

Re:Why use PS3s?

[Anonymous Coward]

Yeah, but a large part of the cost is the bluray player, which is useless in a supercomputer. I guess you could probably sell the drive/laser for 100 bucks to offset your costs.

You could be right that a large part of the cost won't be used by a supercomputer setup, but is the BR player still a large part of the cost?

If you break it down, the only real big difference with DVD is the laser, other than that it probably has a lot of parts comparable with an internal DVD drive.

When you see that stand-alones are going towards $160 (perhaps even lower as I write) and I assume they are not sold at a loss, how much can the internal drive really cost?

http://news.cnet.com/8301-13845_3-10125666-58.html

Don't forget a stand-alone comes with PSU, hardware to decode everything, connectors and chassis, you still think you can get $100 for the drive in the PS3?

Re:Why use PS3s?

[Anonymous Coward]

Hear, hear.

Just one little note, I assume they are running the "stock" version and that means another SPU is lost as it is occupied by the hypervisor.

So, on a PS3 you can play with 6 SPUs under Linux.

Re:Why use PS3s?

[J05H]

When I installed linux+MPI on the test PS3 it recognized all the processors - pretty cool seeing 8 little penguins pop up. From what Chris said the programming is fairly generic C/C++ to utilize the whole console. It's apparently not that hard and PS3s are dirt cheap (compared to supercomputers or even blade servers).

Josh

Re:Why use PS3s?

[J05H]

Depends on the specific code - to my knowledge no one has written/ported a 3d renderer to the PS3Cluster architecture yet - so you should get to work on it. 8) To get realistic textures does require a lot of RAM or a lot of swapping. One thing that could help in this context is to have a big block of NAS on the same network - and treat part of it as a RAM disk or texture buffer. Not necessarily efficient but could work following any of several weird render schemes.

I only suggest the block of external storage because I know that is part of Dr. Khanna's setup and it works.

Josh

Re:Why use PS3s?

[CronoCloud]

Yes, the GPU client is faster, but it's limited to the kinds of WU's it can do, compared to the PS3 client, as the FAH site says:

What type of calculations the PS3 client is capable of running?

The PS3 right now runs what are called implicit solvation calculations, including some simple ones (sigmodal dependent dielectric) and some more sophisticated ones (AGBNP, a type of Generalized Born method from Prof. Ron Levy's group at Rutgers). In this respect, the PS3 client is much like our GPU client. However, the PS3 client is more flexible, in that it can also run explicit solvent calculations as well, although not at the same speed increase relative to PC's. We are working to increase the speed of explicit solvent on the PS3 and would then run these calculations on the PS3 as well. In a nutshell, the PS3 takes the middle ground between GPU's (extreme speed, but at limited types of WU's) and CPU's (less speed, but more flexibility in types of WU's).

It seems that the PS3 is more than 10X as powerful as an average PC. Why doesn't it get 10X PPD as well?

We balance the points based on both speed and the flexibility of the client. The GPU client is still the fastest, but it is the least flexible and can only run a very, very limited set of WUs. Thus, its points are not linearly proportional to the speed increase. The PS3 takes the middle ground between GPUs (extreme speed, but at limited types of WU's) and CPU's (less speed, but more flexibility in types of WUs). We have picked the PS3 as the natural benchmark machine for PS3 calculations and set its points per day to 900 to reflect this middle ground between speed (faster than CPU, but slower than GPU) and flexibility (more flexible than GPU, less than CPU).

The PS3 is outrunning all the rest of the FAH client types. Should I stop my existing PC/GPU/... FAH clients?

NO. The other clients are valuable to us too and we have chosen a points system to try to reflect the relative merits of each different platform to our scientific research. For example, the SMP client has been producing some very exciting scientific results and continues to be very important in our work. By supporting machines with lots of different functionality, we have a very rich set of hardware on which to run varied types of calculations, allowing us to tailor calculations to the hardware to achieve maximum performance.

Re:Why use PS3s?

[jrmcauliffe]

Particularly as PS3's probably hold their value better than these... http://cgi.ebay.com.au/9-x-IBM-eServer-2x-3-2-GHz-Server-Blade-QS20-Rackmount_W0QQitemZ260329549954QQcmdZViewItemQQptZAU_Servers?hash=item260329549954&_trksid=p3286.c0.m14&_trkparms=66%3A2|65%3A1|39%3A1|240%3A1318

Re:Why use PS3s?

[MikeBabcock]

Feel free not to believe it, but actually doing your research might be smarter.

And which part of the GPU not being fully exposed to Linux is relevant to supercomputing exactly?

Re:Why use PS3s?

[Anonymous Coward]

ok - You are bringing up PS3 reoptimization.

I'm going to try to say why you really don't do this.
To me, this would cram rip. You cram it in, but by the way that the XDR simply works, it will not explode. You would possibly see a lot of searing and mislabeled textures.

I think that a function of the laser is to do what this is describing. Remember that you may have a bit of a problem 7200 jiucing the drive.

Re:Why use PS3s?

[Anonymous Coward]

The heap/recursion thing is occasionally annoying but not a blocker. If you can "be smart" when programming a cell you can be smart and unroll your recursion as I did to code non-recursive heap operations w/ CUDA. The payoff is I now have a ridiculously fast and awesome heap. Using hardware acceleration in the first place means you have a problem where you deem the machine time to be more valuable than the dev time needed to code to it over just the CPU. Recursive implementations are dev friendly but not usually the best solution for the machine. If it's worth the dev time to code for the custom hw then it's worth the effort to unroll recursion (that is to say, since you are going for high performance you'd probably unroll your recursion on such an important problem on a CPU also).

Branches - that's really nitpicking & from what I've seen a really modest price. I don't sweat branches - if you need them you need them & none of them have yet killed my performance.

Pointers - huh? CUDA is rife with pointers especially w/ image processing. See the image processing samples in Nvidia's CUDA samples. e.g. boxFilter. the image gets chopped up and stream processors process bits of the image, indexed by pointers into the image. You do a cudaMalloc and it returns a pointer to device memory. Pointer math works fine. It's not a pointer to host memory but it is a pointer.

I think the claim that "CUDA is roughly C" is accurate after having slogged through the APIs for other hardware accelerated solutions. So it's a relative thing. Not perfect but oh yes it's very nice.

Re:Why use PS3s?

[TrancePhreak]

The part that can run extra instructions on the GPU, doubling your available computing power.

Re:Why use PS3s?

[forkazoo]

Not quite. CUDA looks a lot like C in that it has C-family syntax but the biggest limitation it has is that there is no application stack - which means no recursion. CUDA also lacks the idea of a pointer, although you can bypass this by doing number to address translation (as in, the number 78 means look up tex2D(tex, 0.7, 0.8)). The GPU also has other shortcomings, in that most architectures like to have all their shaders running the same instruction at the same time. For this code

if (pixel.r

Yeah, you wind up with some strange idioms on parallel systems like that which would be insanely idiotic on a more ordinary platform... (Just a random example based on your example which avoids branches. Depending on the actual task, there are probably better ways to do this.)

A = (pixel.r pixel.g){
Aresult = //do stuff A
}

B = (pixel.g pixel.b){
Bresult = //do stuff B
}

C = (pixel.g==pixel.b){
Cresult = //do stuff C
}

answer = A*Aresult + B*Bresult + Cresult*C;

That way, you are brute forcing your way through three times as much work that you should need to do, but you avoid branches. If the A condition is false, then you multiply Aresult by zero. The strange hoops you have to jump through on these types of platforms are why you should always take vendor press releases with a certain grain of salt...

Re:Why use PS3s?

[Anonymous Coward]

Right on the head. I bought a PS3 a few years ago specifically for the purpose of rendering using the cell be. What a mistake! You are hobbled at every turn. One SPE is turned against you (virtual baby-sitter making sure you don't access the video sub-system), one is dead from the factory, leaving you 6 of 8. In order to use the SPE's you need IBM software. Getting the software from mothercorp only requires anal probes and promises you aren't a terrorist (and everything in between). If you do get it though, you are stuck with 256MB of ram. You can't upgrade. Its all soldered in, and no one makes bigger chips that will work, so rewrite everything to fit in! 256MB might have been fine 15 years ago, but its a bit tight now. In short, I'm buying some xeons and building a real system.

Re:Why use PS3s?

[Anonymous Coward]

depends on your renderer; for mental ray you could render with a tile size of 15x15 pixels, and your "slave" ps3 would have more than enough resources to deal with it, the master (workstation) computer would need to have greater resources, say 8 gigs of ram would be nice, the more chunks it has to deal with the more memory would be used. The less ram on the slaves, the more network traffic as it juggles data between the sytems.

Re:Why use PS3s?

[myvirtualid]

Why would you want to use PS3s for a homebrew supercomputing cluster

IIRC, a general purpose CPU has a small data cache and a large instruction cache, coz you can never be 100% sure which instruction is likely to be executed next. PS3s have large data caches and small instruction caches, because they spend much of their time executing a small number of instructions over a large set of data, that is, graphics rendering.

If you are doing any sort of mathematical simulation, you can likely express your numerical methods in a relatively small set of instructions. And you are likely going to have hoards and scads and barrels of data.

Machines like the PS3 are perfectly suited to number crunching. If only there were some way to kit them into a Beowulf cluster....

Again, IIRC, this was the original motivation for porting Linux to the PS2 before the official kit came out: Because a cluster of PS2s running Linux made for a far faster number cruncher than anything else available for a comparable price (or a price within an order or two of magnitude).

As for writing and optimizing code, well, let's just say you made my eyes bug out with that one. If you are doing any serious supercomputing, you are ALWAYS writing and optimizing code: The point is to get the fastest possible execution so you can crunch the greatest amount of data possible and get the best possible results from your work.

In serious number crunching, the effort spent coding and optimizing is almost always going to pay off.

Re:Why use PS3s?

[nasch]

How do those numbers show a PS3 is a fourth as powerful as an average GPU?

Re:Why use PS3s?

[Anonymous Coward]

Not strictly true, the latest linux distros for the PS3 can use the GPU's memory, so PS3 linux has 512MB of ram available to it (double than before..).

Re:Why use PS3s?

[powerlord]

Alternatively I wonder if you could just add flashdrives. Since you'd need a fair amount of space for the textures you could cache them in the flash drive for faster access than the hard-drive or (i bet) the network.

Since an 8GB flash drive is now ~30$ this certainly seems possible from the cost perspective (at least).

Re:Why use PS3s?

[Funk_dat69]

It seems like it would be cheaper to build a cluster out of commodity PC parts, and maybe use GPUs+CUDA to get more muscle without having to completely hand-roll your own accelerated computation code (since CUDA is roughly C).

I dont see how that would be cheaper since high end video cards cost as much as a PS3. Also, CUDA may be *rougly* C, but the code for CELL *is* C, just with familiar SIMD extensions a la MMX, altivec, etc. There's no crazy "load matrix / compute matrix" craziness like in CUDA.

Of course there is SPU management code which can be a pain -no one said it's perfect- but the point is that writing CUDA code is at least the same amount of hassle as writing Cell code. I think your only valid argument is the RAM limit.

Re:Why use PS3s?

[ChrisAI]

This is Chris, co-author of the guide. The MPI lib is utilizing each of the cores without Cell optimization required (This is a further step for the best performance gains). So yes you can go the PC route, and I have deployed this same MPI cluster setup on PC infrastructure without problems. But I assure you that for the best performance, and buying 'new', the PS3 route is cheaper hardware due to the battles over the game console market. Write me anytime if anyone needs help. Good luck everyone, and thank you for taking an interest. Thanks, Chris

Re:Why use PS3s?

[Anonymous Coward]

Since its original purpose (gaming) has floundered a bit, it's starting to sound like we have a solution (PS3) in search of a problem.

I don't have one, but a friend of mine claims that the PS3 helps heat a room in his house. If research projects don't pay off, Sony can always try marketing the PS3 as a space heater! ;-)

Re:Why use PS3s?

[Anonymous Coward]

I'm not sure just how much performance you'll get out of the PS3's GPU. It is a 7900 series Nvidia chip and those do not support CUDA.

Re:Why use PS3s?

[CronoCloud]

Yes, the thing does put out heat. It's nice having the heat today with the cold and the ice storm and all. It's also useful for posting to Slashdot (with Firefox under Linux), and playing Oblivion.

Why PS3s?

[whyloginwhysubscribe]

I don't understand why this isn't possible with normal PC hardware - what is special about the PS3 - or is it just because it is better value for money?

Re:Why PS3s?

[Darkness404]

A) Although the cell is a pain to code for, it is much better than whatever PC you can get for ~$400 which will probably contain a mid-to-low-range dual core x86 CPU, whereas the PS3 gives you a Cell CPU which is much, much, faster than the x86 CPU.

B) PS3s are uniform. Other than HD differences, a PS3 built in 2008 will be the same PS3 built in 2012 (assuming the PS3 lasts that long) this allows for a uniform cluster without worrying about differing parts (for example, the Core i7 built in 2008 will not be the same as the Core i7 built in 2012 and getting a 2008 Core i7 is going to be a pain)

C) PS3s are the new fad. It isn't going to be hard to set up a supercomputer cluster with PS3s compared to using a mismatch of older computers because again, the PS3 is uniform.

Re:Why PS3s?

[Anonymous Coward]

A) Although the cell is a pain to code for, it is much better than whatever PC you can get for ~$400 which will probably contain a mid-to-low-range dual core x86 CPU, whereas the PS3 gives you a Cell CPU which is much, much, faster than the x86 CPU.

time is money. The time wasted wrestling with the PS3 is expensive and wastefull and more than justifies increased cost of commodity hardware.

B) PS3s are uniform. Other than HD differences, a PS3 built in 2008 will be the same PS3 built in 2012 (assuming the PS3 lasts that long) this allows for a uniform cluster without worrying about differing parts (for example, the Core i7 built in 2008 will not be the same as the Core i7 built in 2012 and getting a 2008 Core i7 is going to be a pain)

FALSE, the software interfaces are uniform, sony/nintendo/MS have in the past and will most likley in the future continue to make chip changes to reduce there hardware costs, they can do this as nothing accesses the hardware directly.

C) PS3s are the new fad. It isn't going to be hard to set up a supercomputer cluster with PS3s compared to using a mismatch of older computers because again, the PS3 is uniform.

see points 1 and 2. Why waste time/money and resources on something that will give you subpar performance and a lot of pain.

Re:Why PS3s?

[Anonymous Coward]

Um yeah the PS3 in 2008 will be the same PS3 built in 2012. Whereas the GeForce cards will be maybe an order of magnitude faster. You have binary compatibility of CPUs and for GPUs Nvidia promised forward compatibility of CUDA - your implication that you need uniform model #s is false unless you are either building a wicked huge cluster where it's actually relevant requirement that the nodes must be plug & play replaceable, in which case piles of PS3s is not a serious architecture anyway.
I'm not sure why anyone would commit to the PS3 for a long term or large cluster - they're never going to upgrade because they're good enough for their principle market. prices will come down but you're stuck with their current performance, power consumption, etc. whereas with GPUs you get the option to move to new hardware.
Does anybody think we'll actually be reading about someone building a PS3 cluster in 2012? or reading about this cluster having great success over other non-PS3 clusters because of their foresight to go with a uniform hardware solution?

Re:Why PS3s?

[Richard_at_work]

B) PS3s are uniform. Other than HD differences, a PS3 built in 2008 will be the same PS3 built in 2012 (assuming the PS3 lasts that long) this allows for a uniform cluster without worrying about differing parts (for example, the Core i7 built in 2008 will not be the same as the Core i7 built in 2012 and getting a 2008 Core i7 is going to be a pain)

Don't rely on this - there are large hardware differences between early PS2 and later PS2 models as manufacturing tweaks and cost reduction packages were applied to the production process, to the extent where some games refused to run and some features were changed. I don't expect Sony to act any differently with the PS3.

De Ja Vu?

[senorpoco]

I believe it is called 'the turk' and lifted directly from the terminator series.

What about those junk PIIs?

[bogaboga]

I wonder whether this how-to will enable me build a cluster consisting of PIIs. I have 11 lying around.

Re:What about those junk PIIs?

[AvitarX]

11 PIIs is not that much.

Better off to buy a laptop.

Re:What about those junk PIIs?

[armanox]

Yes and no. The concepts are the same, but I'd use a more lightweight distrobution then Fedora to do it on P3's (I've got a stack of P3 1GHz laying around too)

Re:What about those junk PIIs?

[meringuoid]

I wonder whether this how-to will enable me build a cluster consisting of PIIs. I have 11 lying around.

Imagine a Beowulf cluster of those.

Then build one.

I doubt the performance will be awfully impressive by today's standards - you'll be outdone by pretty much any 2008 desktop machine - but it'll be an interesting project anyway. Let us know how you get on.

Limited use

[idiot900]

Couple issues with this as an alternative to the garden-variety x86 cluster connected with InfiniBand:

Slow network interconnect. For problems that are not trivially parallel, network latency is usually a big deal. Ethernet doesn't cut it.
Lack of RAM. 'Nuff said.
Have to care about Cell and PS3 architecture. The codes ("codes" has a slightly different meaning in the context of supercomputing) have to be modified to take advantage of this very specific architecture. Software always outlives hardware, so in the long run the effort may not be worth it.

That said, it's really cheap. If your application isn't held back too much by these issues then enjoy your insanely cheap cluster!

Re:Limited use

[hairyfeet]

I don't see why it is even considered cheap when you can pick up Dual core kits for $200. And since x86 has been around forever you know you'll be able to run your code years from now. So if I wanted a cluster for cheap I would just pick up some of these along with some CUDA capable PCIe cards(which are going down in price as Nvidia and AMD have it out) and be good to go.

Re:Limited use

[thaig]

If you are writing code that only works on x86 then there's a problem straight away.

The code should also probably be written using some library that abstracts some of the details so it should be possible to change hardware at some point.

Then all that matters is whether or not the design of the machine fits the problem. PCs just might not do it.

Re:Limited use

[thaig]

Well there are 7 SPEs on a chip right? So the latency right there must be pretty low?

His problem probably *is* "trivially" parallel so perhaps he was right to do what he did?

Later he can "upgrade" to an IBM PowerXCell 8i based blade.

Re:Limited use

[raftpeople]

This is why:
8 PS3's = 8 cells
8 cells X 7 available SPE's per cell = 56 SPE's
56 SPE's X 4 simultaneous FP calcs = 224 FP calcs per cycle

You would need to get quite a few of those x86 dual core kits to match that performance

Re:Limited use

[MikeBabcock]

I tell you what: you go ahead and buy $4000 of those Dual core kits, and we'll compare your output from a well-written algorithm versus the Cell system designed by this team.

Some interesting code examples for using the Cell have been demonstrated and it has immense processing power that most people don't recognize immediately. Check out this Dr Dobb's Journal article for an example.

Re:Limited use

[Lord Grey]

The link to the actual Dr. Dobb's article is way more readable.

Re:Limited use

[default luser]

You don't seem to understand the Cell's architecture at all.

The cell has 7 SPEs, but one is reserved for the Hypervisor.

Second, each SPE can perform two 128-bit vector operations per-cycle, so long as you do not use double-precision. That's EIGHT 32-bit float operations per SPE, per cycle, excluding the overhead of store operations.

When you do the math, the real performance for a single Cell is between 24 and 48 FLOP per cycle, single-percision (I'd put the number closer to 24). The double-precision performance is about 1/10 the above, as it is not pipelined.

Re:Limited use

[raftpeople]

You don't seem to understand the Cell's architecture at all. The cell has 7 SPEs, but one is reserved for the Hypervisor.

The cell has 8 spe's not 7, the reservation for the hypervisor has nothing to do with "cell's architecture", that has to do with the implementation of software on the PS3 by Sony. Although that is a valid point regarding the number of available SPE's.

Second, each SPE can perform two 128-bit vector operations per-cycle, so long as you do not use double-precision. That's EIGHT 32-bit float operations per SPE, per cycle, excluding the overhead of store operations

You will notice that I said "FP" operations and you will also notice that the FP ALU is on the even pipeline, not both the even and the odd pipeline, that means you can perform 4 single precision FP's per cycle.

Here are a couple tips:
1) In the future, when you correct someone, make sure you are right first
2) Helpful corrections can be good, statements like "don't understand the architecture at all" aren't really valuable

Re:Limited use

[hairyfeet]

But I thought most of the scientific community and the FLOSS guys were ANTI DRM? Isn't it true you have NO access to the GPU at all and only a limited access to the cell to stop "teh evil piratez!" and enforce Sony DRM? And if the Cell is really that wonderful why wouldn't you simply add a few cells to your x86/CUDA cluster and have a supercomputer capable of doing it all? And since the above cell fits into a 1x or 4x PCIe slot, you could simply add more as required or budgeting allows. So the only advantage I can see for Cell is for making a super cheapo cluster.

But if you are pushing the kind of data that needs a super computer anyway I'm sure there are grants and endowments that will help you get the hardware required for your research if you don't already have the money. And with the above solution you have 1)complete control of the hardware 2)The ability to scale the existing units by adding more CPUs/cells/GPUs, and finally 3) The ability to do several kinds of research as your cluster wouldn't need to be tied to the subset of calculations that the cell outperforms on. Oh and lets us not forget gigabit ethernet to tie it all together to help keep the cluster well fed.

So am I missing something here? Or is all this cell talk just because you can pick them up for cheap at Wally World? And since Sony loses serious money on PS3 sales wouldn't a big uptick in those using the PS3 for research seriously damage the company, who last I heard is already bleeding pretty bad?

Re:Limited use

[MikeBabcock]

Thanks, I found the other faster on Google ;-)

Re:Message from Government Man

[johanatan]

Or maybe he did catch it and thinks that sort of thing belongs on Reddit?

Imagine a beowulf cluster..

[Anonymous Coward]

Imagine a beowulf cluster of those!

Re:Imagine a beowulf cluster..

[Arthur Grumbine]

Imagine the ratio of expense of "one" of these to a beowulf cluster of x86s created to provide equivalent computing power. Just imagine!! Hmmm...on second thought it's probably only like 1 to 2.5 or 3...

Might

[CopaceticOpus]

If this catches on for research institutions it may increase Sony's sales, but they might not be seeing the corresponding sale of games spike...

Why "might not"? Are you implying that people may be building PS3 clusters just so that they can sneak into the lab at night and have big gaming parties? Because I can totally see that.

a 2008 supercomputer is 100 teraflops

[peter303]

At any given time a supercomputer is one order of magnitude world fastest computers. This may have been a Year 2000 supercomputer, but far from one now.

Re:a 2008 supercomputer is 100 teraflops

[Anonymous Coward]

And a 2000 computer can still run Starcraft at full specs; I'm sure a 2000 computer is still useful for supercomputing and analysing large datasets. We still have problems in medium-sized datasets that need someone working on them, and if they can do it for $4000 initial + power and get as many runs as they like, they're gonna get a lot more research done on a lot less dollar than at $5000 a run. Hell, we have NEW data since 2000, for NEW problems, that'd be a cinch to crunch on a true blue 2008 supercomputer, but the timesharing is an unbelievable weight on their workload - they don't have time to deal with datasets in the millions, they're working on trillionfold datasets, if not higher. Cheap massively-parallel ("super") computing can only be a good thing.

Even if it does help Sony's profit margins a little.

Re:a 2008 supercomputer is 100 teraflops

[Herby Sagues]

Absolutely! I think these folks are reading the numbers wrong. The slowest BlueGene ever built is 280 TFlops (not 280 GFlops as it is listed in some web sites). A single SPE in a Cell ranks at about one GFlop in double precision (14 GFlops in single, but that doesn't count for these rankings). Eight consoles with seven SPEs would then yield, in the most optimistic calculation possible (very far from reality) 56GFlops, or two orders of magnitude short for being considered a (very low end) supercomputer. A console is a console, even a fast one is a few orders of magnitude short of anything that could be considered in the ballpark of a supercomputer.

Re:Oblig. OMG...

[windsurfer619]

At first I laughed... But then I realized that, no, Vista won't be able to run on this.

Vista doesn't support the PowerPC architecture.

Re:Pretty much useless

[Anonymous Coward]

A supercomputer node (in this case a PS3) handles a small portion of a much larger problem and do not require much RAM or a Blu-ray drive for that matter...

Re:Message from Government Man

[Anonymous Coward]

Luke: Your overconfidence is your weakness.

Emperor: Your faith in your moderators is yours!

Invader Zim is non-free

[tepples]

It's not my fault you didn't catch the Invader Zim reference.

Invader Zim is non-free. It's easier to catch pop culture references if they are pre-1923 or otherwise free.

Re:Pretty much useless

[blind biker]

How is it useless, when the guy who built it, used it already for a month? And it has replaced 200 supercomputer nodes, for his purpose? I'd say that's very fucking useful.

But you know what, maybe you should send him an e-mail and try to convince him how his cluster is useless. Make it a nice, insightful and intelligent e-mail, like your post.

Supercluster is a better word

[Renegade Iconoclast]

After reading TFA, and as a multithreaded software application writer, I don't really care for the word supercomputer to describe this setup.

Sure, it's a supercomputer, if you write a bunch of sophisticated software to use the Cell model of parallelism. I admit ignorance about the Cell model, but I do know that parallelism doesn't come completely free, no matter what the architecture.

I was hoping for a build-it-yourself machine with a whole bunch of cores and a fast bus between them, like a PC, so that you could just write your own threads, and have them assigned by the operating system.

Hell, you could daisy chain Amigas together 20 years ago, and make a render farm with them. Render farms are fairly easily subdivided, because you can just give each node a part of the frame to render. In other words, the task is easily subdivided.

What if you want to beat Kasparov at chess, though? How do you subdivide chess in a way that helps, more than it hurts, if the machines are terribly slow at sharing memory?

The solution is non-obvious, if you spend a bit of time thinking about it, and definitely non trivial.

Universities will have plenty of grad students that can make something like this work, but a hobbyist would be bound to come away disappointed, if they thought this would solve all of their problems on the road to world domination.

Hell, we have supercomputers on our desks now. More CPUs won't make them faster, unless we software writers take advantage of the multiple cpus most of us already have!

Sony owns a movie studio

[tepples]

Most of my friends purchased a ps3 because it's the cheapest profile 2(is that what it's called?) blueray player on the market, they have 1-2 movies each, and just use it as a showcase, sony will never make cash out of them....

Sony owns a movie studio. And as I understand it, Sony also owns a significant share of the patent and trade secret rights in Blu-ray Disc. But if by "Sony" you mean the PlayStation division called "Sony Computer Entertainment", you may be right.

What is up with the article banner text color?

[davidsyes]

You know, it's "NOT NICE to fool Mother Nature..."

My eyes saw:

"Technology: How To Build a Homebrew PS3 Cluster Supercomputer"

"Technology: How To Build a Hebrew PS3 Cluster Supercomputer"

If i wait about 10 seconds and then move my mouse or highlight the text i eventally can read it, but something weird is going on. It's in Firefox AND in Opera...

Re:What is up with the article banner text color?

[Anonymous Coward]

The banner is wonky for me in Opera but not in firefox.

Re:What is up with the article banner text color?

[Renderer of Evil]

"Technology: How To Build a Hebrew PS3 Cluster Supercomputer"

Damn, I saw that too.

  I suppose any PS3 Cluster Supercomputer can qualify as one if you turn it off on Saturdays.

Re:What is up with the article banner text color?

[Anonymous Coward]

Actually you'd turn it off at sundown on Friday, because that's when the Shabbat actually starts

Re:Invader Zim is non-free 23-skidoo!

[hguorbray]

sorry, but that's stupid -how many pop culture references from 1923 are relevant to TODAY's pop culture:

seeya snookums, me and the squeeze are the bees knees in our raccoon coats, we're gonna get jazzed up in our hupmobile on hootch and go check out Mary Astor's horse after we hit the blind pig.

I agree its unfortunate that this stuff is non free, but pre 1923 means that most talkies would be out of bounds as well -including stuff you can see on tv all the time.

I'm just sayin'

WOO!

[azav]

Props for my Alma Mater!!

Drinks at the Sunset Room are on me!

Re:WOO!

[Anonymous Coward]

Get out.

AFRL did it already

[Anonymous Coward]

AFRL built a system early this year with 300 PS3s.

https://www.fbo.gov/index?s=opportunity&mode=form&id=3c9d2849f07a72c7aeca93af2fea0c85&tab=core&_cview=1

I saw a presentation from them on this system at HPEC.

Invited: Case Studies Optimizing Applications for a 50 TFLOPS Cluster of PS3s
Richard Linderman / AFRL
Electronic files not available

At: http://www.ll.mit.edu/HPEC/agendas/proc08/agenda.html

Awesome results.

Re:AFRL did it already

[J05H]

Cool - but is it open source?

Why doesn't Sony make it ?

[zomniac]

I betcha there's enough talent out there to make a small desktop unit. An affordable, out of box, mass produced Sony supercomputer.
Is it possible ? It was done with calculators.

BTW, I still have a sliderule in my desk drawer for some reason.

Re:Why doesn't Sony make it ?

[Malekin]

I betcha there's enough talent out there to make a small desktop unit. An affordable, out of box, mass produced Sony supercomputer./p>

I think Sony feels that's exactly what the PS3 is.

Re:Why doesn't Sony make it ?

[CronoCloud]

Well, you can use a PS3 as a desktop, I do it myself, like I did the PS2 before it. But is this more of the kind of thing you're thinking about:

http://pro.sony.com/bbsccms/ext/ZEGO/ZEGO.shtml

It's a rackmount server though.

Re:Why doesn't Sony make it ?

[Sarusa]

Well the funny thing is they're still losing money on every PS3 sold. At least from the last cost analysis I saw, which was back in July. They are counting on you watching Blu-Ray disks or buying games, and any PS3 in a computing scenario won't be doing any of that (barring someone 'misallocating resources' *koff koff*).

So your idea makes sense. They are partnered with Toshiba to produce the low(er) cost Cell add-ons outside of a PS3. And you should see http://en.wikipedia.org/wiki/Cell_%28microprocessor%29#Possible_applications for some others.

The sad thing is that a Cell is far more suited for supercomputing than it is for writing games on, but Sony seems insistent that the primary purpose is making game devs's lives miserable while IBM and Toshiba seem more focused on using it where it's actually useful.

Re:Why doesn't Sony make it ?

[Anonymous Coward]

I've got about 20 games and 5 blu rays, and that doesn't include the DLC I've picked up. We are out there buying the games, believe it or not!

Re:Why doesn't Sony make it ?

[zomniac]

Wow, that IS interesting !
Thanks for tha link.

http://pro.sony.com/bbsccms/ext/ZEGO/ZEGO.shtml

Power and maintenance?

[lemaymd]

Researchers pay not only for the initial capital outlay required to install a supercomputer, but also for its power, cooling, the building it resides in, and its maintenance. This PS3 cluster might be cheap from the researchers' standpoint if they don't pay for any of these things directly, but I imagine their departments won't be real thrilled if a bunch of researchers start building their own individual "cheap" supercomputers! Those issues aside, it sounds like they're doing pretty cool stuff with those machines, so maybe more supercomputers should be cell-based!

Re:Power and maintenance?

[Anonymous Coward]

Researchers pay not only for the initial capital outlay required to install a supercomputer, but also for its power, cooling, the building it resides in, and its maintenance.

Someone noted that "AFRL did it already". While doing it, they noted and were able to take advantage of:
1) low power consumption compared to a typical 1U unit (or desktop PC) used in clusters. Very low draw when in idle mode.
2) low heat output, and engineered to be tolerant of warm/hot environments found in typical users' homes.
3) low maintenance, with a consumer system designed for low MTBF, and easily/cheaply swapped out if a unit is a lemon.

The AFRL lead investigator also noted that a PS3 ain't the end all/be all. Given the lack of ram, "his" cluster was set to spawn off specific types of processing that a Cell CPU with middling ram could do a good job on, to get a good FLOP/$ ratio. The AFRL cluster mixes in a number of x86 1U systems as job masters.

Re:Power and maintenance?

[CronoCloud]

From what I gather with my limited geekery skills, the PS2 clusters worked the same way, a few x86 boxes to act as masters to keep the PS2's that did the grunt work with their VU's fed with data efficiently.

Re:Power and maintenance?

[ChrisAI]

This is Chris, good point. And yes the power/cooling requirements are non-trivial. And meanwhile, your right about the second assertion. the world's fastest computer (Roadrunner) already uses this same processor. See here: http://en.wikipedia.org/wiki/IBM_Roadrunner Thanks...

Re:Invader Zim is non-free 23-skidoo!

[mollymoo]

sorry, but that's stupid -how many pop culture references from 1923 are relevant to TODAY's pop culture:

A perfect illustration of the fact that copyright terms are way too long.

I wish I had one

[uassholes]

For the dick licks that say it's useless, I guess you missed all the previous articles about scientists who have been doing the same thing:

http://www.physorg.com/news92674403.html

http://dgl.com/itinfo/2003/it030528.html

http://www.lbl.gov/Science-Articles/Archive/sabl/2006/Jul/06.html

http://folding.stanford.edu/English/FAQ-PS3

Re:I wish I had one

[Anonymous Coward]

ahhh. appeal to authority is definitely my favorite sin.

What a ripoff!

[Anonymous Coward]

What a complete farce! Here I was all excited to go see this PS3 cluster "guide". From TFA:

"Found at www.ps3cluster.org, the resource fully illustrates how to create a fully functioning and high performance supercomputer with the Sony Playstation 3."

And what is actually *on* the site?? How to install Linux on a PS3 (as if there weren't any guides for that out there already). Then, they show the magical touch where they download the stock Fedora Open MPI implementation, and configure it using all *TWO THREADS* of the Power PC unit.

No mention that Open MPI doesn't even utilize the synergistic processors on the Cell. No benchmarks. Nada. They can boot Linux, and run a networked application that has absolutely NOTHING to do with the CELL architecture itself.

From the site: "One of the authors (Khanna) estimates that his MPI computations run much faster than on desktop workstation chipsets, and that his original 8 PS3 (i.e. 64 core) Cell cluster had comparable if not better performance to a 200 Node IBM Blue Gene system."

B.S. (And I am being generous.) Their MPI isn't using any 64 processors (when there are actually only 56 available cores for use on the PS3). His data sets may run about as fast as they would on 8 older Apple laptops, but there is no way they're anywhere near a Blue Gene. My tax dollars had better not have been used to fund this "research"....

Re:What a ripoff!

[Anonymous Coward]

It's not cheap, but there's a book on programming the cell processor here.

It doesn't discuss MPI, but it describes IBM's Accelerated Library Framework (ALF), which is a similar methodology of development and deployment.

Re:What a ripoff!

[enslaved_robot_boy]

If you clicked on some of the links you would find some quantitative data hotshot.

ATTN: SHOEBOY

[Anonymous Coward]

vlad farted

Games

[daybot]

If this catches on for research institutions it may increase Sony's sales, but they might not be seeing the corresponding sale of games spike

Come on - that's the whole point. This is what you'll need to run the PS3 version of Crysis!

Re:Invader Zim is non-free 23-skidoo!

[hguorbray]

That's not the point -relevant pop culture deals with stuff happening NOW.

'I like ike' and 'tippecanoe and tyler too' were relevant in their time -not now.

Now its about bling, blogs and obama

if you limit yourself to 50 year old culture references you are missing leet-geek-speak, hiphop/drug culture, simpsons (which is mostly culture references itself) and anything else that has been in the public consciousness for the past 40 years

not that missing some of the above would necessarily be a bad thing...but that's current pop culture.

I'm just sayin'

Imagine...

[toblak]

Imagine a beowulf cluster... oh crap.

yeah

[jaimz22]

but my question is, does it run on linux?

diy distributed computing

[doronbc]

I've thought about Folding @Home and I've always wondered why can't there be a diy distributed computing server that could be setup. Something like this PS3 cluster but could be replicated with any home pc.

My setup for PS3Cluster

[J05H]

I helped Chris with the documentation, testing and image capture on this project. I see some "it doesn't do this!" comments above - please remember this is a young project that started out of one researcher's need to solve a specific type of problem. If you want to see this advance, it's all open source so start hacking.

So my setup:
1 40Gb Playstation3 w/ HDMI cable out and keyboard
Hauppauge HDPVR digitizer
PC running Windoze and Photoshop
TV hanging off the HDPVR for reference

Software as described on PS3Cluster.org including Geoff's Cell libraries, boot image on USB and Fedora 8 for PPC.

Plugged everything together, installed Fedora 6 the first time around since we knew that worked, then redid it with Fedora 8. Added the MPI libraries and ran the little Pi test code. Digitized the whole install as video, proofed out the process in terms of instructions. Did frame grabs from the video, cropped etc in Photoshop. Lots of work, totally worth it seeing the project posted here.

Oh, and it runs X - kinda cool having Firefox running on a game deck.

Enjoy,
Josh

Re:My setup for PS3Cluster

[CronoCloud]

Course it's cool having Firefox on a game console, though I had X and Firefox on my PS2 5 years ago, though it was Firefox was still Phoenix then. :-)

Re:My setup for PS3Cluster

[J05H]

THat's even cooler.

The thing that's nice is that the PS3 is commodity hardware and we know it will be around for a while - and Cell is ridiculously fast for certain operations.

Pretty much useless for most tasks

[melted]

Granted, there are problems which don't require much RAM per core. However, I run my stuff on a 500+ core cluster nightly and boxes there have 16GB of RAM. We're thinking of upgrading to 32GB, actually, to be able to work with larger data sets. 50MB per core is utterly inadequate for 95% of problems out there, and 100mbps interconnect is inadequate for the remaining 4.5%. If this dude falls into 0.5% for whom this could actually be useful, he can pat himself on the back I guess.

Economy of Scale

[inKubus]

Yep, this is a classic example of economies of scale. There are several other versions of the Cell processor out there, many of them designed for high-performance computing (HPC). But they range from 400-3000 bucks a pop. With the PS3, they have basically forced out a much more expensive chip more cheaply because they are delivering so many. And of course they can make up the difference in games.

Memory Bandwidth

[Anonymous Coward]

How much memory bandwidth does a PS3 have? If all SPE's are running code that is not cache friendly, is there even enough memory bandwidth to keep them all happy?

OLD News!!

[Darkk]

Red Dog Linux been doing this for awhile now!

http://www.wired.com/techbiz/it/news/2007/10/ps3_supercomputer

Re:OLD News!!

[Darkk]

Whoops... I mean Yellow Dog Linux... not Red Dog

Déjà vu!

[Kolargol00]

at the Laboratory for cryptologic algorithms.

This is probably a silly question

[Chrisq]

This is probably a silly question but why/how are they running PPC Linux (which is presumably for the Power PC) on PS/3s which have cell processors?

I guess that either the PS3 has a PPC chip as well, or it runs some sort of emulation mode. I can't find either documented.

Re:This is probably a silly question

[brackishboy]

The Cell in the PS3 is basically a PPC core and 7 DSP cores, though I believe they call them SPEs (Synergistic Processing Elements).

Re:This is probably a silly question

[zackhugh]

Actually, there are eight SPEs, but the PS3 only allows users to access six of them.

Re:This is probably a silly question

[Chrisq]

Thanks, that's interesting

Re:This is probably a silly question

[Anonymous Coward]

The Cell processor is just a general-purpose POWERPC plus several SPE's which are vectors processors.

Re:This is probably a silly question

[the 9a3eedi]

The cell uses the same instruction set as PPC, so they're quite compatible.

Re:Message from Government Man

[the_xaqster]

Hmm, (Score:-1, Offtopic).

Looks like the moderators have spoken, it doesn't belong here.

NAMD on CELL

[Pope Raymond Lama]

Since we are talking about this,does any one is using or have any newer news on the molecular simulator NAMD on the CELL Processor? The official development stalled two years ago as its maintainer sinked into other projects, but I do actually help a team with a PS3 cluster which would be very interested in getting NAMD working under full load there.

Imagine...

[danieltdp]

...a beowulf cluster of that. Oh, wait!

Democratic supercomputers for the masses...

[Brad_McBad]

But you can you still not get access to all the cores / whatever they're called?

IBM's version

[sanosuke001]

IBM BladeCenter QS22

If you want your cell system without the PS3's, get a couple of these. Each comes with two Cell 8i CPUs in a 1U case. Upgradeable dedicated processor memory slots and general use RAM slots. A bit more expensive than the PS3's, but might be easier to get the institutions to pay for...

Re:IBM's version

[sanosuke001]

A bit more info I found on the Cell Wikipedia Page

In 2008, IBM announced a revised variant of the Cell called the PowerXCell 8i, which is available in QS22 Blade Servers from IBM. The PowerXCell is manufactured on a 65 nm process, and adds support for up to 32GB of slotted DDR2 memory, as well as dramatically improving double-precision floating-point performance on the SPEs from a peak of about 14 GFLOPS to 102 GFLOPS total for 8 SPEs.

So, I configured 4 QS22's with 32GB RAM each and it came out to $42,144. This would be roughly 10x the price of your PS3 cluster with 8 systems. However, with the updated 8i architecture, you get more RAM as well as the higher performance.

8x PS3s = 112 GFLOPS
4x QS22s = 816 GFLOPS

7.28x the processing power for 10.5x the price. Seems reasonable to me. And if it really does cost them $5,000 per run on the NSF machine, 10 jobs pays for your system.

Re:IBM's version

[Icegryphon]

This seems a little low. I thought the Xblok 360 had more FLOPS then this.

Re:Invader Zim is non-free 23-skidoo!

[meringuoid]

sorry, but that's stupid -how many pop culture references from 1923 are relevant to TODAY's pop culture

More than you'd think. In 1923, Edgar Rice Burroughs published Tarzan and the Golden Lion, A. A. Milne published The House at Pooh Corner, Felix Salten published Bambi, A Life in the Woods, and P. G. Wodehouse published The Inimitable Jeeves and Leave it to Psmith. Pretty much everyone recorded one version or another of Yes, We Have No Bananas, and the year also saw the Charleston dance craze. Charlie Chaplin ate his shoe in The Gold Rush. And to the delight of brewers and distillers everywhere, Brendan Behan was born.

1923-skidoo

[tepples]

sorry, but that's stupid -how many pop culture references from 1923 are relevant to TODAY's pop culture:

Anglophone popular culture is full of allusions to plays written four centuries ago by William Shakespeare. You also find allusions to Twain, Dickens, and other nineteenth-century authors.

but pre 1923 means that most talkies would be out of bounds as well

At least with classic literature, you can quote more of the context to clarify your reference without any risk of crossing the line from fair use to copyvio. Otherwise, if you often allude to works that are still copyrighted, you have to accept the excuse "I don't own a copy of that work."

"I like Ike"

[tepples]

'I like ike' and 'tippecanoe and tyler too' were relevant in their time -not now.

I wouldn't say they're exactly irrelevant. If you know about "I like Ike", you'll better appreciate the cheer for one of the characters in Nintendo's Super Smash Bros. Brawl: "We like Ike! We like Ike!"

if you limit yourself to 50 year old culture references you are missing leet-geek-speak

Not necessarily. Geekspeak is more likely to be under a license for free cultural works.

Old news.

[Anonymous Coward]

This is old, guys. The guys in the office next to me have done this 2 years ago. This is a pretty inefficient machine, mostly due to being utterly complex to program and VERY bad interconnect. The "giga-ethernet" card struggles to get 350Mbps.

Re:Oblig. OMG...

[Anonymous Coward]

Distributed QEMU...
How many PS3s would one need? :)

Cell workloads

[Funk_dat69]

A small handful of workloads for which the Cell chip does very well:

Modified Assignment (MASS, MASSV)
various Fast Fourier Transforms (FFT)
Basic Linear Algebra routines (BLAS)
Image/Video Encoding
Monte Carlo algoritms

It's worth noting that even with CUDA, workloads running on a video card have awful utilization. And general purpose CPUs are not set up to grind out embarrassingly parallel problems.

imagine how much porn I can transcode ?

[Anonymous Coward]

so if I were to set this rig as prescribed, how long would it take me to transcode all the porn on my 1TB hard drive ?

What can I (home user) do with a cluster?

[tbuskey]

I find these supercomputer articles interesting as they touch on techniques that help science.

As a home user, what might I do with a cluster of PCs/PS3s/Cells/GPUs that might be useful?

Are there any "cluster apps" that I might want to run at home?

Re:What can I (home user) do with a cluster?

[hchaudh1]

Folding@HOME comes to mind.