Earth Simulator, G5 Cluster Drop In 'Top 500' List

daveschroeder writes "The November Top 500 supercomputer list has been published at SC2004. Topping the charts is IBM and the US Department of Energy's 'BlueGene/L DD2' beta system, at 70.72 TFlops, followed by NASA's 'Columbia' at 51.87.TFlops. For the first time in several publications of this list, Japan's Earth Simulator is no longer in the number one slot, falling to third. Virginia Tech's 'System X' Xserve G5 cluster, while 20% faster than the original cluster that debuted at number 3 last November, has fallen to number 7 due to the new entries, but remains the fastest supercomputer at an academic institution. Here's an excellent cost comparison (Google cache) of the top machines ('System X' is significantly cheaper than anything else in the top 20, not to mention cheaper than many things far below it in performance)."

Comments

Hmm

[foxalopex]

Err, I'm not sure if the costs can be accurately compared in this way. One needs to remember that a cluster of separate computers acting as a supercomputer compared to a custom designed hardwired system isn't exactly the same thing! Otherwise you can start comparing stuff like SETI which I'm sure is the world's cheapest supercomputer because it technically didn't cost anything to SETI themselves.

Re:Hmm

[Anonymous Coward]

Viruses are the new supercomputers.

Re:Hmm

[Wesley Felter]

All the systems on the Top 500 list are benchmarked running Linpack. If you can run Linpack on SETI@home, you're welcome to count it.

The Dept. of Energy

[aztektum]

I hear they're using it to convert heat into electricity for the rest of the government. Hence their name.

Pizza arguments

[daveschroeder]

A few more thoughts...

Before anyone says "Of course System X is cheaper! Virginia Tech had free student labor to put it together! They paid them in pizza!"

The only thing anywhere close to System X is NCSA's Tungsten, a 2500 processor Pentium IV Xeon Dell Linux cluster. It cost $12 million, just for the asset (comparable to System X's $5.8 million overall price, including the upgrade to Xserve G5s). That's twice the cost, and over 2Tflops less performance. 2Tflops is a top 100 supercomputer...so it's a whole top 100 supercomputer poorer in performance, for an extra $6.2 million.

Another example is PNNL's 1936 processor Itanium2 cluster: 3.5Tflops less performance than System X, for $25 million.

Any way you slice it - no pun intended - System X is still a LOT cheaper, even if you allot, say $2M for professional installation and systems integration - an EXTREMELY liberal estimate, probably by an order of magnitude.

System X also has the highest Rmax per CPU of any system on the list, except for specialty non-commodity systems like Earth Simulator.

And on top of it all, last November, they hit #3 in the world, #2 in the US, and #1 academic, as well as the first academic site to ever exceed 10Tflops, all for less than $7 million in total - including all improvements to buildings, physical plant, and other infrastructure.

That first system might not have had ECC, but what it did do is break into the top 5, following all the rules of the Top 500 organization, for relative pocket change - for a price that was absolutely unheard of, sharing the spotlight with systems that cost $100 million or more - and also catapulted Virginia Tech to a supercomputing center of national prominence overnight, able to attract additional attention, funding, grants, and publicity. Not to mention testing and proving the suitability of a completely new OS, platform, processor, and interconnect for high-performance computing, increasing choice for all (and resulting in new clusters based on the same technology, such as the US Army/COLSA cluster). And even as new systems enter the top ten in the tens and hundreds of millions of dollars, System X retains the title of #1 at any academic institution, and shares the top 10 with the best of the best.

Seems to me that Virginia Tech pulled a real coup here, and a full year later, is still considerably cheaper that anything else. And now, it's being used for real scientific work. To bring a whole new platform onto the scene in essentially under a year and break into the ranks of the supercomputing elite virtually overnight, and to do it significantly, and sometimes ridiculously, cheaper than everyone else, is a feat that can't be ignored.

Re:Pizza arguments

[hernick]

A major cost is power and cooling requirements. According to Apple, a single drive, dual processor 2.0GHz XServe will use about 250W peak. Virginia Tech has 2.3GHz machines and Infiniband PCI-X cards. There's also networking gear and other support equipment to consider. So, we'll use a high figure of 350W per node, giving us a 40% overhead. As for heat production, same overhead, 1200BTU/h per node.

We're going to consider the worst-case scenario, under which we have a 100% load, year round, on all 1100 nodes. That gives us a power consumption of 385kW and 1320kBTU/h of heat generation.

Now, we need to get rid of that heat, and that's going to require a lot of power. My research indicates up to 300kW may be required, but that's a high number and actual requirements may be lower.

So, here we are, with 685kW required for power and cooling. That means a 6000MW/h a year.

Now, the cost of power is high, since you need to amortize and maintain the UPS equipment and the generators. We'll use a figure of 0.15$/kW/h, or 150$/MW/h. Very generous.

So here we are. The absolute worst case for power and cooling. Full load, year round, expensive cooling, overpriced power and amortized UPS and generators.

900 000$/yr. Below a million. It's not that bad, is it ? The real cost is likely below a half-million.

As for the rest, well, how much pizza is really required to entice graduate students and professors to work on that machine ?

Power architecture does well

[blamanj]

5 of the 10 top machines use the Power archtecture, either the Power4 or PPC family.

Re:Power architecture does well

[Brett Johnson]

It is actually 6 of the top 10, and 13 of the top 25.

Re:Power architecture does well

[Brett Johnson]

I think it is interesting that 11% of the top 500 are Power architecture, and 64% of the top 500 are intel based systems. Yet 50% of the top 10 are Power architecture and only 20% of the top 10 are intel architecture. Also interesting is that the Power based systems seem to have twice the Mflop/dollar ratio over the intel systems.

Who has coffee?

[Saeed al-Sahaf]

It's nice that Off The Shelf boxes like Apple and Intel can make a super computer cluster. When do the stories stop? We know that if you put enough PCs together, you get a very powerful machine. What we should be looking at is cutting edge technology in specialized CPUs. Give me 10,000 vanilla boxes and some good custom software, but give me a cutting edge CPU designed for super computing, that's science. We already know that it is possible to fill a fucking building with Pentiums, or better 68000s.

Thanks for that post

[commodoresloat]

That was a close one... I was starting to worry that Apple might be dying again.

Earth Simulator has liberal bias

[Anonymous Coward]

The Earth Simulator computed that global warming will cause major climate change in the next 50 years.

Clearly it suffers from liberal bias.

Re:power costs?

[daveschroeder]

That's easy...the Xserve G5s consume a LOT less power (and therefore generate less heat, resulting in lower cooling costs) than any competitive products (Xeon, Itanium2, Opteron)...and this was true even when they were using the 970 (as opposed to the 970fx they are using now).

Several of the researchers at Virginia Tech have referred to this in various news stories numerous times - one estimate was over two times less power than comparable systems.

Re:VA Tech Supercomputer

[jdog1016]

It's not even technically on campus. Its at the nearby Virginia Tech Corporate Research Facility. And in any case, you can arrange a tour if you want.

It's still running

[Rui del-Negro]

> How much Earth was simulated?

Well, I've noticed a vew glitches (disappearing keys, poor AI in girlfriends, crazy presidents in some countries, etc.), but I'd say most of the Earth has been running reasonably well.

Re:Not to mention that BG beats BigMac in flops/$

[Anonymous Coward]

Recently IBM announced their commercial prices for BG machines (see e.g. theregister.co.uk or news.com.com). Prices start at $1.5 million (1 fully equipped cabinet). Using this price and published linpack figures one arrives at about 2.9 Mflop/s/$, compared to the maximum value of 2.2 Mflop/s/$ he quotes for the best apple system.

Actually, that 2.2MFLOP/s/$ figure for VT's cluster isn't the best for an apple XServe based system. The University of Maine built a 512 CPU cluster of XServes for $680K, or 3.012MFLOP/s/$, according to the price comparison's chart, edging out a BlueGene cabinet.

Of course, now I get to be flamed by a bunch of mac haters who think pointing out a factual error in your statement means I don't know anything else about clustering and will blindly chose a mac above all others.... *sigh*

Re:What is with the Apple fan-boyism?

[xenoandroid]

Apple worship because we all knew they were going to die and are overpriced and whatnot, then we see their products being used to make one of the most powerful clusters in the world for very cheep.

Apple worship because it's a smack in the face to those who still continue to bash Apple for reasons that no longer exist.

"OS 9 sucks!"
"We're on OS X now, and it's unix-like"
"oh...um...Well one button~"
"And yet I'm still more productive on it than my Windows box"
"Well um, I want linu-"
"You can install that too."
"Well, they're slow and-"
"I suggest you actually try using one before saying that."
"They're overpric-"
"Really? I didn't think $900 was that expensive for a mid-range machine."

We do the Apple worship thing just to fustrate the anti-mac crowd even more.