Dell $38m Supercomputer [not] More Costly than VT's G5s

An anonymous reader writes "According to the Austin Business Journal, Dell's 3-teraflop, 600 server supercomputer cluster cost the University of Texas $38 million. As The Apple Turns has pointed out that this is 7 times the cost (and a quarter of the power) of Apple's cluster at Virginia Tech! " Update: 10/14 17:56 GMT by M : worm eater writes "The Register has posted a correction to the widely-reported story that a 3.7 terraflop Dell cluster cost the University of Texas $38 million. As it turns out, the computer cost $3 million, vs. $5.2 million for the 17.6 terraflop Mac G5 cluster at Virginia Tech."

Comments

Here's a copy of the apple turns web page

[eyegor]

Since the Apple site is going to get knocked flat, here's a copy of the page.

Monday, 5:57 PM: Virginia Tech's G5-based supercomputer is (sort of) running-- with 17.6 teraflops of theoretical performance. Meanwhile, Dell tries to build something (sort of) similar, but it winds up with a quarter of the power and seven times the price, and Apple (sort of) announces Xgrid, a product for "parallel and distributed high performance computing"...

Monday, 5:57 PM: Today's holiday episode is now broadcasting. Don't forget to take your shot for a free AtAT shirt (tee or turtleneck) by entering the Q4/03 Beat The Analysts contest; guess closest to Apple's final reported quarterly profit or loss, and you get the garment-- or your choice of creaky old software from the Baffling Vault of Antiquity(TM). You've only got until Wednesday at 4 PM, and in the likely event of a tie the earliest entry wins, so why wait? Enter now!

Up, Running, & Kicking Tail (10/13/03) Fun fact: believe it or not, folks, AtAT's wild success isn't confined to these here United States. No, seriously, it's true! The show actually has semi-regular viewers holed up in such far-flung corners of the world as Iceland, the Dominican Republic, and Delaware-- and for the benefit of those fans, we thought we'd explain that, here in the U.S., today we celebrate a holiday called Columbus Day. Columbus Day, for the uninitiated, is one of our most sacred occasions: a day on which we reflect on the many cultural and technical achievements of the city of Columbus, Ohio. We celebrate Greater Columbus's world-famous quilts, its shrubberies recreating Pointillist masterpieces, and (most importantly) its commitment to the preservation of really old TV sets by wondering why the bank is closed and our mail never came. A good time is had by all.

So if this is such a major holiday, why are we broadcasting, you ask? Well, normally we wouldn't, but faithful viewer Nathaniel Madura pointed out that Slashdot just referenced a BBC World report on that G5 supercomputer down at Virginia Tech, and we're just a little giddy about the existence of a Mac-based cluster than can chew through 17.6 trillion floating point operations per second. Yes, the thing is up and running (at least enough to run performance testing), and reportedly it pumps out 17.6 teraflops of raw perforated aluminum muscle while sucking down enough juice to power 3,000 average homes. Wow, is it getting warm in here, or is it just us? (It's just us-- the G5s are cooled by means of 1,500 gallons of chilled water pumped through every minute. Ooooo, frosty.)

Kudos to the Virginia Tech team who pulled this off, because frankly, this is the sort of technological triumph we'd normally only expect to come out of, say, Columbus, Ohio. Now, what's interesting about that 17.6 teraflop figure is that if you scope out the last compiled list of the world's top 500 supercomputers (from last June), you'll notice that, if 17.6 teraflops is Virginia Tech's "theoretical peak performance" score, it'll probably slot in nicely at number three. (Scores are ranked by "Maximal LINPACK performance achieved," so it's just guesswork so far.) The top-ranked Intel-based cluster is currently ranked at number three, with 2,304 2.4 GHz Xeons and a theoretical peak performance of 11 teraflops. Gee, more processors, a higher clock speed per processor, and 63% of the performance. Now that's efficiency, baby!

We'll have to wait until the next top 500 list comes out in November to see if "Big Mac" (as the VA Techies apparently call it) really takes third place, or if the real-life LINPACK scores push it down lower-- but we figure a top five placement is a safe bet. One of the world's bestest supercomputers made of Macs and running Mac OS X? Why, it's a Columbus Day miracle!

4x The Bang, 1/7 The Buck (10/13/03) Meanwhile, we know that the G5 supercomputer is delivering more pluck per processor than any other supercomputer out there, but what about bang for the b

Re:Here's a copy of the apple turns web page

[IWannaBeAnAC]

Umm, can you provide some links to back this up? NUMA machines certainly offer this capability, but they are relatively rare and this cluster is certainly not NUMA. I have used several high-performance scientific clusters and I have never yet encountered one that is configured in this way. Surely it would require some substantial hardware support?

To the parent, the focus in high performance computing nowdays is on "portable perforamnce"; for the most part, libraries are written so that the performance critical components are either very small, or self-optimizing (or both). For most applications, the most you might need to do is optimize some of the kernels (eg BLAS) for the architecure.

Re:Here's a copy of the apple turns web page

[Anonymous Coward]

Tina Romanella de Marquez, communications and development manager at the Texas Advanced Computing Center (TACC), says that the $38 million mentioned by MacNN is for far more than just a supercomputer.

She said: "The $38M total you refer to was NOT for a single supercomputer. It was announced in February for a total package that included:

"The establishment of the new Institute for Computational Engineering & Sciences (ICES) at UT, including:

four new endowed faculty chairs in ICES at UT
additional funding for the research endowment and the visiting scholars endowment in ICES
the completion of construction of the ACES building (the 4th floor) for use by ICES and TACC

"and the establishment of a terascale distributed computing infrastructure at UT, hosted by TACC, including:

two supercomputers at TACC (the cluster you refer to, and the other IBM system
two massive storage systems at TACC
three leading-edge components to increase UT's networking infrastructure
increases in operations funding over five years for ICES and TACC".

She adds: "There are many more things that were needed to create ICES and establish a terascale distributed computing architecture at TACC. This point was made by TACC Director, Jay Boisseau, during the Lonestar dedication ceremony. The value of the specific computer referred to was approximately $3.0 million. And, no tuition funds were used in this process. Most of the money did not even come from UT. The package included $8M in discounts and donations from about 10 leading technology vendors, and over $15M from a generous foundation." And, she continued: "The VaTech number ONLY includes the actual computer, not the cost of the building, power, cooling, people, or anything else needed to actually operate it."

Hmmm, let's see ...

[Daniel Dvorkin]

UT is in Austin. Dell is in Austin.

Can you say "sweetheart deal," boys and girls? I knew you could.

Shocking!

[Bob9113]

So wait, let me see if I get this straight. Are they acutally implying that supercomputers built starting five years ago are actually more expensive per unit of computing power than supercomputers built today? Why, if that were true, and if the same thing applied to workstations, then you'd be able to get a 2 gigahertz machine today for what a 500 megahertz machine cost five years ago. Ludicrous I tell you. Simply ludicrous.

Paying $126k for each server -- idonfinkso

[gregor_b_dramkin]

$38e6 / 300 servers = 1.2667e+05 $/server

Methinks the price tag includes a lot more than the hardware costs.

The comparison with the VT supercomputer is almost certainly not apples to apples (so to speak)

the extra 30 million

[Triumph The Insult C]

went towards mice with more than one key

Re:Costs

[Eric_Cartman_South_P]

Imagine it, we have ASCII blue, ASCII red and ASCII white guarded by guys with guns... ...Cool.

Wanna really go for cool? Guard the Apples using girls with guns!

$38M wasn't only for the cluster

[jimfrost]

I knew something had to be wrong with that $38M number; 600 Dell servers, good ones, should run in the ballpark of $3M, not $30M. Even with the high-performance interconnect you'd need you ought to be able to do it for under $5M.

So I hit Dell's website and at educational pricing the servers they bought run around $4k apiece. Which means that this solution should be very price/performance competitive with the VT cluster.

I hit the UT page and found that the $38M number came from a press release about their investment in quite a bit of stuff, including the "Institute for Computational Engineering and Sciences (ICES), a new center for interdisciplinary research and graduate study in the computational sciences." I.e. at least one new building.

Re:Costs

[robbieduncan]

Apple did not offer them a special deal on the pricing. They sold them 1100 G5s at standard educational discount. What they did offer was a bump up the queue to ensure that the cluster would be running in time to make the November list.

Re:See?!

[PierceLabs]

And you wouldn't get those same discounts for buying large volumes of PCs? I'm intrigued by this :)

Re:Costs

[sloth jr]

More things that I'm sure people will talk about: The Dells are 1U and 2U boxes designed for rack enclosures meaning they'll be more heat and power effecient not to mention they take up about 1/3 the physical space as the enormous PowerMac G5.

Rack-designed enclosures don't mean that there'll be less heat - 1655MC blade servers, for instance, effectively consume .5 U, but pump out gobs and gobs of heat, and are rated at 6A nominal/12A maximal. Placing 11kw in one 42U rack is enormously difficult to cool. Recent article in ComputerWorld talks about this issue.

Having a tough time accepting the numbers?

[Infonaut]

It's interesting to see how many people in here are getting themselves in a lather refusing to believe that it might cost less to run a supercomputer on Macs than on Dell boxes.

If this were an OS to OS comparison between Windows and OS X, perhaps we wouldn't be getting so frothed up. But this is hardware, and dammit, PC hardware is supposed to *always* be cheaper than Mac hardware!

To summarize what others have said:

1) Dell gave UT a sweetheart deal

2) Apple gave VT a sweetheart deal

3) Nobody has dredged up any information to indicate that the $38M UT spent includes the cost of a building. As csoto pointed out:"A "Center" at UT is a special term for a particular type of organized unit, often a research unit. It does not necessarily mean this place gets its own building. In fact, at UT, space is such a premium that most "Centers" don't have their own (yeah the place is huge, but has lots of people). In fact, I'd venture to guess that NO center has its own building."

4) Hardware is only a portion of the total cost, obviously. UT and VT have set up their supercomputing projects differently. This again is obvious.

5) The really important point of all this is that VT manage to put together a very powerful supercomputing cluster using Macs at a cost that in no way can be considered more expensive than if they'd used PC hardware.

You can argue that costs would have been cheaper had they built their own, or used PCs from some source cheaper than Dell. But they still would have had to deal with labor costs in assembling the PCs, or higher maintenance costs associated with keeping all of those commodity PCs running properly.

UCLA is already using OS X to run Beowulf-style clusters. Tokyo University is replacing over 1,100 Linux PCs with OS X boxes.

Even the total cost of installing, operating, and maintaining large numbers of Macs running OS X is cheaper than either PCs running Windows or PCs running Linux, people often seem incapable of absorbing that information.

You can talk all you want about the Reality Distortion Field, but the truth is that Apple is always working against an incredibly strong bias that says Apple is always more expensive.

That's simply no longer true.