Cray XT-3 Ships

anzha writes "Cray's XT-3 has shipped. Using AMD's Opteron processor, it scales to a total of 30,580 CPUs. The starting price is $2 million for a 200 processor system. One of its strongest advantages over the std linux cluster is that it has an excellent interconnect built by Cray. Sandia National Labs and Oak Ridge National Labs are among the very first customers. Read more here."

Comments

imagine a...

[Anonymous Coward]

single node of those.

Re:imagine a...

[catch23]

Man a single node is $20,000?? I could build a pretty good opteron system for $2000.... so the other 18 goes to the interconnect? $2 million is a lot for only 200 processors.

Re:imagine a...

[Anonymous Coward]

*rolls eyes*

When you have a single CPU, designing the system to be pretty fast is easy. There's no major contention to deal with.

Two CPUs? Slightly harder, but reasonably straightforward. You don't see a 2x improvement in speed over one CPU, but it's around 1.95x, give or take a bit.

Four CPUs? Now you're starting to see less improvement ... probably around 3.2x, because of all the contention issues.

Sixty-four CPUs? You'll be lucky to get a 50x speed up over a single CPU.

When you get to 200 CPUs, the issue of access to shared memory and other shared resources becomes critically important. It's also an issue that most computer buyers don't need to worry about, because they don't have 200 CPUs in their system. This means that you have a lot of highly specialised research going on, and relatively few buyers to spread the cost of that research over.

Two million for a 200 CPU box which has low latency, low contention, and solid reliability is not a lot at all. You might not buy it. That doesn't mean nobody will.

Re:imagine a...

[crimsun]

It's not just hardware: the amount of non-parallelizable code in parallel applications impacts scalability most tremendously.

The upper bound on speedup is generally Amdahl's law. Plainly, the efficiency approaches zero as the number of processes is increased. Generally we consider the major sources of overhead to be communication, idle time, and extra computation. Interprocess communication is considered negligible for serial programs in this context (we consider message passing). Idle time ends up contributing to overhead, because processes idle awaiting information from others. Extra computation is virtually unavoidable at some point; for instance in MPI's Single Program Multiple Data model, each process in tree-structured communication other than the root is eventually idled prior to the completion of computation, and each process determines IPC at some point based on rank.

There are notable exceptions to Amdahl's law, however; Gustafson, Montry and Benner wrote about such in Development of parallel methods for a 1024-processor hypercube, SIAM Journal on Scientific and Statistical Computing 9(4):609-638, 1988.

Re:imagine a...

[ant_slayer]

My apologies, but I couldn't help but think that you'd be *really* lucky to get 50x out of 64 CPUs. Examine the following:

1 CPU @ 1.00x -> 1.00 / 1 = 1.000
2 CPUs @ 1.95x -> 1.95 / 2 = 0.975
4 CPUs @ 3.20x -> 3.20 / 4 = 0.800
64 CPUs @ 50.0x -> 50.0 / 64 = 0.783

Pop that into an OpenOffice.org spreadsheet and look at the graph.

That is not linear, in fact, it's non-linear in the direction that *helps* more and more processors. If the decline from 4 CPUs to 64 CPUs is a mere 1.7% efficiency compared to the 17.5% drop from 2 to 4, then, by golly, I'm going to cram hundreds of CPUs in there and see it tail off. Hello amazing performance.

Instead, reality is that the dynamics change. You can't evaluate "equivalent performance" to a single processor system. There is no reasonable metric with which to do so.

-Ant Slayer-

Re:imagine a...

[thegnu]

Isn't there a point at which spending what we now consider an excess of cpu cycles on directing the flow in processes is more efficient? My oh-so-simplistic analogy is the jump from 2D to 3D graphics took a while, but it became more efficient to create frames from renderings of a massive 3D construct than a frame-by-frame drawing.

So, sure, there's more waste at the current point in the spectrum, but won't there be another curve in the opposite direction at some point? I think just the fact that 5 years ago, 5 processors was less efficient than now is an indication of things to come. There are constantly incredibly brainy people talking about the wall we're pushing up against, and we still haven't hit it.

Are we at the point where we have dedicated process processors (doing load balancing and only load balancing)? Keep in mind I know nothing about the subject at hand.

I'm fairly young, so my exposure to the advancement of computing technology is limited, but I haven't seen the indication that we're hitting a wall really. It might take some reevaluation, but when was that a bad thing?

The Bush Administration sez: "When yer president!"

Re:imagine a...

[Craig Davison]

Come on, he was obviously pulling the numbers out of his ass. Like those people who keep saying "99%" when they mean "the vast majority, but I don't have any hard numbers beyond that".

Comment removed

[account_deleted]

Comment removed based on user account deletion

How big is it?

[rooijan]

I read the article (okay, so I kinda read it :-) ) and it has the speed and specs to be a geek's improvement on sliced bread. But how big is it, physically?

The article doesn't appear to mention its dimensions, and I'm curious to know what kind of space you need to install this baby. Anyone got any idea?

Re:How big is it?

[Anonymous Coward]

Dimensions (cabinet): H 80.50 in. (2045 mm) x W 22.50 in. (572 mm) x D 56.75 in. (1441 mm)

Weight (maximum): 1529 lbs per cabinet (694 kg)

http://www.cray.com/products/xt3/specifications. ht ml

Re:How big is it?

[commodoresloat]

The weight is 1529 lbs per cabinet (694 kg). Imagine lugging that up to your 5th floor walkup apartment....

Re:How big is it?

[Max Romantschuk]

Sibling posts already gave the spec, but I believe the size of a system like this is rather insignificant.

If you're paying 2 million and upwards for a thing like this you probably can afford an appropriate space with appropriate climate control.

(OK, so some people cram ${car_price * 10} worth of HIFI equipment into a ${small_japanese_car}, but I doubt anyone would want one of these installed in their closet...)

Re:How big is it?

[Anonymous Coward]

Dimensions (cabinet): H 80.50 in. (2045 mm)

Wow... for the first time in my life, I couldn't picture 80 inches, but I could 2 meters. I think there may be hope in the metric system afterall.

Re:How big is it?

[Seanasy]

The 2000 processor version (10 teraflop peak) going to Pittsburgh Supercomputing Center is about the size of a living room (12'x28'). Compare that with PSC's current HP machine (~3000 processors, 6 teraflop peak) which is the size of a basketball court.

Re:How big is it?

[afidel]

Actually as IBM is trying to prove with Blue Gene/L thermal density is a limiting factor in compute power. Also roughly 2/3rds of the cost of a supercomputer is operating budget which includes electricity both for the devices and ~3x as much for cooling, plus the cost of the facilities upgrade. In fact I know several academic institutions that have turned down one generation old supercomputers that would have been free because the facilities upgrade and operating costs were too high for their budget.

I'll pass for now.

[mrjb]

This is only the XT-3. I'll wait for the Pentium-3-4.

Re:I'll pass for now.

[Pleione]

Don't you mean "AT/ATX"?

we're getting closer...

[nilbog]

A few more years of advances like this and we might have a machine capable of running Longhorn!

Re:we're getting closer...

[metlin]

Ahh, now that's what I call an optimist.

Re:we're getting closer...

[NanoGator]

"A few more years of advances like this and we might have a machine capable of running Longhorn!"

A few more years of computer advances and this joke will still be modded funny!

Re:we're getting closer...

[Pleione]

Oh, quiet you...it was funny...sort of...well not really but you know what I mean?

Re:we're getting closer...

[NanoGator]

"Oh, quiet you...it was funny...sort of...well not really but you know what I mean?"

It was funny like a year ago. Now it's as overused as an SNL skit.

Re:we're getting closer...

[relaxmax]

You say Longhorn... I say games, OS's, VM's, Office stuff, Audio/Video utilities, yada yada yada.

Gimme back those PC-XT days when we had to boot the computer of a 5.25" floppy, followed by Digdug or Pacman. Even 0 B-KB-MB-GB HDD was enuff...

** sigh **

Re:we're getting closer...

[provolt]

Ah the joys of youth.

Back in my day we spelled "enuff" without the 'f' character and it was good enough for us.

Re:we're getting closer...

[drinkypoo]

A few more years of computer advances and this joke will still be modded funny!

A few more years of computer advances, and this joke will still be based around a kernel of truth.

Re:we're getting closer...

[Tony-A]

The hardware giveth.
The software taketh.
That's the long and the short of it.
Hornwise, that is.

$2 million for a computer?

[commodoresloat]

It better have a lot of good games. How many mouse buttons does it have?

I can't believe people complain about the price of iMacs....

Re:$2 million for a computer?

[Maniac-X]

It's a supercomputer. You use it for taking over the wor- I mean, things like rendering a feature-length CG movie. Or doing the special effects for Star Wars ep. 3 _

Re:$2 million for a computer?

[Klar]

How many mouse buttons does it have?
Please.. it doesn't have any.. it just *knows* what you want to do before you *know* what you want to do..

real FPU operations

[Barbarian]

How are the Opterons at standard FPU operations in double precision? SSE2 and friends are nice, unless you have to make compromises in your simulations.

I ask, because I remember that the Athlons beat the pants off the Pentium 4's in FPU operations, so all the benchmarks were rewritten to use SSE2.

Re:real FPU operations

[jmv]

Opterons beat the pants off the Pentium 4s in x87 (i.e. old) FPU operations. If you want to get good performance, you need SSE/SSE2. Both for AMD and Intel. For pure SSE, the Pentium 4s beat the Opterons mainly because of the clock speed, but for multi-processor systems, the hyper-transport and all more than makes up for that.

Re:real FPU operations

[bersl2]

I seem to remember it being the other way around.

Anyway, it depends on how you're using the floating point numbers: the standard 387 FPU instructions are faster, but the superscalar operations are more efficient when used in their intended role of vector calculations.

Or so I've heard. YMMV.

(BTW: nice sig...)

Re:real FPU operations

[Pleione]

Isn't the P4 a generation behind the Opteron? What about 3DNowPro!, doesn't that stand for something?

Re:real FPU operations

[ceeam]

Yeah, yeah, SSE... Nice, if you can manage your data properly aligned and don't mind manual assembler coding (even intel's compiler does so-so job at vectoring). Still, even then you don't get trigonometry, for instance.

Re:real FPU operations

[Dogers]

I think 3dnow pro doesnt really include enough "extra" stuff to be worthwhile. It's mostly made up of the SSE instructions..
http://www.amd.com/us-en/Processor s/ProductInforma tion/0,,30_118_756_3734%5E3738,00.html

Re:real FPU operations

[jmv]

Both SSE and 3DNow! get you (in theory, at best) two adds and two multiplies per clock cycle, even on an Opteron. So yes, just because of the clock, the P4 beats the Opteron in the case of pure (no memory/cache access, no depencency, nothing else) float operation. Now, in real life, you sometimes spend longer waiting for the data than computing with it and that's how the Opteron quite often comes out on top, especially for multi-processors.

Re:real FPU operations

[jmv]

Couple facts about SSE:
1) You can use it in scalar mode, in which case it's almost like x87, only a bit faster because:
a) It doesn't use a braindead register model (stack)
b) On P4, you can do a mul and an add in parallel with SSE, but not with x87
2) You can use SSE intrinsics. It's not as easy as "normal" programming, but easier than assembly and almost the same speed.
3) Unaligned access is possible. It's slower than aligned access, but overall better than non-vectorized code.
4) Trig is so slow that SSE/x87 doesn't matter (unless you write approximations, in which case SSE will also be faster).

Re:real FPU operations

[Maniac-X]

http://www.tomshardware.com/cpu/20030422/opteron-1 9.html Looks to me like the opterons pretty much beat the pants off of barton 3000+'s in most FPU ops, but a few are only by a slim margin. Maybe that has something to do with the fact that there's two opterons, and only one barton. Its probably better to compare the dual opterons to the dual xeons... in which case the opterons did 2-3x better, in everything but mflops. But that probably has to do with the xeon's having almost twice the frequency of the opterons. (the xeons were 3.06GHz, and the opterons were 1.8GHz)

Re:real FPU operations

[jmv]

Yes, I've been experiencing that too while optimizing Speex. The main reason I think is that the gcc folks have been working on optimizing x87 code for more than 10 years, while SSE is pretty recent. Actually, until around 3.3, -mfpmath=sse would simply generate bad code on Speex. Anyway, I guess it just means that more work should be put on improving scalar SSE code, not just dumping the idea.

Heat...

[Gurezaemon]

I can't even begin to imagine what sort of heat this thing would put out, or what sort of power requirements it would have.

Re:Heat...

[rctay]

These type systems either use floor ducting on a AC system designed for multiple cabinets/rack, or a dedicated freon cooling system. In the days of mainframes you would have a dedicated chiller system just for the computer room. The air and power was routed in by a modular floor grid system.

Just the name brings back memories

[Dancin_Santa]

In this day and age of very fast computers and clusters built in our basements, there sometimes comes along a story that whispers of the computing age of days long past. Cray is one of those names that can drop a jaw just by the mere utteration of the name.

The name is synonymous with speed and power and the unwillingness to cut corners in order to shave a few dollars off the final product. When you buy a Cray, you know you are getting top of the line hardware.

It looks like Sandia wants to build the fastest supercomputer in the world by clustering a few of these monsters, and I have no doubt that they will. Looks like more fun articles about this in the future. :-D

There are two prominent applications for these machines. The first is nuclear weapons simulation. Personally, I don't see the point to that. The other application is in weather prediction. By feeding in current weather variables into a well-written model, a supercomputer is able to predict to a large degree of accuracy the future weather. Such an application will always be welcome.

I think I'm going to have to fire up the old ][e, the nostalgia is killing me!

Re:Just the name brings back memories

[joib]

There are two prominent applications for these machines. The first is nuclear weapons simulation. Personally, I don't see the point to that. The other application is in weather prediction.


Oh, please. Buy a clue, will ya? There's lots and lots and lots of applications that use supercomputers, or could use if they were more affordable. A few examples from the top of my head:

Materials science, that is ab initio simulations, moldyn, you name it. This alone probably uses > 50 % of all supercomputer cpu time in the world. By comparison, weather prediction and nuke simulations is small potatoes (or shall we say, the simulations as such are big, but the number of people engaged in weather prediction or nuke simulation is really small compared to all the supercomputing materials scientists).

CFD, the automobile and aerospace sectors are big users.

Electronic design.

Seismic surveys, the oil industry uses lots and lots of supercomputers to find oil deposits.

Biology. Gene sequencing, moldyn simulations of lipid layers and whatever.

Climate prediction, somewhat related to weather prediction. Official purpose of the Earth Simulator.

All of the examples above could easily use almost any amount of cpu power you can throw at them. The only thing that stands between a lot of scientists and improved understanding of the world is computing power.

Re:Just the name brings back memories

[LiquidCoooled]

There are two prominent applications for these machines.

Wrong! There is a third, more used application: Solitare.

Even super computer coders have to wait for results.

I also asked this recently, but didn't get a reasonable answer, do these beasts have screen savers? if so, Are they just blackout type, or busy 3d rendered whizbang super cool ones "Just because we can"?

(I realise you may not be able to answer that, but someone might)

Re:Just the name brings back memories

[LiquidCoooled]

No, I asked this because this machine cannot be active and at 100% cpu usage all the time.

During the times of no activity what does it do?

Now, since you answered my question (running on x allows any of the screensavers), I will have to thank you Mr AC. :)

A secondary follow on question now springs to mind, how quickly would the SETI saver run on one of these boxes, has anybody benchmarked it?

Re:Just the name brings back memories

[ocelotbob]

I also asked this recently, but didn't get a reasonable answer, do these beasts have screen savers? if so, Are they just blackout type, or busy 3d rendered whizbang super cool ones "Just because we can"?

3d screen savers, of course. In fact, this entire universe is actually a screen saver simulation running on one of these boxes. Just hope that no one decides to move the mou

Re:Just the name brings back memories

[capmilk]

During the times of no activity what does it do?

It creates random noise that is then fed into the Seti project so our computers have something to do in times without activity.

Re:Just the name brings back memories

[fitten]

I dunno... many of the HPC machines I've seen use batch schedulers and do, in fact, run at near 100% CPU utilization for weeks on end. When you have 1000s of scientists from all over the world all queuing up jobs to be run and each job takes a while, the machine tends to stay busy. We even had jobs that took multiple days to run even when running on 64+ CPUs (all of them busy at 100% for as long).

Many of these machines at the national labs (Sandia, ORNL, WES, etc.) are actually used by scientists all over the USA, if not the world, so it isn't like the machine only runs the jobs that the folks at the site need run.

Re:Just the name brings back memories

[Atzanteol]

During the times of no activity what does it do?

Saves energy running idle? This thing takes a lot of power to keep running, and wasting it on a screensaver probably isn't very smart. I'd be surprised if X windows were even installed on this thing. This isn't your home computer!

Re:Just the name brings back memories

[droleary]

There are two prominent applications for these machines. The first is nuclear weapons simulation. Personally, I don't see the point to that.

Well, when you nuke the site from orbit, you do want to be sure don't you?

Re:Just the name brings back memories

[flaming-opus]

Actually, there is no reason to cluster a few of these. If you have a 2000 node xt3 (or t3e, paragon, blue-gene, cm5, insert mesh-structured mpp here) and a 4000 node xt3, you stick them together and make a 6000 node xt3. But that's just picking nits.

Curiously the xt3 IS about shaving dollars off the price. If you go read the origional whitepapers on the system, they go through EXTENSIVE cost-return analysis. They studied their (then-) current generation of cluster systems, as well as future linux/solaris/aix clusters, and rejected them as (interestingly) FAR TOO EXPENSIVE, once the administrative costs are factored in. They then looked at, and rejected, cray's vector solution, the X1. They then decided that the (amazingly) most cost effective solution was to underwrite cray's product development cycle on a wholey new product. Basically they asked for an update to the system they already had. (asci-red i.e. intel paragon++) Nobody was building such a thing. Since cray had a really strong similar product in the 90s. (T3D, T3E) the department of energy asked them to create an update. Some designs never die.

What I'm most interested in is the reliability. One of the biggest difficulties in the T3D engineering cycle was dealing with memory failure. red-storm is going to have 10,000 processors. Lets assume each has 2 banks time 3 dimms (chip-kill) of memory. That means there are 10,000 x 6 x 18 = 1 million+ memory chips in the system. IF 1/100th or a percent of these fail, that's still a lot of memory failures. How well are faults isolated? That's the big question for systems this big.

I'm also a little wary of cray's use of lustre. I've used lustre before, as well as other cluster-FSes. While I'm not aware of other filesystems that will scale to 700+ i/o nodes, I'm not confident in lustre. It's an immature product at best. (I don't mean to disparage the people working on it, it's a neat architecture, but it's a hard problem, and I'm not sure it's ready for prime-time.)

Re:Just the name brings back memories

[LiquidCoooled]

Excellent. Thanks for the insight.
It certainly makes sense.

(As you can probably tell, I haven't had any experience of big rigs)

Re:Just the name brings back memories

[caveat]

...nuclear weapons simulation. Personally, I don't see the point to that.

Would you rather we resume testing to insure the continued viability of the stockpile? I agree, humaity would be better off without the things, but if we're going to keep them around we might as well be as sure as we can that if we do ever have to use them, they're going to work; personally, I'd much rather see the DOE doing things like this that have trickle-down benefit instead of vaporizing small islands in the South Pacific.

Re:Just the name brings back memories

[psetzer]

During times of no activity, they find something for it to do. Typically, if you've got a fast enough computer, there's people lining up just to get their hands on the thing for a few milliseconds. You use something like a workstation to talk to it, and it spends all of its time running FORTRAN or C.

Re:Just the name brings back memories

[Inthewire]

"There's more than just speed"
-- Seymour Cray

Re:Just the name brings back memories

[ScrewMaster]

Yes, Corporal Hicks, indeed you do.

But the real question I want an answer to is: can this thing play Duke Nukem 3D Atomic Edition? On a really really big screen? Now there's a weapons simulation for you.

Re:Just the name brings back memories

[d474]

My angle on this issue of nuclear testing/simulation is that we (USA) tested these weapons for decades both in reality (South Pacific/Nevada/New Mexico)and in more recent decades in supercomputer simulations. How much more testing does it take to come to the conclusion that when you drop a big F'n bomb on a target that the target and everything around it for miles are totally eviserated?

After all, when the proverbial shit hits the fan and these weapons are used in any level of escalation, the subtle minutiae of precise details involving effective yield, fallout, etc., aren't really going to concern us - we'll be dead or dying anyway. Interesting how much money we are all willing to sacrifice to study the forcasts of our own self-imposed demise.

The world is what you make of it.

cray

[Exter-C]

Cray was at one time the fastest system.. then it went belly up.. now its back.. lets hope its here to stay for the long term. With some of the technology flowing through it would be a great thing for the industry.

~If only HP would keep the Alpha on and continue the development rather than crush one of the only real risc procs.

Re:cray

[Anonymous Coward]

Cray never went "belly up". It was acquired by SGI around 1997 or so, then divested and merged with Tera, who renamed the resultant entity "Cray Research".

Although it's true that Cray was not growing strongly before the SGI buy-out, it was not failing either. It could have kept running quite happily for many years, but in the bizarro-world of Wall Street, a company which is not growing is dying. I so love it when economists use biological terminology for corporations. In Wall Street's thinking, the only healthy growth would be a cancerous tumor.

Anyway....

The whole SGI-period of Cray is actually quite fascinating, and I suspect the true story will never be fully known. Lots of SGI engineers had their non-Cray technology branded with Cray marketting names, most egregiously LegoNet becoming CrayLink. Lots of Cray folks - aka. Crayons - felt that the core of their company was gutted by an SGI operation which didn't care for the extreme high-ends of HPC.

One rumor I heard, from a well-placed source, is that the Cray merger with SGI was primarily arranged by the USG. The intelligence services have huge investments in both company's products, so the merger between them made sense. I was told that as a quid-pro quo, the USG had an in-principle agreement to continue purchasing Cray gear to provide enough revenue inside SGI to keep both Cray architectures alive. However, certain parts of SGI felt that the US government didn't live up to their agreement, negotiations to rectify that weren't successful, and so SGI management defunded significant aspects of the Cray engineering work.

Also, FYI, Cray is one of those companies which will never totally go "belly up" anyway. Given the sensitivity of the work which they did, their support databases alone are full of sensitive and/or classified information. Should the company cease trading, it would be acquired by a shelf company whose sole function is to ensure this data would remain private. That's been the fate of almost all of the now-defunct supercomputer and high-end graphics companies who formerly supplied the defence and intelligence market.

Re:cray

[Fred_A]

It seems to be really lacking in the blinkenlight department though.

What good is a supercomputer without blinkenlights ? They just don't make them like they used to...

Re:cray

[flaming-opus]

Pre-SGI cray needed help, badly. They just had too many product lines for a post-cold-war supercomputing company.
They had the C90-to-T90 transition, which they were doing almost completely by themselves. Even IBM had given up on ECL. T90 was fast, but it was really really expensive. 32 processors sharing a flat memory is just too daunting, even without caches.
The J90 was profitable in the end, but took a lot of capital during the design work. Furthermore it detracted from the T90's transition to IEEE fp.
The T3D was a really cool product, but the engineering costs were huge.
The 6400 was a really great product, FOR SUN microsystems. Why did cray think they could sell a database server better than IBM or sun could?

It's not that they didn't have a pot to piss in, it's that they tried to piss in 25 pots all at once. Lo and behold, they got their feet wet. Shock and amazement.

It's too bad that sgi didn't use the cray division very well. They had a really good opportunity to blend some amazing technologies. Cray had the old, wool-suit engineers who had tackled hard problems for a long-long time. SGI had the wild breed of california technologist from the early RISC days. The sum of the parts was huge. Alas the combined whole was a disaster.

Cray today looks like they are doing cool things. xt3 is pretty neat. I'm curious to see how they combine it with the x1 vector stuff. Hopefully their three current product lines can share a lot of technology in future revisions. They just need to avoid getting too big for their britches. The supercomputing market is just not big enough to support anything too massive. Especially when you have to go head-to-head with IBM.

Re:cray

[joib]

Cray today looks like they are doing cool things. xt3 is pretty neat.


XT3 certainly is a very interesting architecture. However, I do wonder how it will succeed economically. It's trying to squeece in on a pretty crowded market, with both clusters on the low end and blue gene very shortly providing superior density at very competetive prices.


I'm curious to see how they combine it with the x1 vector stuff.


Crays roadmap indicates that they plan to share tech as much as possible between the X1 and the XT3 families, to the point that sometimes in the 2010 timeframe the only difference will be the cpu boards. Everything else will be the same, interconnect tech, software, etc. Even going so far that you could have different cpu boards in the same system!


Hopefully their three current product lines can share a lot of technology in future revisions.


Do you mean XD1 as the third line? Personally, I think XD1 is somewhat of the black sheep. As I see some significant overlap between XD1 and the XT3, perhaps they simply bought Octigabay to avoid them meddling in their future XT3 market?

At the moment, I think Crays trump card is the X1 family. Most users have apparently been very pleased with the performance of the systems on real world codes. Also, the only competitor in the vector business is NEC, so they can probably sell X1:s with higher margins than the XT3. Soon they'll have their X1E ready, with 3 times the compute density compared to the X1. That is, the X1E will be within a factor of 3 of the compute density of the Blue Gene! The NEC SX-8 as such seems like a very competetive cpu to the X1E cpu, but NEC has an appalingly low density of only 8 cpu:s per cabinet whereas Cray crams 128 in a single X1E cabinet.

Re:cray

[flaming-opus]

Octiga bay is certaintly an odd buy, but it doesn't seem to get in the way of the big guns. It really is more of a linux networx sort of product, with a similar level of sophistication. They can keep the division largely seperate and be happy, I would think. The cray name adds credibility to the product, and it addresses a market segment that isn't served by the current cray offerings. Obviously the institutional investors agreed with you, and dropped cray stock 20% after the acquisition. I definately agree that it's a bit of a black sheep, but maybe that's okay.

I think the X1 is an amazing machine. However I don't think it's true that X1 is only competing with NEC. Most big codes have been ported from vector archs to MPP or cluster code. If you read the comparisons Oak Ridge did, they are comparing X1 to altix and IBM p-series. The X1 does very well compared to scalar systems, but they also have to compete on cost. They need to keep up a critical mass of sales to pay for the custom engineering.

They also need to do something about the terrible, terrible scalar performance of the MIPS-derived X1 cpus. While it would be cool to have a mix-match system with some vector boards and some scalar boards (presumably the vector boards have multiple parallel interconnect networks), I'd rather see an X1 with an opteron as the scalar processor. (Not that this seems reasonable, as the scalars in the X1 can access the vector registers in 3 cycles, or something like that. totally impossible even on hypertransport)

How big it is

[commodoresloat]

from TFA -

Dimensions (cabinet):

H 80.50 in. (2045 mm) x W 22.50 in. (572 mm) x D 56.75 in. (1441 mm)

Sorry to reply twice but I forgot this detail.

You don't have to begin to imagine

[commodoresloat]

You could just read on the spec page: Power: 14.8 kVA (14.5 kW) per cabinet. Circuit Requirement: 80 AMP at 200/208 VAC (3 Phase & Ground), 63 AMP at 400 VAC (3 Phase, Neutral & Ground) Cooling Requirement: Air Cooled, Air Flow: 3000 cfm (1.41 m3/s) Intake: bottom, Exhaust: top.

Re:You don't have to begin to imagine

[fbform]

More interesting is this spec:

Acoustical Noise Level: 75 dBa at 3.3 ft (1.0 m)

For comparison, that's roughly the same as an average vacuum cleaner when you're operating it, or maybe a good-sized pickup truck passing you in the next lane.

And remember, this value is *per cabinet*. You have to do a weighted sum over all the cabinets in an installation to get a true dB level. I wonder whether the maintenance people will have to use noise-level exposure limits for this baby.

And here I was, complaining about the quiet whine of my PC's fan.

Re:You don't have to begin to imagine

[pchan-]

Power: 14.8 kVA (14.5 kW) per cabinet.

that's amazing. how did the cray guys get a kilovolt-ampere that is not equal to a kilowatt? just goes to show you the power of fast interconnects.

Re:You don't have to begin to imagine

[Anonymous Coward]

it is quite possible for kVA != kW, it all depends on the relative phase of the current and the voltage...

14.5kW would just be the real component of power, i.e. heat dissipated
this leaves - using pythagoras' theorem - an imaginary component of power of 2.96kVA

Re:You don't have to begin to imagine

[wronskyMan]

Disclaimer: IANACEBIATAPEC (I Am Not A Cray Engineer But I Am Taking A Power Engineering Course)
It's fairly common to get a KVA !=KW.
Overall power used by a load is expressed as S=P+jQ, where P is the "real" power and Q is the reactive power (capacitive/inductive from motors, fluorescent lamp ballasts, etc).

While the "units" of S, P, and Q are power=voltage*current, S is generally expressed in VA, P in W, and Q in VAR(volt-ampere reactive) to differentiate the variables. Because the magnitude of S=sqrt(P^2+Q^2), S will always be greater than or equal to P (in this case, 14.8kVA=sqrt((14.5kW)^2+(+-2.965kVAR)^2)

Re:You don't have to begin to imagine

[ch-chuck]

you can even buy power factor correction power supplies if your Q gets too big, as the power company does charge for imaginary power.

Re:You don't have to begin to imagine

[ppz003]

OSHA requires hearing protection for those continuously exposed to 85 dBa and above. So, not quite.

Re:You don't have to begin to imagine

[ZigMonty]

Look up the term power factor. This system has a power factor of 14.5/14.8 = 0.98, which is surprisingly good for a computer, IIRC.

Re:You don't have to begin to imagine

[HawkingMattress]

Not that noisy, we have two HP ML370 here that makes at least as much noise as your typical vacuum cleaner (didn't measure it, but still...). (They have debian running , and hp fan control stuff will only work on redhat or suse, so they're always blowing at full capacity for now. And there are around 15 fans in there). Still, they're just very high availibility pcs, so it wouldn't have shocked me to hear that the crays make as much noise as a jet engine...

Opterons and PowerPC together

[Henriok]

It seems that the XT-3 not only use Opteron processors but they also use PowerPC 440 co-processors from IBM to off load inter-processor communication from the main computing CPUs. Quite an interessting set up.

The XT-3's biggest comptetitor in this segment must be the BlueGene/L type super computer made by IBM. The processors in Blue Gene/L is a custom built dual core version of the PowerPC 440 with built in high speed interconnects.

Just like IBM have a finger in all the future game consoles, they seem to have a finger in several of the next generation super computers also. Nice going IBM.

Re:Opterons and PowerPC together

[Shinobi]

No. The biggest competitor to the XT3 will be machines like the NEC SX-8, their own X1 family or the IBM p690's. They are all shared memory systems, while the Blue Gene family is not. And therein lies a whole world of difference.

Re:Opterons and PowerPC together

[evilviper]

Just like IBM have a finger in all the future game consoles, they seem to have a finger in several of the next generation super computers also. Nice going IBM.

It's not that they're the best thing since sliced bread, it's mainly that all their competition went down the chute for one reason or another.

HP/Compaq/DEC was the king of supercomputers. Now they're only supporting their formerly glorious products, with practically nothing new comming to replace it.

Sun seems to really be sitting on their ass.

Intel was trying, but screwed the pooch with Itanium/Itanic.

SGI was a competitor, but they've just faded out.

Motorola could compete if they put some effort into it, but they've been out of it for some time.

etc.

Re:Opterons and PowerPC together

[pchan-]

let's see what you're missing:

* first, sgi still makes and sells supercomputers, they are far from faded. they also own cray (or did).
* tandem, bought by compaq, we all know what happened there.
* hp sells a superdome once in a while. but nobody seems excited about their itanic systems.
* sun, rotting with their out of date cpus.
* fujitsu is doing well in the supercomputer market.
* nec is also successful.
* ibm, of course.

and you mentioned motorola? you're joking, i hope.

the largest purchasers of supercomputers in the world - national labs and the nsa, like to buy american hardware. they've always had a hand in keeping the industry afloat. notice that the big labs tend to round-robbin their supercomputer vendors so that they buy a machine from each vendor.

Re:Opterons and PowerPC together

[Dusabre]

etc.

Sounds like an opening for...

Google!

Now that I think about it... they have massive experience with huge data systems!

Re:Opterons and PowerPC together

[flaming-opus]

Basically this is the same off-load as blue-gene. One processor with a MPI off-load engine. In the case of blue gene, the main cpu is another 440, while xt3 uses a much stronger opteron. (of course the IBM solution is less expensive, and much denser).

The real difference in this system is the high bandwidth shared memory. Blue Gene has hardware support for shared memory, but the software appears to be strictly MPI based. (at least in the first revision, and according to what I've read, this may be preliminary data only).

Re:Opterons and PowerPC together

[Wesley Felter]

Neither the XT3 nor BlueGene/L have shared memory. The XT3 supports Cray's "SHMEM" API, but that's not really shared memory as we know it.

Re:Opterons and PowerPC together

[Bert64]

And according to the last benchmarks i saw, the Alpha EV68 and EV7 chips, despite no development for years, beats the itanic at the same clockrate..
Ofcourse, HP are trying to de-emphasize the Alpha and won't be publishing the benchmarks showing what it can truly do..
As for huge multiprocessor systems, the interconnect has a HUGE impact on the overall performance, especially linpack benchmarks

Re:So......the cost compared to?

[the_2nd_coming]

X-serve clusters would be cheaper, but I think that Cray has the edge n the interconnect tech. So, you need massive bandwidth in the system, get the Cray. you need next best bandwidth at a low price, get the Xserve cluster.

Wow!

[pyro jackelope]

$2 million? Oh, sure...I'll take like...5 of those. They'll go good with the particle accellerator and mock hubble telescope I have in my backyard =). Anyways, this sounds really cool. A person could crunch so many numbers with one of those, or perhaps play a game of solitaire?

Re:Wow!

[Maniac-X]

I wonder how much the 30,000-odd CPU model would cost. Imagine the solitaire on that!

Sic transit gloria mundis

[oakad]

It seems like Cray is not capable of sustaining its heritage. Buying cheap AMD processor and connecting them with customized HT interconnect is not enough to build a machine capable of record-breaking single-task performance, old Crays exhibited. When one could be sure with Cray XMP that he has the best machine money can buy (with outstanding scalar and vector abilities), new Cray is just another loosely-coupled AMD cluster. Thanks god it's not a NEC clone (at least).

Re:Sic transit gloria mundis

[Anonymous Coward]

It's not a customized HT interconnect. There's a dedicated SeaStar router chip that connects via HT to the uniprocessor Opteron + RAM node, but the actual fabric connecting the SeaStars is proprietary (each SeaStar connecting to six others via 7.6 GB/s interconnects, forming a 3D grid fabric expandable to 30K+ nodes).

That's why they use mere 100-series Opterons: they need only one HT link per CPU. Because the whole is not based on HT interconnects.

Really, loosely-coupled cluster my ass. This machine *is* capable of record-breaking single-task performance. Read the product pages again.

Re:Sic transit gloria mundis

[Durinia]

Cray does have its own fully custom system still, its called the X1 - a highly scalable Vector machine (thats far from a NEC clone...)

I'd link it, but the site is down...

yum!!

[commodoresloat]

From the spec page:

Operating System:

UNICOS/lc--Components include SUSE(TM) LINUX(TM), Cray Catamount Microkernel, CRMS and SMW software

I remember installing SuSE linux long ago on a ppc. The first thing I did once I got X running was fire up the gimp and doctor their logo so it said "Welcome to DuDE Linux...."

To answer your question, it looks like they've patched SuSE to run on the Cray Catamount Microkernel. Since there's no way in hell I'm going to buy one of these for my modest word processing and web browsing needs, I'm too lazy to look up what that actually is, of course.

Anyway, I run OS X these days. When I need Cray computing power, I don't need to spend no $2 million. I just fire up Desktop Cray and go to work!

The first test of the new Cray

[teamhasnoi]

they simulated a woman who posts to Slashdot and is waiting for her Centris running PearPC on Debian to boot OS X.

Strangely, it took roughly a week. The second test was a simulation of the moderation results of this post.

It received a +5 Funny, which puzzled researchers, as it is currently modded -1 Offtopic.

Damn you Schroedinger!

MP performance overhead

[Durzel]

I was under the seemingly somewhat misguided assumption that multi-processor systems incurred a non-trivial performance overhead from their application - i.e. a dual processor system doesn't technically perform at 200% the speed of the equivalent single because of the inherent overhead in the MP architecture and implementation. From memory generally speaking it's more like 90% (or even less).

Wouldn't this overhead rise exponentially as you added more processors to the point where - unless the machine had a compartively small form factor (vs an equivalent number of Blade servers for example) - it would become a Law of diminishing returns?

Re:MP performance overhead

[Anonymous Coward]

It doesnt work that way because the applications this thing will be running will be highly specialised for the architecture.

The algorithm will decompose the workspace so as each node can compute its own slice of the problem almost independently.

The is only a slight communicatiosn overhead depending on what application is run. This is also offset a bit by the supposedly fast interconnect hardware.

Re:MP performance overhead

[Big Mark]

If Crays were built the same was as desktop dual-proc machines, then yes, the multi CPU overhead would cripple it. Fortunately, it's designed completely differently - e.g. they use PowerPC chips to handle almost all of the inter-processor communication.

You can't really compare something that can hold thousands of CPUs to something powered by Abit that can hold two, anyway. It's like comparing apples and a strange bug thing with tentacles.

Re:software

[Coryoth]

what kind of operation system runs on this beast?

UNICOS is usually a safe bet. In this case the specs say UNICOS/lc, which is made up of "SUSE(TM) Linux(TM), Cray Catamount Microkernel, CRMS and SMW software"

I'm not entirely clear how to interpet that, but I think it runs as follows: It runs the Catamount Microkernel as the kernel, and uses SUSE for everything else (so we have SUSE Linux, without the Linux - all of a sudden that GNU/Linux stuff starts to make sense). The CRMS is their interconnect management and monitoring software, and SMW is the System Management Workstation - which I'm guessing is their administration frontend.

It's worth noting that that's some pretty serious software there (because Cray has a lot of experience dealing with large systems) - you can bet that the management and monitoring software is some very serious stuff.

This thing is to a beowulf cluster what a dual G5 PowerMac is to homebuilt PC system running Linux From Scratch. It's going to work flawlessly "out of the box" with a smooth and polished interface that lets you get done everything you want to do simply and easily. You can of course make your home built PC with LFS work just as well, it's just going to take you an awful lot of effort.

Jedidiah.

Nuclear Simulations

[dynamic_cast]

The point is that if you simulate the effects of age on a warhead you don't have to do any realworld testing. It does no good to have a nuclear arsenal that is not capable of being used.

Re:Nuclear Simulations

[October_30th]

you don't have to do any realworld testing

I admire your positive outlook on the prospects of simulations, but as an experimentalist, I find this "soon we won't need experiments at all" (see Rev. Mod. Phys. 64, 1045-1097 (1992), for instance) attitude very dangerous. Simulations and models, even at the first principles level, should never be trusted implicitly. They only sure way to tell how nature works is via experimentation.

I can sort of understand simulating nuclear explosions, but simulating the aging process of a warhead doesn't make that much sense to me - unless the simulations are accompanied by direct observation of the (accelerated) aging of a warhead.

Re:Nuclear Simulations

[klorentzj]

The reason for simulating nuclear reactions instead of directly observing the effects is because of things like the nuclear test ban treaty.

Re:Nuclear Simulations

[October_30th]

Are you talking about nuclear reactions as in "fission/fusion explosion" or as in "radiation induced damage to surrounding materials". Both explosions and the aging of warhead can be simulated. Both processes involve nuclear reactions and, as far as I know, only the former is subject to the nuclear test ban. I don't see how investigating radiation damage to materials, for instance, would constitute a breach of the treaty.

Re:Nuclear Simulations

[klorentzj]

I guess I was talking about fission/fusion reactions, although I have to say that I hadn't really considered radiation damage.

The extent of my knowledge on this is that I spent a summer working on supercomputing at Los Alamos, and this was the explanation I heard for why simulating nuclear explosions was necessary.

Not having a security clearance, I didn't get any more details than that. But, I have to agree with the original post. At least in terms of how the research grants are justified, nuclear simulation (due to restrictions from the test ban treaty) and weather/climate modeling are two major applications for supercomputing.

I didn't pursue the field at the time, because I wasn't really interested in either of those applications. So, I would be very interested to hear more about other applications that you know of for supercomputing. (I'm talking about high-end massively parallel number-crunching...I would put things like computational genetics in a different category, because they are really more of a database problem)

I'd also imagine that there are applications that are not tractable now, but will be when the horsepower is increased. What are these applications of tomorrow?

Re:Nuclear Simulations

[dynamic_cast]

Well, the idea is you don't have to do any testing...

Personally I think we oughta pop one off every fourth of July or so. Only way to be sure.

newfangled buzz.

[colaboy]

Yes, all that mumbo jumbo about CPUs and interconnects is fine, but what we really want to know is how fast it can run Mac OS X under PearPC... Can anyone tell us an XBench score?

Re:newfangled buzz.

[adzoox]

Ha - well I'm sure the guy behind CherryOS will have a press release that it runs The Mac OS at 30 Terahertz.

Re:newfangled buzz.

[MarcQuadra]

This is ging to become the computing equivalent to 'about the size of a volkswagen beetle' isn't it?

I can just hear it:

NEW SUPERCOMPUTER RUNS OS X AT 2.7x NATIVE SPEED UNDER PEARPC!!!

Leather seats?

[gspr]

Do they have leather seats for the operators like the 1980s models did?

Re:So......the cost compared to?

[Coryoth]

So, how does this compare to running Apple's Xserve? Bang per buck? Heat? Space? Etc etc....

There's not a lot to compare. We're talking apples and oranges. It's like asking to compare a PowerMac G5 with a bunch of PC parts scattered on the floor as desktop machines. Sure, you can put the PC together, load it with Linux, tinker with it to get everything working, etc. but that's a fair amount of work compared taking the PowerMac out of the box, plugging it in, turning it on, and having everything work perfectly.

Read the specs, particularly with regard to the interconnect, system administration, and hardware and software reliability features. This thing is seriously engineered to be massively parallel system with top of the line hardware and software to support and maintain that, as well as extremely impressive reliability features.

Jedidiah.

Anyone know the IP?

[Lord Kano]

My Distributed.net stats could use the boost.

LK

AMD gets about...

[BrookHarty]

So 96 processors, AMD gets about 144K per PE node at 1500 per cpu, or does Cray get a discount?

Also, a 30,000 cpu complex, AMD must be making a tidy sum.

Re:AMD gets about...

[Big Mark]

I highly doubt Cray are paying retail prices for these CPUs. I'd imagine they buy them wholesale from AMD directly, and probably a further bulk discount on top of that.

Re:AMD gets about...

[mikera]

AMD may even be gving Cray an extra discount/kickback for the publicity value - not uncommon for this kind of cutting edge stuff.

My old company managed to get some seriously expensive enterprise software for just 10% of the retail price because we were able to convince them that having us as a client would be a publicity coup for them....

Not even trying...

[F'Nok]

Using AMD's Opteron processor, it scales to a total of 30,580 CPUs.

They must be in a rush to market... Clearly the target was to have 32,768 CPU's.

I mean, what the hell am I going to do with only 30,580 CPU's?!

Re:Not even trying...

[jimicus]

Run Duke Nukem 3D on Longhorn?

Re:Not even trying...

[LiquidCoooled]

I think you would have a hard time running Minesweeper on LongHorn with "only" 30,580 cpus!

Dont even consider doing 3d work until you get over 65535 processors.

Intersting note

[floydman]

from their Tech.sheet they are using the Luster file system

This is the first time i see a shipped linux with this file system. Now the intersting part is that lusterfs is made for linux clusters, but this monster is not a cluster... any body can shed some light?

Re:Intersting note

[flaming-opus]

It's not a cluster.....

well, sort of.

There are thousands of compute nodes, all of which get i/o services from dozens, or hundreds of i/o nodes. These i/o nodes run linux, several instances of linux. Basically the i/o nodes ARE a cluster, though not a compute cluster, and not necessarily a symmetric cluster. The i/o nodes run lustre in very much the same way that a cluster system would (though they can take advantage of hardware features not present on commodity clusters).

The real difference in this system is that nodes are not peers, as they are in commodity clustes. Each node has only one function, and the software is tuned to provide that one function very effectively.

$2 million for a 200 processor system ?

[Professeur Shadoko]

I can't find this price anywhere.
And it seems _really_ low.
I would expect a price at least twice higher.

Ok, $2 million is starting price, but on Cray's website they say the configuration can be as "small" as 96 CPUs.

So it's maybe $2 million for 96 CPUs.
(Still fairly cheap for a Cray, if you ask me)

Mirror Links

[Agret]

In case original page gets slashdotted: http://www.cray.com.nyud.net:8090/products/xt3/ind ex.html

Re:My new dream toy

[Guppy06]

Maybe if you included promises of free iPods...

Re:So......the cost compared to?

[Lussarn]

So what you say is that not even the macintosh can touch the 30000 proc cray, but it has the second place pretty much secured. And to a better price. Dream on.

Re:fucking death labs

[Zog The Undeniable]

Ah....for tactical nuclear weapons simulation, no doubt. And we thought General MacArthur was bonkers.

scary thought

[nounderscores]

how would you accelerate the aging of a warhead?
make the plutonium pit out of plutonium mixed with plutonium decay products, layered to match the cross-section of a genuine old warhead pit?

make the explosive pannels that are supposed to all go off perfectly semmetrically out of aging, unstable, unreliable explosives?

make the wiring exclusively out of decaying cables, which have the insulation falling off?

Hmm. better dismantle and scrap these guys as soon as any one part begins to go.

All the real life testing on even slightly aged warheads was done in the 1970s, and by the french in the 1990s. They now keep the models in sillico, and don't let the warheads age past the point where the model has been shown to hold by recycling them for components.

I really don't want to be anywhere near an artificially aged warhead.

Re:scary thought

[Anonymous Coward]

how would you accelerate the aging of a warhead?

Just give it a couple of little baby warheads to look after.

Re:So......the cost compared to?

[adzoox]

You say it's comparing Apples to Oranges but its not really ...

The VT Supercomputer specs vs the Cray specs page you pointed to:

CRAY 460 GFLOPS per cabinet (96 processors @ 2.4 GHz)

Apple - if my math is right - 420 GFLOPS (100 processors @ 2.0Ghz)

The new specs for the specialized VT Supercluster are pretty impressive.

Their throughput and interconnect is most likely weaker - but still VERY strong with fiber channel.

Re:The math for a comparable Xserve system

[joib]

What a value!!


That is, until you throw a tightly coupled problem at it and the Cray is 10 times faster because it has much better internode bandwidth and lower latency.

And, you forgot to count the cost of the InfiniBand interconnect that the VT cluster used? That's a couple grand per node.

Bottom line, apples and oranges. If your applications is easily parallelizable (i.e. doesn't require much communication between the nodes) you'd be stupid to piss away your money on a "real" supercomputer instead of a cluster. And vice versa.

Re:So......the cost compared to?

[Coryoth]

You say it's comparing Apples to Oranges but its not really ...

Oh, why is that?

The VT Supercomputer specs vs the Cray specs page you pointed to...

Right, so if you compare Virginia Tech forking over money not just for Xserves, but also for all the interconnects, the cooling, setting up the system, developing maintenance, monitoring and administration software etc. to buying a Cray and plugging it in they have almost comparable performance?

If I buy a cheap AMD CPU, cheap motherboard etc. overclock it, install a Linux From Scratch system on it over a couple of weeks, and get a desktop running (that all the Apple fans deride), that's comparable to buying a G5 PowerMac and plugging it in, because, you know, the performance specs are similar, and you can fiddle the Linux desktop so it looks like OS X...

Cray is to supercomputers what Apple is to desktop machines. If you want to argue that a bunch of PC parts is comparable to an Apple desktop, go for it.

Jedidiah.

Re:So......the cost compared to?

[ozbird]

There's not a lot to compare. We're talking apples and oranges.

No, we're talking Apples and Crays... Didn't you read the post before replying? ;-)

Re:My new dream toy

[Anonymous Coward]

Gamer's dream machine? This is a supercomputer, not a game console. I bet this thing doesn't even have a graphic card.

700kgs, 75dB and 14kW...

[Alkonaut]

...Sadly I think that beats my Volkswagen on all three

What?? no usb?

[Anonymous Coward]

Where do i plug my 128 usb pendrive to store
results from my computations... :)

Finally ...

[Zurd3]

We'll be able now to install Gentoo in just a few days !

Cool!

[Dorsai65]

Doom3 here I come!

Re:software

[marsu_k]

It runs the Catamount Microkernel as the kernel, and uses SUSE for everything else (so we have SUSE Linux, without the Linux - all of a sudden that GNU/Linux stuff starts to make sense)

So should we call it GNU/Catamount?

At last

[SFSouthpaw]

Now I can run every game under MAME @ full speed.

No, what stands in the way is price

[Moraelin]

The real problem that stands between scientists and them having lots of shiny toys is funding.

E.g., yeah, having a 30,000 CPU super-computer to simulate your gene model on would be nice. Forking over half a billion for it, well, it's suddenly not that nice any more.

Having one of those to simulate an electronic circuit, now that would probably rock. Again, paying half a billion for it, suddenly isn't that attractive.

The real question isn't how nice a toy you'd like to have, it's ROI. (Unless you work for the government, and just have a budget you _have_ to blow on stuff, whether you need that stuff or not.)

And in that context, you'd be surprised what you _can_ do with a lot less expensive toys.

Having Cray's custom interconnects sure is impressive, but for a lot of problems they're not even needed any more. _That_ is what killed Cray.

Most RL problems are not really the kind described as "_one_ huge indivisible data set, that you have to process in _one_ huge batch process." They're more like "we have this process with a small data set that we have to run 100,000,000 times." Most design problems or biology problems are really of that kind: run the same thing 100,000,000 times with different parameters.

And as Seti@Home or Folding@Home proved, a helluva lot of those don't really need _any_ kind of shared memory or fancy interconnects. The real ticket is noting that instead of accelerating the batch run 200 times, you could just split it into 200 smaller batches ran on 200 single-CPU machines.

The super-computer solution costs 2,000,000 just for the machine alone, while the 200 PCs solution costs 200,000 or so. I.e., 10 times cheaper. Better yet, the 200 PCs solution is also far cheaper to program. (Anyone can program a non-threaded batch app.) _And_ for that kind of a problem the 200 PCs solution would actually finish faster, since it has no contention issues whatsoever.

Again, that's what really killed Cray and the super-computers. They're techologically impressive, they're a geek's wet dream, but... for 99.9% of the problems out there they're just not worth the price any more.

... Back in my day .... young whippersnapper

[ebooher]

So come on, ante up. How many remember being awed at the mere sight of old Crays back in the day? Like the Cray-3? I remember the first time I saw a Cray .... thing was in an anti-static environment. To access it, one had to pass through an airlock and be "decharged" or "depolarized" etc. Basically they some how charged the air to get rid of static electricity. Then you had this system that was running *in* liquid! Take that "Oh I'm so cool cause I have a l337 haX0r water cooled CPU" overclockers

They (Cray) were so proud of this accomplishment that the upper portion of the cabinet was some kind of plexiglass so you could see the fluid as it moved, and moved wiring and what not with it. Very surreal feeling, almost like the thing was breathing.

And what about the Cray-1? Wasn't that a true testiment to 70's *art* and sculpture? The thing looks like some kind of freaky bus station bench with it's odd red and white panels and black base. Though, I don't know if they all looked like that, maybe you could get them in other colors?

Ahh .... those were the days.

Re:... Back in my day .... young whippersnapper

[flaming-opus]

well, when the computer costs 10-15 million dollars, you can afford to spend twenty thousand on making it look really cool.

compared to ASCI-red, the system that red-storm is replacing, xt3 looks increadible. Yes it's a long row of rectangular racks, but at least they are stylish racks. Intel built asic-red in beige box style. Oh well. function over form I suppose.

Re:... Back in my day .... young whippersnapper

[mrdogi]

Then you had this system that was running *in* liquid!

Before that was the Cray-2 (a.k.a World's most expensive aquarium")? In case anybody's interested, I believe they used Fluorinert as the liquid, as it wouldn't swell the PC boards, short anything out, or cause anything to corrode.

A note, the Cray-3 was created by Cray Computer Corporation of Colorado, whereas the Cray-1 was made by Cray Research of Wisconsin. In ~1990, Seymore wanted to start working on computers using gallium arsenide instead of silicon, since they could switch faster. Cray Research didn't want to try anything so revolutionary, so Seymore headed to Colorado with a group of people and started CCC. Unfortunately, they apparently made exactly one Cray-3, then folded.

Seymore Cray was quite the Übergeek.

Re:... Back in my day .... young whippersnapper

[ebooher]

Well .... it's actually pretty funny that you would mention that. After my own statement about what color the Cray-1 was I decided to run around the net like a mad hen ..... which is where I started finding pics ..... nothing like seeing a Cray-1 in Lemon Yellow. Anyway ....

From: http://www.spikynorman.dsl.pipex.com/CrayWWWStuff/ Cfaqp3.html#TOC19

Could you choose the colour of your Cray machine ? In the early days of the company yes, there was even rumour of a cowhide covered XMP delivered to a Houston oil company. As time went on colouring your computer was dropped as a customer option. Well almost - when there is that much money changing hands, if enough fuss is made, the exterior panels would revisit the paint shop. This did not apply to Els which were all black and red. Well almost - one customer which had just upgraded a pair of YMPs (one green, one blue) for a C90 and an EL did manage to get the EL painted a rather fetching sky blue colour.

As for XMP/EA, sn501 an internal contact reports "We lobbied to have it done up in denim (like the denim Jeans) & have a little red Levis tag attached. ... Management was not amused & it never happened."

One second user customer did have a bit of a surprise when their second user C90 arrived in a lurid deep rose/pink colour. The top of the C90, being a convex shape happened to sit just a couple of feet under a set of strip lights and resulted in lovely pink glow over a whole section of the machine room. Rumour has it that the colour matched the previous owners, girlfriends' nail varnish.

The Bell Labs Cray (XMP) was a wallpapered IC design.

Yeah, it's gotta be awful

[thegnu]

Last time I bought a Cray super-computer, I was kicking myself for weeks about the 2 million dollars I wasted.

Next time, I'm just gonna build a beowulf cluster out of 200 overclocked AMD Barton 2500s. I shall NOT be suckered again!

Re:Yeah, it's gotta be awful

[minus9]

I've got a bag full of thinnet coax you can have, complete with T pieces, you might have to find your own terminators though.

Re:Yeah, it's gotta be awful

[charleste]

While at one of those jobs that I'd have to kill you if I told you what I did (even though I didn't know really), they had a nice "old" Cray on display in the lobby. They had got a Cray 2, so they were "obsoleting" the Cray. My thought: Awesome fishtank! Put a lavalamp in the middle, pull the boards and seal it up. It's already plumbed for hoses (liquid cooled). If only I had like $100,000 at the time to buy military surplus.... (sigh)

I saw a demo of one...

[Frank of Earth]

... and when you turn it on, a crackly computer generated voice says, "Would you like to play a game?"

Cray and blinkenlights

[rdunnell]

Cray never really put much in the way of blinkenlights on their systems.

Of course, most models look like obscure modernist furniture or art exhibits, some of the older ones had windows that allowed you to see the liquid coolant, etc. But they definitely did not have a lot of flashy light thingies. Then again, do you need them? It's a Cray! You're supposed to be stunned and look at it in silent reverence. It's like telling a Mafia boss that he has a nice suit but he needs more jewelry.

Thinking Machines, well, that was different. Too bad it didn't do more, but it was covered in LEDs.

hybrid system with multiple kernels

[Dink Paisy]

From the documents, it looks like it runs Linux on the management nodes and Catamount on the compute nodes. The idea is you can do what you like with the general purpose nodes, but for the compute nodes, you run a lightweight operating system that has low overhead, minimal services and predictable scheduling. BlueGene/L works the same way; it runs Linux on the management nodes and a custom operating system on the compute nodes. Compute nodes likely provide scheduling for only the number of threads that run on the node, communication through MPI and some proprietary API, and basic debugging facilities. Compute nodes probably lack normal OS services like network, disk, or even a console.

It's probably more than one rack, though.

[chipmeister]

Cray, and most other large system manufacurers, usually put four CPUs on a single board. Each board plugs into a backplane. I'm just guessing here, but you can probably only fit 256 or so in a single cabinet.

Too bad...

[TheVidiot]

Too bad Doom3 can only use the first processor, leaving the other 199 for eDonkey and SETI....

But...

[_undan]

... will it run PearPC fast enough?

Re:The math for a comparable Xserve system

[selderrr]

And vice versa

If your application is not parallelizable, the supercomputer pisses away on you ?

Re:Internode bandwidth - cheap solutions?

[SuiteSisterMary]

'Super-computing on a budget' is kind of an oxymoron. You either pony up, or you're not super-computering; you're parallel processing on a big cluster.

Ever wonder why....

[Gentlewhisper]

They didn't do it using Intel Xeons? =)

are you sure you remember seeing the Cray 3 ?

[bmajik]

Because, IIRC, that was the one that they were only building one of, and when the govt cancelled the order, thats when Cray Research went under.

Re:are you sure you remember seeing the Cray 3 ?

[ebooher]

Well, anything is possible. It may not have been the 3 I saw. Though I seem to remember being able to see over it, and I think almost every other Cray was at least 6 foot tall. But I don't know, it could have been recessed into the floor. I was quite a bit younger at the time, obviously. I was more focused on the tour guide guy showing off the fact that electronics were in liquid. At the time I thought if it was wet, it conducted electricity, so that kind of blew my mind.

I'd love to find an old Cray, like maybe a C90. Something big enough to be retrofitted ... man talk about cool factor plus 50. If a geek can build a bar/fridge out of an old VAX just think what a Cray could do. Obviously it wouldn't be economical to actually use the thing as a computer anymore. Too expensive. But what a garage fridge the thing would make.

Re:are you sure you remember seeing the Cray 3 ?

[Durinia]

wrong company - Cray 3 was being built by "Cray Computer", a different company that Seymour started after he left the original Cray Research. He's most likely talking about the Cray 2, which even came with its own Waterfall

Re:are you sure you remember seeing the Cray 3 ?

[ebooher]

Looked around on the net, as well as a couple other /.'rs here, and someone posted a link here to a 2 and I found a pic of a 2 with the waterfall system that was mentioned by another person, and I must accept defeat within the loosened strands of my unraveling mind.

It was indeed a Cray-2 that I remember so vividly. Nevertheless, still an extremely exotic machine. Very much the Ferrari F40 or McLaren F1 of super computing. You've seen pics, maybe even seen one at a car show, but you know you'll never be allowed to touch one. It had as much class as an Italian sports car too.

I find myself wondering how many /. geeks it would take at what $$ amount to colocate a community Cray somewhere ...... be like going in with 100 friends to buy a Ferrari though. Who gets to keep the keys?

Re:are you sure you remember seeing the Cray 3 ?

[ebooher]

Again, kind of answered my own question here. Check out http://www.cray-cyber.org/general/start.php/ for a hosting provider in Germany that happens to have a Cray Y-MP EL online for external users with free accounts. Though, you do have to tell them what you want to use the account for and what not, so if you tell them to root kit their l33t 5up3rc0mpu73r they may not let you play with their toys.

Re:The math for a comparable Xserve system

[joib]

If your application is not parallelizable, the supercomputer pisses away on you ?


Only in Soviet Russia.

My dumb math.

[JoshRoss]

That works out to $305,800,000.00 (before taxes) for the 30,580 proc version. What a deal! $10k per processor. We are reaching Sun(TM)* territory.

*Sun is a registered trademark of some company.

You can get to their website?

[Roadkills-R-Us]

Obviously they aren't using one of these for their webserver. Or if they are, they need something more than a modem for their internet connection!

Re:You can get to their website?

[rooijan]

I happened to see the story quite early - I got in, read it and got out before the sound of a million slashdotters could be heard approaching on the horizon.

Re:The math for a comparable Xserve system

[SuperQ]

You're leaving out a lot of stuff necessary to make a cluster:

#1 RAM: $3000 for the G5 cluster node includes 512mb ram. Most places demand atleast 2gb ram per CPU, we require 3GB ram per CPU in all new system purchases. This brings the node price (from apple.com) to $6500
200x $6500 = $1,300,000

#2 Racks and power: Each rack can hold about 32 machines (without getting way to hot/dense) for 200 nodes, this would be about 7 racks.
7x $1200 = $8400

#3 Interconnect: No HPC system is usefull without an interconnect. An 80 node myrinet system was $250,000, so at $3125/node you're looking at:
200x $3000 (estimate) = $600,000

#4 Networking: you need a network switch and cabling to connect all the nodes... gige is a must these days. Let's say we go cheap with HP ProCurve 2848 Layer2 managed for $3300 each we need 7 of those, one for each rack cabinet.. with trunking we can get 4gb back to a central switch. not too bad. Say we add $10/cable for pre-made patch cables, (length averaged) that's about $2250 in cables.
7x $3300 + $2200 = $25,300

#5 Disk: You quoted a bunch of XservRaid's without any kind of apple care.. with IDE raid.. I'm not going without some kind of support on it. Oh wait.. 1 file server is NOT enough to handle 200 nodes of HPC.. and apple doesn't have a clustered filesystem. You're going to have to go with Linux/Intel with RedHat GFS for that one (yes, there are other options, but I know GFS)
Say we do 4 XserveRaid's with applecare:
4x $16,000 = $64,000
We also need for dual whatever intel machines: (i'll be nice and include F-C cards in the price)
4x $3000 = $12,000
We also need a F-C switch to link all the nodes:
SanBox 8 port $5200 and 8x SFP modules $750 = $11,200
I'll pretend like we don't need GFS software support, but most places would want it. (it's another $20,000 or so, but eh.. we want cheap solution)
Disk total comes to: $87,200

Price so far: $2,020,900

And that doesn't even include setup!

No one does trig with x87 anyway.

[Ayanami Rei]

You're supposed to use lookup tables, recurrence relations and interative refinement to get the precision you need when you need it.
The only time you should be using fsincos (SLOW) is when you need to build a table or populate variables accurately before a loop.

Re:software

[flaming-opus]

This split microkernel architecture has been in use for a long time on big mpp systems like the paragon and the t3e. The software base (catamount/linux) is new, but the design is old.

catamount is the kernel that runs on the compute nodes. IT's a tiny kernel that packages up the OS service requests, and sends them, over the interconnect, to an OS or I/O node, which does the real work of the operating system. catamount is a descendant of PUMA, which came from Cougar. These are heavily derived from work done at caltech. (I believe CMU, and one of the UTexas schools also played a role, but am not sure). The idea is that the microkernel is small and unobtrusive, and it gets the hell out of the way so the application can use the CPU as much as is possible.

The OS and I/O nodes run linux, and provide services to the compute nodes. This is probably, but it could just as easily be running as a user-space daemon on the OS node. (Though you might have to do some mem-copys that way, which would lower performance)

NOTE: Though these nodes take advantage of some of linux's features (like the lustre file system) they do NOT necessarily implement these features for the system as a whole. They probably provide a minimal set of features necessary for the sorts of problems that the xt3 runs. All the scheduling work that has gone into more recent linux kernels is of little use, as the compute nodes have their own scheduler, probably more closely tied to the batch dispatcher than to the linux kernel. To say that the system runs linux is true, but a little misleading. It's a very different linux than what runs on my desktop, and it's used in a very different way.

Was it the 1a or 2 maybe?

[caveat]

1a
2

Active / 100% CPU usage vs. Screensaver.

[Ayanami Rei]

Errr...
No computer in existance runs a screen saver because the CPU usage is less than 100%. Desktop computers run screen savers to prevent a still image from burning onto your monitor or LCD (or to lock the terminal if it thinks you've left). This has everything to do with no use of mouse or keyboard, not CPU usage.

When any computer has nothing to do in a timeslice, it generally calls the HALT instruction, which puts it into a low power state until a timer interrupt or something comes along and wakes it up to do something else.

This is why CPUs that are idle generally take less power than one running Prime95 or 128 of them rendering a 3d movie.

Sandia's data sheet on Red Storm

[scoobrs]

Here's all the theory and architecture information you might want on Red Storm including a discussion of how Amdahl's Law is wrong. They add overhead due to communications to the equation and make some very interesting proposals about scalability. http://www.lanl.gov/orgs/ccn/salishan2003/pdf/camp .pdf

Their bottom line to the research: "A well-balanced architecture is nearly insensitive to communications overhead. By contrast a system with weak communications can lose half its power for applications where communications are important."

Conclusion: "For most large scientific and engineering applications the performance is more determined by parallel scalability and less by the speed of individual CPUs. There must be balance between processor, interconnect, and I/ O performance to achieve overall performance. To date, only a few tightly-coupled, parallel computer systems have been able to demonstrate a high level of scalability on a broad set of scientific and engineering applications."

Re:Sandia's data sheet on Red Storm

[Mistah Blue]

I think this can be applied to the CPU's themselves. My very unscientific observation has been that a 1.4GHz Pentium-M spanks a 2.8GHz P4 on one web based application I work with. I have to say I am very pleased with the 1.4GHz Pentium-M and feel no burning desire to upgrade my Latitude D-800 to a faster one. It is plenty fast for me, and I suspect it's power draw is less than the faster ones.

Also, eye for expandability...

[Ayanami Rei]

By "limiting" themselves to the 1xx opterons, they could move to 2xx or 8xx later on. It'd then be easy enough to connect 8 or 12-way by hooking up more interconnects per board, adding a second CPU on each board, etc. etc. by utilizing the additional HT links.
And then when the dual cores come out... hehehe.

Wall Street

[Moraelin]

You have to understand though that the stock market's expectations have nothing to do with whether the company is doing well or not.

Surrealistic point in case: at one point 3Com had a lower market value than the Palm daughter-company. Basically if you subtract the value of the Palm shares, the whole rest of 3Com was actually worth a _negative_ value for the stock market.

And we're talking divisions which were making a tidy profit. Yet they were apparently worth a _negative_ number.

No, it's not a joke. Roll it around a bit in your head to fully grasp how completely sad and idiotic that is. Real profits, real assets, worth a negative number of dollars. Stupid.

Or at the other end of the spectrum you have Microsoft whose stock market value is _way_ above the value of its assets. Without paying any dividends or acquiring much in the way of long term assets, people just flocked to drive the price up and make Bill Gates rich. Basically to give their money to Bill Gates and not even get a Windows CD in return.

The thing is, however, the stock market value has _nothing_ to do with a company's value or profits. The value of a share is only worth as much or as little as people want to believe it is. It is like Monopoly (the board game, not MS;) money: if tomorrow we decide that the blue bills are worth 10% more and the red bills are worth 10% less, who's to argue with that.

The _only_ reason the stock market on the whole goes up is basically because yearly people dump more money into it. Basically it goes up just because people want to believe it's going up, and put their money where their belief is.

And the way those values fluctuate, now that just has to do with hype and greed.

The stocks worth buying are those who'll make you a profit: typically meaning they'll raise in value. The stocks worth selling are those who don't.

Except with no intrinsic value it becomes a game of guessing what the other lemmings will buy (driving the price up), and what the other lemmings will sell (driving the price down.)

One thing that makes lemmings buy is the prospect of growth. Hence, hype is good. Hence, yes, shares in a cancerous tumor would sell like hot cakes and rocket sky high in price.

Hence, conversely, shares in a company which doesn't grow or otherwise cause more lemmings to buy, are not worth holding on to. Because they won't bring a profit. If Microsoft truly plateaued and didn't pay dividends either, regardless of how much profits it made at that point, its shares would plummet. Because between holding onto a share in MS that doesn't bring a profit, and investing in some startup that grows quickly, the second promises more of a ROI.

Now that's all a bit of an over-simplification.

Of course, there are other factors. Like just paying dividends to give people a reason to hold onto your shares even without massive hype and growth. (See why MS started doing that when its market explosion slowed down.) Or like fraud: "analysts" just telling lemmings what to buy, and thus drive up the price of the shared owned by the "analyst" and his/her clients. Etc.

But as a quick intro to the madness of the stock market, it will have to do.

The SSI is configurable...

[Ayanami Rei]

... that is a bunch of "service nodes" will be lashed together into an SSI. These all run their modified SuSE. So if you have two I/O nodes and two login nodes and 4 Network nodes, that might be an "8-way" SUSE image. They communicate with other lashed-together service nodes with Luster. The compute nodes are part of a huge loosely coupled SSI running the microkernel... which are submitted jobs by one the SSIs running on the service nodes.

Someone correct me if I'm off the mark here.

Not really true, but true.

[tod_miller]

Yep, system design is harder, but if you have custom hardware and special programming, you can litterally run all these bitches at full steam.

If you need some inter process traps and doohickies, then you have to try and program code which is harder than licking you own tonsils going via your rectum.

Also, you have to realise that each one of these 200 processors have about 512kb of cache in total, and you can write a bloody good program using 100gb of CPU cache, especially if it is massively parrallella..lell...isable....

Now this reasearch may become prevalent as we try and run consumer based parellel processors, with multiple cores, as lithography sucks, and nanotech is going nowhere because of grey goo panic and Dexters Lab, and Smalle Pox.

Or so I heard.

Of interest to Cray-3 info

[ebooher]

Cray-3 memories by Steve Gombosi From a comp.unix.cray posting

Graywolf ("S5") was installed at NCAR. Like all NCAR supercomputers, until fairly recently, it was named after a Colorado locale.

This was the *only* Cray-3 shipment, installed in May 1993, the machine was a 4-processor, 128 Megaword system.

Two problems in the Cray-3 system were uncovered as a result of running NCAR's production climate codes (particularly MM5): a problem with the "D" module causing intermittent problems with parallel codes, and an error in the implementation of the square root approximation algorithm which caused incorrect results for certain data patterns (kinda like the Pentium divide bug ;-) ). These were rectified and replacement CPU modules were installed, although I can't remember the date.

The machine ran NCAR production until CCC folded in March, 1995. Since NCAR never paid for it, at some point we reduced the CPU count to 2 and let the machine run essentially unattended. I'm not too sure when that happened, although it marked the end of my regular commuting between Colorado Springs and Boulder.

There were a total of 7 Cray-3 "tanks" constructed. S1-S4 were single "octant" tanks (the smallest that could be constructed) which accomodated up to a 2 processor/128MW configuration. S5 and S6 were two-octant tanks. S7 was a four-octant tank which we used as a software development and benchmarking platform. S6 was chiefly used for system testing.

S1-S3 were diverted to Cray-4 testing once the Cray-4 project built up steam. S4 was diverted to the quite possibly suicidal Cray-3/SSS project after S7 became available (S4 was previously our software development machine).

For those of you who have Cray-3 posters lying around (by the way, I took all the photos on that poster as well as the Cray-3 and Cray-4 brochures and all the annual reports except the first two):

1) The big photo is of S5
2) Seymour is leaning on S5 (and you have no idea how hard it was to get him to hold still that long while wearing a suit...or to talk him into that particular pose)
3) The two "cooling system" photos are S6
4) The hand holding the module is mine ;-)

Cray-3 modules were 4x4x0.25 inches in size. Each module consisted of a multi-layer "sandwich" of PC boards (69 electrical layers), with 2 layers of 16 1x1 inch stacks. The stacks were the circuit boards containing the actual circuits (GaAs for logic, SRAM for memory modules). There were 16 bare GaAs chips mounted to each side of a logic stack. I think there were 12 bare SRAM chips on each side of a memory stack (the logic chips were square, the memory chips were rectangular).

It was a customer requirment.

[chipmeister]

Cray built this system to spec for a large customer who demanded the use of the Opterons. They decided to market it sonce they were going to build it anyway. If that customer had not requested this i doubt Cray would have built it that way. look for iformation on Red Storm.

Re:So......the cost compared to?

[Jeremy Erwin]

What's this about fibre channel? System X uses Infiniband as its primary communications fabric.

Infiniband for communications fabric

twenty-four 96-port switches with 10/20 Gb fabric (depending on duplex)

fat-tree topology

choke point is the bus: it is faster to send data across the room than the motherboard. Half the nodes can talk to the other half at the same time: 1.25GB/s

Ultra low latency fabric (less than 8s)

Each card has two ports, they use one.

source

Think on this

[phorm]

A 3Ghz single CPU machine with 1GB of RAM will still about 2-5 minutes or more to render a single frame of a raytraced animation

Let's say for a moment you have a 88 minutes for something like "Toy Story 2." Now, let's say it runs at 30fps. So we have about 88min * 60sec/min * 30fra/sec = 158400 odd frames

Now, 158400 frames * 5min/frame = 792000 minutes to render on my 3Ghz machine

Now go with the concept that a complex image at very high quality might take longer to render, but even at this rates we're looking at
13200h
550d
1.5 YEARS to render an animated movie. Now, obviously there are things that require more power. Think tactical simulations with AI, galaxy simulations with huge numbers of factors, etc etc

Some of the above need results in a timely manner, your 3000Ghz P4 isn't going to cut it, even your business class server isn't going to cut it, and your clustered business servers while perhaps having the power/resources might not be able to do things in the time needed.

For general-computing there's a PC, for blow-your-mind power there's Cray...

Re:So......the cost compared to?

[the_2nd_coming]

you idiot!!!! I am talking about clustering. if you clustered 30,000 CPUs of XServes together, you would get a really good bang for the buck because it would come in way cheaper than the cray or comparable systems.

reading comprehension must have been low priority at your school.

Depends on the code

[Dog135]

That all depends on what you're doing. Lets say you're using it to render a movie. Lets say one frame takes 5 hours to render and 1 minute of data transfer (round trip). Not counting the dedicated computer for pushing the data to the nodes, you get pretty much a 1x increase in speed for each node you add. At 5 hours (300 minutes) per frame, you can have up to 300 nodes without a drop in performance. After that, you'd have to sacrafice a node for pushing data.

So in short, it really depends on what your "work:traffic" ratio is.

Re:My new dream toy

[wolrahnaes]

I bet this thing doesn't even have a graphic card.


Oh, well that's fine...I'll just devote a few hundred processors to software rendering.

Something tells me that 256 Opterons working together could smoke a GF6800Ultra

Shipping ... but are any systems deployed?

[RageEX]

I guess they'll be setting their sights on SGI, which currently has the 'fastest computer in the world' bragging rights (well not officially, gotta way for the Nov. top 500).

Folding@home

[d3ity]

So, how long do you guys think this thing would take to solve all the workoad on folding@home, and find intelligent life via seti@home?

Re:In other news... Bush declares war on..

[narcc]

They confiscated my sliderule and calculator at the airport...called them weapons of math instruction...

Doom3

[tpillon]

But will it run Doom3 on super high quality settings?

How HyperTransport interconnect is different ...

[BitMan]

Power never offered before in a commodity interconnect, HyperTransport is changing how systems are designed.

Sys Admin 2004 November: Dissecting PC Server Performance

Re:The math for a comparable Xserve system

[SuperQ]

I guess I didn't know about Apple's Clustered FS.

We have clustered filesystems on all of hour HPC systems. We have GFS on our "small" 80 node linux cluster, 3 GFS+NFS servers to the nodes.

We use GPFS on our Power4/AIX systems, and we have CXFS on our SGI Altix systems.