Slashdot Mirror

Btw, it should be noted that Cray has something very similar in their Cray XD1 - http://www.cray.com/products/xd1/acceleration.html But the virtex is connected via their proprietary interconnect.

yes this has been done before (different socket for sure). Most of them have failed. But this is getting picked up by others lately and seems to have legs (technologically speaking).

http://www.cray.com/products/xd1/index.html

oh BTW a single 3U is around $45k. For certain memory bound calculations and some sequential algorithms, HFFPGA work well (high frequency FPGA).

What do you mean? http://www.cray.com/ is very much alive....

Steve Scott serves as Chief Technology Officer responsible for designing the integrated infrastructure that will drive Cray's next generation of supercomputer. Dr. Scott, who joined Cray in 1992, was formerly the chief architect of the Cray X1 scalable vector supercomputer and was instrumental in the design of the Cray/Sandia Red Storm supercomputer system. Dr. Scott holds fourteen US patents in the areas of interconnection networks, cache coherence, synchronization mechanisms, and scalable parallel architectures. He received his Ph.D. in computer architecture from the University of Wisconsin, Madison in 1992, where he was a Wisconsin Alumni Research Foundation and Hertz Foundation Fellow. Dr. Scott has served on numerous program committees and as an associate editor for the IEEE Transactions on Parallel and Distributed Systems. He is a noted expert in high performance computer architecture and interconnection networks, and was the recipient of the 2005 ACM Maurice Wilkes Award.

Hmm. Can't wait until Apache becomes mature and pragmatic. Or Debian, and Slackware too. And all those immature projects on Sourceforge. When that happens well, by golly, you'll probably see these eccentric oddities at Lawrence Livermore Labs or running on Cray hardware.

If they were really mature and pragmatic, they just might make it into government use, or even become more commonplace.

We can only patiently wait for that wonderful maturity and pragmatism to blossom. Until then we should be thankful that we are skillfully guided by the benign monopolists. They only have our best interests at heart.

As pointed out by someone else in this thread, Cray Canada makes the XD1 Opteron cluster (http://www.cray.com/products/xd1/index.html) with FPGAs built in. Might cost a little more than a PCIe card though. :-) They can even be programmed in C now, rather than a full-blown hardware design language like Verilog, since Cray bundles a special C-to-FPGA compiler.

I also went to demo of the SRC6 (SRC = Seymour Roger Cray, Cray's last spinoff in Colorado that has never quite taken off) here at the U of I in Urbana-Champaign. It appears to have similar capabilities and is quite neat, but is a smaller system. (http://www.srccomp.com/HardwareElements.htm) Somebody who knows more about these machines could probably point out more differences...

But consider that the Cray X1E can have up to 64 cabinets, each of which can use 65kW (see this), so it's still "fiddling small change" to paraphrase Douglas Adams.

It was the most advanced computer of its day.

Funny, I always thought the Cray-2, also released in 1985, held that title.

Cray uses Opterons. Such as in this product.

These ranking are based on LINPACK doing traditional operations like solving linear equations, so supercomputers like the Cray MTA aren't even listed even though for some grand challenges they destroy everything else, for example when doing dynamic mesh weather simluations. Each processor on the memory grid has 128 processor threads where the active thread switches every cycle (so memory fetch has huge latency). This lets it have a unified memory model and still have extremely high throughput.

So the MTA can adjust the mesh to compute the tornado in very fine detail while using far fewer points for the huge swaths of calmer weather around it. Traditional supercomputers can't do that well since just distributing the data points to each processor is so much overhead.

This sounds like the stuff that Tera was working on with their MTA back in the 90s (see this or for more techincal details here). Basically, a processor that could handle up to 128 threads at a time, with almost zero-latency switching among threads. These processors could be easily interconnected to scale up to whatever the customer (e.g., Sandia, Los Alamos, LLL) wanted. From perusing Cray's website, though, I don't see any current machines that appear to be using that architecture, so I assume it didn't play out somehow.

A few more:
- Garbage (the band)
- The Zucker Brothers (and Jim Abrahams)
- Cray Compter
- The Dells
- The Marquette Golden Eagles... Uh... Gold... Uh... Warriors? Hilltopers? Uh... Dwyane Wade went to school here!
- Fighting Bob La Follette
- Sewer Socialists
- Harley-Davidson
- They brought Arnold Schwarzenegger to America
- Douglas MacArthur
- Bratwurst
- Numbered Highways
- Birthplace of The Republican Party
- Svetlana Alliluyeva (Stalin's Daughter)
- Frank Lloyd Wright
- Heather Graham

Where do they get their information that "Modern Systems Operating at 1 teraflop fill an entire room!". Um, *modern* systems operating at up to 147 teraflops, able to be ordered right from cray.com occupy 35.5 in. x 59.75 in. (.9 m x 1.5 m). Those stats taken from: http://www.cray.com/products/x1e/specifications.ht ml I'm real thrilled they feel they have done something important, but why falsely hype it by misrepresenting the existing systems? Perhaps they are referring to NASA's 1970's-era "modern" supercomputers that are undoubtetly still in use and occupying the space of a small house with their PVC-pipe reminiscient conduits.

Once the 64-node machine is built, the designers will try to transfer several existing supercomputer programs onto the new hardware using these tools. "If we can get these [programs] to work, we'll know that we have a general purpose solution," Parsons says.

So, this is still vapourware.

LARC, at NASA, built an FPGA supercomputer. Here's a link to a related paper from 2002. Note, its a PDF.

Additionally, Cray builds an FPGA using supecomputer in its XD-1. It's definitely a nonvapourware project since they've sold over 15 of them. Yes, yes, it also uses Opterons, but they're paired with FPGAs.

Additionally, prior to Seymour Cray's death at the hands of a drunk driver, he was looking into FPGAs as his next stab at supercomputing.

Besides, FPGAs have two issues that make them good only for a very specific set of apps. Number 1, they don't currently have great floating point performance - this is a killer for most scientific apps. Number 2, they are hard to feed because the rate they can compute at versus the rate memory can feed them is quite skewed. Regardless, they're still very promising. The reconfigurable computing team at LANL (http://rcc.lanl.gov/) has done some very cool things with FPGA based systems.

The closest I've heard of is the Cray X1E, but even that only claims 147 TFLOPS.

Meanwhile FPGAs have displaced DSPs, FFTs and are overtaking CPUs for embedded applications. There are even rumblesof FPGAs seriously impacting the HPC market. Times are a changin so I'm not surprised to see traditional CPU-based CS research being downsized in response to this paradigm shift. Perhaps we need to take VIVA seriously just as Cray, SGI, Starbridge Systems, SRC, Nallatech and others are doing.

Yes, but...
From the CRAY homepage, they announce a computer, the Cray X1E Supercomputer, which tops at 147 TFlops.

But this can be that this computer was shipped AFTER the last Top 500, and that before the next Blue Gene will come with the double of nodes it has now... But CRAY still seems to be in the fight !

Sun's CNP is modeled after Tera's MTA architecture (now named Cray again), which trades memory latency for throughput. Basically, in MTA (massively threaded architecture) each of 128 processor threads issues a few memory fetch instructions and waits for the memory to arrive (dozens to hundreds of cycles). This happens for every thread so the effect is that memory fetches and execution time are separated... iow time=max(execution,fetch) vs time=exeuction+fetch of normal processors. This also makes having a pipeling irrelevant so no effore is wasted in branch prediction.

That's great for scientific apps since they are massively parallel... Sun has taken the same idea and scaled it down to 4 overlapping threads so normal applications can benefit. While it can be used to run 4 separate process threads at a time, at least the MTA's is fine-grained so that what really happens is that the compile changes a for (;;i++) loop into four (;;i+=4) loops and runs them in parallel.

This technology done right means a massive performance boost (as in like 25-50%) while also simplifying the processor. Contrast that this Hyperthreading, which complicates the processor and only gets ~5-8% benefit on average... it's mostly designed to minimize context switch times.

No they don't have top of the line anything.
I bet this is better, faster and more stable.

Actually this is a very good point. Cray computers use Linux, Quadrics computers use Linux, and I'm sure anyone who has made an "Imagine a Beowulf cluster" post has read this. If I was managing Microsoft, I'd want people to think Windows is a powerful OS, so the last place I'd want it to be overtaken on is the world's more powerful computers. Especially by a freely available OS like Linux. Thus...the upcoming supercomputer Windows.

BTW the Windows-for-supercomputers story is already rising up on the Google lists. Check on or before the 7th link.

It's an old Cray supercomputer. http://cray.com/about_cray/history.html/

Not just round to int. FP rounding mode refers to how all operations round the bottommost bit of precision--IEE754 specifies that operations must (effectively) be copmuted with sufficient extra precision at the bottom to then round in the various directions.

You might not but Cray would.

the process seems to have started already:
Cray SX-6 System

from their Tech.sheet they are using the Luster file system

This is the first time i see a shipped linux with this file system. Now the intersting part is that lusterfs is made for linux clusters, but this monster is not a cluster... any body can shed some light?

So, how does this compare to running Apple's Xserve? Bang per buck? Heat? Space? Etc etc....

There's not a lot to compare. We're talking apples and oranges. It's like asking to compare a PowerMac G5 with a bunch of PC parts scattered on the floor as desktop machines. Sure, you can put the PC together, load it with Linux, tinker with it to get everything working, etc. but that's a fair amount of work compared taking the PowerMac out of the box, plugging it in, turning it on, and having everything work perfectly.

Read the specs, particularly with regard to the interconnect, system administration, and hardware and software reliability features. This thing is seriously engineered to be massively parallel system with top of the line hardware and software to support and maintain that, as well as extremely impressive reliability features.

Jedidiah.

what kind of operation system runs on this beast?

UNICOS is usually a safe bet. In this case the specs say UNICOS/lc, which is made up of "SUSE(TM) Linux(TM), Cray Catamount Microkernel, CRMS and SMW software"

I'm not entirely clear how to interpet that, but I think it runs as follows: It runs the Catamount Microkernel as the kernel, and uses SUSE for everything else (so we have SUSE Linux, without the Linux - all of a sudden that GNU/Linux stuff starts to make sense). The CRMS is their interconnect management and monitoring software, and SMW is the System Management Workstation - which I'm guessing is their administration frontend.

It's worth noting that that's some pretty serious software there (because Cray has a lot of experience dealing with large systems) - you can bet that the management and monitoring software is some very serious stuff.

This thing is to a beowulf cluster what a dual G5 PowerMac is to homebuilt PC system running Linux From Scratch. It's going to work flawlessly "out of the box" with a smooth and polished interface that lets you get done everything you want to do simply and easily. You can of course make your home built PC with LFS work just as well, it's just going to take you an awful lot of effort.

Jedidiah.

Check it out in this video.

Cray's phase change uses Fluorinert, while the average PC uses Freon.

I went with an XP-90 to air cool my new Athlon 64. The heatpipes arguably make it passive phase change cooling.

In reply to my own post..

The Cray SX-6 System runs the UNIX-based SUPER-UX operating system.

Sorry about that. Maybe they ported GFS.. dunno.

It's very impressive and all, but how is this going to benefit me down the line? It's not like they're affordable to small/medium businesses like the Cray or HP's highly valued Alpha DEC workstations.

We are stiffling progress at the lower level by pricing these systems well beyond the reach of the average researcher or multi-national oil conglomerate.

Why is this?

Cray not-too-long-ago had major announcements with the RedStorm project. I believe that system is supposed to be a single image 10,000 CPU AMD based rig. There are some oddities friends have pointed out, like the OS is based on IRIX I believe...

Yea check this out:

Cray Unicos/mp"

Actually that references the X1, which is not based on PeeCee stuff, but actually a 8 core MPM.

Sad thing is, even with Red Storm I think IBM will remain on top as their contract calls for 130,000 of their powerPCs on one system?

It would be nice to see Cray on top, with something other than a commoditiy processors. I realize the T3D and T3E were both Alpha based systems.

PS, I still have a J932se 32 proc Vector Cray ( for sale ) if anyone wants a Cray for home. $4500, real deal 3 cabinet Cray from 97', most likely used for gov't nuclear energy something-or-other. Located in Southeastern Virginia.

Thats what it says on its specs sheet "Cray HPC-enhanced Linux, Kernel version 2.4.21"

I think its wise they went with the 2.4 kernel though, but i wonder what is this cray linux, never heared of it.

However, part of me is a bit saddened by seeing the Cray name attached to X86s. Yes, I felt the same thing with SGI, DEC, and Sun. Yes, I need to get over it and move on. :-)

No need to worry -- if you want a Cray vector processor-based supercomputer, you can still buy one. You can also source the parts for your own Earth Simulator from Cray, as well.

However, part of me is a bit saddened by seeing the Cray name attached to X86s. Yes, I felt the same thing with SGI, DEC, and Sun. Yes, I need to get over it and move on. :-)

No need to worry -- if you want a Cray vector processor-based supercomputer, you can still buy one. You can also source the parts for your own Earth Simulator from Cray, as well.

already exists

Cray already makes 12 processor optetron systems

It's called XD1. Used to be manufactured by Octiga Bay, but that company has been absorbed into Cray.

The Itanium can scale up to 8, 16, 32 or even 64 processor modules ...

Bear in mind it's not a COTS ( Commodity Off The Shelf ) system like a lot of clusters, it's not even a typical cluster like the kind I have at work ( oodles of dual CPU nodes connected via Ethernet ). SGI Altix machines scale up to 256 CPUs and 4 TB of memory in a single system image, which is one very large SMP machine ( actually, CC-NUMA ). I'm not sure how many CPUs per node ( OS image ) this machine will have, as the article didn't coherently state it, but bear in mind that a cluster of these is more like a cluster of IBM p690s than anything else. Massive memory available per node, very high speed interconnect, etc.

As far as I know, the only company doing anything similar w/ Opterons is Cray's XD1 systems, and they're along the same line as an SGI Altix.

So it's not actually a waste of taxpayer money. As much as I love linux clusters composed 1-4 CPU nodes connected w/ a commodity interconnect ( as you can almost consider, GigE, Myrinet, Quadrics, SCI, etc. - when compared with the proprietary interconnects used in SGIs, IBM SP switches, etc. ), there's some things that you can do with greater efficiency by having 128 processors and 2 TB of memory in a single system image.

That would be Fluorinert. Not a bad idea.

All the good parts of the Cray T90 were immersed in a big tank of Fluorinert.

Don't be silly, Apple is a very small fish in a very big pond compared to real supercomputer companies like SGI.

What the big companies do is let you login to their machines free of charge, over the internet, and take their machines for a spin. See e.g. http://www.testdrive.compaq.com

On top of that, what the companies with a *serious interest* in high performance computing (like SGI and Cray) do is let you email your program to them: they will spend days or weeks (up to you) tuning it, telling you how it works on their hardware, and even telling you that you're better off with the competition, if that ends up being the case. What these companies understand very well, that Apple doesn't, is that people who spend $10M+ on computing equipment, typically with public money, need to show due diligence in their choice of hardware. This means THOROUGH evaluation: benchmarks, benchmarks, benchmarks - and not _standard_ benchmarks, but the benchmarks that matter: the user's own software.

Take a look here to see a real-life snapshot of this kind of process.

Any company that tried to "fudge the numbers" would be caught out, and that looks VERY bad. So the companies instead do all they can to help with the evaluation process, and hope that they get chosen. If not, there's always the next sale. A big supercomputer is sold somewhere every day or two, after all.

Virginia Tech's "X" doesn't come into this category: for example, no actual scientific work has been performed on the machine so far apart from benchmarks/system development, etc. Dr Varadarajan is probably going to get a pretty nasty grilling in a couple of years, when the University asks "so, what did we get for our money?" Some P.R., and not a whole lot more it would seem. I wish them luck...

Apple is not really a supercomputer company: and really, who cares? Apple has been, is, and will probably remain a great computer company for a good long while yet. But don't expect Apple to help you out a great deal if you pop them an email going "say, how does the G5 run on the ASCI Purple benchmarks?". That's just not what Apple are about. (example: they are happy to sell you a box with a 64-bit CPU, but they could care less that they don't have a 64-bit OS to go with it..)

Dude, it's no secret:
http://www.cray.com/craydoc/manuals/S-2346-23/html -S-2346-23/z1019077484.html

Great video with a fair bit of detail on the X1's cooling system, chip density, etc.

View it with mplayer:

Cray X1

Posted Anon to not karma whore.

So John Carmack are you listening? Any chance this can be included in DOOM3?

Sure, add it all in! That way when DOOM3 is finally released it will only be available bundled with a new Cray X1.

Please, mod the parent UP !
and everybody else, please take 30 seconds to read the XD1 datasheet

"I seem to remember hearing..."

Maybe this glossary is where the person who told you read it from. Under PFIB it says:

"Perfluoroisobutylene. A toxic, colorless, odorless gas that can be produced when Fluorinert liquid thermally decomposes when exposed to open flames, glowing electric heating elements, electric arcs, or temperatures above 200 degrees Celsius."

From the description there it appears that while this guy is bashing Linux's ability to cluster, they seem to have no problem with it's ability to SMP against a specialized chassis. It also appears that their system a half-breed between what clustering offers and what a single computer can offer. Honestly, pretty cool technology, but still no excuse to bash the capabilities of Linux clustered.

Too bad that they couldn't have released with the much faster (for SMP, especially) 2.6 Kernel.

I wonder where we can download their Kernel Patch sets.

Hello,

actually, if you read the datasheet, the XD1 runs Linux 2.4.21 with some modifications (see xd1_datasheet.pdf

So does the SGI Itanium machine. What sets these computers apart is that they offer better interconnections between the processors than clusters do.
The Bigmac has 1.2GB/s between two nodes through Infiniband whereas an SGI machine has 6.4GB/s.

As a summary, in a cluster you use slower links with higher latency and your processors communicate through messages.
In a SGI or Cray machine, you use fast and expensive links (think more wires, more expensive controllers) and your processors can work as though they all shared the same memory.

SGI sells systems with 128 processors where there is only ONE Linux kernel (as opposed to 128 in a Linux cluster).

I have a rack right here housing a linux cluster. 36 1U dual-Xeon servers. On the Cray XD1 site it details the "Exceptional Performance" of the XD1 system. It details the performance of a system with 12 AMD Opteron processors, and the performance of a rack of systems with 12 AMD Opteron processors. I understand that the underlying architecture of those servers may be vastly different than the servers in the rack next to me, but fundementally aren't they both multi-processor PC servers operating in a cluster? If so why does their rack full of multi-processor systems qualify as an HPCbut mine does not?