New SGI Altix 3000

dlloyd writes "SGI has just publicly announced the Altix 3000 series of computers that can scale from 4 to hundreds of processors, with up to 64 processors per single system image. Processors each come in a C brick that has 4 CPUs. I/O is done though IX and PX bricks (12 PCI slots per brick, IX bricks have a base I/O controler and two ultra 160 disks inside), just like on the Origin 3900 series. Anything more than 8 CPUs (2 C bricks) is connected by R bricks, which route the NumaLink packets between nodes. The NumaLink network is good for an aggregate 6.4 gigabytes/sec to *each* node. That scales as you add more C and R bricks. Basically, you can think of this as SGI's origin 3000 series, except that it runs Linux and has Itanium2 processors. The performance and scalability is like nothing that has ever run Linux and is *far* ahead of the competition. For those of you who wonder why anyone would need a 64 processor Linux machine, many scientific and technical customers prefer running their code on large, single system image machines. Large single system image machines are also less labor intensive to maintain and admin, plus they work much better on code that needs to share memory and pass messages between threads (even myrinet and mpi is glacial compared to the SGI numalink network and running code multithreaded)."

Comments

Re:Isn't the issue in this area $/MIPS?

[XaXXon]

Yeah, you're wrong. This isn't a beowulf, it's a multiprocessor box. It runs standard software. That means you don't have to re-write everything to support clustered solutions.

Lots of people don't understand that a 1024-processor beowulf won't run battlefield 1942 (if you've ever played it, you understand what I'm talking about), because it's not like a 2-processor workstation box. You have to write your software so that discrete pieces can be offloaded to other nodes and have the results posted back. A beowulf cluster is similar to SETI@Home or whatever distributed computing project you like. Though the interconnects are faster, the general idea behind how the software works is similar.

With this SGI system, it's like a 2-processor workstation on steroids. You can run standard multi-threaded code on it and actually use 1024 processors (and could possibly run battlefield 1942).

Huh? it uses Itanium 2

[green pizza]

SGI's Altix machines use Itanium 2 CPUs (up to 150 watts per CPU). They have **VERY** advanced cooling subsystems. This is not the MIPS/IRIX Origin series, this is the Itanium2/Linux series.

Re:Isn't the issue in this area $/MIPS?

[Zathrus]

No, because $/MIPS is a misrepresentation. Heck, MIPS alone is meaningless, because all it does is take a theoretical maximum of CPU speed. MIPS doesn't take into account anything beyond CPU speed - like memory speed, backplanes, drive arrays, etc.

If you have heavily interrelated datasets, like in just about any thermal dynamics/plasma/weather problem, then there is so much interdependancy between adjacent "cells" that each work unit needs information from adjacent work units constantly. Spread that system out on a cluster solution and you're DOA because your communications between boxes are horrendously slow, with latencies measured in milliseconds instead of nanoseconds. So while you may have some absurd number of MIPS, the reality is that the CPUs are sitting idle 90% of the time waiting for data from some other CPU/memory block.

Take all those CPUs, all that memory, put them in a single box and do the backplanes and memory interfaces right (this is where the cost comes in by the way) and your latency becomes reasonable and you actually get all those MIPS.

It boils down to what the problem set is. If you need an obscene amount of transactions or have a highly interdependant problem set then you're better off with a single large box. If you can break up the problem set and minimize interactions then clustering is your friend.

There's also the issue of maintainance, and while the hardware costs may be lower for a large cluster, the time spent fixing the hundreds of boxes may kill you. Have a single box that's designed for redundancy and you'll pay a fortune for the support contract, but you won't spend an appreciable amount of your time on hardware support on the rare occasions it actually needs something.

Re:Itaniums?

[jbischof]

Itaniums are marketed toward high-end servers only. Generally not available to, and too expensive for, the general public (if that is what you mean by regular consumer).

In Intel's mind, the Itanium doesn't compete with the Opteron. Opteron will be at Xeon's throat, trying to tear up some of the 95% market share that xeon has in corporate and other mid-range servers.

Re:Correction

[fgodfrey]

I think you're confusing NUMA with message passing. NUMA stands for Non-Uniform Memory Access (actually, this machine is cc-NUMA - Cache Coherent NUMA, but I digress). NUMA means that when I do a standard memory reference it will go faster or slower depending on where in memory that reference goes. This is accomplished by having a group of processors and a group of RAM DIMMs tied to each other with a memory controller that is also a router. If you want someone else's memory, you go over the router to the other memory controller and it returns the answer. That takes longer than your local memory (longer vs. shorter is not uniform), hence the machine is NUMA. IIRC, this machine is NUMA after 2 processors up to the max system size. Despite running multiple Linux kernels, all the memory is visible to all the processors even outside your own kernel. It seems they've picked 64p as the maximum useful size for a single kernel.

What is your quad P4? That's SMP - symetric multiprocessing. Symetric means that all memory accesses take the "same" amount of time since there is only one pool of memory for all the processor and no processor is closer to it than any other. SMP systems larger than around 32 processors are rare since your single memory subsystem needs to feed *all* theprocessors.

So what is a Beowulf cluster then? A typical Beowulf cluster (well, just a cluster in general) is a group of nodes which can't directly address each other's memory and hence have to send a message to the other guy to read/write his memory. Cards like Myrinet exist to try to get some form of shared memory between the nodes in the cluster to varying success. Compared with this, they are low bandwidth and high latency. (Of course compared with a Cray X1, this machine is low bandwidth, but I'm biased :)

There have been a variety of NUMA machines released over the years. Highlights other than this thing include the Thinking Machines boxes, the Cray T3D and T3E, the SGI Origin series, and the Cray X1.