New SGI Altix 3000

← Back to Stories (view on slashdot.org)

Posted by ryuzaki0 on Tuesday January 7, 2003 @05:57AM from the heavy-metal dept.

dlloyd writes "SGI has just publicly announced the Altix 3000 series of computers that can scale from 4 to hundreds of processors, with up to 64 processors per single system image. Processors each come in a C brick that has 4 CPUs. I/O is done though IX and PX bricks (12 PCI slots per brick, IX bricks have a base I/O controler and two ultra 160 disks inside), just like on the Origin 3900 series. Anything more than 8 CPUs (2 C bricks) is connected by R bricks, which route the NumaLink packets between nodes. The NumaLink network is good for an aggregate 6.4 gigabytes/sec to *each* node. That scales as you add more C and R bricks. Basically, you can think of this as SGI's origin 3000 series, except that it runs Linux and has Itanium2 processors. The performance and scalability is like nothing that has ever run Linux and is *far* ahead of the competition. For those of you who wonder why anyone would need a 64 processor Linux machine, many scientific and technical customers prefer running their code on large, single system image machines. Large single system image machines are also less labor intensive to maintain and admin, plus they work much better on code that needs to share memory and pass messages between threads (even myrinet and mpi is glacial compared to the SGI numalink network and running code multithreaded)."

34 of 225 comments (clear)

SGI is still in business? by yppiz · 2003-01-07 06:00 · Score: 5, Interesting

What is keeping SGI afloat? Service contracts on existing machines?

--Pat / zippy@cs.brandeis.edu
1. Re:SGI is still in business? by yppiz · 2003-01-07 06:13 · Score: 5, Interesting
  
  Thanks for the suggestion. So I did take a look, and the answer to the question "how are they staying afloat?" is "they aren't. They are steadily hemmorhaging money."
  I posted my question because, as early as 1996 or 1997, it was clear that commodity machines were going to kill SGI. I'm just amazed that they're still alive.
  Here's their income statement, balance sheet, and cash flow statement. As of today, For three out of the previous four quarters, they had sales growth (sic) of -20% or worse. The three analysts who cover this stock have a hold rating, which in analyst-speak means sell.
  --Pat / zippy@cs.brandeis.edu
2. Re:SGI is still in business? by Nynaeve · 2003-01-07 06:17 · Score: 3, Informative
  
  From the NASDAQ Summary:
  Revenue: $1.3 billion for fiscal year 2002
  servers accounted for 38% of fiscal 2002 revenues; Global services, 34%; Visual workstations, 18% and other, 10%
  To answer your question, the revenue from the sales of services is only about one-third of their total revenues. I don't know if this is considered a lot or not.
  IBM has a similar report: global services accounted for 41% of 2001 revenues. This is before the purchase of PWC, so it is probably going to be higher in 2002.
3. Re:SGI is still in business? by kelzer · 2003-01-07 06:19 · Score: 5, Funny
  
  Hmmmm, you seem a little touchy about the subject, and you posted as an AC. One might be inclined to think you had an interest in SGI.
  
  Perhaps you've purchased a LOT of SGI stock in the last year, MR. ROBERT R. BISHOP! Ha! Thought you could sneak in an anonymous post, but you underestimated the analytical and investigative abilities of us Slashdotters.
  
  Nice try.
  
  So, got any old O2's laying around gathering dust, that you don't need anymore?
  
  --
  
  ---------------------------------------------
  SERENITY NOW!!!!!!!!!!!!!!!!
Why Linux? by HeelToe · 2003-01-07 06:00 · Score: 4, Interesting

I still don't understand why SGI has foregone such a great OS as IRIX. Why go with Linux? Just trendy, or does it really offer advantages for scientific computing?
1. Re:Why Linux? by larien · 2003-01-07 06:02 · Score: 5, Insightful
  
  Yup, IRIX was good, but maintaining a full OS takes a lot of money. This way, they can piggy-back on investments made by other people & companies while still having a modern OS. They've already integrated XFS into linux, and it wouldn't surprise me to see other SGI/IRIX technologies coming into linux in the same way. Similarly, IBM have migrated JFS into linux.
2. Re:Why Linux? by drinkypoo · 2003-01-07 07:21 · Score: 3, Informative
  
  IRIX has a terrible name among geeks who have only dabbled with it (like me) who have noticed their amazing track record with security: Amazingly goddamned stupid. Executing "xhost +" by default every time you log into X is the most idiotic thing I can think of. The patch cluster for IRIX 5.3 is bigger than IRIC 5.3! And so on. Given how bad they've been it's hard to trust them now.
  On the other hand, Linux has two things which give it a better name that IRIX; it's open/free, which is obvious, and it's new, so it has an excuse to have some 'issues'. Meanwhile IRIX was around a long time and still horribly, terribly insecure. It's not worth it to me to dress up Unix prettily and easy to use in exchange for security; I frankly want both.
  Linux is also growing in leaps and bounds and implementations using it like this one only serve to prove this; support for vast numbers of processors is one place Linux has traditionally flailed, but as time goes by and manufacturers expend more money on making Linux scale, the last few blocks to running the same OS on a PDA and a supercomputer (IE, from a common, unpatched codebase) are going away. That is undoubtedly powerful because your code will (theoretically) work on any platform large enough to contain it with nothing more than a recompile.
  Granted, there are attempts to do that at the application level rather than the OS level -- and I'm talking about Java and .NET here -- But they fit a somewhat different need, and they will likely never be as compatible between disparate platforms as having the same operating system underneath your program (duh.)
  
  --
  "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
The older the boy... by RebelTycoon · 2003-01-07 06:00 · Score: 5, Funny

IX, C and R bricks

The more expensive his LEGO gets...

--
Tournament Management Online &
1. Re:The older the boy... by fgodfrey · 2003-01-07 10:13 · Score: 4, Insightful
  
  ...and you thought you were joking. "Lego Hardware" was an internal code name for this machine, and the Origin systems at one point. Rumor was that they wanted to use it publicly but LEGO(tm) objected. Not sure if the rumor had any truth to it, though, since the name was dropped before I started working for SGI/Cray.
  
  --
  Go Badgers! -- #include "std/disclaimer.h"
Shoot me by Znonymous+Coward · 2003-01-07 06:01 · Score: 3, Funny

Imagine a... Bang!

--
Karma: The shiznight, mostly because I am the Drizzle.
passing messages by 56 · 2003-01-07 06:03 · Score: 3, Funny

...and pass messages between threads
Is that something you would like to share with the class, Altix?
SGI is finally making some new products by ozzee · 2003-01-07 06:04 · Score: 5, Insightful

Scientific computing has allways been SGI's niche. They unfortunately stumbled around the time that Belluzzo took the helm and wasted the entire internet bubble recovering from the mess that caused.
It's great to see that they're finally back and doing some really serious new stuff.
It's a shame though that they won't be running the AMD 64 bit chips, although, I'll be someone is looking into that.
Congrats SGI !
1. Re:SGI is finally making some new products by Jeff+DeMaagd · 2003-01-07 06:41 · Score: 3, Insightful
  
  Actually, the packaging itself sounds interesting, something that others haven't done yet.
  
  Even if it does use chips made by someone else, there is a lot of work that goes into making those chips work together, and probably a fair amount of work adjusting the Linux kernel into something that scales so large.
  
  I bet they had a lot of work cut out into making this thing modular like it seems to be. Probably has a lot of work just in the chipset too.
  
  It is unfortunate that it uses Itanium2 chips. I don't think competing Opteron systems will be available for at least little while yet. I think I heard from an AMD rep that Cray had commited to making systems based on Opteron.
Don't forget by PD · 2003-01-07 06:05 · Score: 5, Insightful

These machines support 512 GB of RAM in one chunk. A Linux cluster might outperform this thing, but you'll need to chunk your data up to fit into the individual nodes' memory. Sometimes this can be a pain in the neck to do, hence the market for something like this.

--
If tits were wings it'd be flying around.
Brick processor by intermodal · 2003-01-07 06:13 · Score: 3, Interesting

I rather like the concept of this...no more trying to pair up older processors when you run across a board someone is getting rid of a few years down the road. I recall getting a couple dual-PII workstations a year or so ago, and finding a pair of matching (and working) processors to put in them was hell...this way I could have just searched Ebay or my parts stash for a single old part.

--
In SOVIET RUSSIA... erm...NSA AMERICA, the Internet logs onto YOU!
Isn't the issue in this area $/MIPS? by mckwant · 2003-01-07 06:14 · Score: 3, Interesting

I don't do this for a living, but it seems that $/MIPS is the only benchmark even worth discussing, so shouldn't one be able to put together massive clusters of boxen to do the same thing, only without the SGI price tag?

Correct me, because I'm almost certian I'm wrong.

--
ceci n'est pas un sig.
1. Re:Isn't the issue in this area $/MIPS? by XaXXon · 2003-01-07 06:20 · Score: 5, Informative
  
  Yeah, you're wrong. This isn't a beowulf, it's a multiprocessor box. It runs standard software. That means you don't have to re-write everything to support clustered solutions.
  
  Lots of people don't understand that a 1024-processor beowulf won't run battlefield 1942 (if you've ever played it, you understand what I'm talking about), because it's not like a 2-processor workstation box. You have to write your software so that discrete pieces can be offloaded to other nodes and have the results posted back. A beowulf cluster is similar to SETI@Home or whatever distributed computing project you like. Though the interconnects are faster, the general idea behind how the software works is similar.
  
  With this SGI system, it's like a 2-processor workstation on steroids. You can run standard multi-threaded code on it and actually use 1024 processors (and could possibly run battlefield 1942).
2. Re:Isn't the issue in this area $/MIPS? by Zathrus · 2003-01-07 06:37 · Score: 5, Informative
  
  No, because $/MIPS is a misrepresentation. Heck, MIPS alone is meaningless, because all it does is take a theoretical maximum of CPU speed. MIPS doesn't take into account anything beyond CPU speed - like memory speed, backplanes, drive arrays, etc.
  
  If you have heavily interrelated datasets, like in just about any thermal dynamics/plasma/weather problem, then there is so much interdependancy between adjacent "cells" that each work unit needs information from adjacent work units constantly. Spread that system out on a cluster solution and you're DOA because your communications between boxes are horrendously slow, with latencies measured in milliseconds instead of nanoseconds. So while you may have some absurd number of MIPS, the reality is that the CPUs are sitting idle 90% of the time waiting for data from some other CPU/memory block.
  
  Take all those CPUs, all that memory, put them in a single box and do the backplanes and memory interfaces right (this is where the cost comes in by the way) and your latency becomes reasonable and you actually get all those MIPS.
  
  It boils down to what the problem set is. If you need an obscene amount of transactions or have a highly interdependant problem set then you're better off with a single large box. If you can break up the problem set and minimize interactions then clustering is your friend.
  
  There's also the issue of maintainance, and while the hardware costs may be lower for a large cluster, the time spent fixing the hundreds of boxes may kill you. Have a single box that's designed for redundancy and you'll pay a fortune for the support contract, but you won't spend an appreciable amount of your time on hardware support on the rare occasions it actually needs something.
3. Re:Isn't the issue in this area $/MIPS? by pclminion · 2003-01-07 07:32 · Score: 3, Interesting
  
  If you have heavily interrelated datasets, like in just about any thermal dynamics/plasma/weather problem, then there is so much interdependancy between adjacent "cells" that each work unit needs information from adjacent work units constantly. Spread that system out on a cluster solution and you're DOA because your communications between boxes are horrendously slow, with latencies measured in milliseconds instead of nanoseconds.
  You've obviously never actually researched how distributed finite-element simulations work, because you're absolutely wrong.
  In most physical FE methods, each cell interacts only with its 6 nearest neighbors. Yes, the computation requires information that spans across cells, but there's no reason to assign a different CPU to each cell. The cells can be grouped into blocks and assigned to processors that way.
  Remember that surface area increases slower than volume. As the sizes of your cell groups increase, there is more volume within them per unit surface area. And since data only needs to be communicated across the SURFACE of each cell group per simulation timestep, the method actually gets MORE efficient as you make the simulation bigger.
  So your "obscene" number of transactions turn out to be highly localized in space, which minimizes communication overhead. In fact, if your cell blocks are box-like in shape, then each block requires only 6 logical communication links to the adjacent boxes. This could be realized by a traditional switching fabric, or with actual physical links.
Looks nice too... by Midnight+Thunder · 2003-01-07 06:21 · Score: 4, Insightful

One thing that I have always liked about SGI systems, is that not only do I get a high performance system, but I also get something that looks good design wise. Other companies, such as IBM give me the feeling that I am buying, in equivalent terms an F1 car with the body of a Lada. If I pay top of the line prices, I also like to have something nice to show off.

--
Jumpstart the tartan drive.
Altix? by Gudlyf · 2003-01-07 06:21 · Score: 5, Funny

With all those bricks invovled, maybe they should call it the SGI Tetrix.

--
Trolls lurk everywhere. Mod them down.
Correction by XaXXon · 2003-01-07 06:25 · Score: 3, Informative

Now that I've RTFM, let me correct my previous comment -- The Altix3000 runs a single Linux image over up to 64 processors and 512 GB RAM. After that, it's NUMA.

It can, however, do high-speed shared memory over all nodes in the cluster, allowing you to store HUGE shared data sets. Here's a link to the info on the memory.
1. Re:Correction by fgodfrey · 2003-01-07 13:29 · Score: 4, Informative
  
  I think you're confusing NUMA with message passing. NUMA stands for Non-Uniform Memory Access (actually, this machine is cc-NUMA - Cache Coherent NUMA, but I digress). NUMA means that when I do a standard memory reference it will go faster or slower depending on where in memory that reference goes. This is accomplished by having a group of processors and a group of RAM DIMMs tied to each other with a memory controller that is also a router. If you want someone else's memory, you go over the router to the other memory controller and it returns the answer. That takes longer than your local memory (longer vs. shorter is not uniform), hence the machine is NUMA. IIRC, this machine is NUMA after 2 processors up to the max system size. Despite running multiple Linux kernels, all the memory is visible to all the processors even outside your own kernel. It seems they've picked 64p as the maximum useful size for a single kernel.
  What is your quad P4? That's SMP - symetric multiprocessing. Symetric means that all memory accesses take the "same" amount of time since there is only one pool of memory for all the processor and no processor is closer to it than any other. SMP systems larger than around 32 processors are rare since your single memory subsystem needs to feed *all* theprocessors.
  So what is a Beowulf cluster then? A typical Beowulf cluster (well, just a cluster in general) is a group of nodes which can't directly address each other's memory and hence have to send a message to the other guy to read/write his memory. Cards like Myrinet exist to try to get some form of shared memory between the nodes in the cluster to varying success. Compared with this, they are low bandwidth and high latency. (Of course compared with a Cray X1, this machine is low bandwidth, but I'm biased :)
  There have been a variety of NUMA machines released over the years. Highlights other than this thing include the Thinking Machines boxes, the Cray T3D and T3E, the SGI Origin series, and the Cray X1.
  
  --
  Go Badgers! -- #include "std/disclaimer.h"
Huh? it uses Itanium 2 by green+pizza · 2003-01-07 06:25 · Score: 4, Informative

SGI's Altix machines use Itanium 2 CPUs (up to 150 watts per CPU). They have **VERY** advanced cooling subsystems. This is not the MIPS/IRIX Origin series, this is the Itanium2/Linux series.
Re:Cool by Quill_28 · 2003-01-07 06:40 · Score: 3, Insightful

Why would SUN and HP have to release a version of linux on their high-end systems?

SUN cost for Solaris wouldn't be higher than SGI to port Linux to their systems.

It would seem that people producing high-end systems costs would go down, using a no-cost license fee os like Linux.
IBM is doing something similar by cant_get_a_good_nick · 2003-01-07 06:44 · Score: 3, Informative

From the Register
MPI? by Alex+Belits · 2003-01-07 06:45 · Score: 4, Interesting
even myrinet and mpi is glacial compared to the SGI numalink network and running code multithreaded

Don't mix shitty parallel computation libraries and actual performance. Multithreaded applications without MPI are, of course, faster than anything with MPI, however it says absolutely nothing about:
1. Multithreades vs. multiple processes.
2. Myrinet
3. Network programming
4. Clustering
5. NUMA implementations that reduce everything to SMP with a cache that gets flushed a lot
--
Contrary to the popular belief, there indeed is no God.
Re:Itaniums? by jbischof · 2003-01-07 06:48 · Score: 5, Informative

Itaniums are marketed toward high-end servers only. Generally not available to, and too expensive for, the general public (if that is what you mean by regular consumer).
In Intel's mind, the Itanium doesn't compete with the Opteron. Opteron will be at Xeon's throat, trying to tear up some of the 95% market share that xeon has in corporate and other mid-range servers.
Re:Huh? (was Re:SGI is still in business?) by kelzer · 2003-01-07 06:48 · Score: 3, Funny

Of course, if the investing public at large learns this, then the analysts will have to readjust their ratings, introducing a new one - "Extra Strong Buy" that simply means buy, while "Strong Buy" will mean hold, "Buy" will mean sell, and so on.

And once the public catches on to that, . . .

--

---------------------------------------------
SERENITY NOW!!!!!!!!!!!!!!!!
Re:SGI processors.. by Wesley+Felter · 2003-01-07 07:08 · Score: 3, Informative

The above comment is a cut-and-paste from the R16000 story.
Re:Cool by larien · 2003-01-07 07:41 · Score: 3, Interesting

A Sunblade 100 with minimum spec is $995 list. You can buy a PC for half that with similar or better performance. To get performance better than a P3/4, you're really looking at something like a blade 1000 which starts off at around $5,000 (off the top of my head).
Yup, the blade 100 is nice, but it doesn't have price/performance of Intel yet. What it does have is binary compatibility with an F15K, so you can run the same 64-bit app on your cheap-ass desktop as you do on the big iron in the data centre.
The poster is right; most hackers don't have a Sun on their desk as they are comparatively expensive. Given the choice between a $1000 Sun which plays very few games and has OK performance or a $1000 PC which will play 99% of games, what do you think the average hacker is going to buy?
Re:IRIX is better for computational graphics than by fitten · 2003-01-07 07:41 · Score: 3, Interesting

Well... our experience with the O2K was a bit different (haven't touched an O3K though so I dunno much about it). With the Origins 2K and NUMA, if you ran large simulations, the distant memory would flat kill your performance so you had to make sure your working sets fit on a single node. Also, even though each processor board in the O2K had 2 CPUs, if you actually ran an even somewhat memory intensive process on each CPU, your app slogged because the memory bandwidth only supported about 1.4X the bandwidth of a single CPU. Another bad habit was that if someone snuck onto the machine when you were running your 64 process job and they fired up something like emacs, it would cause this nasty shifting around of memory that got you into the NUMA state mentioned above. Later, SGI came out with some tools that would minimize this effect though, which helped a lot.

So... in order to support the kind of stuff we did and run it on 64 nodes, we would have had to buy a 128 processor system because we would only run one process per node and all the nodes were two processors.

So, we also looked at the Sun10K. While the cpu to cpu comparison of raw crunch was lower, the memory bandwidth was uniform so the programs behaved predictably and were almost as fast in any case. It also had the benefit of running 1 process (as far as performance per process) wasn't noticably different from running 64 processes (on a 64 processor machine). At the time, the Sun10K met our needs much better.

Now, of course, all those have been shoved out the door and replaced by something even faster.
You guys are missing the main point! by LinuxParanoid · 2003-01-07 07:49 · Score: 5, Insightful

You guys are all missing the main point!

SGI is the first billion-dollar systems vendor to move their totally high-end million dollar hardware to run Linux, and not just to run Linux poorly, but instead their mega-boxes *require* Linux to performe excellently (unlike, say IBM "Linux/390" mainframes where Linux is not really the native OS supporting all the hardware features and is mostly a curiousity or very expensive Apache server.)

The other vendors, Sun, HP, DEC, IBM have not been nearly as aggressive and are depending on their own UNIXes to remain on their high-end boxes.

SGI is depending on Linux and has tweaked it enough to run huge, 64-way complex NUMA systems. This is a major infrastructure bet on Linux, and (assuming this is a shipping, working product) a huge mark of progress for Linux that it can, today, support this sort of high-end scalable hardware.

We all knew it *could*, in theory, but SGI has invested in making sure that *it does*!

This marks a major shift of SGI to an Intel/Linux pure play. It's not just a bunch of low-end Linux server boxes (which they've done before, and Sun/HP/IBM also do), or boxes that you can run either Linux or some proprietary UNIX. It's a full-scale massive 64-way NUMA SMP server that is optimized to run Linux.

Hats off to SGI, I say.

(I wish they had better business prospects but its hard to do that with a niche sort of product like high-end SMP/NUMA technical computing. We'll see if they can push it into a broader customer base with sufficient application support.)

I wonder how Oracle would do on this sort of puppy?

--LP
Re:Price? by afidel · 2003-01-07 08:28 · Score: 3, Informative

Those prices really aren't bad. $70,000 for 4 cpu's and 32GB of ram is almost exactly what we paid for our 4 cpu 32GB Sun V440 a couple months ago and this thing has more cpu power and a lot lot lot more expandability. $1.1 million for 64 cpu's is pretty cost competitive with Sun and IBM too.

--
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.