Slashdot Mirror
How to get 1.5 TeraFlops from Linux
Oak Ridge National
Lab has purchased from SGI an Altix
3000 (flash movie). This
article claims that:
SGI Altix 3000 is recognized as the first Linux cluster that scales up to 64 processors within each node and the first cluster ever to allow global shared memory access across nodes.
There is more here,
here,
and here.
"SGI Altix 3000 is recognized as the first Linux cluster that scales up to 64 processors"
SCO will be all over your ass now!
Do not look into laser with remaining eye.
After all the beowulf cluster jokes, I am still incredibly curious about them. My goal is to build a small 5-6 node cluster by the end of the summer. The thing is, I still know very little about them. Every jokes about them, but no one puts any useful information. Are there specific langauges one must program in to tak advantage of the multiple processors? Or does the OS take care of that? How much speed can you actually get out of them? Is it pure processing power? Or is there more? I'm very curious and want to know.
Oh great... I can see Jobs wringing his hands already.
"Now how am I going to make the G5's look faster than THIS?"
they going to release their kernel that allows them to globally share memory? or is it more of a hardware thing, than software?
"Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms,
You're better off using mosix. It'll allow for more normal (ie, not beowulf specific) applications to thread across computers. I'd imagine that an open-mosix setup (like the ones using the knoppix boot CDs tailored to it) could probably make for a fairly powerful computing cluster very easily.
Do not look into laser with remaining eye.
(Inside joke for my ol' friends at ORNL...)
It's funny that Microsoft always tries to downplay Linux's enterprise capabilities, when Linux has been scaled to far more power then Microsoft's best offering for years now. Windows 2003 is a clumsy, bloated, closed source chunk of green crap.
...now you get obscene frame rates on quake III while searching for those pesky pockets of natural gas!
HPC Wire had an article that I referenced in my journal on 6/30.
It's an interesting machine. I'd love to get one to play with. I'm sure our benchmarkers will have some even more interesting comments once they're done. Expect teething problems, folks. Systems of this size and complexity take time to break in.
Do you know why the road less traveled by is littered with the bones of the unwary?
makes me just wanna turn off the lights and look at all those LED's blinkin!
OK, so the moderators are on crack today. What's with all these obviously "funny" posts getting moderated as "insightful?"
Guess it's time to meta-moderate!
Just be sure to wear the gold uniform when you beam down -- you know what happens when you wear the red one.
The machine has 256 processors for 1.5 teraflops, not 64.
How hard would it be to /. one of these things??
What's that in bogomips?
Perhaps this will finally get SGI's Open Source Software efforts in the spotlight.So far every other major hardware vendor has jumped the bandwagon making a lot of noise, and trying to get free publicity. SGI however has always quietly contributed large amounts of knowledge but always in a modest or even shy way (sometimes even publicly denying involvement, but working in secret :) ).
In the meantime their additions have contributed quite a bit to open en free thinking in software, take OpenGL and open Inventor, or even to the kernel directly as with the XFS filesystem.
I always liked this approach more than the hyping others have done with linux, but unfortunately this has kept them unadorned within the community. With the Altix cluster (as with their GNU/Linux workstations,which unfortunately failed) I think they have shown that they put their money where their mouth isn't.
I think it's only fair that when we are talking about the large coorporate players in the OSS field SGI at least deserves a footnote for their efforts instead of just hammering exclusively on IBM,Sun etc..as the great backers.
I know, I know. It's a coorporation, so they inherently put money over freedom, it's just something I noticed because of the lack of their name in any high-profile discussions, which I think is unfair.
without SCO's help?
Mod parent +1 Funny!
http://use.perl.org
Um..
I always liked Irix, and everyone I ever talked to who used Irix liked it. The GUI is about 500x more usable than the horrors of OpenWindows or CDE on Solaris.. bleugh.
"Hey! Unless this is a nude love-in, get the hell off my property!!"
What I find amazing is that the Cell is supposed to run up to a TeraFlop when it reaches production. That compared to a 64 processor Linux cluster.
thats 64 processors per node
We just got ours installed yesterday. I'm still installing software and am starting benchmarks. It's only the deskside version (12 cpus, 24GB RAM, 1TB disk), but still more powerful than the 4-cpu SGI Origins that we have been using.
It is the first one that the regional SGI reps had actually installed, but since it is almost exactly the same as the MIPS-based origin 3000 servers (with the exception of the obviously different Itanium 2 cpus and supporting chipsets), they ran into almost no problems getting it online. I have also been suprised as to how many commercial codes have already been ported to the platform.
The main reasons we purchased this machine is for the ease in parallelizing code and the floating point performance of the Itaniam 2 cpus. We're computational materials engineers and the less time we have to spend optimizing codes so that the nodes of a cluster are always kept busy and minimizing I/O bottlenecks gives us more time to concentrate on the theoretical issues.
It runs RedHat 7.2 with some tweaks by SGI called SGI ProPack. The Propack modifications come on separate CDs, with the proprietary software on separate CDs from the open source software. So far, from the command line, everything works just like my PC. It's kind of strange running Linux on a >$100K machine, but it sure beats dealing with the annoying differences between IRIX and Linux. Now to see if it performs as well as we expect...
or, try Quantix, which is derived from cluster knoppix. A self-booting ISO with data analysis software, based on Knoppix. This is geared more for scientific apps; it doesn't come with open office, etc, which cluster knoppix does.
Eat Lamb, 1 million coyotes can't be wrong
Main product page: http://www.sgi.com/servers/altix/
and here there are bunch of PDFs to download: http://www.sgi.com/servers/altix/datasheets.html
for example:
SGI Altix 3000 Family of Servers and Superclusters (172K)
Linux Software for the SGI Altix 3000 Family (50K)
SGI Technology Solutions for Linux (48K)
This Like That - fun with words!
Kernel compile time with make -j?
Hmm, what are some other good ones?
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
The thing about Mosix is the costs of process migration.
First, you have to understand process migration. In a mosix cluster, a running process can be moved, lock stock and barrel, from one CPU to another. All that is left behind is a "stub" process that forwards all file I/O across the network to the new location. So, if the program was a 3D raytracer that had the source description file and the output file open, after migration all file accesses to those files would be forwarded over the network to the stub (since you cannot guarantee that the remote machine can access those files in the same way.)
Now, this is great for programs that do little file I/O but lots of computing (for example the ray tracer I just described.)
However, the process must be set up on the local node first, then migrated. If the process has a 3 G core image (is taking up 3G of memory), then 3G of stuff has to be shoved across the wire, while the program is frozen. Thus, migrating a process is expensive.
Now, if you have a bunch of long-running compute bound processes this is a net win (for example, rendering a movie might benefit). But something like building the Linux kernel won't benefit, since what you have is a bunch of short running, high I/O jobs.
We have a Mosix cluster at work. I tried using it as a compile farm, and the results were disappointing. Not surprising - I was NOT using it for what it was designed for.
However, if we can ever get the FPGA synthesis tools running natively under Linux, the hardware types are going to be quite happy....
www.eFax.com are spammers
'Cause it is surviving a /.ing with a Flash intro even!
From excellent karma to terible karma with a single +5 funny post...
I hated it, if it helps.
I always liked Irix, and everyone I ever talked to who used Irix liked it. The GUI is about 500x more usable than the horrors of OpenWindows or CDE on Solaris.. bleugh.
I vastly prefer 4DWM to GNOME or KDE as well. I'm helping a coworker set up a Dell inspiron 7500 (P3-700) with Linux, and he immediately complained that KDE was far too slow. I switched to WindowMaker, and he immediately noticed the difference. This is a three-year-old machine, with tons of memory and a reasonable processor, and it crawls with KDE3. Pathetic.
Meanwhile, you can run the latest version of Irix on a seven-year-old SGI box (and even older) and it'll still be smooth. My Indy at home feels just as responsive as any PC I've ever used. I wouldn't call it *fast* by any stretch of the imagination, but the OS alone does not cripple the computer. I'm a huge Linux fan, but there are tons of examples like this where it just hasn't caught up to the more polished offerings.
$500 for the scalp of anyone who says the words "Beowulf" and "cluster" in the same post in response to this article.
"All great wisdom is contained in .signature files"
You can find a list here. For most computations and most hardware, you are probably still better off with MPI or PVM rather than shared memory.
Note also that there are several high speed interconnects for Linux clusters available from many different vendors, including InfiniBand, Gigabit Ethernet, FireWire, and Myrinet.
SGI systems (Origin and Altix) have massive interconnects that hold together the single-system architecture. They're fast for shmem-type shared memory apps, but also for MPI. In fact, SGI keeps tweaking their MPI implementation with every release of IRIX and the Linux ProPack, even though MPI is not the "best" way to run apps on their systems.
The interconnects in most Origins and Altix systems are 3.2 gigaBYTE per second with extremely low latency. I don't know about Infiniband, but I do know that GigE is only 125 MB/sec with really high latency... FireWire 800 is 100 MB/sec with better latency.... and I think the bst version of Myrinet is 500 MB/sec (4 gigabit) with about 5x the latency of SGI's 'numalink'.
The smaller Altix systems (and supposedly, future Altix and Origin systems this fall) can be double cabled or can run at a higher speed... for 6.4 gbyte/sec per interconnect.
Also, the Altix can handle up to 64 processors per single machine / single node (or 128 with a very beta set of patches). The cluster in the article is actually four Altix systems, each with 64 processors. The Origin 3800/3900 can handle 512 processors per node (or 1024 with a special "XXL" IRIX kernel).
Great stuff for I/O intensive tasks, but massive overkill for 3d rendering or calculating pi.
This does not seem to have been mentioned before:
Niflheim at Danish University of Technology