Linux Clusters Finally Break the TeraFLOP barrier

cworley submitted - several times - this well-linked submission about a slightly boring topic - fast computers. "Top500.org has just released its latest list of the world's fastest supercomputers (updated twice yearly). For the first time, Linux Beowulf clusters have joined the teraFLOP club, with six new clusters breaking the teraFLOP barrier. Two Linux clusters now rank in the Top 10: Lawrence Livermore's "MCR" (built by Linux NetworX ) ranks #5 achieving 5.694 teraFLOP/s, and Forecast Systems Laboratory's "Jet" (built by HPTi) ranks #8 reaching 3.337 TeraFLOP/s. Other Linux clusters surpassing the teraFLOP/s barrier include: LSU's "SuperMike" at #17 (from Atipa ), the University at Buffalo at #22 and Sandia National Lab at #32 (both from Dell ), an Itanium cluster for British Petroleum Houston at #42 (from HP ), and Argonne National Labs at #46 (from Linux NetworX ) reached just over the one teraFLOP/s mark with 361 processors. In the previous Top500 list compiled last June, the fastest Intel based Netfinity 1024 processor clusters from IBM were sub-teraFLOP/s and the University of Heidelberg's AMD based "HELICS" cluster (built by Megware ) held the top tux rank at #35 with 825 GFLOP/s."

Comments

Enough Links?

[rob-fu]

It's going to take me 4 hours to read all of this.

Re:Enough Links?

[handsomepete]

Imagine a Beowulf cluster of all those links!

It would probably end up linking to the greatest pr0n site of all time...

Re:Enough Links?

[jdkincad]

Look on the bright side, it might just spread out the /. effect enough to keep all the linked sites on line.

Re:Enough Links?

[Jugalator]

What he said! Holy crap! This is the main thing I don't like about slashdot, I can hardly ever tell what the main point of the post is if I have to figure out what link to click first.

Just act like the average Slashdot member. Never click any links to read the articles and just post your thoughts regarding the subject. :-)

Everything get so much easier that way!

Question?

[beldraen]

How long until computing powerful enough to render the probability thought patterns of a manager? That's what I want to know..

Re:Question?

[Guppy06]

"How long until computing powerful enough to render the probability thought patterns of a manager? That's what I want to know.."

Good luck. Last I checked, that one falls under Heisenberg's Uncertainty Theorem.

Re:Question?

[Masa]

How long until computing powerful enough to render the probability thought patterns of a manager?

That shouldn't be too hard... I bet that my Palm Pilot has enough power to predict exactly, what my boss is going to say in the next meeting tomorrow.

If it's about schedules, he'll say:

Work...

1. harder
2. smarter
3. cheaper
4. faster

In that order.

If it's about project goals, he'll ask me to:

Make...

1. miracles

If it's about specifications, he'll say: "I have no idea. You find out yourself." And for anything else it would be just blank. All blank.

On the other hand... if a manager actually has any real thoughts... Well, that would be as easy as to predict patterns from a pure chaos.

Re:Question?

[AndroidCat]

It can already be done by taking a Tamagotchi and relabeling the buttons: Feed = Overtime, Attention = Ego Stroke. Cleaning up the poop remains the same.

If it dies, you're fired.

can you imagine...

[Anonymous Coward]

a single node from one of these clusters?

(hey what else can I say, it's already a cluster)

Have another coffee while it Boots

[hopbine]

I have often wondered how long it takes to boot one of these things. In the HP-UX world I know how long it takes for a K class (sometimes more than 20 minutes). Superdomes are sometimes faster, but not by much.

Re:Have another coffee while it Boots

[jelle]

It's a cluster, so I can imagine the nodes can boot individually, in parallel. Plus I can imagine the system never goes down as a whole, just some nodes may go down when parts break or other maintenance. 1 bootup per lifetime...

Perhaps the boot speed is limited by the ramp-up speed of the local power plant.

Re:Have another coffee while it Boots

[hopbine]

Next time this happens, turn the key fully clockwise and on the console do "CTL B". This means it will stop at the BCH prompt - this ONLY works on the K boxes unfortunately, on all others watch the screen for the "hit any key or wait 10 seconds" message. At the BCH prompt type SER. This will do a search of all potentially bootable devices and should include all disc drives. No drive= bad drive, SCSI cable or SCSI controller. By the way, on any HP computer do not disable the memory check unless your troubleshooting, the extra few minutes might save you problems later on.

Wow!

[miffo.swe]

1 NEC Earth-Simulator 35860.00
2 Hewlett-Packard 7727.00 Los Alamos

The distance from the first to the second is pretty impressive. What on earth did NEC really do over there?

Re:Wow!

[girouette]

The Earth Simulator is a 640-node computer, housed in a building the size of a stadium. Each node is
a 64 GFlop Nec SX-6 supercomputer. (5,120 CPUs total).

It has its own dedicated, custom-built power station. 'nuff said.

Google is your friend, but for starters:

http://www.sw.nec.co.jp/hpc/sx-e/sx6/index.html

http://www.nec.co.jp/press/en/0203/0801.html

Re:Wow!

[CBNobi]

"Simulating the Planet Earth", an article about the Earth-Simulator, has some good information about the system.

One big item to note is that many of the supercomputers built in the US are for weapons research; as opposed to the NEC supercomputer, which deals with, obviously, changes of the earth.

More links:
Press release for the Earth Simulator, dated March 8, 2002
General system information on the cluster

Beowulf clusters aren't so tricky...

[caluml]

I built a small Beowulf cluster. It was actually very easy, apart from writing the MPI enabled code.

Step 1: Install the lam packages on all the nodes
Step 2: Create an account on all nodes, and use a passphrase-less ssh key to avoid prompting.
Step 3: Compile your code with mpicc (rather than gcc)
Step 4: Copy to all nodes.
Step 5: mpirun C ./your-prog

Admittedly it was only a 4 node cluster, but hey ;)

Please, someone break it to me gently if this wasn't actually a Beowulf cluster ;))

Re:Beowulf clusters aren't so tricky...

[That_Dan_Guy]

I have some students from Boeing that are just in love with Linux. The Engineer department there just set up a Linux Cluster with 120 nodes for around 100,000 US dollars. They were running tests on it and found it was much faster than the Cray they had previously been using to do the same things.

The main comment that struck me was how easy it was to set up. The Engineer IT department is mostly Unix (they're all in retraining becuase they are dumping Sun Stations for Intel based systems running XP beleive it or not- becuase Intel chips are so much faster and machines running XP are much cheaper than Sun Sparcs (plus the software they want runs on XP)) so it was of course easy to set up for them.

Next they'll be setting up another LINUX cluster with maxed out dual or quad processor machines with more RAM. They're really excited.

Re:Beowulf clusters aren't so tricky...

[dsfd]

We built and mantain a Beowulf with about 70 nodes. We use Debian GNU/Linux.

I agree with you, in principle, it *is* easy to do but the problems increase with the number of nodes. IMHO, the main problems are:

-Administration effort per node has to be almost zero. Beyond a number of nodes you definitely need things like fully automatic instalation, automatic power control, automatic diagnostic tools, a batch system, etc. All these tools already exist but you need some know-how to put all them together.

-You need a large enough room with a cooling system that gives at least 100 W per node, 7kW in our case. Room temperature has to be about 20oC.

-Low cost PC hardware is not allways reliable enough for this application. If you have codes that run 24x7 for months in a large number of processors, the probability to have a hardware problem is very high.

We have found that our hardware suppliers do not carry out extensive tests on the systems they sell. This is because "normal" users run low quality OSs and they assume that it is normal that the computers just hang from time to time. Therefore, they do not allways detect failures in critical components such as RAM.

-Of course, your application has to be suitable for parallel computing, specially if your cluster uses a low cost 100Mb/s network. In this case, compared to a "conventional" parallel computer (eg Cray T3E), the processors are roughly equivalent but the network is about 10 times slower and is easily the bottleneck of the system.

Having said that, despite all the problems, I love Beowulfs. They have totally changed high performance computing, and they are definitely here to stay.

All this has been possible thanks to free software, so thanks Mr. Stallman/Torvalds and many others...

Re:Links

[isorox]

Is that enough links there? Glad this isn't that impressive to me.

Will we be able to slashdot every one of them though? PErhaps someone should post some mirrors

Re:It seems like the Apple Mac.....

[xenode]

now why not try using macs for your supercomputers?

I know that they arn't as scalable

I think you answered your own question there.

EARTH-SIMULATOR

[Anonymous Coward]

Read it again. What does it say? EARTH-SIMULATOR

It's gonna take some CPU power to simulate earth, don't you think??

Re:EARTH-SIMULATOR

[gl4ss]

int simulate_earth()
{
sleep(years_to_ms(30000));
r eturn 42
}

dunno what they need the computing power for..'
oh yeah, to generate the program to call that.

Re:EARTH-SIMULATOR

[asavage]

What is interesting about the earth simulator is for the first time the world's fastest supercomputer is located outside the USA. Short CNN article here

Impressive numbers

[DrunkenPenguin]

Impressive numbers. I suggest you go take a look at that hardware that runs the Earth Simulator (#1 on the top 500 list). That flash movie is impressive. .. But don't forget that you got a helluva lot faster CPU inside your head - your brains beat all that expensive hardware all the way.
----

Re:Impressive numbers

[glwtta]

But don't forget that you got a helluva lot faster CPU inside your head - your brains beat all that expensive hardware all the way.

Meh, it's got pretty impressive image recognition, but can't do math for shit - all in all, I'm not amazed.

Re:How many FLOPS

[taviso]

pfft, FLOPS are for weenies - real men use bogomips. ;)

$ grep bogomips /proc/cpuinfo
bogomips : 2962.22

Re:FLOPs

[jelle]

Since nobody is answering your question: The Top500 supercomputers are ranked by the results of the LinPack benchmark.

LinuxBIOS

[bstadil]

This is not such a dumb question. The LinuxBIOS project was started by and for the Los Alamos National Lab. One of the nifty things this allows them to do is change Kernel without taking the machines down. You can then switch to a kernel compiled for different purposes.

Article /.ed

[Alu3205]

I hope none of those super computers was the webserver or else it's just the top 499 now. :p

Re:It seems like the Apple Mac.....

[ikekrull]

Ah, that would be because Apples 'supercomputer on the desktop' marketing drivel was just that.

Hell, the Sony Playstation 2 was subject to export restrictions because it was 'too powerful', which was driven by/followed with the requisite marketing drivel, but you don't see and PS2 clusters in the 'Worlds fastest supercomputer' list either.

It has been a long time since Apple PPC was competitive in terms of price/performance with x86s. Of course thats not the only reason to buy a computer, i don't want to get the apple-zealots panties in a bunch.

It's just that Intel/AMD didn't make a song and dance about breaking the GFLOP barrier, since that happened way back with the P3/Athlon 600-800, hardly cutting edge chips.

Hell, a 600Mhz Alpha had GFLOP performance years before either the G4 or the x86s.

The PPC has a nice vector processing unit (Altivec), which could make it a good choice in some situations, but given the premium you pay for Beowulf nodes (Xserves?) from Apple, you will, in general, get a lot more bang for the buck from x86.

Flops is not everything

[Bender_]

Measuring MFlops does not mean a lot - even if it is from a "real life" benchmark. The TOP500 might look much worse for linux-clusters, if more communication-latency dependent benchmarks were used. Linpack, which works mainly on very large matrices, shifts the benchmarks results a lot towards linux-cluster solutions.

A real supercomputer supports much faster I/O, higher interconnection bandwidth and lower interconnection latency.

And btw. the new Cray X1 delivers the performance of a all but the largest linux-clusters in a single cabinet (820 GFlops peak that is..). In terms of computing efficiency it makes even the Earth Simulator look pale. I am really looking forward to the next iteration of the TOP500, when the first X1 machines are included.

Re:Flops is not everything

[Flat5]

It depends on what you're trying to do. An awful lot of supercomputer sites *are* solving, more or less, very large matrices. In that case it means everything.

Some applications scale on these kinds of clusters and some don't. But to say that "MFlops does not mean a lot" is just as silly a blanket statement as pretending that the Linpack benchmark is "the speed" of the computer.

That Cray does look pretty awesome, btw.

Flat5

All Shook Up

[SEWilco]

"Linux Clusters Finally Break the TeraFLOP barrier"

As when other barriers are broken, a bit of a shock wave was created.
Windows machines for miles around were rattled.

Re:It seems like the Apple Mac.....

[miradu2000]

Actually, Mac's are used in super computer clusters. JPL has an intresting benchmaark of 33 Xserves. They get 1/5th of a TeraFLOP of performance. Not bad, considering how cheap they are.

Re:FLOPs

[alannon]

Why don't they write it: FLOP/s?

Because FLOPS means FLoating point Operations Per Second

'/' means 'per'.

FLOP/s would mean FLoating point Operations Per Per Second

FLO/s doesn't seem like a very good idea, except for cleaning your teeth.

Does Microsoft compete in this space?

[e_n_d_o]

Are there any Microsoft Windows-based systems that qualify as supercomputers?

(This is a serious question, I have no idea if they do or do not.)

Re:Ah ha!

[dsoltesz]

Well, the fact is I've been trying to submit the story about MCR (which was hoped to make #4, but Los Alamos submitted two halves of the same computer as two identical computers, bumping MCR to #5) for several months. Obviously some of us do not find cluster news boring.

"Slightly Boring"

[msheppard]

Did I miss the sarcasm tags on the "slightly boring" comment or something? I think there's a large audience on slashdot who are all very excited about high speed computing. Overclockers aside, I know I hate waiting for a compile.

Latley though, I feel the things I'm waiting for my computer are not a function of how fast the CPU can run, but how poorly the software is written. Can someone can tell me why my windoze machines sometimes block for up to a min when I try to click the "Location" box on the top of the file browser common dialog control? Or the oft-complained about boot time for most everything? Or the time it takes almost any program to load up the first time you load it?

Anyone else think it's time to start over, and not just assume the fater and faster machines can deal with the laziness we program into the systems we build?

M@

Imagine....

[DarkHelmet]

All those computers that meet Doom3's system requirements...

... And they're used for trivial things like finding aliens, weather prediction and unified theory.

1. The weatherman is usually wrong.
2. Aliens are abducting us. We need to send radio signals to Fife, Alabama, not out into space.
3. Unified Theory is based on Heisenburg's stuff... You can have relativity and quantum mechanics... but not both at the same time. Damn, that guy was a genius. By the way, the unified theory is:

   e = 42; // always 42.

Of course, I'm sure Doom3 has this somewhere in its source code, so ummm... go crunch 40 TFLOPS on that ;)

</humor>

Check out...

[Kashif Shaikh]

Myrinet Software. Not only does it support Windows plus a whole range of *NIXes.