Slashdot Mirror

It's not that crazy

Anyone wanna do the quick calculations on how many procs it would take to break into the Top 500 Supercomputers"? I think the lowest is 128x 2.4Ghz Xeons

You're spot on. In fact some of the biggest iron in the top 500 are simulating weather.

The publicly posted LIST would be a good place to start. The Virginia Tech computer ranks third.

Here is the current list of 500, last updated in November of '03.

Sigh.

Please try to remember that entry to the 'Top 500' list is as much about your interconnect topology and technology as the capabilities of the processors used.

It is a measure of one, and exactly one benchmark, LINPACK

Machines which are not well suited to this benchmark, or do not have network technologies/topologies well matching linpacks requirements will perform poorly at it, but possibly very well for their chosen purpose.

Good examples of this are the WETA digital clusters used in parts of the LOTR films, which are great for rendering, but hampered seriously in their linpack result by their 100MBit standard ethernet connections.

Another good example of this is the Virginia Tech G5 cluster, which gets a LARGE boost from it's infiniband interconnects (well, it will when Apple finish giving them the new machines... eventually..).

Not that I am defending SPARC's rather lackluster performance these days, just making a rather important point.

Those SPARC boxes better get a LOT cheaper VERY fast if they intend to find any real home in HPC.

Cheap ass $5k per box? Where you do piss you boss's money away? It's not 1998 anymore when HP could catch $50k for four times more power than a $2k PII with Linux.

With Linux, that $5k box can easily saturate your 10Gbit LAN backbone. Actually, per individual box $2k tops is probably 80% of the market. Even tungsten (see below) is built with boxes that go for around $2k, but they probably spent more than that per box on the interconnect.

On the big and powerful side, Linux's claim to fame is its significant and growing presence list of biggest irons in the world, note number four and five running Linux and Thunder's speed and estimated ranking as an indication of what the next list will look like. Now that is worth some fame.

Cheap ass $5k per box? Where you do piss you boss's money away? It's not 1998 anymore when HP could catch $50k for four times more power than a $2k PII with Linux.

With Linux, that $5k box can easily saturate your 10Gbit LAN backbone. Actually, per individual box $2k tops is probably 80% of the market. Even tungsten (see below) is built with boxes that go for around $2k, but they probably spent more than that per box on the interconnect.

On the big and powerful side, Linux's claim to fame is its significant and growing presence list of biggest irons in the world, note number four and five running Linux and Thunder's speed and estimated ranking as an indication of what the next list will look like. Now that is worth some fame.

Cheap ass $5k per box? Where you do piss you boss's money away? It's not 1998 anymore when HP could catch $50k for four times more power than a $2k PII with Linux.

With Linux, that $5k box can easily saturate your 10Gbit LAN backbone. Actually, per individual box $2k tops is probably 80% of the market. Even tungsten (see below) is built with boxes that go for around $2k, but they probably spent more than that per box on the interconnect.

On the big and powerful side, Linux's claim to fame is its significant and growing presence list of biggest irons in the world, note number four and five running Linux and Thunder's speed and estimated ranking as an indication of what the next list will look like. Now that is worth some fame.

Clicky clicky

Top 500

Cornell

NEC Earth Simulator

Interesting statistics:
cluster database

I expect it'll fall somewhere between the two extremes. On the linux cluster of Itanium2 processors as similar as it gets to this new one (mpp2), there are 1794 processors currently running jobs-

one 600 processor job
three 256 processor jobs
one 120 processor job
... and 16 more jobs ranging from 2 to 40 processors totaling the remaining 306 processors.

The largest job currently in the queue requests 900 processors for 24 hours. (it's rare for people to request more than 72 hours for a single job)

So, it's true that the majority of the jobs running are small (40 processors or less)... but three quarters of the processors are doing jobs of at least 256 processors. Most of the clock cycles do go to larger jobs, but it's far from a single job using the whole machine for months limit. Some of the few-processor jobs are run on mpp2 because of unusually large memory/disk requirements which make them impossible or inefficient to run on smaller, cheaper clusters; many also are test runs for larger jobs. (Most people running on mpp2 also have access to more typical smaller clusters and use those for easier calculations.)

This is the official top 500 list of supercomputers (not updated yet although thunder is mentioned as '*possibly* the second-most powerful computing machine on the planet'). Linux moving up to second place (from fifth a bit ago, iirc), woohoo! Only one left to beat!

This is the official top 500 list of supercomputers (not updated yet although thunder is mentioned as '*possibly* the second-most powerful computing machine on the planet'). Linux moving up to second place (from fifth a bit ago, iirc), woohoo! Only one left to beat!

It's possible. IBM has been doing it for some time with the Power4+ chip, as seen here. It's a form of Multi-Chip Module. You can see a picture of one here.

and for a limited time only... things change tooooo quickly these, todays supercomputer is tomorrows laptop

It is fascinating how low the treshold for Top 500 is. All you need is a hundred or so Xeon (or Opteron) CPUs, and with couple of hundreds you can be in Top 300.

BTW they rank Weta at #45 with 1080 CPUs (Xeon 2.8GHz). *If* now they 3300 of them (and all work as single cluster), that should definitely put them to Top 10 (#4 is NCSA with 2500 Xeon 3.06, and #7 is 2304 Xeons 2.4 GHz).

But their Rmax score with 1080 CPU cluster is 1755 Gflops, if we multiply it by 2.8 (3300/1080, and assuming that all CPUs are 2.8GHz in average), it gives 4914 Gflops, only enough for #12.

WETA's cool - but not in supercomputer/HPC land.
Actually they are. The Top500 supercomputer site lists them as number 44.
Which is kind of odd especially considering the fact that the article clearly says they're not on that list. /shrug

The article says that they used ttcp which is a memory-to-memory bandwidth testing program. Most would consider that unrepresentative of reality. On the other hand, today's supercomputers have a tremendous amount of memory (1.2 TB, 6 TB, 10 TB, 33 TB, etc.) so memory to memory is possible.

Others have suggested that disk speeds cannot sustain that rate. However, supercomputer disk arrays can easily keep up (4 GB/s or 32 Gb/s).

Finally, it is possible to achieve nearly the same result (multiple streams instead of a single stream) transfering real data (23.23 Gb/s).

[Bias alert: I am a member of the team that set a previous Internet2 Land Speed Record, Guinness World Record and won the "Bandwidth Lust: Distributed Particle Physics Analysis Using Ultra-High Speed TCP on The Grid" or "Moore's law move over" award at SC2003.]

Now, before you complain that the technology is not available to "mere mortals," let me point out that we first started experimenting with 1 Gb/s Ethernet at work 5 years ago. Now it is readily available at reasonable prices for consumer desktop machines. (Apple has had it standard in G4 desktops for 4 years.) The problem is not with consumer hardware, it is having access to true broadband (not cable modem or DSL), at least in the USA. Although your LAN may support 1 Gb/s, your download speed is limited to 1-3 Mb/s (cable) or 256 -786 Kb/s (DSL). (Your upload speeds are significantly lower.) Since the link provider has very little incentive to upgrade service, I doubt that will change very quickly.

So, yes it is possible. No you can't have it (yet)!

since the parent posted a password protected version of the page, go here for the regular one

Earth Simulator - 5120

LANL / ASCI Q - 8192

LLNL / ASCI White - 8192

NERSC / LBNL / Seaborg - 6656

Nice research, BBC.

Saying the WETA render farm is the third largest machine in the world based on the number of processors is wrong. Just check the latest top 500 list and a quick skim points out that Lawrence Livermore National Lab's ASCI White (8192) and ASCI Blue Pacific (5808), Lawrence Berkeley National Lab/NERSC's seaborg (6656), Sandia National Lab's ASCI Red (9632), and Los Alamos National Lab's ASCI Blue Mountain (6144) all have more processors as well as the two already listed.

Also interesting that WETA Digital is listed as #44 on the list too, huh? They only listed a Xeon cluster though with 1080 processors. (prolly not be the same machine, but...).

I love technical articles from the popular press about technical subjects. They do soooo much in depth research. I hope that they don't hurt themselves.

*Disgusted look*

Look at the difference in R_max (maximum achieved performance in a real LINPACK benchmark) and R_peak (theoretical maximum) - you can see that it is not a machine built for distributed calculations.

Granted, this list http://www.top500.org/list/2003/11/ is a couple of months old but it shows WETA much lower in the rankings.

*cough* Top500 List *cough*

Altrix is NOT a cluster. It has a single OS footprint for all processors (though I would bet that they could be subdivided like an IBM 445). Clusters CAN make a supercomputer, in fact ALL top 5 supercomputers are indeed cluseters. There are NO sparc supercomputers in the top 5 and doesn't even make an appearance until 151st place. A altrix (linux/commoditty processor) appears in the 41st place.

SMP scalabilty has NOTHING to do with the type of kernel (monolithic/micro) but has more to do with how it handles said SMP. Solaris is nothing special here. Solaris does not appear in the top 5, while linux does.

Schwartz is wrong in every sense of his statement. Their is a lot more work going on for the linux kernel than solaris. The resources being thrown at each kernel is decidedly in linuxes favor.

I thought LFNW was awesome. I especially enjoyed the Pacific Northwest National Labs High Performance Linux Cluster talk given by Timothy A Witteveen of PNL. It is one hell of a machine. It is one hell of a machine. 9.4 teraflops and a 53 terrabyte SAN running NWLinux. It placed 5th on the last Top 500 List.

That 53 terrabyte SAN is one contiguous filesystem using lustre. The use of QSNet2/Elan4 interconnects make the use of terbyte data sets with lots of internode communication more efficient than past machines. These interconnects provide a peak bandwidth of 340 MB/sec in each direction. But even more impresive than the bandwidth is the latency, between 2 us and 5 us. Compare that with ethernet latency measured in tens or even hundreds of ms.

During the presentation Tim went over two examples of simulations performed on their cluster that could not be accomplished on other machines. These examples were outside of my domain of knowledge, but one involved simulating the behavior of water molecules and the other was an extremely detailed protein folding simulation.

One last bit that was interesting was their methodology for updating the machines. They have over a thousand and took some time to determine an efficient means to keep the machines up to date. It was determined that reimaging the maches was faster than applying patches. They utilize a multicast approach in which allows them, theoretically, to reimage all of the machines in 28 minutes. They do not always reach this theoretical maximum, but they reimage a thousand machines PDQ.

The meet & greet in the commons was fun. Pogo Linux had a free drawing for a loaded AMD64 system. There was plenty of swag to be had from all sorts of folks. It was cool to stop and chat with one of the Helix developers. All in all there was a really good group of folks gathered.

I know Slashdot is full of trolls, but I must say I am a little suprised at how many crappy comments LFNW is getting. All I have to say to those filled with negativity is screw you. Here in the Pacific Northwest we have some cool stuff going on. Whether it is the 3 new lugs in Seattle, OSDL in Portland, the Linux Cluster at PNL, SeattleWireless, PersonalTelco or LFNW there are exciting things happening up here. I think a couple people on this site need to take their heads out of their arses and take a look around. Folks up here are using linux, and getting stuff done.

I mean compared to this other G5 cluster (see below), wouldn't this thing qualify as up there (yeah yeah, it doesn't have anything more than Gethernet for backplane communication, but still, thats a whole lot of processing going on!

If someone more gungho on a monday morning wants to flesh out the Tflops this thing could crunch, as well as potential requirements for so much juice? Perhaps it loads the ENTIRE film into the cluster's memory and does some wacky matrix transforms across the entire bolus of data? Remember that's 24fps*seconds which is generally around 8 million frames * 4000x3000 pixels * 24 bit resolution == 288000000000000 bytes == 288TBytes of data ... that's some kind of matrix! (btw I didn't dbl check the math!:^)

I hope someone can dig up some more stuff on this, as using all this horsepower is either doing something very cool that is not obvious, or else it sounds like more sizzle than steak...

* http://www.top500.org/dlist/2003/11/
3 Virginia Tech
United States/2003 X
1100 Dual 2.0 GHz Apple G5/Mellanox Infiniband
4X/Cisco GigE / 2200
Self-made NOW - PowerPC
G5 Cluster Academic
10280
17600 520000
152000

All tests for the top 500 supercomputers are done solving a problem using Linpack, not some trivially parallel code such as raytracing 100,000 frames of a movie.

Message passing is the biggest issue with such solvers, and in a way, cray was absolutely right about Linux, although misleading. There are some tests going on now with a modified Linux kernel for doing true HPC, and it's been done in the past (I know, I've used it). Things like disk swapping pretty much immediately disqualifies you for high performance computing. It has its place of course, such as trivially parallelizable codes is one example (Pixar).

Myrinet was out before Gbit ethernet was really available, and also has some nifty routing capabilities. And since the bottleneck for HPC is usually message passing, high performance computing will better realize its theoretical performance as the communication speed catches up to the processor speed.

But, to Cray's discredit, making a blanket statement that Linux can't do HPC is like saying Macintoshes can't do HPC.

All tests for the top 500 supercomputers are done solving a problem using Linpack, not some trivially parallel code such as raytracing 100,000 frames of a movie.

Message passing is the biggest issue with such solvers, and in a way, cray was absolutely right about Linux, although misleading. There are some tests going on now with a modified Linux kernel for doing true HPC, and it's been done in the past (I know, I've used it). Things like disk swapping pretty much immediately disqualifies you for high performance computing. It has its place of course, such as trivially parallelizable codes is one example (Pixar).

Myrinet was out before Gbit ethernet was really available, and also has some nifty routing capabilities. And since the bottleneck for HPC is usually message passing, high performance computing will better realize its theoretical performance as the communication speed catches up to the processor speed.

But, to Cray's discredit, making a blanket statement that Linux can't do HPC is like saying Macintoshes can't do HPC.

All tests for the top 500 supercomputers are done solving a problem using Linpack, not some trivially parallel code such as raytracing 100,000 frames of a movie.

Message passing is the biggest issue with such solvers, and in a way, cray was absolutely right about Linux, although misleading. There are some tests going on now with a modified Linux kernel for doing true HPC, and it's been done in the past (I know, I've used it). Things like disk swapping pretty much immediately disqualifies you for high performance computing. It has its place of course, such as trivially parallelizable codes is one example (Pixar).

Myrinet was out before Gbit ethernet was really available, and also has some nifty routing capabilities. And since the bottleneck for HPC is usually message passing, high performance computing will better realize its theoretical performance as the communication speed catches up to the processor speed.

But, to Cray's discredit, making a blanket statement that Linux can't do HPC is like saying Macintoshes can't do HPC.

doesn't this CTO of cray remind you of someone?
"There IS no Linux in high-performance clusters."

"There IS no Americans in Iraq."

OMG! It's the former Iraqi mis-Informed-ation minister!

Especially when 2004 has been dubbed the year of the penguin, it's wreckless to claim that Linux can't be used in HPC's.
Hell, just look at the current top500 list. There's no Cray in the top 10 but there are two Linux based clusters there (and one based on OSX [FreeBSB based]).

Here's a few:
NCSA's IA32 Linux cluster
NCSA's IA32 Linux cluster
Space Simulator Clust at Los Alamos (SS51G based; makes me proud as I have a SS51G too)
Beowulf - used in many Linux clustering projects
Linux clusters at Los Alamos (they seem to have more than one)
Virginia Tech's Supercomputer X

I'd like to see Paul Terry say this in front of everybody at the Super Computing conference where they announce the Top 500 Computers. Its worth noting that he is not bashing Linux per se, but "Linux Clusters", which is pretty arbitrary, because he should be saying "all clusters", because the OS really doesn't have too much to do with it. Supercomputing apps run in userspace, not kernel space, and the hardware, including interconnects or some kind of interprocessor communication drive the performance.

The Cray XD1 looks like a nice system, but there are only theoretical performance values given, and noone can go out and buy one of these things yet. I also don't know how much these guys cost.

I love this statement:

Linux clusters do have a place. "For applications that require low performance, they are a cheaper solution," said Terry.

Yeah, when we spend a million+ dollars on a supercomputer, we are thinking of low performance, because our applications require it. Thanks.

I'm guessing this guy is a wannabe marketer who got stuck in a CTO position. There are plenty of HPC vendors out there, and trust me if this XD1 has a good price/performance and they work (this is key), then people will buy them with little questions asked. Otherwise, this whole article is just an advertisement that makes many statements without any evidence that the XD1 is any better than 4 Xboxes connected together over a serial connection. Next....

They're just pissed because clusters are 7 out of the 10 top supercomputers, as noted here

Quite true considering that a glance at the Top 500 List shows Linux holding spots at 4, 6 and 7 in the top 10 and Cray not appearing until slot 19. Someone should ask Dr. Terry why they don't have 3 slots in the top 10...

How could Cray be wrong. I mean just becuase linuxis running some of the top 500 computers there is no reason to consider HPC right. What a self serving statement Cray makes....they still dont get it .... there way is a dead-end...

Oh, here's the TOP500 list, btw.

How could Cray be wrong. I mean just becuase linuxis running some of the top 500 computers there is no reason to consider HPC right. What a self serving statement Cray makes....they still dont get it .... there way is a dead-end...

Oh, here's the TOP500 list, btw.

Despite assertions made by Linux vendors, a Linux cluster is not a high performance computer, said Dr. Paul Terry, CTO of Cray Canada. "At best, clusters are a loose collection of unmanaged, individual, microprocessor-based computers."

I guess they're not happy about being only #19 on the Top 500 Supercomputer List. Linux is considered faster than they are according to the list.

The 'ol ad-hominem attack of "if you can't beat them ligitimately, attack them personally" just doesn't cut it Paul. Build a better computer.

It would be more accurate to claim that "x86 can't compete with SPARC in a massively parallel environment".

Yet more than half of the computers in the top 500 list are x86. So, to be charitable, the statement is outdated.

Remember that when Apple & Virginia Tech designed Big Mac, both firewire and gigabit ethernet were built-in, but rejected as being too slow.

Meanwhile, we're all ignoring that "supercomputing for the masses" is already here. The original Cray-1 supercomputer in 1976 ran at a whopping 75MHz with 160 megaflops. Today you can get that much power in a palmtop.

"If you were plowing a field, which would you rather use? Two strong oxen or 1024 chickens?"
Seymour Cray

And history has placed this quote firmly into the company of other "forward thinking" quotes such as the one about the world only needing 5 computers, or 640K being all the RAM you need.

MPP computers are doing rather well, actually.

Here is one Check out entries 4 and 6. Same type of interconnect between nodes, the Opteron cluster has 316 CPU's more, and is slower.

Here

The first example is scientific code, where communcation is high both between CPU's and between nodes

Why select a slower, more expensive platform and take on the cost of porting one's in-house software to yet another platform, when multi-processor AMD-64 chips running GNU/Linux are a dime a dozen?

And what price list are you referring to? It cost Virgina Tech $5.2 million US for a #3 spot in the worlds fastest computers.

On the other hand, an equally equipped Opteron (albeit an extra 600 processors over Big Mac) cluster cost about $10 million and trails Big Mac at #6.

My math may be a bit rusty, but isn't 3rd place better than 6th place, and 5 million less than 10 million?/p.

It has done nothing of the sort. It got a great score on a very old benchmark: LINPACK benchmark results were first published in 1979 - LINPACK itself dates back to the early '70s.

It is at best a very rough approximation of a systems performance on a very specific type of problem. At worst, it is completely useless - there are so many different factors that contribute to LINPACK performance, and these factors often have *nothing* to do with the performance of a system on a real problem (in that the limiting factors in a *real* problem don't come into play)

For example: looking at the top 500 list, you'd think that the Earth Simulator was only three times faster than X. On its actual workload, however (simulating global weather patterns with resolutions of 100s of metres or less) it is very roughly 20 times faster than X. Why? It has about *60* times the memory bandwidth of X. But you'd never know that, looking at only the LINPACK figure.

Subtleties of benchmarking aside, the only other comment I'd like to make is that "Big Mac" hasn't proved anything: the machine was dismantled without any interesting scientific work performed on it whatsoever, as far as I can tell. I could be wrong here: can you point us to even a single application of this "very powerful" cluster?

I'm pretty sure that the only application so far has been some cheap PR for Apple (basically, the markdown on the 1000 G5 systems from new. vs the "ex-big mac" price that's $200 less, if I remember correctly...) $200,000 is not a lot of the kind of publicity they ended up getting. (And continue to get, judging by your post, even though the machine doesn't exist any more!)

The folk at Apple contribute an insanely small amount to Linux development overall (they contribute, not saying they don't, but spitting in the ocean doesn't add that much water overall).

Apple never ported shake to Linux, never will.

Everyone looks at bang-per-buck. Opterons on Linux are cheaper and a lot more bang-per-buck.

The other issue is that Linux is OSS. Need a change? Fix it to meet your needs. Can you do that as easily with OSX? No. You have to tell a mac developer who may or may not make the change you need to best suit your needs.

Apple bad for Linux? Here's where you can find the most beautiful Linux box you could ever have hoped for. Cluster them if it makes you happy---it's supported. But you say Unix based operating systems are more scalable for clusters and render farms. What do you think Mac OS X/Darwin is? Do you want to look at the source? Try starting here.

What propaganda are you talking about, anyway? Are you a troll or could you really just be this stupid? The Virginia Tech cluster was not made at the prompting of Apple, but some researcher did his homework and decided to use it. They came up with something that worked better than anything for the money and also landed third place in the Top 500 honestly. That's not just marketing spiel. A third party decided to use Macs for their cluster, and a third party that ranks these things honestly gave the cluster a well deserved third place. Do you honestly think Apple has no right to use this fact to promote their product?

As for the media thing, I don't know how anyone could honestly argue that Linux is easier to use for photography and movies than the Mac with its native software. What FUD has Apple spread about Linux with respect to media? Why would they have to? In this area, they don't even need to so much as acknowledge the existence of Linux because the people using Linux for media would use it anyway and no one else would bother using Linux for that. Life's too short.

I'd think they'd just keep the extra horsepower around. I mean why throw the baby out with the bathwater? Rather than replacing existing capacity, add to the capacity they already have (yadda yadda).

It wouldn't surprise me at all though if their G5 implementation is deliberately set up to fully outshine everything they've currently got and then some. Hell they may even be able to surpass the Virginia Tech supercomputer that cost (a mere) $5.2 mil, since they're directly related to the supplier of the hardware.

Could Jobs be aiming for an implementation that could surpass the Virginia Tech computer, giving Apple two places on the Top 500?

I know if I was the CEO of two companies, one that needs obscene computing power, and the other that can deliver obscene computing power, I'd have.. well.. some obscene computing power.

Except that theory takes a severe beating. Compare entries 4, 5 and 6 here at Top 500, especially the number of CPU's. Notably, entries 4 and 6 use the same kind of high-speed interconnects between the nodes, so the difference can't be blamed on that.

The problem with the AMD approach is that you get the NUMA drawbacks not only against other nodes, but internally on the node. If the data CPU 1 needs isn't in it's own memory banks, it's got to request them from CPU 2, 3 or 4, with a latency penalty(Letting the memory controllers read/write freely from each other's banks doesn't sound like a good idea, really). It works well with databases, serving websites, file servers etc, where the processes don't need to share memory, talk to each other a lot etc, but for physics and chemistry simulations etc, you get some penalties

On Intel Xeons, noless.

Why couldn't VT hold their horses?

My guess is that both they and Apple wanted to have a spot on the last Top 500 list, with all the associated press at the Supercomputing 2003 conference. Apple's been trying to convince somebody, anybody to build a large HPC cluster with their hardware since the G3 came out. Until the G5 came out, it made very little sense economically -- the per-system price for Apple kit was 30-40% more than comparable Intel-based stuff, and the memory bandwidth and 64-bit floating point performance was the same or worse. The G5 fixed that, for the most part

Nobody in their right mind wants to build a cluster out of machines in desktop/deskside chasses. We've done it once, with the first generation Itanium systems where there was no rackmount option for a 2-way box, and we'll never do it again -- remote management of those machines was and is actively painful. (Our 1st-gen Itanium cluster is out of production service now, but it's been partitioned up into smaller clusters at universities around the state as part of the Cluster Ohio project, which we still manage.)

--Troy

After all, they have the a crapload of machines, they seem to have a nice fat pipe - and they obviously have the knowledge to build the worlds 180th fastest supercomputer in 2002.

They went with the PowerMacs only to upgrade to the Xserves a few months later so that they would meet the deadline for the Top 500 listing.