The Supercomputer Race

CWmike writes "Every June and November a new list of the world's fastest supercomputers is revealed. The latest Top 500 list marked the scaling of computing's Mount Everest — the petaflops barrier. IBM's 'Roadrunner' topped the list, burning up the bytes at 1.026 petaflops. A computer to die for if you are a supercomputer user for whom no machine ever seems fast enough? Maybe not, says Richard Loft, director of supercomputing research at the National Center for Atmospheric Research in Boulder, Colo. The Top 500 list is only useful in telling you the absolute upper bound of the capabilities of the computers ... It's not useful in terms of telling you their utility in real scientific calculations. The problem with the rankings: a decades-old benchmark called Linpack, which is Fortran code that measures the speed of processors on floating-point math operations. One possible fix: Invoking specialization. Loft says of petaflops, peak performance, benchmark results, positions on a list — 'it's a little shell game that everybody plays. ... All we care about is the number of years of climate we can simulate in one day of wall-clock computer time. That tells you what kinds of experiments you can do.' State-of-the-art systems today can simulate about five years per day of computer time, he says, but some climatologists yearn to simulate 100 years in a day."

Comments

The true best measurement

[Anonymous Coward]

Is how many libraries of congress it can read in a fortnight.

Re:Weather Day After Tomorrow

[Fishbulb]

Don't hold your breath; it'll disrupt the predictions.

Simulation

[gringer]

Simulate 100 years of climate in a day? Here's my code:

echo -e "sunny\nrainy\ncloudy" | rl -rc 36525

Flops not useful?

[zippthorne]

But.. The whole point is to test the model, and the models change, don't they? Surely we're not just simulating more "years" of climate with the current batch, but improving resolution, making fewer simplifying assumptions, and hopefully, finding ways to do the exact same operations with fewer cycles.

How can you possibly evaluate supercomputers in any other way except how many mathematical operations can be performed in some reference time? And.. some serial metric if the math is highly parallel, since just reducing the size of vectors in those cases wouldn't actually result in those flops being useful for other tasks.

Re:Flops not useful?

[geekoid]

That's just the problem, people want to hear raw numbers, but those are useless.
How well can it do the specific task it needs to do is the actual question. It's a hard one, to be sure.

Re:Flops not useful?

[corsec67]

Flops wouldn't test how well the interconnects work.

Since you say "increase the resolution of the model", you are expanding the size of the model, and how much data must be used by all of the nodes of the computer.

Since how important the interconnect properties are is dependent on the model, with almost no communication needed, like for F@H, to a problem that needs all of the nodes to have access to a single shared set of data, it would be very hard to quantify performance in one number.

Unfortunately, there are more than a few fields where marketers want a single number to advertise in a "mine is bigger than yours" competition, and come up with a metric that is almost worthless.

Re:Flops not useful?

[Salamander]

How can you possibly evaluate supercomputers in any other way except how many mathematical operations can be performed in some reference time?

Simple: you evaluate how much actual work it can perform across the entire system per unit time, where "actual work" means a mix of operations similar to some real application of interest. The whole problem here is that practically no real application is as purely focused on arithmetic operations as Linpack. Even the people who developed Linpack know this, which is why they developed the HPCC suite as its successor. It's composed of seven benchmarks, including some (e.g. stream triad) that mostly stress memory and some (e.g. matrix transpose) that mostly stress interconnects. If you want to get an idea how your application will perform on various machines, you determine what mix of those seven numbers best approximates your application, assign appropriate weights, and then apply those weights to the vendor numbers. Then you negotiate with the two or three most promising vendors to run your application for real. SPEC should have put an end to simplistic "single figure of merit" comparisons, or if not them then TPC, SPC, etc. Sadly, though, there's still always someone who comes along and tries to revive the corpse.

Re:Flops not useful?

[jd]

To be honest, I thought most people already knew about and used HPC Challenge, which produces 7 different benchmarks covering different types of mathematical problem, memory bandwidth and communications bandwidth. I also imagined people would use MPI-IO for measuring MPI performance, that the numbers on the Top500 was simply because it's hard to track a vast number of stats in a meaningful way.

Of course, if it's actually the case that people are dumb, lazy or in marketing, then that would explain why we don't get a full range of stats, even though the tools have existed for many years and are certainly widely known.

Benchmark your application

[straponego]

A quality HPC vendor will give you the opportunity to benchmark your application before you buy a system or cluster. Most will have standard codes installed, but you should also be able to arrange for a login to build and run your own code on their test clusters. This is the only way to guarantee you're getting the best bang per buck, because the bottleneck in your particular applications may be memory, IO, interconnect, CPU, chipset, libraries, OS... An HPC cluster can be a big purchase, and it performance and reliability can make or break careers. Don't trust generalized benchmarks unless you know that they accurately reflect your workload on the hardware you'll be purchasing.

Re:Weather Day After Tomorrow

[smittyoneeach]

The mondo-flop race,
As the hair on your face,
You yearn to displace,
So do it with grace.
Burma Shave

Re:Weather Day After Tomorrow

[geezer nerd]

I can remember when the big desire of weather simulation supercomputers was to take less than 24 hours to do a 24-hour forecast. IIRC back in the second half of the '70s there was a big government-funded effort to build special fluid-dynamics oriented new machines to break that barrier.

44 years ago 1-5 megaflops was hot! What excitement we felt when the CDC6600 was installed at my university!

Back in '85 I was part of a startup building a mini-Cray, reimplementing the Cray instruction set in a smaller, cheaper box. I remember we focused on the Whetstone benchmark a lot, and it turned out that the Whetstone code really was bound up by moving characters around while formatting output strings, etc. We paid very careful attention to efficiently coding the C library string handling routines, and that got us more performance payback than anything we could do to optimize the arithmetic. One needs to understand the benchmark being used.

Well, let's see

[Louis Savain]

It's about a half a petaflop... but guess what? It runs Linux!

This sounds kind of nice but why should this make it any easier to write parallel programs for it? You still have to manage hundreds if not thousands of threads, right? This will not magically turn it into a computer for the masses, I guarantee you that. I have said it elswhere but parallel computing will not come of age until they do away with multithreading and the traditional CPU core. There is a way to build and program parallel computers that does not involve the use of threads or CPUs. This is the only way to solve the parallel programming crisis. Until then, supercomputing will continue to be a curiosity that us mainstream programmers and users can only dream about.

Re:Uhh, do you have a model?

[TapeCutter]

I just threw away a couple of mod points to bring you this announcement: Climate != weather, climate is the long term statistics of weather. Two different numerical analysis models, both computationally expensive.