The Problem With the Top500 Supercomputer List

angry tapir writes "The Top500 list of supercomputers is dutifully watched by high-performance computing participants and observers, even as they vocally doubt its fidelity to excellence. Many question the use of a single metric — Linpack — to rank the performance of something as mind-bogglingly complex as a supercomputer. During a panel at the SC2010 conference this week in New Orleans, one high-performance-computing vendor executive joked about stringing together 100,000 Android smartphones to get the largest Linpack number, thereby revealing the 'stupidity' of Linpack. While grumbling about Linpack is nothing new, the discontent was pronounced this year as more systems, such as the Tianhe-1A, used GPUs to boost Linpack ratings, in effect gaming the Top500 list." Fortunately, Sandia National Laboratories is heading an effort to develop a new set of benchmarks. In other supercomputer news, it turns out the Windows-based cluster that lost out to Linux stumbled because of a bug in Microsoft's software package. Several readers have also pointed out that IBM's Blue Gene/Q has taken the top spot in the Green500 for energy efficient supercomputing, while a team of students built the third-place system.

Comments

Quelle surprise!

[Elbart]

Now that the Chinese are ahead, there's suddenly a problem with the list/benchmark.

Re:Quelle surprise!

[Daishiman]

Agreed. It seems like the issue is "big enough" only now that other people are catching up.

Re:Quelle surprise!

[Macman408]

+1.

There's nothing that's inherently "cheating" about using GPUs in a supercomputer. If your problem maps well to the hardware they have (and many large scientific and engineering workloads do), they can provide a huge speedup at a relatively low cost and relatively high performance per watt. After all, a GPU floating point throughput can be around 20 times faster than on a CPU; they're designed to do many things all at once (high throughput, high latency), while a CPU is designed to do one thing really really fast (lower throughput, lower latency). Recently, with multicore CPUs, the extra cores add performance very similarly to how a GPU would. If having a GPU is cheating, I'd surmise that having a multi-core CPU is cheating too.

It is true that LINPACK doesn't measure everything - it doesn't put a heavy stress on the interconnect, for example. Though if your problem is compute bound, you'd probably do well to find a way to minimize interconnect use to begin with. In any case, LINPACK measures *something* - it's a place to start comparing speeds, not the absolute truth of who will always be the fastest.

Besides, what's so important about the Top500? It gives somebody bragging rights for 6 months, or, if you're very lucky, a year or two, before something bigger comes along and squishes you. Not to mention, there are many supercomputers not on the list. If the NSA builds the world's largest supercomputer, they're probably not going to brag about how much compute power they have. They prefer the mystery, so outsiders have no idea what is within the realm of possibilities for them. I was at Cray once, and was told that they sometimes sell supercomputers into such secretive areas. The government (or whoever) will send a few guys to get trained about the computer, then it gets loaded onto trucks, and Cray never hears a thing about the computer ever again. No support calls, no upgrades, no idea where it even went to or what it's used for.

Re:Quelle surprise!

[inhuman_4]

The Linpack complaining has been going on for years. I remember this coming up with the NEC earth simulator, and other ASIC based systems.

Here are some interesting numbers:
AMD Radeon HD 4870X2 ~2.4 teraFLOPS
Intel Core i7 980 XE ~107.55 gigaFLOPS

According to this the AMD is 20x faster then the Intel; and this is true, but only in some cases. If what is need is graphic processing the AMD will crush the Intel. But if you need anything else (I am ignoring GPGPU for simplification) the AMD doesn't just lose, it doesn't run. This is a problem for all ASIC based systems, GPU ones are just the newest to come out.

So this new Chinese supercomputer (and other ASIC based supercomputers) score very high in Linpack because the ASICs are designed to be good at this type of task. This makes for a non-general purpose, but very cost effective solution.

But this then means that a supercomputer that cannot use GPUs for its intended task, score very low because they are general purpose machines. Because the Top500 is based on one benchmark (Linpack) you end up with a car to pickup-truck comparison; sure the car is faster, but what about towing capacity?

The end result is the supercomputer analog of the megahertz-myth, people like bigger numbers. A high score proves that is it faster at somethings, but not that it is faster in general.

Re:Quelle surprise!

[natet]

Agreed. It seems like the issue is "big enough" only now that other people are catching up.

I call bullsh*t on this comment. Around 8 years ago, the top computer on the list was a Japanese machine, and it rode atop the list for 3 years straight. Those of us who have worked in high performance computing have known for years that the top 500 list was a load of crap. It's something to write a press release about so that the people that give us money to build the big computers feel like their money is well spent. I worked on a top 5 computer at one time, but our focus was always the science that we wanted to do on the computer. Running the linpack benchmark for the top 500 list was an afterthought (though it was a pleasant surprise to score as well as we did).

I Don't See A Problem

[sexconker]

I don't see a problem with using GPUs.
They do lots of parallel shit really fast.

It's no different than slapping on a math coprocessor, or adding a block of hardware to accelerate common encryption/decryption functions.

Missing the Point

[Lev13than]

As the article alludes, the big problem with ranking supercomputers via Linpack is that it doesn't advance supercomputer design. The net result is a pissing match over scalability, where winning is dependent upon who can cram the most cores into a single room. The real innovatiors should be recognized for their efforts to reduce space, power and cost, or finding new algorithms to crunch the numbers in more efficient or useful ways.

Re:Missing the Point

[timeOday]

The real innovatiors should be recognized for their efforts to reduce space, power and cost, or finding new algorithms to crunch the numbers in more efficient or useful ways.

And NHRA should start awarding drag-racing championships on fuel efficiency rather than quarter-mile times.

Look, the Top500 is about performance, as in speed. There are other metrics for flops/watt or flops/dollar, or whatever. If those were the lists that managed to draw competitors and eyeballs, then nobody would care about Top500 and we wouldn't have to quibble about whether Linpack is a representative benchmark of what it claims to measure: speed.

Heard at Microsoft headquarters:

[pushing-robot]

Good news, everyone! Our supercomputer OS only lost because it's buggy!

Well, there's a non-notable point!

[icannotthinkofaname]

In other supercomputer news, it turns out the Windows-based cluster that lost out to Linux stumbled because of a bug in Microsoft's software package.

As it should. That's not news; that's how the game is played. If your software is buggy, and those bugs drag your performance far enough down, you don't deserve a top500 spot.

If they fix their software, rerun the test, and perform better than Linux, then they will have won that battle (the battle for the top500 spot, not the battle for market share) fair and square.

Re:Why is being on the the Top500 important?

[SnarfQuest]

I have always wondered why being on the Top500 list of supercomputers that important for those on the list.

I always choose my supercomputers from the Bottom500 list.

I will be better served by being told the advantage(s) or edge those who've been on that list have gotten since they got onto the list. Thanks.

At the price level these things cost, you can probably list your own requirements instead of accepting the vendors.

If you are purchasing a SuperComputer, you are looking for something to do raw number crunching. You aren't worrying about how well it will run MicroSoft Word, or how many frames/second you'll get out of doom.

Re:Good to hear

[timeOday]

OK, so you think only algorithms requiring uniform memory access are valid benchmarks. How uniform does it have to be? Real world problems do have structure, they do have locality, and an architecture that fails to exploit that is going to lose out to those that do.

Sure, your point is taken, otherwise you could say "my benchmarks is IOPS" and "my computer is every computer in the world" and win. But Linpack is not that; you can't score well without a fast interconnect, because what it measures is a set of computations that are actually useful. (Which is why the quip about a beowulf cluster of Android smartphones is stupid... because it couldn't actually be done. Go ahead and try to get on Top500 with a seti@home-type setup.)

Re:Why is being on the the Top500 important?

[Jeremy Erwin]

And that's why Top500 should use another benchmark. If the beancounters use Top500 to allocate resources, and the supercomputing companies use the beancounter's allocations to determine the future direction of their products, the scientists lose out. It's not so much that Tianhe-1 gamed the benchmark, it that's this gaming could lead to a machine that's not very useful.