Slashdot Mirror
The Problem With the Top500 Supercomputer List
angry tapir writes "The Top500 list of supercomputers is dutifully watched by high-performance computing participants and observers, even as they vocally doubt its fidelity to excellence. Many question the use of a single metric — Linpack — to rank the performance of something as mind-bogglingly complex as a supercomputer. During a panel at the SC2010 conference this week in New Orleans, one high-performance-computing vendor executive joked about stringing together 100,000 Android smartphones to get the largest Linpack number, thereby revealing the 'stupidity' of Linpack. While grumbling about Linpack is nothing new, the discontent was pronounced this year as more systems, such as the Tianhe-1A, used GPUs to boost Linpack ratings, in effect gaming the Top500 list."
Fortunately, Sandia National Laboratories is heading an effort to develop a new set of benchmarks. In other supercomputer news, it turns out the Windows-based cluster that lost out to Linux stumbled because of a bug in Microsoft's software package. Several readers have also pointed out that IBM's Blue Gene/Q has taken the top spot in the Green500 for energy efficient supercomputing, while a team of students built the third-place system.
The guide has this to say about supercomputers: "Supercomputers," it says, "are big. Really big. You just won't believe how vastly, hugely, mindbogglingly big they are. I mean, you may think your SGI Challenge DM is big, but that's just peanuts to supercomputers, listen..."
The Top500 has the problem in that many of the systems on there aren't super computers, they are clusters. Now clusters are all well and good. There's lots of shit clusters do well, and if your application is one of them then by all means build and use a cluster. However they aren't supercomputers. What makes supercomputers "super" is their unified memory. A real supercomputer has high speed interconnects that allow direct memory access (non-uniform with respect to time but still) by CPUs to all the memory in the system. This is needed in situation where you have calculations that are highly interdependent, like particle physics simulations.
So while you might find a $10,000,000 cluster gives you similar performance to a $50,000,000 supercomputer on Linpack, or other benchmark that is very distributed and doesn't rely on a lot of inter-node communication, you would find it falls flat when given certain tasks.
If we want to have a cluster rating as well that's cool, but a supercomputer benchmark should be better focused on the tasks that make owning an actual supercomputer worth it. They are out there, that's why people continue to buy them.
The Linpack complaining has been going on for years. I remember this coming up with the NEC earth simulator, and other ASIC based systems.
Here are some interesting numbers:
AMD Radeon HD 4870X2 ~2.4 teraFLOPS
Intel Core i7 980 XE ~107.55 gigaFLOPS
According to this the AMD is 20x faster then the Intel; and this is true, but only in some cases. If what is need is graphic processing the AMD will crush the Intel. But if you need anything else (I am ignoring GPGPU for simplification) the AMD doesn't just lose, it doesn't run. This is a problem for all ASIC based systems, GPU ones are just the newest to come out.
So this new Chinese supercomputer (and other ASIC based supercomputers) score very high in Linpack because the ASICs are designed to be good at this type of task. This makes for a non-general purpose, but very cost effective solution.
But this then means that a supercomputer that cannot use GPUs for its intended task, score very low because they are general purpose machines. Because the Top500 is based on one benchmark (Linpack) you end up with a car to pickup-truck comparison; sure the car is faster, but what about towing capacity?
The end result is the supercomputer analog of the megahertz-myth, people like bigger numbers. A high score proves that is it faster at somethings, but not that it is faster in general.