Slashdot Mirror
What Makes a Valid Benchmark?
An anonymous reader writes "Benchmarks can make a big difference if they are accurate in predicting performance. That's simple enough to describe; it's not nearly so simple to implement. Benchmarks can be an excellent tool for predicting performance and estimating requirements, but they also can be misleading, possibly catastrophically so. This article looks at benchmarks; the good, the bad and the ugly."
The best benchmarks are those that lead their respective industries.
As manufacturers seek to maximize benchmark scores, they end up improving their products in ways that make the product more useful to the consumers.
One example of a bad benchmark: For the longest time, cpu frequency has been a sort of benchmark easily understood by the buying public. But it was a very poor one, leading Intel to maximize cpu frequency at the cost of almost all else -- actual computational performance fell behind, power efficiency became even worse with chips becoming mini-furnaces.
We can easily produce results in favor of your company provided you pay our staff's salary and my bonus.
Aside from the usual High School "Scientifically fair test" basic stuff: like for like testing with control options and keeping other non calculated factors constant, the best benchmark is the application you plan to run with the hardware, or very close. Nothing can tell you exactly how well it will perform except the actual app you are running, becuase now days setups are so complex and individual hardware and software components rely on so many different factors that it's easy to produce a test which looks like it might give a good gauge of performance in similar apps, but actually doesnt.
My 3D Texturing Skinning work (under construction)
are real world apps that your audience will be using. Whenever I read a review of a new CPU or graphics card, I always skip synthetic benchmarks (PCMark, 3Dmark, etc.) and go straight to the real world stuff like media encoding, and gaming benchmarks. Synthetic benchmarks tend to be little more than dick waving contests and have little bearing on the real world. If I see 4000 3Dmarks, its a meaningless number. If I see 58 fps in F.E.A.R. or 45 seconds in Photoshop, I immediately have a decent idea of how the computer is going to perform in real world use.
Let's not forget the "optimizations" that both ATI and nVidia engaged in. The ones where you'd take quake3.exe and rename it quack3.exe and your framerate would suddenly drop by 20fps. You see, Quake 3 was the defacto benchmark of the era...
Interestingly, despite this special treatment, the game ran better under Wine, and still better as a native Linux program.
Don't thank God, thank a doctor!
"Offer valid in the US states and territories except where prohibited by law. Void in connecticut."
#ifdef USA
#ifdef CONNECTICUT
void benchmark(int throwaway);
#else
int benchmark(int inputdata);
#endif
#endif
Scope means defining clearly and specifically what your benchmark measures and what it does not measure.
Repeatability means being able to run the benchmark many times under the same conditions and getting statistically consistent results.
Transparency means having the details of the mechanics of the benchmark, so that the results can be completely analyzed and understood.
A republic cannot succeed till it contains a certain body of men imbued with the principles of justice and honour.
This reminds me of a comparison I saw in Circuit City once... (Warning: I'm not going to talk about a computer hardware benchmark.) They were trying to sell the insanely expensive Monster video cables by comparing the Monster cables to standard cables on identical TV's. The screen with the monster cables looks hella better than the other monitor. The difference was so astounding that I just had to look at the back of the TV... The Monster TV was hooked up with an HDMI cable..... The other a FRIGGIN UNSHIELDED COMPOSITE VIDEO CABLE. Apples and oranges, apples and oranges...
it has taken almost five years to double performance on the SysMark 2004 SE Office Productivity: CPUs are just now doubling the 2.0GHz P4 from 2001!!!!!!!!
i tracked Content Creation Winstone 2000 scores up until the first P4s came out, and scores were taking about 30 months (two and a half years) to double.
Mathmatica 5.x doubling time? about 27 months.
i disable sigs
Why, everyone knows that the only valid benchmark is bogomips!!!
How to enable garbage collection on a system without protected memory: #define malloc() ((void *) rand())
As long as your piece of hardware is faster in some subset of some test in some benchmark, you'll be able to advertise "xx% faster". It's not limited to computer gear either, every car is "best" in this or "first" in that...
...is in my pants.