What Makes a Valid Benchmark?

An anonymous reader writes "Benchmarks can make a big difference if they are accurate in predicting performance. That's simple enough to describe; it's not nearly so simple to implement. Benchmarks can be an excellent tool for predicting performance and estimating requirements, but they also can be misleading, possibly catastrophically so. This article looks at benchmarks; the good, the bad and the ugly."

Comments

The best benchmarks

[Clockwurk]

are real world apps that your audience will be using. Whenever I read a review of a new CPU or graphics card, I always skip synthetic benchmarks (PCMark, 3Dmark, etc.) and go straight to the real world stuff like media encoding, and gaming benchmarks. Synthetic benchmarks tend to be little more than dick waving contests and have little bearing on the real world. If I see 4000 3Dmarks, its a meaningless number. If I see 58 fps in F.E.A.R. or 45 seconds in Photoshop, I immediately have a decent idea of how the computer is going to perform in real world use.

Three things.

[aquabat]

Scope, repeatability and transparency.

Scope means defining clearly and specifically what your benchmark measures and what it does not measure.

Repeatability means being able to run the benchmark many times under the same conditions and getting statistically consistent results.

Transparency means having the details of the mechanics of the benchmark, so that the results can be completely analyzed and understood.