Slashdot Mirror
Benchmarking the Benchmarks
apoppin writes "HardOCP put video card benchmarking on trial and comes back with some pretty incredible verdicts. They show one video returning benchmark scores much better than another compared to what you get when you actually play the game. Lies, damn lies, and benchmarks."
Benchmarking using actual games is, of course, important. But part of the reason a lot of us buy video cards and such isn't JUST about the performance on today's games, but for how they'll play the games coming out in the next few months. Synthetic benchmarks often implement advanced features not currently seen in today's games, but which will be implemented in just-over-the-horizon games. So while clearly one ought not judge a card purely on 3DMark or similar benchmarking suites, they do have their uses.
The problem is that it's hard to objectively score performance by "running things on it." Benchmarks are nice because they run the exact same tests every time. You can't just turn on FPS display and walk around in the game to measure performance--your actions may not be the same each time, and slight variations could cause drastically different results.
Benchmarking provides potential customers with a metric to compare potential purchases.