Benchmarking the Benchmarks

apoppin writes "HardOCP put video card benchmarking on trial and comes back with some pretty incredible verdicts. They show one video returning benchmark scores much better than another compared to what you get when you actually play the game. Lies, damn lies, and benchmarks."

Comments

Re:back in my day...

[SQLGuru]

And, on top of that, they are on your lawn....

Layne

Benchmarks

[Anonymous Coward]

Duh, a benchmark is a controlled test performed "on a bench" - meaning, in a controlled environment with specific, well-described procedures.

You must perform the same exact test on all video cards, disclose any variables, and you must not "pick a subset of completed tests to publish". You must not compare tests performed using different procedures, no matter how slight the deviation of the procedures are.

One cannot draw conclusions about "real world" performance from a benchmark. The benchmark is merely an indicator. A "real world" test that uses the strong, formalized procedures of a benchmark IS a benchmark - and suddenly, the benchmark is not "real world" - because the "real world" doesn't have formal procedures for gameplay.

Haphazard "non-blind" gameplay on a random machine is NOT a benchmark, and it can not provide useful, comparable numbers.

A good benchmark is one where (1) most experts agree that it has validity, and (2) one where the tester cannot change the rules of the game.

The numbers of a benchmark are meaningless, except in terms of being compared to one another using the same exact procedure.

Re:OSS

[snoyberg]

PS yes...release your rage and mod me down.... just makes my post more Insightful.

Translation: if you mod me down, I will become more insightful than you can possibly imagine.