Slashdot Mirror

← Back to Stories (view on slashdot.org)

Examining Benchmarking

Posted by michael on from the no-benches-involved dept.

VL writes "Benchmarks exist to determine how a particular piece of hardware performs in relation to itself, and to others. Question is, are readers getting the information they really need?"

3 of 95 comments (clear)

  1. Goedel says benchmarks are inherently flawed. by Peter+Cooper · · Score: 5, Interesting

    Benchmarks are inherently flawed for the reasons stated in the posts. Comparing hardware to itself and similar hardware means there's no external reference point. Comparing one thing to another is okay, but you can't get absolute numbers in a closed Platonic system.

    Goedel's Incompleteness Theorem states that you can't define a system entirely in its own terms, and that any system needs to be defined by terms outside of it.

    So, how can you accurately rate hardware based on similar hardware? To meet the GIT (Goedel's Incompleteness Theorem), you would need to compare the hardware with something outside of the system, so you have an external reference point. For example, if you're benchmarking graphics cards, you need to also compare them to something outside of that area of hardware.. so.. say, a graphics tablet, or an iPod.

    So, say that the first graphics card is 0.7% compared to the iPod, we now have an external reference to use with the other graphics cards.. so a better card might be 10% compared with the iPod, or a few percent compared to the graphics tablet, which proves that the second card is better than the first, due to the respective ratings compared to the external objects.

    This is just regular math. I have to say, it's pretty amazing what you can apply regular math to.. yes, even benchmarks!

  2. Prefer multipl e benchmarks, or your own 'problem' by Anonymous Coward · · Score: 5, Interesting

    It all depends on the range of excercise-able aspects of some hardware a particular benchmarking suite excercices. That's why you prefer a suite rather than a stand-alone benchmark. For instance, Top500.org ranks HPC machine according to LINPACK, for which the ES (earth simulator) of course does well due its vectorization capabilities.

    So, if you want to know about your hardware, you better run more than one benchmark, and more importantly, your 'problem code'. Yes, you want hardware that performs well for you problem. Something that can be good in general, is ratrher rare.

  3. Hard to care. by xanderwilson · · Score: 5, Interesting

    My favorite computers haven't been the fastest. In fact, I've been the most productive on systems that were objectively less impressive.

    My favorite Operating Systems haven't been the ones with the best selection of software.

    My favorite games haven't been the ones with the best graphics.

    The reviews I find most valuable don't have the most complete set of numbers of why something's the best or worst.

    It's interesting that the goal of benchmarks is to be objective as possible, when it's the subjective that makes me want to buy or not buy something. But meanwhile the more the objectivity of the benchmark tests are in doubt, the less important the tests become. So I guess that means benchmarks don't mean anything to me one way or the other, huh?

    Alex.