Slashdot Asks: What's Your View On Benchmark Apps?

There's no doubt that benchmark apps help you evaluate different aspects of a product, but do they paint a complete picture? Should we utterly rely on benchmark apps to assess the performance and quality of a product or service? Vlad Savov of The Verge makes an interesting point. He notes that DxOMark (a hugely popular benchmark app for testing a camera) rating of HTC 10's camera sensor is equal to that of Samsung's Galaxy S7, however, in real life shooting, the Galaxy S7's shooter offers a far superior result. "I've used both extensively and I can tell you that's simply not the case -- the S7 is outstanding whereas the 10 is merely good." He offers another example: If a laptop or a phone does well in a web-browsing battery benchmark, that only gives an indication that it would probably fare decently when handling bigger workloads too. But not always. My good friend Anand Shimpi, formerly of AnandTech, once articulated this very well by pointing out how the MacBook Pro had better battery life than the MacBook Air -- which was hailed as the endurance champ -- when the use changed to consistently heavy workloads. The Pro was more efficient in that scenario, but most battery tests aren't sophisticated or dynamic enough to account for that nuance. It takes a person running multiple tests, analyzing the data, and adding context and understanding to achieve the highest degree of certainty. The problem is -- more often than not -- gadget reviewers treat these values as the most important signal when judging a product, which in turn, also influences several readers' opinion. What's your take on this?

Comments

You also have to consider cheaters

[dargaud]

Case in point: ADSL line speed. I've had several different ADSL providers, and living somewhat far out, the speed is consistently bad, sometimes awful. But if I try one of the many 'ADSL speed test' websites, the results are always in line with the promised speed. I once routed one of those through a proxy, just for the name change, and the speed was one tenth, same if I accessed it simply by the IP number ! Benchmarks are too easy to cheat. Wasn't it Intel who was caught doing that a few years ago ?

Re:You also have to consider cheaters

[ArmoredDragon]

Case in point: ADSL line speed. I've had several different ADSL providers, and living somewhat far out, the speed is consistently bad, sometimes awful. But if I try one of the many 'ADSL speed test' websites, the results are always in line with the promised speed.

Not every place you visit (in fact, likely most places) will fully saturate your downstream link. They might have the bandwidth to be capable of doing so, but they ration it on a per-session (sometimes per-IP) basis so that everybody who happens to access the site can get a reasonable speed. (By the way, this is the principle that so called "download accelerators" take advantage of -- they combine multiple sessions into one. But they won't work on a per-IP basis unless you are able to do i.e. multipath TCP.)

On a gigabit link that's just mine and only mine, few sites have saturated my downstream. Steam is probably the fastest one, which pumps out 33MByte/s but no more than that. The only way I can saturate my link without a bandwidth benchmarking site is with bittorrent on a torrent that has plenty of seeders, so you might try that.

The Benchmark Lifecycle

[Prien715]

A good benchmark -- in cameras, CPUs, GPUs, cars, anything really -- is ideally a set of tests which contains a random sampling of real-world scenarios. In the beginning, the benchmark is good precisely because the vendors are unaware of it and don't spend a bunch of time trying to optimize for it specifically.

Once a benchmark becomes popular, companies try to make their product better for the benchmark ("See PHB! I increased our PCBench score by 10%!") but CAN ultimately end up doing so in a custom way that doesn't represent real-world performance (e.g. Volkswagen). Because the company is now specifically trying to optimize for a specific use-case, the benchmark is no longer random and thus no longer representative of real-world use.

Enter a new benchmark, which is really good, and better mirrors real-world performance and the cycle begins anew.

Build to the benchmark

[MetaKey]

An unpleasant side effect of benchmarking is when manufacturers start building products to do well on the benchmark to the detriment of other, also important, specs. So, while the product may kick ass on the common benchmarks it may not be so great because other important stuff gets neglected.. The benchmark process starts steering the design "committee"..

Re:Build to the benchmark

[somenickname]

Companies have been known to take this even further. You can probably find plenty of compilers that have something like, "if(this_looks_like_benchmark_x) emit_special_code_for_benchmark_x". I know for a fact that the old Sun compiler could detect a matrix multiply and would emit hand tuned, parallelized assembly when it detected it.

Vendors will always play games with benchmarks and customers will always read things into benchmarks that aren't true. That's not to say that benchmarks aren't useful but, if you are making decisions based on benchmarks, you really need to understand what is being benchmarked, who did the benchmarking and what (if anything) the benchmark results mean.

My view is:

[QuietLagoon]

Benchmark tools do well when they are used for what they are designed to measure. Benchmark tools go off the rails when they are seen and interpreted as some kind of all-purpose suitability tester.

Reviewes

[110010001000]

"however, in real life shooting, the Galaxy S7's shooter offers a far superior result."
Says who? The reviewers "objective" opinion? These are the same guys that say a $10,000 audio cable produces "warmer" sounds than a $5 one.

It's a benchmarck, not God's Score

[gurps_npc]

You are not looking at God's manual for existence, to check a score, like some kind of video game.

It's just the results from a test - helpful, but not perfect. Luck, design for the test, and many other factors may affect it.

If all you do is look at the benchmark, you deserve to be screwed over. Doing so is like looking at new lawyers grades in law school and making the highest score a partner right off the back.

Benchmark = a standard or point of reference

[Proudrooster]

Benchmark: a standard or point of reference against which things may be compared or assessed.

Yes, benchmarks do a good job of comparing two pieces of hardware, especially tests which involve the entire system. I use benchmarks all the time for hardware comparison and system optimization/overclock comparison. Without benchmark tools we couldn't effectively compare changes to setting or in hardware speed specifically raw CPU, raw GPU, raw RAM, and raw DISK I/O speeds.

Benchmark tools also help determine system stability by pushing the hardware to the limit and taking it to it's thermal throttling speed.

So people ship custom hardware to vendors to cheat on benchmark? Yes.

Will these cheats show up in the reviews on NewEgg, Amazon, and Tom's Hardware when they can't be replicated? Yes

So please, benchmark away. Publish the results. Keep the data in a table for all to view. Benchmarks keep everyone honest in the end.

Trust

[wickedsteve]

I can't trust benchmarks unless they are actually doing what my device is for. I have a gaming pc so I trust a benchmark tool that actually renders scenes like the games I play. The benchmark records things that apply to my enjoyment of games like frames per second under various settings. If a tool just gives me a grade on some arbitrary scale then it is no use to me.