Slashdot Mirror
Slashdot Asks: What's Your View On Benchmark Apps?
There's no doubt that benchmark apps help you evaluate different aspects of a product, but do they paint a complete picture? Should we utterly rely on benchmark apps to assess the performance and quality of a product or service? Vlad Savov of The Verge makes an interesting point. He notes that DxOMark (a hugely popular benchmark app for testing a camera) rating of HTC 10's camera sensor is equal to that of Samsung's Galaxy S7, however, in real life shooting, the Galaxy S7's shooter offers a far superior result. "I've used both extensively and I can tell you that's simply not the case -- the S7 is outstanding whereas the 10 is merely good." He offers another example: If a laptop or a phone does well in a web-browsing battery benchmark, that only gives an indication that it would probably fare decently when handling bigger workloads too. But not always. My good friend Anand Shimpi, formerly of AnandTech, once articulated this very well by pointing out how the MacBook Pro had better battery life than the MacBook Air -- which was hailed as the endurance champ -- when the use changed to consistently heavy workloads. The Pro was more efficient in that scenario, but most battery tests aren't sophisticated or dynamic enough to account for that nuance. It takes a person running multiple tests, analyzing the data, and adding context and understanding to achieve the highest degree of certainty. The problem is -- more often than not -- gadget reviewers treat these values as the most important signal when judging a product, which in turn, also influences several readers' opinion. What's your take on this?
Case in point: ADSL line speed. I've had several different ADSL providers, and living somewhat far out, the speed is consistently bad, sometimes awful. But if I try one of the many 'ADSL speed test' websites, the results are always in line with the promised speed. I once routed one of those through a proxy, just for the name change, and the speed was one tenth, same if I accessed it simply by the IP number ! Benchmarks are too easy to cheat. Wasn't it Intel who was caught doing that a few years ago ?
Non-Linux Penguins ?
A good benchmark -- in cameras, CPUs, GPUs, cars, anything really -- is ideally a set of tests which contains a random sampling of real-world scenarios. In the beginning, the benchmark is good precisely because the vendors are unaware of it and don't spend a bunch of time trying to optimize for it specifically.
Once a benchmark becomes popular, companies try to make their product better for the benchmark ("See PHB! I increased our PCBench score by 10%!") but CAN ultimately end up doing so in a custom way that doesn't represent real-world performance (e.g. Volkswagen). Because the company is now specifically trying to optimize for a specific use-case, the benchmark is no longer random and thus no longer representative of real-world use.
Enter a new benchmark, which is really good, and better mirrors real-world performance and the cycle begins anew.
-- Political fascism requires a Fuhrer.
I just want to know I get what I pay for.
Boot up time and Photoshop filters. Use a bittorrent client to measure internet speeds. "Speed test" web sites are dogged down by traffic.
“He’s not deformed, he’s just drunk!”
Just buy one of each at Best Buy and then return the ones you didn't like. Or return them all and buy the one you do like from Newegg.
Benchmark tools are exactly that... a benchmark of that specific machine at that specific time under those specific stresses. Not all that much different than the ACT, SAT or similar standardized tests. Other than computer tests tend to be a bit more accurately repeatable.
As with published machine specs, and user and industry reviews, they can be helpful to size up one machine over another. That is about it.
It's like getting in a religious argument and saying 'My Ghod can beat up your Ghod!'
Not any different than buying a car, washing machine, or cow.
Look for what you need it to do.
Look for what features you need, then extras you would like to have / can afford.
Determine which manufacturer you are willing to shell out your shekels for.
Roll the dice and take your chances along with the rest of us the sheep.
FredInIT
An unpleasant side effect of benchmarking is when manufacturers start building products to do well on the benchmark to the detriment of other, also important, specs. So, while the product may kick ass on the common benchmarks it may not be so great because other important stuff gets neglected.. The benchmark process starts steering the design "committee"..
Bench Marking Tools. I usually use a mechanical pencil myself.
>> What's Your View On Benchmark Tools?
So..who are the "tools" - the shysters creating the benchmarks or the rubes consuming them?
That's the conclusion I've mostly come to, at least for complete consumer products.
When I look at the latest Dell, Apple, etc desktop or laptop I already see the figures available from the maker, and often there's at least a few choices in terms of CPU, RAM, or SSD options. The only way performance from one item to another would be considerably different would be if one OEM made a major error.
On the other hand there are things that are hard to tell from the spec sheet that make a huge difference for me:
Is the keyboard any good?
What kind of fan does this system use?
How bad is the glare on the included screen?
Does the case feel like it will fall apart on the first tweak?
More so for laptops and tablets...
How long can the system maintain a safe temperature while running a stress test, or a lesser stress of an intense game?
And leave the epeen contests to the hipsters. If you can't afford pimp, it's irrelevant anyway.
Benchmark tools do well when they are used for what they are designed to measure. Benchmark tools go off the rails when they are seen and interpreted as some kind of all-purpose suitability tester.
"however, in real life shooting, the Galaxy S7's shooter offers a far superior result."
Says who? The reviewers "objective" opinion? These are the same guys that say a $10,000 audio cable produces "warmer" sounds than a $5 one.
You are not looking at God's manual for existence, to check a score, like some kind of video game.
It's just the results from a test - helpful, but not perfect. Luck, design for the test, and many other factors may affect it.
If all you do is look at the benchmark, you deserve to be screwed over. Doing so is like looking at new lawyers grades in law school and making the highest score a partner right off the back.
excitingthingstodo.blogspot.com
If you want to use a test result, you must first understand what the test is measuring. It isn't ever going to be as simple as "Laptop A got 536 and laptop B got 642, therefore laptop B is better at everything." This same thing applies to medical diagnostic tests, or academic test, or product quality tests. Unfortunately, this is hard. Because statistics is hard. And science is hard.
Sorry. :-(
as someone who has recently been through evaluation at dxomark and with the tech media, I can say that the tech media (e.g., the verge) really don't have much of an idea what they're talking about.. Evaluating camera performance is such a subjective topic (and one that people become very passionate about -- see iPhone), that what one reviewer likes, another reviewer may hate.
The benchmarks at least provide a repeatable set of tests which are designed to measure specific capabilities.... not just "it doesn't take pictures of my cat as good as the iPhone". For example: sharpness. I would rather rely on a review that measured the sharpness to a quantifiable number, rather than see a photo on a review site that gives no context (were you walking? was the subject moving? did you use a tripod? etc.)
The more troubling aspect of a review lab like DxO is that the same company also sells imaging consulting services to device manufacturers.
If I compared a pencil to a thin laptop, there would be many overlapping capabilities and features. Some areas the pencil would excel (subjectively and measurably) and others where the laptop would "win". If a test was only designed to compare thickness, both items may be seen as equal. Color could also be identical, as well as temperature in standby mode. Benchmarks should include a number of tests, and some factors remain subjective, especially when it comes to user preference...but the final choice is up to the customer/consumer.
Benchmark apps do nothing except paint a picture of performance of your current hardware in the current configuration. Unless you are willing to do the legwork and RTFM to figure out the optimal configuration, then you are just wasting time with the benchmark.
Most people that are technically savvy enough to build their own PC are also savvy enough to manually do the math and figure out the optimal configuration. Benchmark software is a tool for beginners.
Benchmark: a standard or point of reference against which things may be compared or assessed.
Yes, benchmarks do a good job of comparing two pieces of hardware, especially tests which involve the entire system. I use benchmarks all the time for hardware comparison and system optimization/overclock comparison. Without benchmark tools we couldn't effectively compare changes to setting or in hardware speed specifically raw CPU, raw GPU, raw RAM, and raw DISK I/O speeds.
Benchmark tools also help determine system stability by pushing the hardware to the limit and taking it to it's thermal throttling speed.
So people ship custom hardware to vendors to cheat on benchmark? Yes.
Will these cheats show up in the reviews on NewEgg, Amazon, and Tom's Hardware when they can't be replicated? Yes
So please, benchmark away. Publish the results. Keep the data in a table for all to view. Benchmarks keep everyone honest in the end.
Systematic review is very important; however, in most cases, the system used to review is not complex enough to effectively qualify what's being reviewed.
It's like any system used to summarize data: fundamentally you're going to get a flawed diagnosis, because it's summarized. Unless you're dealing with a huge amount of data, and the analysis thereof, the answer is almost always "it depends".
And then there is the 'bias review' introduced in a lot of these benchmark tools. It's why open source benchmark methodology has arisen over the years for desktops, by and large.
It wasn't long ago when many of the popular benchmark systems biased against AMD, versus Intel, or one GPU over another. There was also no way you could do anything about it - the Intel C compiler at one point would "do shit compilation" when a non-Intel CPU was detected.
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
I can't trust benchmarks unless they are actually doing what my device is for. I have a gaming pc so I trust a benchmark tool that actually renders scenes like the games I play. The benchmark records things that apply to my enjoyment of games like frames per second under various settings. If a tool just gives me a grade on some arbitrary scale then it is no use to me.
DxOMark is indeed a perfect example of elaborate benchmarking and what can go wrong with it. To make a streamline and objective test they only measure the few things that are the easiest to measure objectively over various cameras. In the end they seem to just combine these test scores and come up with a number that makes no sense if you look at real life performance, since not only they do not measure a multitude of things that also affect performance, but in addition, the way they combine the things they measure is not helping things. For example they measure color depth in bits and they actually say that "22 bits is excellent, differences under 1 bit are hard to detect" and yet, since it is only one of the few things that they can measure, if a camera has a bit depth of 22.9 and another 22.4, with other things being equal the former will get a much higher score, even though they do admit both are excellent and their difference is not perceptible! And in their scores from what I can gather, they don't seem to include resolution, since I see cameras with the same stats getting the same overall score despite one managing exactly the same noise/bit depth etc at twice the resolution!
So, what good is DxOMark? Like every benchmark it is perfect for what it measures. In fact from the posted SNR curves people have calculated things like the read noise and even the quantum efficiency of sensors with what seems to be good accuracy (if you compare the values for the few sensors which have a published or calculated otherwise QE), but the "overall score" is rubbish. The car analogy is of course being given cars to benchmark and you have excellent facilities to measure the breaking distance, the acceleration and the engine noise with great accuracy. You make them into "scores" and add them to give the "overall score" of a car! And to make the analogy more accurate, you measure two basically inaudible electric engines and your sensitive equipment gives you a few db difference in SPL, but it translates to a greater score to the one as inaudible to humans as the other electric car and because it is 1/3rd of the score, it becomes an "overall better" car.
The CPU performance benchmarks are also the same thing. They will tell you how good a CPU runs the benchmark. Hence, when my CS lab for example wanted to get a new processing cluster I made benchmarks out of the projects that would be running on the cluster and Dell, HP and Sun gave us access to sample units to run them. If you don't have access of course things get a lot harder...
Violence is the last refuge of the incompetent. Polar Scope Align for iOS
It can be the case that you have a horrible product that outperforms another. For instance it might be a buggy piece of shit. Or in other cases there may be a lesser performing product that is more powerful in other ways. Such as capabilities being added as a the result of source code being released under a free software license. I wouldn't even begin to consider hardware that was dependent on proprietary components. No performance test is going to rate that and I'm going to be a bit sceptical of someone recommending a Windows laptop that performs well in Windows when I run GNU/Linux. Plus different drivers on each OS can impact such things. It might be the Windows drivers are better today- but what about tomorrow? I want to know that my experience is smooth. That upgrades don't result in loss of support for hardware and other such issues.
Benchmarks are necessary, but not sufficient way to test things.
The reason for benchmarks is simple - you want a scientifically repeatable test that can be used to compare things with each other. This limits the benchmark's utility as a real-world test because it's inherently limited in what it can test. All it gives is how your thing measures up to all the other things out there. And yes, benchmarks will be gamed, doesn't matter the field (see VW, Mitsubishi and everyone else with diesel engines). However, that doesn't mean their utility is null - it's a comparison tool. Just like how your fuel consumption figures are based on somewhat unrealistic test scenarios, they're like that because they have to be repeatable and comparable.
But on devices which are complex, a benchmark will never cover all the use cases and will never cover everything.
(E.g., an audio amplifier is really simple and a benchmark can cover everything because its job is to increase signal strength, so all you need as a benchmark is how far the output waveform deviates from the input waveform. But a preamp that say corrects for room deficiencies cannot be tested by benchmarks alone because its too complex).
So for complex measurements (or things not fully quantifiable, e.g., "image looks better" or "clearer" or "faster", then you need more tests.
Although, for the imaging system a benchmark should be good enough - as all it needs to do is take a photo of a calibration chart and measure the final photo output for errors. Other aspects like lag can be measured as well. In which case if they produce the same results, they should be just as good. (This will be analyzing the image itself internally or via the same screen). If the S7 images are "more vibrant" then perhaps it's the screen itself since OLEDs are known to oversaturate and produce nice looking, but completely color inaccurate photos.
Oh the old days where you could rate a system relative to IBM/XT... and even then people had the same discussion.
When the copyright term is "forever minus a day", live every day like it's the last.
3dmark? pretty pictures
iobench? now that's useful
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Benchmarks are great tools since they are repeatable and give you a picture of what your hardware, phone, etc is capable of.
However I've learned never to rely on the benchmarks alone as they normally don't mimic real world usage scenarios.
Tl;dr, great for reference and stress test, bad for real world usage.
Have you ever fallen asleep at the keybhanusdiog?