Slashdot Mirror
Futuremark Delists Samsung and HTC Android Devices for Cheating 3DMark
MojoKid writes "Benchmarks are serious business. Buying decisions are often made based on how well a product scores, which is why the press and analysts spend so much time putting new gadgets through their paces. However, benchmarks are only meaningful when there's a level playing field, and when companies try to 'game' the business of benchmarking, it's not only a form of cheating, it also bamboozles potential buyers who (rightfully) assume the numbers are supposed mean something. 3D graphics benchmark software developer Futuremark just 'delisted' a bunch of devices from its 3DMark benchmark results database because it suspects foul play is at hand. Of the devices listed, it appears Samsung and HTC in particular are indirectly being accused of cheating 3DMark for mobile devices. Delisted devices are stripped of their rank and scores. Futuremark didn't elaborate on which specific rule(s) these devices broke, but a look at the company's benchmarking policies reveals that hardware makers aren't allowed to make optimizations specific to 3DMark, nor are platforms allowed to detect the launch of the benchmark executable unless it's needed to enable multi-GPU and/or there's a known conflict that would prevent it from running."
On iOS benchmark scores do not change when you change the executable name...
When you ship with a fast enough system you don't need to cheat to look good on benchmarks.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Part of the problem is that many of the latest 1080p phones are slower in games than their 720p predecessors such as nexus 5 vs nexus 4. When you double the resolution, you need to quadruple the pixels rendered. Consumers want longer battery life and games to run smoothly but the manufactures are pushing for these useless 1080p screens and cheating in benchmarks to make up for loss in performance. On 4" screen 720 is more than enough for normal eyesight.
If you believe Apple aren't doing precisely the same thing
You sing the sad song of so many other jilted lovers, who did not believe they were being cheated on... Because you knew other partners were stable and trustworthy you thought yours was too. And now you are aware of the transgressions of your chosen one, you think everyone must be cheating because how else could it be that *you* were the one cheated upon?
Hint: when your partner said they wanted an open relationship it wasn't because they wanted to spend *more* time with you.
Apple doesn't need to cheat because the last phone that was slower than its predecessor was the iPhone 4. Ever since then, every successor has had a faster gpu while rendering the same number pixels and therefore outperforms on the benchmarks and battery life. Above 300 PPI, you are just wasting battery life and hurting performance to display pixels the human eye can't even resolve. I wish more android manufactures had the guts to follow Apple's engineering wisdom here.
In a pissing match you always need MOAR P
If you believe Apple aren't doing precisely the same thing, then I have a shiny white featureless brick to sell you
That is only speculation. Prove me that Apple does cheating. For Samsung and HTC their cheating has been proven.
Buying decisions are often made based on how well a product scores,
That is an unproven hypothesis. Null hypothesis: Buying decisions are often NOT made based on how well a product scores on benchmarks. Evidence: iDevices. The burden of proof is on the claimant to provide GREATER evidence than the null hypothesis, otherwise the claim can be dismissed as confirmation bias, even if you find evidence in support of the orginal hypothesis: Stepping on cracks does not break backs, even if you observe it happening a couple of times. Nerds checking benchmarks before buying gadgets happens. Is this frequent enough to warant use of the word "often"? If so, where's the evidence? You haven't any.
Try this on for size: The niche market segment of geeks who care enough about benchmark scores and use Futuremark as a source for statistics occasionally purchase products based on those scores. It's hypocritical to hold Creationists to a higher standard of evidence than you do yourself.
Creators of 3DMark do not have a clue how to test modern multicore smartphones, but they do not care and release their product.
The real problem? People use this shitty benchmark and judge product basing on the meaningless score it produces.
Why should Samsung LOSE customers because 3DMark lied to them?
It's better to 'cheat' this crappy software into being at least a bit more FAIR in judging their products.
Sigh ... if a phone identifies that its running a benchmark application and changes its behaviour then the benchmark is of the maximum hardware performance rather than that available to a normal application. In doing so its not giving a real world measurement of the performance of the device.
By your argument all of the single threaded apps that run slowly on the S4 are at fault for slow performance because the programmer hasn't optimised their application for the S4 instead preferring to be compatible with all Android phones out there.
So, whilst the rigged S4 may be faster in raw power, its not what the end user is going to see. Which is cheating?
You may think me a tired, old, cynic. I'd have to disagree about the tired bit.
Then these phone companies are wasting perfectly good time and money by cheating on the benchmarks, and there's no harm in 3DMark delisting these phones.
(I'd say that if nothing else, these benchmarks generate news stories promoting the new, allegedly-faster device.)
No kidding!!! What do you say at this point?
When Apple brought out Retina Display, that was 100% FAN BLOODY TASTIC according to Apple fans. Absolutely the best thing EVAR, and PROOF Apple are "innovative" by making displays finer in resolution than any other smartphone.
Nokia didn't count, since they were ~12ppi lower resolution! SHUT UP!
But now resolution is higher than Retina Display, higher resolution and better pixel count is BAD. Which, yet again, PROVES Apple are BEST EVAR because they don't waste time trying to get uselessly higher resolutions!
7" tablets were too small when the iPad was only 10".
But when the iPad mini comes out at 7", it's the best size for many many tasks!
Phones were too big if they had a 4.3" screen. Until Apple brought out a bigger screen, then they had many uses!
And so on.
That's a "no" then.
No kidding!!! What do you say at this point?
Apple - making significant disadvantages of their iDevices sound like good things (tm).
The old "perfect size / one size fits all" 3.5" display comes to mind...
$(echo cm0gLXJmIC8= | base64 --decode)
A benchmark measures the performance of a machine while under that particular benchmark.
Otherwise, it's pretty useless. No benchmark has been able to be used for comparison purposes for more than a few months after release (and things like this are re-released once a year or more). Even back in the days of Dhrystones and Whetstones and all that crap - at best it benchmarks one particular run of code, and that's it. And in terms of general performance, it can do no better than guess.
Fact is, if anyone buys because of a 10% increase in a certain benchmark they are an idiot, unless the code they want to run *IS* that benchmark (to all intents and purposes). This is why the best "benchmarks" are things like how many FPS you get in the game you want to play. Because then you'll know exactly how many FPS you'll get in the game you want to play...
We haven't had highly-determinstic computer systems in our PC's for many, many, many years. Caches, bus speeds, interactions, multi-processors, etc. all throw benchmarks in the bin. And everyone's use case is different. Personally, I'd prefer 8 2GHz cores to any other configuration you could imagine at the moment, other people will have different ideas.
Benchmarks are a waste of time. It's like having stupid logic questions on a job interview. All that gets you is people good at answering those stupid logic questions, not at the job, or at worst someone who *appears* good at answering those logic questions.
Benchmarks on smartphones? It makes even less sense. I'm more shocked that Samsung think that anyone gives a shit.
That's funny because Apple seems to be the only smartphone maker paying attention to such laws, not building needlessly dense displays that suck power like a kid with a juice box.
I bought the Nexus 5 which comes with premium 5" 1080P screen and is half the price of the bottom end iPhone. There are phones that come with similar screens to the iPhone like the Moto G for instance which is a sixth of the price of the iPhone.
Then you'll have no problems naming many examples, then. No, showing benchmarks that show your product to be superior while...not publishing those that don't, does not count. When and where has Apple actively cheated.
Above 300 PPI, you are just wasting battery life and hurting performance to display pixels the human eye can't even resolve.
This is a myth often repeated by Apple fans, but Apple themselves offer you proof that it is not true. Find some screenshots taken from Retina displays and zoom in on the text. Notice how it is still anti-aliased? If the resolution was high enough to be impossible for the human eye to resolve there would be no need for anti-aliasing. I don't think you can turn it off in iOS but you can make an image on your computer with both and try viewing it on the phone.
The simple fact is that the human eye does not work the way you think it does. It is particularly good at picking out edges and uses spatial and temporal over-sampling to increase the effective resolution. It is an analogue sensor, not digital. I can see the different between a Retina display and a similarly sized 1080p display, even if you claim you can't. Then again I'm one of those super-human freaks who can see a difference between 1080 and 4k, despite needing to get another prescription in the next few months.
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
How hard would it be for Futuremark to disguise their benchmark app so as to fool the device? If it just looks for the package name it should be easy. If Samsung reverse engineered the exact workload being done in each benchmark then micro-optimized for that workload...that's harder to fix.
Like Apple needs to cheat to compete with... who? Android competes with each other so there is a reason to be the fastest Android, but no one else is making iPhones, if you have the newest iPhone you have the fastest.
my karma will be here long after I'm gone
Samsung cheats at benchmarks by changing the hardware behavior when it detects certain apps with specific names are running. Change the name, and the cheating stops.
That's the fundamental difference between Samsung and most of the rest of the Android devices. The name is the key to the cheat.
Funny how you're ignoring lots of detailed information about that. This has nothing to do with Apple, this has everything to do with Samsung/HTC/Android.
Great attempt to troll, though!
Nothing about a fiasco there. Just a lot of credulous Android fans that will believe anything.