Slashdot Mirror
Futuremark Delists Samsung and HTC Android Devices for Cheating 3DMark
MojoKid writes "Benchmarks are serious business. Buying decisions are often made based on how well a product scores, which is why the press and analysts spend so much time putting new gadgets through their paces. However, benchmarks are only meaningful when there's a level playing field, and when companies try to 'game' the business of benchmarking, it's not only a form of cheating, it also bamboozles potential buyers who (rightfully) assume the numbers are supposed mean something. 3D graphics benchmark software developer Futuremark just 'delisted' a bunch of devices from its 3DMark benchmark results database because it suspects foul play is at hand. Of the devices listed, it appears Samsung and HTC in particular are indirectly being accused of cheating 3DMark for mobile devices. Delisted devices are stripped of their rank and scores. Futuremark didn't elaborate on which specific rule(s) these devices broke, but a look at the company's benchmarking policies reveals that hardware makers aren't allowed to make optimizations specific to 3DMark, nor are platforms allowed to detect the launch of the benchmark executable unless it's needed to enable multi-GPU and/or there's a known conflict that would prevent it from running."
If you believe Apple aren't doing precisely the same thing, then I have a shiny white featureless brick to sell you
Do not tell the Apple kids; let them lose their money so they can look cool to the other Apple kids. YOU BUY APPLE; YOU TAKE DICK IN ASS AND SMOKE IT AFTER.
On iOS benchmark scores do not change when you change the executable name...
When you ship with a fast enough system you don't need to cheat to look good on benchmarks.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Part of the problem is that many of the latest 1080p phones are slower in games than their 720p predecessors such as nexus 5 vs nexus 4. When you double the resolution, you need to quadruple the pixels rendered. Consumers want longer battery life and games to run smoothly but the manufactures are pushing for these useless 1080p screens and cheating in benchmarks to make up for loss in performance. On 4" screen 720 is more than enough for normal eyesight.
If you believe Apple aren't doing precisely the same thing
You sing the sad song of so many other jilted lovers, who did not believe they were being cheated on... Because you knew other partners were stable and trustworthy you thought yours was too. And now you are aware of the transgressions of your chosen one, you think everyone must be cheating because how else could it be that *you* were the one cheated upon?
Hint: when your partner said they wanted an open relationship it wasn't because they wanted to spend *more* time with you.
Can you please stop dropping me in beta.slashdot for random articles ? It sucks to have 2 interfaces. Make up your mind. And please, don't let it be beta.
But we must have MOER 'P'. And with 4K being the new 1080p, you can in fact expect more of them to be here soon.
Incidentally, Skyfall would have been a better movie if it were about a plot to switch all smartphone manufactures to power draining 1080p displays without them realizing until it was too late.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Ha Ha, you said "EPA" and "Integrity" in the same post!
I have not been here for 3 years. I left because it sucked really bad, and was thick with fools and kiddies and libs. Is it better?? Will check back tomorrow, thnx.
The next thing to remember is to put next things next.
How dare you tell me that my particular hardware-centric algorithm doesn't precisely measure the quality and capability of your product!
3DMark does not understand what it it measuring, gives scores out of thin air, and blames companies for trying not to LOSE points.
Imagine this situation:
Samsung Galaxy S4 with 8 cores 1.6GHz each: to say it simply: it has algorithms to suppress usage of all of them for 'normal' applications to save battery.
And a ShittyPhone with a dual-core 1.8GHz each and no optimisations for battery life whatsoever.
Which one is better?
Wrong.
3DMark will give more points to the ShittyPhone, because as a single application it cannot utilize the power of Galaxy S4.
What does it mean? That 3DMark is just a shitty benchmark, and that's what it is.
This example is just for CPU power testing, but you can find such flaws in almost all 3DMark tests.
Creators of 3DMark do not have a clue how to test modern multicore smartphones, but they do not care and release their product.
The real problem? People use this shitty benchmark and judge product basing on the meaningless score it produces.
Why should Samsung LOSE customers because 3DMark lied to them?
It's better to 'cheat' this crappy software into being at least a bit more FAIR in judging their products.
... they just obfuscate (or rather encrypt and then decrypt) everything, including every single executable, under the iOS.
In doing so, nobody could be absolutely certain what the fuck is going on within the iOS kernel and the Apple hardwares when the benchmark routine is being run.
Muchas Gracias, Señor Edward Snowden !
I have not been here for 3 years. I left because it sucked really bad
Trust me, /. has gone from bad, to WORSE !!
Buying decisions are often made based on how well a product scores,
That is an unproven hypothesis. Null hypothesis: Buying decisions are often NOT made based on how well a product scores on benchmarks. Evidence: iDevices. The burden of proof is on the claimant to provide GREATER evidence than the null hypothesis, otherwise the claim can be dismissed as confirmation bias, even if you find evidence in support of the orginal hypothesis: Stepping on cracks does not break backs, even if you observe it happening a couple of times. Nerds checking benchmarks before buying gadgets happens. Is this frequent enough to warant use of the word "often"? If so, where's the evidence? You haven't any.
Try this on for size: The niche market segment of geeks who care enough about benchmark scores and use Futuremark as a source for statistics occasionally purchase products based on those scores. It's hypocritical to hold Creationists to a higher standard of evidence than you do yourself.
I guess that it's easier to delist Android, especially when the US government is against them because Apple is paying for lobbying against them.
Benchmarks on buying decisions are for CPUs and GPUs. They are for people building high end machines, or people trying to get the best processing bang for buck.
These are phones. What sells is screen size, phone style, and feature list. No one cares how many points a phone has in benchmarks except for some reviewers. People want to know if it takes good photos, how well the hover features work, if it's 3G or 4G, hell most recently buyers have been more interested in if it comes in white or black rather than the processor.
My phone is pretty much the only "computer" in my entire house where I could honestly not tell you the clock speed. It runs Plants vs Zombies 2 as fast as just about any other phone.
Quote :"Buying decisions are often NOT made based on how well a product scores on benchmarks. Evidence: iDevices"
Except that your evidence is patently wrong as iDevices are amongst the very fastest. I don't know where you got the idea from that iDevices are somehow slow. They are not. Probably from the stupid assumption that core number X and frequency Y automatically means faster. Which is wrong.
Then these phone companies are wasting perfectly good time and money by cheating on the benchmarks, and there's no harm in 3DMark delisting these phones.
(I'd say that if nothing else, these benchmarks generate news stories promoting the new, allegedly-faster device.)
No kidding!!! What do you say at this point?
When Apple brought out Retina Display, that was 100% FAN BLOODY TASTIC according to Apple fans. Absolutely the best thing EVAR, and PROOF Apple are "innovative" by making displays finer in resolution than any other smartphone.
Nokia didn't count, since they were ~12ppi lower resolution! SHUT UP!
But now resolution is higher than Retina Display, higher resolution and better pixel count is BAD. Which, yet again, PROVES Apple are BEST EVAR because they don't waste time trying to get uselessly higher resolutions!
7" tablets were too small when the iPad was only 10".
But when the iPad mini comes out at 7", it's the best size for many many tasks!
Phones were too big if they had a 4.3" screen. Until Apple brought out a bigger screen, then they had many uses!
And so on.
A benchmark measures the performance of a machine while under that particular benchmark.
Otherwise, it's pretty useless. No benchmark has been able to be used for comparison purposes for more than a few months after release (and things like this are re-released once a year or more). Even back in the days of Dhrystones and Whetstones and all that crap - at best it benchmarks one particular run of code, and that's it. And in terms of general performance, it can do no better than guess.
Fact is, if anyone buys because of a 10% increase in a certain benchmark they are an idiot, unless the code they want to run *IS* that benchmark (to all intents and purposes). This is why the best "benchmarks" are things like how many FPS you get in the game you want to play. Because then you'll know exactly how many FPS you'll get in the game you want to play...
We haven't had highly-determinstic computer systems in our PC's for many, many, many years. Caches, bus speeds, interactions, multi-processors, etc. all throw benchmarks in the bin. And everyone's use case is different. Personally, I'd prefer 8 2GHz cores to any other configuration you could imagine at the moment, other people will have different ideas.
Benchmarks are a waste of time. It's like having stupid logic questions on a job interview. All that gets you is people good at answering those stupid logic questions, not at the job, or at worst someone who *appears* good at answering those logic questions.
Benchmarks on smartphones? It makes even less sense. I'm more shocked that Samsung think that anyone gives a shit.
All phones are slow when they have been on the market for 6 months, there is no special magic pixie dust in the iPhones that makes them faster. But people still buy them, same with Androids..
Except that your evidence is patently wrong as iDevices are amongst the very fastest.
Only two core Apple has the slowest processors on the market for its none pemium model, and during its lifetime is outpased by the competition 2-3 times, and is considered dlow by Apple by the time its put into a plastic case (as opposed to simply discounting it).
The original poster is wrong though iDevices success is due to its Brand in countries where it is heavily subsidied in its country of Manufacture it sells only 1% of Phones.
That's funny because Apple seems to be the only smartphone maker paying attention to such laws, not building needlessly dense displays that suck power like a kid with a juice box.
I bought the Nexus 5 which comes with premium 5" 1080P screen and is half the price of the bottom end iPhone. There are phones that come with similar screens to the iPhone like the Moto G for instance which is a sixth of the price of the iPhone.
POS $89 Chinese tablets or something
Ironically for you the iPad is a POS $89 Chinese tablet with a massive mark-up. Although seriously Apples sales are not only destroyed by Android, they are flat(actually dropping) despite explsive market growth. Perhaps they should compete on price.
If you buy a device based on benchmark scores, you're a fucking idiot. Period.
How hard would it be for Futuremark to disguise their benchmark app so as to fool the device? If it just looks for the package name it should be easy. If Samsung reverse engineered the exact workload being done in each benchmark then micro-optimized for that workload...that's harder to fix.
Benchmarks are serious business.
For a tiny segment of the population, maybe. For the rest of the world, raw benchmarks don't matter at all. It's all perception and other features over raw framerate. Normal humans can't really detect anything above 50 or 60 FPS. So if you are proud your phone gets 150FPS, congratulations! You got that going for you, which is nice. I guess.
Seriously, target-specific optimizations are as old as benchmarks themselves. If you're going to start penalizing companies for that, nVidia should've been delisted since 3DMark 99.
Samsung cheats at benchmarks by changing the hardware behavior when it detects certain apps with specific names are running. Change the name, and the cheating stops.
That's the fundamental difference between Samsung and most of the rest of the Android devices. The name is the key to the cheat.
Funny how you're ignoring lots of detailed information about that. This has nothing to do with Apple, this has everything to do with Samsung/HTC/Android.
Great attempt to troll, though!
Buying decisions are often NOT made based on how well a product scores on benchmarks. Evidence: iDevices
Except they win most benchmarks handily.