How Many Android OEMs Cheat Benchmark Scores? Pretty Much All of Them

An anonymous reader writes "After Samsung got caught out cheating on benchmarks (Note 3, Galaxy S4) AnandTech has done a detailed analysis of the state of benchmark cheating amongst Android OEMs. With the exception of Motorola, literally every single OEM they've looked at ships (or has shipped) at least one device that does benchmark-specific CPU optimizations. AnandTech also thinks it will get worse before it gets better. 'The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0 - 5%, and somewhere south of 10% on our GPU benchmarks as far as we can tell. I can't stress enough that it would be far less painful for the OEMs to just stop this nonsense and instead demand better performance/power efficiency from their silicon vendors.' The article notes that Apple doesn't do any of the frequency gaming stuff."

Comments

Easy solution

[TheRealMindChild]

The benchmark software should randomize the process name on launch

Re:Easy solution

[hairyfeet]

Even easier...don't use benchmarks which are easily rigged, choose real world applications and cook up some tests using them!

The problem is ALL the benchmarks have been rigged by somebody for ages...X86? Any benchmark compiled using ICC is rigged against AMD and boosts Intel by 30% (look up "Intel Cripple Compiler" if you want to learn more, end result is benchmarks are completely useless on X86 unless compiled with GCC) and as we see here ARM? Rigged by everybody. This kind of crap has been going on since "Quack.exe" and is why you shouldn't trust ANY benchmarks. I'll always remember one of the reviews before the guy was told about ICC when he was testing netbooks "These new netbooks with AMD seem to be MUCH snappier...yet it loses to the low end Atom by nearly 30%, and I just don't know why"...the reason why is the benchmarks are all rigged!

So if you want to bench an ARM chip? Root the phone, install Bash, and run a script that runs real applications in real world scenarios. there are more than enough apps out there that cooking up their own informal benches should NOT be hard and until they do? Just remember to replace the word benchmark with bullshit every time you see one of these articles.

Or, alternatively

[Truth_Quark]

The phone manufacturers should not be dicks.

Re:Or, alternatively

[L4t3r4lu5]

Pfff. Car manufacturers tape up the air intakes and door seams on their cars to do fuel economy runs, just to eek out the every last 0.1mpg. Running your car like that for any reasonable period of time would wreck the engine pretty quick.

Benchmarks are about as useful as manufacturer spec sheets. Take both with a a few metric tonnes of salt.

Re:Or, alternatively

[gnasher719]

On the contrary, the testers should just be bigger dicks. "We detected benchmark-specific optimizations in products #1, #2 and #3, so they all got zero points."

That seems quite arbitrary. What about "To test the battery, we tested how many minutes the battery lasts while running benchmark X". The cheaters will get shorter battery life.

Re:Or, alternatively

[ftobin]

Actually, manufacturers do report their own efficiency numbers, and the EPA spot-checks them.

http://business.time.com/2012/12/10/more-reason-to-be-skeptical-about-new-car-mpg-claims/

And Apple

[Anonymous Coward]

With the exception of Motorola...

And Apple. Apple and Motorola/Google are the only two companies that don't boost their devices for benchmark tests. If you're going to give credit to one, please do be fair and give credit to the other.

I respect both of them for that level of integrity and I hope they stick to their guns and remain honest.

I may be an Apple fanboy (and I am) but I'm really looking forward to seeing what Motorola starts releasing in about a year once Google's able to, as they said, flush things out of the system and start releasing truly Google-designed products.

Re:And Apple

[Dixie_Flatline]

It's not a mid-range device. It's only mid-range if you look at the spec sheet and nothing else. Its (non-gamed) benchmarks are actually pretty good for all this talk of 'mid-range'. They did the same thing Apple did and tried to balance out performance with battery life. They didn't put the biggest screen in it, and they have optimised silicon to listen for commands without keeping the CPU on all the time.

Specs aren't the war that anyone should be trying to win in the mobile space. That kind of thinking is why there are phones that only last half the day.

If you measure it, it gets better

[tompaulco]

If you measure it, it gets better. Nobody ever stops to think why. They mostly just think they are good managers. But in fact, if you create a measuring tool to measure qualities of a device, the manufacturers will work to make that measurement better. If you make a measurement to determine how your employees are performing, they will perform better according to that measurement. That's just the way it works.
You can't measure everything, so you're best bet is to try to keep the measurement methods secret and change them frequently. Unless, of course, your measurements are intended to improve a particular area, then by all means, measure on.

Re:"Pretty Much All of Them"

[flimflammer]

No, not really. Comparing iOS and Android directly on performance is silly. They're two totally different ecosystems and hard performance numbers don't change much. That's like a typical user picking a Mac or Windows PC because one performs 5% better at random tasks, ignoring the fact that the offerings between each machine is radically different and pure performance numbers are only a tiny part of the whole picture.

Apple has no reason to cheat because they have no competition that merits the risk of cheating on. It might have been a different story had iOS hardware been available from multiple vendors.

Re:Wrong, they are boosting clock speed above norm

[ifiwereasculptor]

either the GPU or CPU cock increased.

Whoa, can cell phones do that now? I hold these things to my ear, for Christ's sake!