How Many Android OEMs Cheat Benchmark Scores? Pretty Much All of Them

← Back to Stories (view on slashdot.org)

How Many Android OEMs Cheat Benchmark Scores? Pretty Much All of Them

Posted by Soulskill on Friday October 4, 2013 @01:34AM from the at-least-they're-consistent dept.

An anonymous reader writes "After Samsung got caught out cheating on benchmarks (Note 3, Galaxy S4) AnandTech has done a detailed analysis of the state of benchmark cheating amongst Android OEMs. With the exception of Motorola, literally every single OEM they've looked at ships (or has shipped) at least one device that does benchmark-specific CPU optimizations. AnandTech also thinks it will get worse before it gets better. 'The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0 - 5%, and somewhere south of 10% on our GPU benchmarks as far as we can tell. I can't stress enough that it would be far less painful for the OEMs to just stop this nonsense and instead demand better performance/power efficiency from their silicon vendors.' The article notes that Apple doesn't do any of the frequency gaming stuff."

25 of 189 comments (clear)

Min score:

Reason:

Sort:

Easy solution by TheRealMindChild · 2013-10-04 01:38 · Score: 4, Interesting

The benchmark software should randomize the process name on launch

--

"When life gives you lemons, don't make lemonade. Make life take the lemons back!" -- Cave Johnson
1. Re:Easy solution by hairyfeet · 2013-10-04 02:19 · Score: 4, Insightful
  
  Even easier...don't use benchmarks which are easily rigged, choose real world applications and cook up some tests using them!
  The problem is ALL the benchmarks have been rigged by somebody for ages...X86? Any benchmark compiled using ICC is rigged against AMD and boosts Intel by 30% (look up "Intel Cripple Compiler" if you want to learn more, end result is benchmarks are completely useless on X86 unless compiled with GCC) and as we see here ARM? Rigged by everybody. This kind of crap has been going on since "Quack.exe" and is why you shouldn't trust ANY benchmarks. I'll always remember one of the reviews before the guy was told about ICC when he was testing netbooks "These new netbooks with AMD seem to be MUCH snappier...yet it loses to the low end Atom by nearly 30%, and I just don't know why"...the reason why is the benchmarks are all rigged!
  So if you want to bench an ARM chip? Root the phone, install Bash, and run a script that runs real applications in real world scenarios. there are more than enough apps out there that cooking up their own informal benches should NOT be hard and until they do? Just remember to replace the word benchmark with bullshit every time you see one of these articles.
  
  --
  ACs don't waste your time replying, your posts are never seen by me.
2. Re:Easy solution by hairyfeet · 2013-10-04 03:41 · Score: 3, Informative
  
  If you are trying to say AMD's compiler rigs? Sorry you are full of shit, the reviewer that first found the cripple code in ICC did rigorous testing of AMD's compiler and found it was merely GCC with some extra X64 flags added to better support 64 bit, that is it. Oh and you can download the source for the ACC and see for yourself, its based on GCC and is fully GPL V2 compliant.
  
  --
  ACs don't waste your time replying, your posts are never seen by me.
Or, alternatively by Truth_Quark · 2013-10-04 01:44 · Score: 5, Insightful

The phone manufacturers should not be dicks.
1. Re:Or, alternatively by Anonymous Coward · 2013-10-04 02:06 · Score: 3, Interesting
  
  On the contrary, the testers should just be bigger dicks. "We detected benchmark-specific optimizations in products #1, #2 and #3, so they all got zero points."
2. Re:Or, alternatively by L4t3r4lu5 · 2013-10-04 02:33 · Score: 4, Informative
  
  Pfff. Car manufacturers tape up the air intakes and door seams on their cars to do fuel economy runs, just to eek out the every last 0.1mpg. Running your car like that for any reasonable period of time would wreck the engine pretty quick.
  
  Benchmarks are about as useful as manufacturer spec sheets. Take both with a a few metric tonnes of salt.
  
  --
  Finally had enough. Come see us over at https://soylentnews.org/
3. Re:Or, alternatively by gnasher719 · 2013-10-04 03:14 · Score: 4, Insightful
  
  On the contrary, the testers should just be bigger dicks. "We detected benchmark-specific optimizations in products #1, #2 and #3, so they all got zero points."
  That seems quite arbitrary. What about "To test the battery, we tested how many minutes the battery lasts while running benchmark X". The cheaters will get shorter battery life.
4. Re:Or, alternatively by ftobin · 2013-10-04 03:34 · Score: 4, Informative
  
  Actually, manufacturers do report their own efficiency numbers, and the EPA spot-checks them.
  http://business.time.com/2012/12/10/more-reason-to-be-skeptical-about-new-car-mpg-claims/
5. Re:Or, alternatively by Pieroxy · 2013-10-04 03:49 · Score: 3, Insightful
  
  Just buy an iPhone. Apple doesn't cheat apparently.
  
  --
  Write boring code, not shiny code!
And Apple by Anonymous Coward · 2013-10-04 01:50 · Score: 5, Informative

With the exception of Motorola...
And Apple. Apple and Motorola/Google are the only two companies that don't boost their devices for benchmark tests. If you're going to give credit to one, please do be fair and give credit to the other.
I respect both of them for that level of integrity and I hope they stick to their guns and remain honest.
I may be an Apple fanboy (and I am) but I'm really looking forward to seeing what Motorola starts releasing in about a year once Google's able to, as they said, flush things out of the system and start releasing truly Google-designed products.
1. Re:And Apple by Anonymous Coward · 2013-10-04 01:55 · Score: 3, Interesting
  
  Posting anonymously since I'm a motoroogle employee... you'll be disappointed. I certainly am. At this point, I expect google to shut us down or spin us off.
2. Re:And Apple by h4rr4r · 2013-10-04 01:57 · Score: 3, Insightful
  
  I fear Motorola is dead.
  The Moto X was supposed to be a device with high amounts of post Google input. It is a high priced mediocre mid range device. They still cater to carriers unlike Apple. They need to compete with the 5S, One X and S4. Instead they are competing with last years product.
  Moto needs to copy Apple in some things. Make 1 main device, sell the old one or a cheaper version as well. Max 2-3 devices. Refresh them once a year. This lets third parties make all kinds of doodads for the device. Put them on all carriers and update them at the same time. If need be get the FCC to lean on Verizon's C block requirements. Other than that, unlock the bootloaders and make donations of devices to the popular mod groups. Most users will never care, but the PR among the tech crowd would be huge and they influence other buyers.
3. Re:And Apple by Dixie_Flatline · 2013-10-04 02:07 · Score: 5, Interesting
  
  It's not a mid-range device. It's only mid-range if you look at the spec sheet and nothing else. Its (non-gamed) benchmarks are actually pretty good for all this talk of 'mid-range'. They did the same thing Apple did and tried to balance out performance with battery life. They didn't put the biggest screen in it, and they have optimised silicon to listen for commands without keeping the CPU on all the time.
  Specs aren't the war that anyone should be trying to win in the mobile space. That kind of thinking is why there are phones that only last half the day.
4. Re:And Apple by JanneM · 2013-10-04 02:20 · Score: 3, Interesting
  
  I've heard a number of people I trust comment on how the Moto X just feels really good in the hand and how the screen and size is just right. So the phone is certainly not bad at all, and most of us don't actually buy the high-end phones any more than we buy the high-end cars or bikes. A really good mid-range phone is exactly what I want; I'm already making calf-eyes toward the coming Sony Z1 Mini.
  What's hurting Motorola for me, personally, is that it's simply not on sale in most of the world, and seems unlikely to ever be. It's not just about being able to get it where I live, but having a phone designed from the start to be usable in all major regions of the world as I travel.
  
  --
  Trust the Computer. The Computer is your friend.
5. Re:And Apple by king+neckbeard · 2013-10-04 02:21 · Score: 3, Informative
  
  Perhaps you should take a refresher course on reading comprehension. The first sentence mentions that Android OEMs were analyzed. Apple is not an Android OEM, so they were not included at that point.
  
  --
  This is my signature. There are many like it, but this one is mine.
6. Re:And Apple by Karlt1 · 2013-10-04 03:04 · Score: 3, Insightful
  
  Apple is not the exception. They just fudge the numbers in a different way. Really, they're all bastards though and I don't trust them any further than I can throw them.
  And a little deeper analysis will show....
  a) Apple accounted for numbers the same way they always did.
  b) Apple does not count phones as sold if they are still in Apple Stores inventories.
  c) If Apple were channel stuffing, phones would readily be available at carriers.
  The fact is that the "analyst" were wrong and are trying to cover their tails.
  Do you really believe that there is not enough of a demand in the launch countries to sale 9 million phones in one weekend?
  Do you think that Apple will be forced to take a write down (like MS and BlackBerry) for unsold inventory in the channel?
7. Re:And Apple by Trillian_1138 · 2013-10-04 05:52 · Score: 3, Interesting
  
  Just picked up a Moto X to replace my aging Galaxy Nexus. I'm on Verizon, which makes me skeptical I'll see another Nexus phone anytime soon. I like the near-stock Moto X setup, and the little tweaks (active notifications, touchless control, etc) are pretty fun. It's a pretty great phone, and I hope it helps boost Motorola in the Android world. I do think they priced it too high (although I still bought it, so take that as you will..) but I highly recommend the phone to anyone who plays the occasional, not to graphic-intensive games, wants their phone to be quick and responsive, likes stock Android, etc. It's an awesome phone.
Re:Probably wont get better by h4rr4r · 2013-10-04 01:58 · Score: 3, Insightful

They started that game long before you could get 1 GB hard drives.
Hilarious... by Savage-Rabbit · 2013-10-04 02:05 · Score: 3, Insightful

The hilarious part of all of this is we’re still talking about small gains in performance.
The hilarious part of all this is that most people really don't give a rat's as about performance when selecting a phone or even a tablet. The criteria are things like: how does it handle? How intuitive is the UI? Can I watch my favorite online video feeds on this thing? Are any buttons in annoying palaces? What's the price? Does this thing have software to view and edit MS Office files I get sent by mail? The only performance tests these smartphone and tablet things usually get is playing around with a display example in the shop and seeing if the UI is nice and snappy. Nobody excepts tech nerds gives a rats ass that a Samsun Galaxy 4 get a few more FPS in Modern Combat than an iPhone 5.

--
Only to idiots, are orders laws.
-- Henning von Tresckow
1. Re:Hilarious... by Savage-Rabbit · 2013-10-04 03:32 · Score: 3, Insightful
  
  Apple decided the only way they can differentiate their iPhone 5s amidst all the comments that they are no longer innovative is to create a competitive market based on useless CPU performance numbers, just like what Apple did with Retina displays. Before Retina, nobody cared about pixel density. Before A7, nobody cared about CPU performance or its bittyness. Before the iPhone 5s camera, nobody cared about the size of the CCD pixel on their phone camera.
  Dude, It's not that long ago that articles like this slashvertisment were plastered all over the web accompanied by comments filled with enthusiastic boasts by hoards of Android fans detailing how iPhone performance sucks ass. Even if blisteringly fast benchmark performance is pretty low down on the list of most people out to buy a smartphone I still can't fault Apple for trying to put a sock in the collective mouth of the Android community. There is a certain personal satisfaction to be had from making the choir of hard core Android fans shut up about benchmarks until Samsung comes up with a still faster device (hopefully free of benchmark cheating this time) even if the customers will probably hardly notice this stupid pissing contest.
  
  --
  Only to idiots, are orders laws.
  -- Henning von Tresckow
The best part by aitikin · 2013-10-04 02:19 · Score: 3, Funny

Is that the table is titled "I Can't Believe I Have to Make This Table". Made me smile.

--
"Don't meddle in the affairs of a patent dragon, for thou art tasty and good with ketchup." ~ohcrapitssteve
If you measure it, it gets better by tompaulco · 2013-10-04 02:25 · Score: 4, Insightful

If you measure it, it gets better. Nobody ever stops to think why. They mostly just think they are good managers. But in fact, if you create a measuring tool to measure qualities of a device, the manufacturers will work to make that measurement better. If you make a measurement to determine how your employees are performing, they will perform better according to that measurement. That's just the way it works.
You can't measure everything, so you're best bet is to try to keep the measurement methods secret and change them frequently. Unless, of course, your measurements are intended to improve a particular area, then by all means, measure on.

--
If you are not allowed to question your government then the government has answered your question.
Re:"Pretty Much All of Them" by flimflammer · 2013-10-04 04:00 · Score: 4, Insightful

No, not really. Comparing iOS and Android directly on performance is silly. They're two totally different ecosystems and hard performance numbers don't change much. That's like a typical user picking a Mac or Windows PC because one performs 5% better at random tasks, ignoring the fact that the offerings between each machine is radically different and pure performance numbers are only a tiny part of the whole picture.
Apple has no reason to cheat because they have no competition that merits the risk of cheating on. It might have been a different story had iOS hardware been available from multiple vendors.
Re:Wrong, they are boosting clock speed above norm by ifiwereasculptor · 2013-10-04 04:03 · Score: 4, Funny

either the GPU or CPU cock increased.
Whoa, can cell phones do that now? I hold these things to my ear, for Christ's sake!
Re:"Pretty Much All of Them" by triffid_98 · 2013-10-04 05:00 · Score: 3, Funny

No, Apple already has that part down to a science.

Step #1
Hype new iPhone
Step #2
Release new iPhone
Step #3
Immediately release new iOS update.
Step #4
Watch existing iPhone users complain after the iOS update cripples older models.
Step #5
Laugh maniacally after existing iPhone users stand in lines waiting for new uncrippled iPhones.
Step #6
PROFIT