How Many Android OEMs Cheat Benchmark Scores? Pretty Much All of Them

An anonymous reader writes "After Samsung got caught out cheating on benchmarks (Note 3, Galaxy S4) AnandTech has done a detailed analysis of the state of benchmark cheating amongst Android OEMs. With the exception of Motorola, literally every single OEM they've looked at ships (or has shipped) at least one device that does benchmark-specific CPU optimizations. AnandTech also thinks it will get worse before it gets better. 'The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0 - 5%, and somewhere south of 10% on our GPU benchmarks as far as we can tell. I can't stress enough that it would be far less painful for the OEMs to just stop this nonsense and instead demand better performance/power efficiency from their silicon vendors.' The article notes that Apple doesn't do any of the frequency gaming stuff."

Comments

Easy solution

[TheRealMindChild]

The benchmark software should randomize the process name on launch

Re:Easy solution

[h4rr4r]

So then you just look for behavior. Oh look a bunch of unnatural activity, change the governor to performance instead of on_demand.

Re:Easy solution

[Charliemopps]

you're assuming the benchmark authors aren't working directly with the hardware manufacturers.

Re:Easy solution

[JanneM]

So, make the benchmark software resemble the composite behaviour of common classes of apps. OGL benchmarks effectively act as the "typical" 3D game and so on.

At which point cheating becomes pointless, as any tweaking in favour of performance on those benchmarks immediately hit you as worse batterly life and high temps for all users running any similar kinds of apps, including your reviewers.

Re:Easy solution

[hairyfeet]

Even easier...don't use benchmarks which are easily rigged, choose real world applications and cook up some tests using them!

The problem is ALL the benchmarks have been rigged by somebody for ages...X86? Any benchmark compiled using ICC is rigged against AMD and boosts Intel by 30% (look up "Intel Cripple Compiler" if you want to learn more, end result is benchmarks are completely useless on X86 unless compiled with GCC) and as we see here ARM? Rigged by everybody. This kind of crap has been going on since "Quack.exe" and is why you shouldn't trust ANY benchmarks. I'll always remember one of the reviews before the guy was told about ICC when he was testing netbooks "These new netbooks with AMD seem to be MUCH snappier...yet it loses to the low end Atom by nearly 30%, and I just don't know why"...the reason why is the benchmarks are all rigged!

So if you want to bench an ARM chip? Root the phone, install Bash, and run a script that runs real applications in real world scenarios. there are more than enough apps out there that cooking up their own informal benches should NOT be hard and until they do? Just remember to replace the word benchmark with bullshit every time you see one of these articles.

Re:Easy solution

[erroneus]

Better solution. No benchmark test should be accepted unless the software was written AFTER the release of the hardware being tested. And then, yes, randomize various things including the compiled code.

Re:Easy solution

[Sockatume]

You want something that's consistent and reliable from device to device and run to run; that consistency dooms any attempt to cloak the behaviour of the app to failure.

Re:Easy solution

[Sockatume]

The idea that people should have to come up with less-reliable, improvised tests because the hardware manufacturers are going to crawl over each other to cheat any consistent, scientific one is kind of depressing. You kind of know that all these phone companies are first-rate bullshit artists, with Samsung running at the fore, but the idea even that their engineers are going out of their way to fake their product to the top of the specs pile is just sad.

Re:Easy solution

[Anonymous Coward]

Additional solution -

Combine the performance tests with a battery rundown test and combine it into a composite score.

i.e. setup the benchmark to loop run until the battery dies.

Then if anyone decides to do any funky cheats, it will kill the score because it will ruin the battery life.

Re:Easy solution

[larry bagina]

Why don't you just use AMD's compiler? Oh, that's right.

Re:Easy solution

[realityimpaired]

Oh look a bunch of unnatural activity, change the governor to performance instead of on_demand.

That shouldn't make that much of a difference... if the governor is in the low_power setting, limiting the CPU to minimum clock speed, then there'll be a performance hit on benchmarks, but with on_demand, by the time the system has realized it needs to change the governor to performance, the clock speed will have already been increased to maximum due to CPU load. Meanwhile, you're wasting clock cycles monitoring what stuff is doing....

Re:Easy solution

[Bill, Shooter of Bul]

That sounds like a great idea, but that would just favor phones with larger batteries. At the end of the day, the benchmark wouldn't accurately represent real life use. Which is the only reason to look at them, unless you're a fan of bar charts and statistics.

Re:Easy solution

[h4rr4r]

I was thinking to cheat battery test for example or have it lift the max CPU clock above what is normally used.

Re:Easy solution

[Zero__Kelvin]

"So then you just look for behavior. Oh look a bunch of unnatural activity, change the governor to performance instead of on_demand."

That's an excellent idea. Now all you have to do it look for a way to do it with almost no overhead.

Re:Easy solution

[mwvdlee]

The goal is to make detection so complicated that the simple act of detecting a benchmark reduces performance to levels impossible to compensate by cheating.

Re:Easy solution

[mwvdlee]

Alternatively, can we get an app that changes the name of any game to one of the benchmarks' names?

Re:Easy solution

[hairyfeet]

If you are trying to say AMD's compiler rigs? Sorry you are full of shit, the reviewer that first found the cripple code in ICC did rigorous testing of AMD's compiler and found it was merely GCC with some extra X64 flags added to better support 64 bit, that is it. Oh and you can download the source for the ACC and see for yourself, its based on GCC and is fully GPL V2 compliant.

Re:Easy solution

[realityimpaired]

Fair enough, And not an unreasonable measure.

Truthfully, and I know I'm in the minority in a discussion about benchmarks, cell phones have been "good enough" for quite a while, and I bought mine based on feature list and price. I bought the cheapest phone that had all of the features I wanted: FM radio, GPS, bluetooth, wifi tether, and that's it. Didn't even care whether it had 4G connectivity (though it did). Even a bottom-end $100 smart phone with an 800MHz single core processor is in the performance range that will be good enough for most of us, and for the serious gamers, they can worry about getting the most expensive quad core on the market, or, like I did, get a good quality tablet to accompany the phone. With entrants like the Asus MeMo Pad HD 7" (a sub-$200 quad core tablet with plenty of memory, there are others but that's the one I bought), it's getting ridiculously easy to get good performance in your devices for just about anything you throw at it in the real world.

Re:Easy solution

[hairyfeet]

Yes but until these companies stop cheating, just as Intel to this very day cheats with ICC then what choice do you have? If every scale is being rigged by the manufacturer then the only choice is to not use their scale, correct?

And in both cases since you have access to a CLI writing a script that calls real world applications shouldn't be very hard. as long as all the corps are rigging there really isn't a choice.

Re:Easy solution

[h4rr4r]

I totally agree. I want a better display, more color accurate and such more than faster CPU.

Re:Easy solution

[ifiwereasculptor]

OIn the contrary, they should come up with more meaningful tests. On the x86 side, there are quite a few staples of testing that are also real-world scenarios, like RAR decompression, encryption and video and mp3 encoding. For GPUs, an assorted bundle of real games are great for measuring performance.

Re:Easy solution

[tlhIngan]

The benchmark software should randomize the process name on launch

Can you even do that on Android? Generally to rename the package on Android you need to take the APK apart and rename the DEX files there.

Right now, it looks like the most impressive tests are ones done with the browser purely because to cheat three means generally it works for everyone.

(Though even then I can see why they'd cheat - an iPhone 5s dual core processor can keep up with quad core SoCs running nearly twice as fast (1.3GHz A7 vs. 2.3GHz Snapdragon).).

Re:Easy solution

[sjames]

Battery life measurements should take place with the benchmark app running.

Re:Easy solution

[hairyfeet]

Exactly and all of those can be trivially automated with scripts so they can be just as accurate as any benchmark WITHOUT allowing the companies to rig the test!

At the end of the day if everybody has rigged the test then the test is worthless, that's it, no more argument on the subject, the test is worthless and meaningless if everybody cheats and as we have seen both on X86 and on ARM cheating is rampant, so what to do? There is only one thing you CAN do and that is to throw out the test and use real applications to measure real performance. this way they can't rig as there are dozens of tools that do the same job, from WinRAR to PKZip, from Handbrake to WinDVD, so by using real world applications not only do they give you actual results without cheats but they also give you more of an overview of what the device is really like on day to day usage. Seems like a win/win to everyone but the companies selling benchmark software to me.

Or, alternatively

[Truth_Quark]

The phone manufacturers should not be dicks.

Re:Or, alternatively

[Anonymous Coward]

On the contrary, the testers should just be bigger dicks. "We detected benchmark-specific optimizations in products #1, #2 and #3, so they all got zero points."

Re:Or, alternatively

[Nerdfest]

I'm a little confused about this. I was under the impression that what they were doing was ensuring that the clock speed was running at full, not slowed down for power saving, etc. From my point of view, that would just give a consistent reading of how fast the phone could run and wouldn't be considered 'cheating' unless they were boosting the clock speed up over normal running speeds. I assume some games and other applications also force the processor to full, but perhaps this is not the case. The whole thing seems to be a bit of an over-reaction, but perhaps I'm missing some information.

Re:Or, alternatively

[L4t3r4lu5]

Pfff. Car manufacturers tape up the air intakes and door seams on their cars to do fuel economy runs, just to eek out the every last 0.1mpg. Running your car like that for any reasonable period of time would wreck the engine pretty quick.

Benchmarks are about as useful as manufacturer spec sheets. Take both with a a few metric tonnes of salt.

Re:Or, alternatively

[NatasRevol]

You think there's fanbois here, imagine how torn up they'd get if they did that.

Re:Or, alternatively

[wonkey_monkey]

It makes comparing two phones harder because you won't know which of them is "cheating."

Re:Or, alternatively

[idontgno]

Pfff. Car manufacturers tape up the air intakes and door seams on their cars to do fuel economy runs, just to eek out the every last 0.1mpg. Running your car like that for any reasonable period of time would wreck the engine pretty quick.

Does any manufacturer report their own measured mileage? Within the US, the only number that matters is measured by the EPA, on a dynamometer.

Re:Or, alternatively

[gnasher719]

On the contrary, the testers should just be bigger dicks. "We detected benchmark-specific optimizations in products #1, #2 and #3, so they all got zero points."

That seems quite arbitrary. What about "To test the battery, we tested how many minutes the battery lasts while running benchmark X". The cheaters will get shorter battery life.

Re:Or, alternatively

[smash]

Pretty much.

Re:Or, alternatively

[ftobin]

Actually, manufacturers do report their own efficiency numbers, and the EPA spot-checks them.

http://business.time.com/2012/12/10/more-reason-to-be-skeptical-about-new-car-mpg-claims/

Re:Or, alternatively

[Pieroxy]

Why would they get shorter battery life?

They can't artificially boost performance without a drain on the battery.

Re:Or, alternatively

[Pieroxy]

Just buy an iPhone. Apple doesn't cheat apparently.

Re:Or, alternatively

[dj245]

Actually, manufacturers do report their own efficiency numbers, and the EPA spot-checks them.

http://business.time.com/2012/12/10/more-reason-to-be-skeptical-about-new-car-mpg-claims/

Not only that, but they are allowed to use the same numbers if the drivetrain and weight of the vehicle is the same as to a previously tested vehicle.

Re:Or, alternatively

[Anonymous Coward]

Your analogy does not apply.

They are artificially boosting performance by clocking their CPU faster than normal. This necessarily has negative effects on battery life (or else they'd be running their CPU normally at THAT speed... and if they boosted clock speed higher than that, it'd have lower-than-normal battery life too).

There's actually the additional factor of heat - serious PC overclockers generally spend money on aftermarket coolers. That could be an additional method of detecting this kind of cheating?

Re:Or, alternatively

[armanox]

I find that I get better MPG on I-95 than on I-495 or I-695.

Re:Or, alternatively

[LordLucless]

They don't need to. When you don't offer end users any choices, comparisons are meaningless.

Re: Or, alternatively

[armanox]

I am referring to the DC and Baltimore beltways (495 and 695, respectively)

Re:Or, alternatively

[Zero__Kelvin]

Since different phones from different manufacturers have different battery lives, what would that tell us? For the record, there is no over-clocking going on. They are CPU frequency throttling, which isn't the same thing. To overclock you need to use non-standard hardware. Also, battery life from one battery to the next, on the same phone running exactly the same software will differ. Your proposed test tell absolutely nothing from a benchmark/comparison standpoint.

Re:Or, alternatively

[kllrnohj]

(The device clocks aren't increasing, the priority of the processes is changing.)

This is why you don't comment on an article without reading it - you look stupid as a result.

The clocks *ARE* being increased, and the device is being kept awake artificially regardless of work load.

Re: Or, alternatively

[armanox]

Yes. Live just north of Baltimore. Previous job was a DOE contractor. Had a friend (female) that I used to see in Springfield, VA. Well versed in this area's traffic.

Re: Or, alternatively

[OneAhead]

Yes. Live just north of Baltimore. Previous job was a DOE contractor. Had a friend (female)

Hold it right there! You almost had me. How much of your part of this discussion am I supposed to believe if you're making such outrageous claims?!

Re: Or, alternatively

[armanox]

I said she was a friend, not a girlfriend....

Re: Or, alternatively

[GrahamJ]

Not only that but the 5S, at least, blows the doors off even most of the cheating scores.

"Pretty Much All of Them"

[Flavianoep]

...and some do it better than others.

The article notes that Apple doesn't do any of the frequency gaming stuff.

...for the obvious reason that Apple is the only manufacturer of iOS compatible devices.

Re:"Pretty Much All of Them"

[K. S. Kyosuke]

They could still cheat on the comparisons of new and old devices. But yeah, they really don't need to. It's not like this would significantly change the demand.

Re:"Pretty Much All of Them"

[Sockatume]

What difference does that make?

Re:"Pretty Much All of Them"

[Sockatume]

There are cross-platform benchmarks comparing performance across different phone OSes; it would be in Apple's best interests to cheat on those.

Re:"Pretty Much All of Them"

[Sockatume]

I was asking: what difference does it make that only Apple makes iOS devices, when those devices run the same benchmarks as anyone else?

You seem to have answered another question entirely.

Re:"Pretty Much All of Them"

[Bill, Shooter of Bul]

Well, to answer your question, The difference is that Android manufactores compete with other Android manufactorers. There is more pressure to compete within the android ecosystem, than the Apple. Right now, most people decide Apple or Android, then choose the particular device within the ecosystem. That's where they would turn to a benchmark, if they were to use one at all. IMHO, they're stupid to use one at all because they don't measure real world performance.

Re:"Pretty Much All of Them"

[Sockatume]

I dunno, for the areas where compute performance matters the most - i.e. games - I find that there's a lot of parity between the software ranges on Android and iOS, so there's a lot of reasons to compare across ecosystems. I'm an iOS owner now and I'm being swayed by the choice between a Nexus that's guaranteed to have top of the line specs and therefore a long gaming life ahead of it, or paying the same money for a rather crusty older-model iPhone.

Re:"Pretty Much All of Them"

[flimflammer]

No, not really. Comparing iOS and Android directly on performance is silly. They're two totally different ecosystems and hard performance numbers don't change much. That's like a typical user picking a Mac or Windows PC because one performs 5% better at random tasks, ignoring the fact that the offerings between each machine is radically different and pure performance numbers are only a tiny part of the whole picture.

Apple has no reason to cheat because they have no competition that merits the risk of cheating on. It might have been a different story had iOS hardware been available from multiple vendors.

Re:"Pretty Much All of Them"

[triffid_98]

No, Apple already has that part down to a science.

Step #1
Hype new iPhone
Step #2
Release new iPhone
Step #3
Immediately release new iOS update.
Step #4
Watch existing iPhone users complain after the iOS update cripples older models.
Step #5
Laugh maniacally after existing iPhone users stand in lines waiting for new uncrippled iPhones.
Step #6
PROFIT

Re:"Pretty Much All of Them"

[jonesy16]

Why would it be "silly"? If the point of benchmarking is to compare "like" things, and the same game is written for both ecosystems, why wouldn't the concerned consumer want to know that game X runs 20% faster on device Y, regardless of whether device Y is android or iOS? The only people concerned with these benchmarks must be looking for that 5% difference. So if that's what they want, then knowing that another platform gets them that 5% should be just as important as knowing the performance spread among devices of the same OS.

Re: "Pretty Much All of Them"

[sl149q]

IOS 7 runs on pretty much everything newer than the iPhone 3GS.

iPhone 4 runs IOS7 (2010 design).

iPhone 4S runs IOS7 (2011 design) and is still being sold.

What 2011 Android phone runs the most current version of Android and is STILL being sold?

Re: "Pretty Much All of Them"

[triffid_98]

The problem isn't that Apple refuses to update old iPhones, the problem is that they will update old iPhones and cause them not to work very well due to inferior/missing hardware.

In contrast, the 2011 Android phone isn't updated when a new device comes out, and... it continues to work exactly the way it did before the update.

Re:"Pretty Much All of Them"

[Sockatume]

And yet, we see iPhone-Android benchmark comparisons quite frequently.

Re:"Pretty Much All of Them"

[Plumpaquatsch]

What difference does that make?

The difference is that if one vendor of an Android device cheats, the Press and Apple fanboys can jump up and scream "OMFG ANDROID IZ CHEETZ THA SKORZ" even if every other Android vendor is "playing fair".

OMFG, are we now so far that even the headlines are too long to read?

Re: "Pretty Much All of Them"

[GrahamJ]

I'd agree for iOS 7 on the iPhone 4 but it runs really well on 4S+. On Android you're lucky to EVER get any update, even security patches.

And Apple

[Anonymous Coward]

With the exception of Motorola...

And Apple. Apple and Motorola/Google are the only two companies that don't boost their devices for benchmark tests. If you're going to give credit to one, please do be fair and give credit to the other.

I respect both of them for that level of integrity and I hope they stick to their guns and remain honest.

I may be an Apple fanboy (and I am) but I'm really looking forward to seeing what Motorola starts releasing in about a year once Google's able to, as they said, flush things out of the system and start releasing truly Google-designed products.

Re:And Apple

[Anonymous Coward]

Posting anonymously since I'm a motoroogle employee... you'll be disappointed. I certainly am. At this point, I expect google to shut us down or spin us off.

Re:And Apple

[R.Mo_Robert]

With the exception of Motorola...

And Apple. [...] If you're going to give credit to one, please do be fair and give credit to the other.

They did, and it's in the summary. You didn't even have to read TFA. Additionally, the headline narrows this down to Android OEMs, so that's why Apple was excluded from the discussion until the very end.

The article notes that Apple doesn't do any of the frequency gaming stuff.

Re:And Apple

[h4rr4r]

I fear Motorola is dead.

The Moto X was supposed to be a device with high amounts of post Google input. It is a high priced mediocre mid range device. They still cater to carriers unlike Apple. They need to compete with the 5S, One X and S4. Instead they are competing with last years product.

Moto needs to copy Apple in some things. Make 1 main device, sell the old one or a cheaper version as well. Max 2-3 devices. Refresh them once a year. This lets third parties make all kinds of doodads for the device. Put them on all carriers and update them at the same time. If need be get the FCC to lean on Verizon's C block requirements. Other than that, unlock the bootloaders and make donations of devices to the popular mod groups. Most users will never care, but the PR among the tech crowd would be huge and they influence other buyers.

Re:And Apple

[Dixie_Flatline]

It's not a mid-range device. It's only mid-range if you look at the spec sheet and nothing else. Its (non-gamed) benchmarks are actually pretty good for all this talk of 'mid-range'. They did the same thing Apple did and tried to balance out performance with battery life. They didn't put the biggest screen in it, and they have optimised silicon to listen for commands without keeping the CPU on all the time.

Specs aren't the war that anyone should be trying to win in the mobile space. That kind of thinking is why there are phones that only last half the day.

Re:And Apple

[Anonymous Coward]

I did read the article. I just don't understand why mentioning Apple was a footnote thought. The article is about how virtually every OEM inflates their scores, except Motorola and Apple. That's how it should be presented. Presenting it the way it was implants the thought that, as I quoted, "with the exception of Motorola..." and that's not accurate. It's "with the exception of Motorola and Apple..."

You know, had the summary been written "With the exception of Apple..." with a little footnote "oh, and Motorola doesn't do it as well" at the end, people would claim the summary was biased and complain so, sorry, expect the same when it's done the other way.

I may be an Apple fanboy, and I am, but I'm more interested in fair and unbiased coverage of news than anything else. Fair and unbiased means "with the exception of Motorola and Apple..."

Yes, yes. I must be new here what with wanting fair and unbiased Trust me, however, I'm not - I'm not logging in because I'm at work but I've been around Slashdot for many, many, many years. I know what the site is - I just wish it was better...

Re:And Apple

[JanneM]

I've heard a number of people I trust comment on how the Moto X just feels really good in the hand and how the screen and size is just right. So the phone is certainly not bad at all, and most of us don't actually buy the high-end phones any more than we buy the high-end cars or bikes. A really good mid-range phone is exactly what I want; I'm already making calf-eyes toward the coming Sony Z1 Mini.

What's hurting Motorola for me, personally, is that it's simply not on sale in most of the world, and seems unlikely to ever be. It's not just about being able to get it where I live, but having a phone designed from the start to be usable in all major regions of the world as I travel.

Re:And Apple

[king neckbeard]

Perhaps you should take a refresher course on reading comprehension. The first sentence mentions that Android OEMs were analyzed. Apple is not an Android OEM, so they were not included at that point.

Re:And Apple

[Sockatume]

It's funny that the number of iPhones those analysts claim Apple really sold, is exactly the same as the amount of iPhones they blind-ass-guessed would sell before the sales figures came out. It's almost like they were trying to defend their original estimates and therefore their reputations as analysts.

Re:And Apple

[gaiageek]

Posting anonymously since I'm a motoroogle employee... you'll be disappointed. I certainly am. At this point, I expect google to shut us down or spin us off.

Care to elaborate on why we'll be disappointed? I ask because this thread is focused on (perceived) performance of smartphones, and I know Motorola caught some criticism for not giving the Moto X better silicon to compete with other flagships, but personally, I think their tactic is pretty smart: focusing on functionality for the average user instead of performance capabilities which most people don't care about. I'd actually consider buying a Moto X if it weren't for the non-removable battery (deal breaker for me).

Re:And Apple

[h4rr4r]

It is a mid range device if you hold one. Go get one and tell me it is not a midrange device.

I am not suggesting chasing specs, I am suggesting building a premium device.

Re:And Apple

[Karlt1]

Apple is not the exception. They just fudge the numbers in a different way. Really, they're all bastards though and I don't trust them any further than I can throw them.

And a little deeper analysis will show....

a) Apple accounted for numbers the same way they always did.
b) Apple does not count phones as sold if they are still in Apple Stores inventories.
c) If Apple were channel stuffing, phones would readily be available at carriers.

The fact is that the "analyst" were wrong and are trying to cover their tails.

Do you really believe that there is not enough of a demand in the launch countries to sale 9 million phones in one weekend?

Do you think that Apple will be forced to take a write down (like MS and BlackBerry) for unsold inventory in the channel?

Re:And Apple

[Dixie_Flatline]

It can't be worse than the Samsung's I've handled. And the HTC One is premium and beautiful, but that doesn't seem to be working out.

I will accept that you personally are calling it a mid-range device on the build quality alone, but so much of the commentary has focussed on how it has 'mid-range specs' and is overpriced for the screen/CPU/GPU that's in there.

There are trade-offs to be had. iPhones are top-tier devices, but they've got smaller screens than the top-tier Android phones. The S4 is a plasticky top-tier device with a big screen and a whole bunch of useless software features. Given these things, I'm not convinced the Moto X (and I admit, I haven't sat down and played with one) deserves its mid-tier accusations.

Re:And Apple

[UnknowingFool]

So Apple is reporting sales after an opening weekend like they always have done in the same manner as everyone else? And the alternative would be? Unless Apple (and any other manufacturer) can get real time sales figures from their retailers, they can only report shipped. End of quarter numbers are different. Apple does report actual sales as there is enough time to get these numbers from their retailers.

Re:And Apple

[h4rr4r]

The One is a really nice device. If they would kill SenseUI already it would be the best Android device on the market.

Not only build quality but some specs matter, think less about things like resolution and more about how accurate colors are or the white balance on the device is. Pentile vs RGB IPS is a good example of this sort of thing.

The S4 might have a lot of plastic, but it makes it rather durable. They should have had the S4 Active as the only model. Instead they made that AT&T exclusive. It is not only less plasticy but also has some water resistance.

iPhones have gotten bigger and I think they might do so again. A slightly larger device has advantages, especially in markets outside the West. Not only due to income meaning some folks will want a large phones vs buying a smaller one and a tablet but because of the belief that a large phone makes your face look smaller. Which is what is selling many of these huge devices in Asia.

Re:And Apple

[aristotle-dude]

So you have a bunch of brits complaining about the iPhone 3G not being fast on their slow networks? The retina display, when held at a normal viewing distance by the average person has a pixel density that makes it appear as those there are no pixels composing the glyph of a font. That is a statement of fact.

I'm not going to bother to follow the rest of your links sorry.

Re:And Apple

[aarroneous]

Apple reports sales after their opening weekend like they've always done, however, it is in a manner DIFFERENT from everyone else. Everyone else indeed counts total shipped, whereas Apple has always reported total sold. Apple's largest retailer happens to be... Apple. I think they have a very good handle of what they sold themselves.

Re:And Apple

[UnknowingFool]

In years previous, shipped == sold because the new iPhone was sold out everywhere. This year the 5S appeared to be sold out but the 5C is not.

Re:And Apple

[Trillian_1138]

Just picked up a Moto X to replace my aging Galaxy Nexus. I'm on Verizon, which makes me skeptical I'll see another Nexus phone anytime soon. I like the near-stock Moto X setup, and the little tweaks (active notifications, touchless control, etc) are pretty fun. It's a pretty great phone, and I hope it helps boost Motorola in the Android world. I do think they priced it too high (although I still bought it, so take that as you will..) but I highly recommend the phone to anyone who plays the occasional, not to graphic-intensive games, wants their phone to be quick and responsive, likes stock Android, etc. It's an awesome phone.

Re:And Apple

[psithurism]

Good point, the article should have congratulated all the companies that were not caught cheating on android performance tests:

Microsoft,
Walmart,
Berkshire-Hathaway,
Lockheed Martin,
that "self employed" beggar across the street,
Burger King,
Ford Motor Company, ...

That could take awhile actually, lets just congratulate the company you're a fan-boy of then shall we?

Re:And Apple

[TrueSpeed]

Posting anonymously since I'm a motoroogle employee... you'll be disappointed. I certainly am. At this point, I expect google to shut us down or spin us off.

I'm sure if you trace his IP (with a GUI written in VB, of course) you'll find that it originated from a basement.

Re:And Apple

[samwichse]

Ask and you shall receive:
https://play.google.com/store/devices/details?id=htc_one

Re:And Apple

[h4rr4r]

If it worked properly on T-mobile that would be better.
I am still considering it though.

Re:Probably wont get better

[h4rr4r]

They started that game long before you could get 1 GB hard drives.

CPU Benchmark Shenanigans ROI??

[Anonymous Coward]

'The hilarious part of all of this is we’re still talking about small gains in performance.'

The even more hilarious part is that OEMs are going to the trouble to do this when CPU benchmark scores are a very small factor in the decision of most consumers to buy these phones. I doubt that the ROI is higher than say, oh, improving the user experience of the GUI or call QoS.

Cheating or just gaming the benchmark?

[kaalon]

Not convinced that it is cheating. For cheating I would have expected to see some evidence of not running a part of the benchmark to get a higher score. But here it looks just to be the OEMs trying to modify the power consumption to game the score. Heh, with that said I guess it is a little swarmy that all of them took the time to change from the default settings just for a bump in the ratings.

Re:Cheating or just gaming the benchmark?

[Sockatume]

I'm not sure that your argument makes any sense.

"You cheated. You copied the answer key."

"I did the whole test, though!"

Re:Cheating or just gaming the benchmark?

[DJRumpy]

You do realize that this is a battery operated device, and that such 'tweaking' dramatically impacts that battery life. Couple that with them not reporting the actual battery life while running the GPU/CPU coverclocked, and you are essentially lying out of you eye teeth.

Re:Cheating or just gaming the benchmark?

[zieroh]

The key fact is that none of the other applications get to run under those conditions (though Samsung gets squishy on this point) and so the cheated benchmark ends up representing an unrealistic performance level.

Re:Cheating or just gaming the benchmark?

[kaalon]

Thanks, I agree with that. Cheating it is.

Re:Cheating or just gaming the benchmark?

[kaalon]

Yeah that is the swarmy part. I agree cheating it is.

Re:Cheating or just gaming the benchmark?

[dj245]

You do realize that this is a battery operated device, and that such 'tweaking' dramatically impacts that battery life. Couple that with them not reporting the actual battery life while running the GPU/CPU coverclocked, and you are essentially lying out of you eye teeth.

This may not be as straightforward as you think- when batteries are cold, they have a different "extractable" capacity compared to when they are hot. Running the CPU/GPU at full tilt is going to warm up the battery. It is kind of a guess as to how much is actually left in there, which could easily explain this.

Re:Probably wont get better

[serviscope_minor]

Was it really as many gigabytes as it was advertised?

Yep.

You're not confusing gigabytes with gibibytes are you?

The only people who ever used powers of 1024 were RAM manufacturers since it makes sense there.

Do you also complain that 100mbit ethernet is not 100*1024*1024 bits per second?

Hilarious...

[Savage-Rabbit]

The hilarious part of all of this is we’re still talking about small gains in performance.

The hilarious part of all this is that most people really don't give a rat's as about performance when selecting a phone or even a tablet. The criteria are things like: how does it handle? How intuitive is the UI? Can I watch my favorite online video feeds on this thing? Are any buttons in annoying palaces? What's the price? Does this thing have software to view and edit MS Office files I get sent by mail? The only performance tests these smartphone and tablet things usually get is playing around with a display example in the shop and seeing if the UI is nice and snappy. Nobody excepts tech nerds gives a rats ass that a Samsun Galaxy 4 get a few more FPS in Modern Combat than an iPhone 5.

Re:Hilarious...

[Sockatume]

Nobody cares about the benchmarks quantitatively, but when you're putting down something like $700 on a computing device getting something that's "the fastest" is a big driver of decisions. I'm sure that guaranteeing a bunch of "Samsung Galaxy blah takes performance crown from Apple iPhone whatevs" stories in the tech press on the basis of a 0.5% difference in the Zootybench score is exactly why Samsung does this crap.

Re:Hilarious...

[coolsnowmen]

I might not care as much about performance as I care about battery life, and a rating of that is performance/Watt.

Re:Hilarious...

[tompaulco]

The hilarious part of all this is that most people really don't give a rat's as about performance when selecting a phone or even a tablet. The criteria are things like: how does it handle? How intuitive is the UI? Can I watch my favorite online video feeds on this thing? Are any buttons in annoying palaces? What's the price? Does this thing have software to view and edit MS Office files I get sent by mail? The only performance tests these smartphone and tablet things usually get is playing around with a display example in the shop and seeing if the UI is nice and snappy. Nobody excepts tech nerds gives a rats ass that a Samsun Galaxy 4 get a few more FPS in Modern Combat than an iPhone 5.

Don't forget "Does it start with an 'i'". For some people that is the strongest factor. And it could be a plus or a minus factor.

Re:Hilarious...

[the_humeister]

Here are some tests I've done on my devices compiling Povray 3.6 (single threaded) and rendering the benchmark scene. I was rather surprised that my new Nexus 4 (Qualcomm S4) is slower per core than my SGS II (Samsung Exynos 4). Yet the Nexus 4 doesn't feel any slower and is my main phone now due to the larger and higher resolution screen.

Athlon II x4 (2.8 GHz): 179.82 pps ; 64.22 pps/GHz
Exynos 5 (1.7 GHz): 77.36 pps ; 45.51 pps/GHz (-mfloat-abi=hard -mcpu=cortex-a9 -mthumb -mthumb-interwork)
PowerPC 750 (700 MHz): 20.47 pps ; 29.25 pps/GHz

Exynos 4210 (1.2 GHz): 29.90 pps ; 24.91 pps/GHz (-mfloat-abi=hard)
Pentium 4m (1.5 GHz): 36.24 pps ; 24.16 pps/GHz
S4 Pro APQ8064 (1.5 GHz): 31.81 pps ; 21.20 pps/GHz (-mfoat-abi=hard -mcpu=cortex-a15)
Atom N270 (1.6 GHz): 28.96 pps ; 18.10 pps/GHz

Unfortunately can't post all the data I have because this site thinks there're too many 'junk' characters already...

Re:Hilarious...

[TheSkepticalOptimist]

Naw, I would argue that having the fastest phone was only a deciding factor when Apple started boasting about their new 64bit CPU in the iPhone 5s. Prior to this month the average consumer would not even know or care if their phone had 1, 2 or 4 cores or how many goggleflops it was able to perform.

Apple decided the only way they can differentiate their iPhone 5s amidst all the comments that they are no longer innovative is to create a competitive market based on useless CPU performance numbers, just like what Apple did with Retina displays. Before Retina, nobody cared about pixel density. Before A7, nobody cared about CPU performance or its bittyness. Before the iPhone 5s camera, nobody cared about the size of the CCD pixel on their phone camera.

Apple creates these vapid competitive advantages and then the consumers latch on to them and, unfortunately, the competitors feel the need to have to follow suit.

Re:Hilarious...

[Savage-Rabbit]

Apple decided the only way they can differentiate their iPhone 5s amidst all the comments that they are no longer innovative is to create a competitive market based on useless CPU performance numbers, just like what Apple did with Retina displays. Before Retina, nobody cared about pixel density. Before A7, nobody cared about CPU performance or its bittyness. Before the iPhone 5s camera, nobody cared about the size of the CCD pixel on their phone camera.

Dude, It's not that long ago that articles like this slashvertisment were plastered all over the web accompanied by comments filled with enthusiastic boasts by hoards of Android fans detailing how iPhone performance sucks ass. Even if blisteringly fast benchmark performance is pretty low down on the list of most people out to buy a smartphone I still can't fault Apple for trying to put a sock in the collective mouth of the Android community. There is a certain personal satisfaction to be had from making the choir of hard core Android fans shut up about benchmarks until Samsung comes up with a still faster device (hopefully free of benchmark cheating this time) even if the customers will probably hardly notice this stupid pissing contest.

Re:Hilarious...

[Savage-Rabbit]

I might not care as much about performance as I care about battery life, and a rating of that is performance/Watt.

Ditto, that's probably the only part of a benchmark test that I really care about.

Re:Hilarious...

[samwichse]

My guess would be because the GPU is significantly better in the Nexus 4, along with twice the RAM, and I heard that Google went to some trouble to source the fastest flash memory for it.

Re:I got a bridge to sell, cheap

[Sockatume]

Yes, I'm sure it's all lies where it conflicts with $priorbeliefs.

Re:Not cheating

[Sockatume]

The performance quoted simply is not available to apps that are not on a whitelist of benchmark applications. It literally does not represent any part of the phone's non-benchmarking performance.

Re:Probably wont get better

[stewsters]

Yes. Yes I do.

The best part

[aitikin]

Is that the table is titled "I Can't Believe I Have to Make This Table". Made me smile.

If you measure it, it gets better

[tompaulco]

If you measure it, it gets better. Nobody ever stops to think why. They mostly just think they are good managers. But in fact, if you create a measuring tool to measure qualities of a device, the manufacturers will work to make that measurement better. If you make a measurement to determine how your employees are performing, they will perform better according to that measurement. That's just the way it works.
You can't measure everything, so you're best bet is to try to keep the measurement methods secret and change them frequently. Unless, of course, your measurements are intended to improve a particular area, then by all means, measure on.

Re:Probably wont get better

[KiloByte]

Do you also complain that 100mbit ethernet is not 100*1024*1024 bits per second?

My ISP, as sucky as it is, doesn't cheat here. Heck, it's only disk manufacturers who routinely do.

You don't change an unit in wide use for more than six decades just because some committee feels they don't get enough attention.

Re:Probably wont get better

[Sockatume]

The discrepancy is deliberate. For a long, long time drive capacity was quoted in the same units that the computer used for storage: binary SI prefixes, not decimal ones. The change to "1 megabyte = 1 million bytes" didn't set in until the 2000s.

Kudos to Apple for making their specs and their OS use consistent units, but it's still a marketing bullshit decision.

Re:I got a bridge to sell, cheap

[UnknowingFool]

This is somewhat reminiscent of the MHz wars. The OEMs use all the same OS and they feel they have to distinguish themselves in some way. Except for Samsung, Apple, and Motorola, these OEMs buy their processors from someone else.

Wrong, they are boosting clock speed above normal

[SuperKendall]

I was under the impression that what they were doing was ensuring that the clock speed was running at full, not slowed down for power saving, etc.

No. They are running at a clock speed that no real application will see under any circumstance, either the GPU or CPU cock increased.

It is within what the parts are rated for but not what the device was built to run at normally.

I assume some games and other applications also force the processor to full

There is no way to build a game on Android that can run at the speed the benchmarks are getting run at on each of the devices "cheating".

Re:Wrong, they are boosting clock speed above norm

[drerwk]

There is no way to build a game on Android that can run at the speed the benchmarks are getting run at on each of the devices "cheating".

What happens if you name your game to match the benchmark that the phone is looking for?

Re:Wrong, they are boosting clock speed above norm

[Nerdfest]

Thanks, as I said, I was under the impression they were just ensuring it was not reduced. If there's no way an app could get the same speed then it is most definitely cheating. I'm surprised they don't overclock. If you're going to cheat you might as well do it right.

Re:Probably wont get better

[zieroh]

The only people who ever used powers of 1024 were RAM manufacturers since it makes sense there.

Well, them and every software engineer on the planet.

(Yes, I think in K. And yes, I will continue to define 1k as 1024, no matter how many pedantic monkeys object)

Re:Wrong, they are boosting clock speed above norm

[Anonymous Coward]

No, he was right. The phones CAN and DO reach that clock speed. Read the AnandTech article. The graph of CPU speed shows it quite clearly.

Re:Probably wont get better

[UnknownSoldier]

Considering that hard disk sectors have ALWAYS been a power-of-2 even though the encoding of them may not, using a power of 10 was pure marketing bullshit.

> You're not confusing gigabytes with gibibytes are you?
That stupid term was invented years after the fact of common usage. You can piss off with that retarded term.

> Do you also complain that 100mbit ethernet is not 100*1024*1024 bits per second?
That includes meta-data such as parity, error detecting, etc. There is NO confusion because it has always been "How many bits can I send over the wire reliably knowing full well that I'm going to get LESS then N/8 bytes out."

Re:Not cheating

[MachineShedFred]

So it's not cheating to detect when a specific benchmark app is running, and then clock the device to a setting that in no way will ever be set when not running those specific benchmark apps.

Okay. I guess it's not cheating to pump yourself full of steroids right before the start of a professional baseball season either.

Meh.

[DarthVain]

This has been going on since benchmarks were a thing. So a long time.

GPU in particular had a bad rep for this. However they actually got the benchmark software altered!

Simply checking for a process name and OC, isn't exactly all that sneaky. Maybe a bit unethical, but still. Whoever is running the benchmark could easily check for that. Of course it depends on how closed down the OS is for inspection also.

Re:Meh.

[SuiteSisterMary]

quack3.exe

Re:Wrong, they are boosting clock speed above norm

[ifiwereasculptor]

either the GPU or CPU cock increased.

Whoa, can cell phones do that now? I hold these things to my ear, for Christ's sake!

Mechanical Turk Benchmarks

[swb]

Instead of automated benchmarks of hardware, why not real world human benchmarking where a group of people is given a set of tasks to do on a given cell phone platform and see who can do them faster?

Automated technical benchmarks make sense when the workloads more or less approximate the benchmark -- video gaming, 3D modeling, disk throughput, etc.

But unless I'm living totally in the dark, most people aren't buying cell phones oriented towards single-task performance (eg, gaming). They get used for many tasks and the fact that I can run some obscure CPU task faster than some other model doesn't tell me if it makes it faster to open an app, etc.

Apple does not need to lie.

[codepunk]

Native Code Execution, it runs as fast as it is going to run.

Re:Wrong, they are boosting clock speed above norm

[Dogtanian]

either the GPU or CPU cock increased.

Whoa, can cell phones do that now? I hold these things to my ear, for Christ's sake!

Yeah, they'll f**k your brains out... literally.

Re:Wrong, they are boosting clock speed above norm

[DMUTPeregrine]

Overclocking rooted Android phones has been around for some time. SetCPU is a popular app for it, and many custom roms have a built-in way to control clock speeds and governors.

Re:Wrong, they are boosting clock speed above norm

[DMUTPeregrine]

aaand I missed the typo.

That said, given the internet I'm sure someone has "overcocked" their phone, and stuck it in their ear. Probably in Japan.

Re:Wrong, they are boosting clock speed above norm

[antdude]

Mmm, cooked roosters. I wonder how they taste compared to female chickens.

Nothing new here, move along

[hazydave]

GPU companies have been cheating on benchmarks for decades. Back in the PCI and AGP days, it was just pretty much expected that graphics cards would include a hack to lock themselves on the shared bus for as long as they felt they needed. Sure, this wasn't benchmark specific, but it was also destructive -- it killed the realtime performance of your PC. Specifically in those days, audio workstation software was broken by these hacks... leading a bunch of counter-hacks necessary to defeat these (and sure, the occasional GPU company savvy enough to offer a non-broken version of the driver online).

And then there was the Intel C compiler.. the one that basically had a "detect Dhrystone" function. And over the years, a whole series of cheats that either made AMD look bad or Intel amazingly good... enough to be much of the meat of AMD's suit against Intel in 2005... which lead to a court order for Intel to stop their compiler producing code that intentionally ran slower on non-Intel chips. And more recently with Intel's AnTuTu cheat, making Intel smartphones look magically better than AMD.. but only on that one benchmark.

And of course, nVidia. nVidia's been cheating at benchmarks for years... though at least some of that came in response to AMD/ATi getting caught cheaping on Quake 3 performance. And recently, too... earlier this year, they had a driver that detected 3DMark (and perhaps other benchmarks) and cheated, by switching to custom PHYSX code rather than running the DLL everyone else runs. And the various cheats on benchmarks for Tegra 2 and Tegra 3 SOCs.

I mean, this is one reason we speak of lies, damn lies, and benchmarks. Everybody cheats at this stuff. Apple used to cheat in the latter days of the PowerPC, even on published SPECmarks, as their stuff kept falling behind Intel.