Slashdot Mirror
How Many Android OEMs Cheat Benchmark Scores? Pretty Much All of Them
An anonymous reader writes "After Samsung got caught out cheating on benchmarks (Note 3, Galaxy S4) AnandTech has done a detailed analysis of the state of benchmark cheating amongst Android OEMs. With the exception of Motorola, literally every single OEM they've looked at ships (or has shipped) at least one device that does benchmark-specific CPU optimizations. AnandTech also thinks it will get worse before it gets better. 'The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0 - 5%, and somewhere south of 10% on our GPU benchmarks as far as we can tell. I can't stress enough that it would be far less painful for the OEMs to just stop this nonsense and instead demand better performance/power efficiency from their silicon vendors.' The article notes that Apple doesn't do any of the frequency gaming stuff."
The benchmark software should randomize the process name on launch
"When life gives you lemons, don't make lemonade. Make life take the lemons back!" -- Cave Johnson
The phone manufacturers should not be dicks.
With the exception of Motorola...
And Apple. Apple and Motorola/Google are the only two companies that don't boost their devices for benchmark tests. If you're going to give credit to one, please do be fair and give credit to the other.
I respect both of them for that level of integrity and I hope they stick to their guns and remain honest.
I may be an Apple fanboy (and I am) but I'm really looking forward to seeing what Motorola starts releasing in about a year once Google's able to, as they said, flush things out of the system and start releasing truly Google-designed products.
They started that game long before you could get 1 GB hard drives.
'The hilarious part of all of this is we’re still talking about small gains in performance.'
The even more hilarious part is that OEMs are going to the trouble to do this when CPU benchmark scores are a very small factor in the decision of most consumers to buy these phones. I doubt that the ROI is higher than say, oh, improving the user experience of the GUI or call QoS.
The hilarious part of all of this is we’re still talking about small gains in performance.
The hilarious part of all this is that most people really don't give a rat's as about performance when selecting a phone or even a tablet. The criteria are things like: how does it handle? How intuitive is the UI? Can I watch my favorite online video feeds on this thing? Are any buttons in annoying palaces? What's the price? Does this thing have software to view and edit MS Office files I get sent by mail? The only performance tests these smartphone and tablet things usually get is playing around with a display example in the shop and seeing if the UI is nice and snappy. Nobody excepts tech nerds gives a rats ass that a Samsun Galaxy 4 get a few more FPS in Modern Combat than an iPhone 5.
Only to idiots, are orders laws.
-- Henning von Tresckow
Is that the table is titled "I Can't Believe I Have to Make This Table". Made me smile.
"Don't meddle in the affairs of a patent dragon, for thou art tasty and good with ketchup." ~ohcrapitssteve
If you measure it, it gets better. Nobody ever stops to think why. They mostly just think they are good managers. But in fact, if you create a measuring tool to measure qualities of a device, the manufacturers will work to make that measurement better. If you make a measurement to determine how your employees are performing, they will perform better according to that measurement. That's just the way it works.
You can't measure everything, so you're best bet is to try to keep the measurement methods secret and change them frequently. Unless, of course, your measurements are intended to improve a particular area, then by all means, measure on.
If you are not allowed to question your government then the government has answered your question.
I was under the impression that what they were doing was ensuring that the clock speed was running at full, not slowed down for power saving, etc.
No. They are running at a clock speed that no real application will see under any circumstance, either the GPU or CPU cock increased.
It is within what the parts are rated for but not what the device was built to run at normally.
I assume some games and other applications also force the processor to full
There is no way to build a game on Android that can run at the speed the benchmarks are getting run at on each of the devices "cheating".
"There is more worth loving than we have strength to love." - Brian Jay Stanley
You do realize that this is a battery operated device, and that such 'tweaking' dramatically impacts that battery life. Couple that with them not reporting the actual battery life while running the GPU/CPU coverclocked, and you are essentially lying out of you eye teeth.
No, he was right. The phones CAN and DO reach that clock speed. Read the AnandTech article. The graph of CPU speed shows it quite clearly.
No, not really. Comparing iOS and Android directly on performance is silly. They're two totally different ecosystems and hard performance numbers don't change much. That's like a typical user picking a Mac or Windows PC because one performs 5% better at random tasks, ignoring the fact that the offerings between each machine is radically different and pure performance numbers are only a tiny part of the whole picture.
Apple has no reason to cheat because they have no competition that merits the risk of cheating on. It might have been a different story had iOS hardware been available from multiple vendors.
either the GPU or CPU cock increased.
Whoa, can cell phones do that now? I hold these things to my ear, for Christ's sake!
Instead of automated benchmarks of hardware, why not real world human benchmarking where a group of people is given a set of tasks to do on a given cell phone platform and see who can do them faster?
Automated technical benchmarks make sense when the workloads more or less approximate the benchmark -- video gaming, 3D modeling, disk throughput, etc.
But unless I'm living totally in the dark, most people aren't buying cell phones oriented towards single-task performance (eg, gaming). They get used for many tasks and the fact that I can run some obscure CPU task faster than some other model doesn't tell me if it makes it faster to open an app, etc.
No, Apple already has that part down to a science.
Step #1
Hype new iPhone
Step #2
Release new iPhone
Step #3
Immediately release new iOS update.
Step #4
Watch existing iPhone users complain after the iOS update cripples older models.
Step #5
Laugh maniacally after existing iPhone users stand in lines waiting for new uncrippled iPhones.
Step #6
PROFIT
You do realize that this is a battery operated device, and that such 'tweaking' dramatically impacts that battery life. Couple that with them not reporting the actual battery life while running the GPU/CPU coverclocked, and you are essentially lying out of you eye teeth.
This may not be as straightforward as you think- when batteries are cold, they have a different "extractable" capacity compared to when they are hot. Running the CPU/GPU at full tilt is going to warm up the battery. It is kind of a guess as to how much is actually left in there, which could easily explain this.
Even those who arrange and design shrubberies are under considerable economic stress at this period in history.