Slashdot Mirror
Samsung Fudging Benchmarks Again On Galaxy Note 3
tlhIngan writes "A few months ago, Samsung was caught gaming benchmarks on the Galaxy S4 (International version). They would lock the GPU at a higher-than-normal frequency when certain applications were run, including many popular Android benchmarking programs. These had the expected result of boosting the performance numbers. This time, the Galaxy Note 3 was caught doing the same thing, boosting CPU scores by 20% over the otherwise identical LG G2 (which uses the same SoC at the same clock). Samsung defends these claims by saying the other apps make use of such functionality, but Ars reversed-engineered the relevant code and discovered it applied only to benchmark applications. Even more damning was that the Note 3 was still faster than the G2 when run using 'stealth' (basically renamed) versions of the benchmarking apps which did not get the boost."
If Apple did this, people would be up in arms!
The real Sig captains the Northwestern. This one captains
If someone is surprised Humans are willing to cheat, rip off, etc to get ahead... well you haven't really been paying attention.
Fixed that for you.
I wouldn't want people to unfairly categorize you as a racist moron.
#DeleteChrome
And 95% of consumers could care less as long as the screen looks nice and the battery lasts more than 2 hours.
There's lies, damned lies, statistics, and vendor performance numbers.
I'm a little disappointed that there isn't actually any penalties for fudging your benchmarks -- it's blatantly lying to consumers about your product.
And to me, that seems like it's bordering on fraud.
Lost at C:>. Found at C.
Wait, what? How is that more damning? It sounds like that means the benchmark is faster even without cheating, which means that they've changed the kernel scheduler/idle timers/clock stepping in a way that, at least for the sorts of tests performed in the benchmark, improves performance—presumably because their case design and/or battery capacity is better, allowing them to get away with less processor throttling. That sounds like it is almost inarguably a good thing. And that's coming from somebody who has dealt with several of Samsung's products and hated almost all of them. What's with the hate?
Unless, of course, they're being too aggressive about keeping the clock speed high, in which case you might argue that their battery life isn't what it should be... but that's pretty subjective.
Check out my sci-fi/humor trilogy at PatriotsBooks.
That's fair comment on the original post, but let's narrow it down a bit...
"If someone is surprised that a manufacturer with a track-record of fudging benchmarks is willing to cheat, rip off, etc to get ahead... well you haven't really been paying attention"
Not all humans are morally and ethically bankrupt. Samsung (as a corporate entity) is though.
Simon
Physicists get Hadrons!
The benchmarking itself seems to be flawed. Samsung wants to benchmark the devices at their full capacity, to see what they are capable of (the higher setting is reached in normal use of some apps anyway). The testers would probably like to do real world comparison tests (and not rely just on numbers). I don't see Samsung doing anything wrong here, even though the benchmarking apps are specifically chosen.
Are you implying that us android users wouldn't be "up in arms"?
No implication is needed, we can see quite plainly there is very little outcry over this, just as there wasn't before. Android users simply accept this is the way things are, in a way they do not with any Apple problem whatsoever.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Not sure how this is "damning". I'd have thought it would prove the principle that the optimizations aren't app specific.
What am I missing?
It's not app-specific, it's app *name* specific. It's analogous to the Quake/Quack benchmark scandal years (OMG, more than a decade...time flies) ago. Samsung wrote this boosting protocol to enable itself when running benchmarks and *only* when running benchmarks. There is no legitimate way to invoke it, so no user will ever see the benefit of it when running any app *other than* the benchmark itself.
For the inevitable car analogy: you take a Samsung car for a test drive, and when you floor it you feel 200hp worth of acceleration. Since the car is identical in almost every other aspect to competing HTC cars and Motorola cars (same price, similar trim, same engine) but they only make you feel about 150hp worth of acceleration, you opt for the Samsung car. Only when you drive it off the lot, you only feel 150hp worth of acceleration. You take it back to the dealer thinking something's wrong only to be informed the car will only give you 200hp when in "test drive by prospective customer" mode, and now that you've bought the car you're no longer in that category and cannot invoke it.
In the end they will lay their freedom at our feet and say to us, Make us your slaves, but feed us. - Fyodor Dostoyevsky
It's hard to find info on it, and it was at least 8 years ago. So you're saying that Samsung's benchmark juicing today is like Apple choosing the Intel compiler with extra options back in the day?
This is what Samsung does, in pseudocode:
if app.name == benchmark speed up
This is what Apple did on its benchmarks:
# for G5
cc test.c -altivec
# for x86
gcc test.c
If you can't tell the difference between the two, you're either stupid, or Samsung.
They're changing the operating profile of their phone to inflate the benchmark results. In real life you would never be able to achieve numbers like that on a phone with a halfway decent battery life, even on applications that behave similarly to the benchmark. No normal use would achieve numbers found in the benchmark, they can't because they involve running the phone super hot and burning through the battery in no time, even when idle.
If the hack were to go to full power when plugged into the wall, then I could maybe see a case for this being legitimate, since it would mean you could theoretically achieve the same results by simply playing your game or whatnot while plugged in, but because they're only switching on full power mode for a handful of specific benchmark applications there is just no excuse.
I read the internet for the articles.
Battery life still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58409.png
You are comparing a phone with a 4 inch screen, with a "phone" that has a 5.7 inch screen. You can't compare battery life when the screen is what uses up most of the power. If you want a huge screen you have to compromise on battery life (and many other things - seriously, the note is ridiculously big to use as an every-day phone).
Browser speed still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58440.png
I don't suppose Samsung can do much about that. It is quite possible that with the same CPU, an Android would still be slower than an iOS device. Sure, Google has made a fast Java VM, but it still is a Java VM, right? For example, I had a Nokia N9 running Meego/Maemo. It could run circles around Android phones with the same CPU.
Graphics performance still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58425.png
Ehm, this result (to which you cleverly linked directly - hiding the context) is ran in native resolution. The Note has almost 3x the iphone's resolution, so it would be pretty strange to come on top in fps. But in all the other GPU benchmarks which are ran at 1080p it does come on top of the iphone.
But in any case I personally prefer a phone that has a good battery life, it can fit in my hand and lets me do whatever I want with it. So that rules out the note and the iphone ;)
Violence is the last refuge of the incompetent. Polar Scope Align for iOS