Samsung Fudging Benchmarks Again On Galaxy Note 3

← Back to Stories (view on slashdot.org)

Samsung Fudging Benchmarks Again On Galaxy Note 3

Posted by timothy on Tuesday October 1, 2013 @05:28AM from the selective-activation-sounds-nicer dept.

tlhIngan writes "A few months ago, Samsung was caught gaming benchmarks on the Galaxy S4 (International version). They would lock the GPU at a higher-than-normal frequency when certain applications were run, including many popular Android benchmarking programs. These had the expected result of boosting the performance numbers. This time, the Galaxy Note 3 was caught doing the same thing, boosting CPU scores by 20% over the otherwise identical LG G2 (which uses the same SoC at the same clock). Samsung defends these claims by saying the other apps make use of such functionality, but Ars reversed-engineered the relevant code and discovered it applied only to benchmark applications. Even more damning was that the Note 3 was still faster than the G2 when run using 'stealth' (basically renamed) versions of the benchmarking apps which did not get the boost."

41 of 258 comments (clear)

Min score:

Reason:

Sort:

If this was Apple... by thestudio_bob · 2013-10-01 05:30 · Score: 2, Insightful

If Apple did this, people would be up in arms!

--
The real Sig captains the Northwestern. This one captains /.
1. Re:If this was Apple... by Bill_the_Engineer · 2013-10-01 05:34 · Score: 5, Insightful
  
  Are you implying that us android users wouldn't be "up in arms"? It affects us more. While apple sells two models of the iPhone, we have a multitude of android phones to choose from. Samsung messing with the benchmark has the potential to cause a customer to chose samsung over HTC, Google Nexus, or Motorola phone.
  
  --
  These comments are my own and do not necessarily reflect the views or opinions of my employer or colleagues...
2. Re:If this was Apple... by Anonymous Coward · 2013-10-01 05:38 · Score: 3, Insightful
  
  Why would you even consider synthetic benchmarks when choosing a phone? Planning on trying some bitcoin mining on the go?
3. Re:If this was Apple... by dgatwood · 2013-10-01 05:46 · Score: 3, Insightful
  
  Samsung are still very much the good guys, and their hardware is still clearly the best.
  You've obviously never used their Blu-Ray players.
  
  --
  Check out my sci-fi/humor trilogy at PatriotsBooks.
4. Re:If this was Apple... by Anonymous Coward · 2013-10-01 05:46 · Score: 2, Funny
  
  If Apple did this, people would be up in arms!
  Both Apple and Android use ARM CPUs.
  Now, if it had been happening on the Surface tablets (non-RT, of course) people would be up in Atoms.
5. Re:If this was Apple... by DickBreath · 2013-10-01 05:48 · Score: 5, Funny
  
  > If Apple did this, people would be up in arms!
  
  If Apple did this, they would be suing Samsung for copying.
  
  --
  
  I'll see your senator, and I'll raise you two judges.
6. Re:If this was Apple... by waddgodd · 2013-10-01 05:49 · Score: 4, Funny
  
  Wrong, they DID care about benchmarking once, while Macs were still Power PC based, then they got caught fudging the benchmarks approximately like Samsung did. They suddenly decided that benchmarks didn't matter soon thereafter. They ought to sue Samsung again, because business methods are patentable...
  
  --
  Just because you're paranoid doesn't mean they aren't out to get you
7. Re:If this was Apple... by Space+cowboy · 2013-10-01 05:52 · Score: 4, Informative
  
  Citation required, because all I can find is: http://www.theregister.co.uk/2003/06/25/apple_denies_fiddling_g5_xeon/ ... which seems to be refuting the claim...
  Simon.
  
  --
  Physicists get Hadrons!
8. Re:If this was Apple... by Old97 · 2013-10-01 05:53 · Score: 5, Informative
  
  My you are full of unsupported assertions today. "Most every tech company has been caught". Really? Any evidence? "Samsung ... the good guys" You must be kidding. They copy and they clone. Apple does gold so Samsung does gold. Apple sells a 64-bit phone with a 64 bit operating system and conversion tools to take advantage of it. Samsung announces that they'll be building 64 bit phones too, one day. Of course unless Android is converted to 64 bit that will be pointless and there is nothing from Google indicating that is going to happen any time soon. Chrome OS seems to be more important to them these days anyway. And finally, "their hardware is still clearly the best". Evidence or just your opinion based on your limited experience? I've tried Apple, Nokia, HTC and Samsung and liked Samsung the least hardware wise. Consumer Reports and other customer satisfaction survey's I've seen don't rate Samsung all that highly. Apple leads the pack in every survey I've seen.
  
  --
  Very often, people confuse simple with simplistic. The nuance is lost on most. - Clement Mok
9. Re:If this was Apple... by mspohr · 2013-10-01 06:01 · Score: 5, Informative
  
  Not so much Samsung in the Nexus dept.
  Nexus One was by HTC
  Nexus 4 was by LG
  Nexus 7 is by Asus
  Nexus 10 is by Asus
  Anyway, it doesn't matter to me who makes the phone, I look at the features, OS and apps. Samsung has done a good job of marketing the Galaxy series. Some people buy because of good marketing. I still hate it when the manufacturer or the telecoms giant mess with the interface and applications... it's usually not an improvement.
  
  --
  I don't read your sig. Why are you reading mine?
10. Re:If this was Apple... by h4rr4r · 2013-10-01 06:07 · Score: 2
  
  Nexus 5 is LG as well.
  Galaxy Nexus was Samsung.
11. Re:If this was Apple... by Wookact · 2013-10-01 06:14 · Score: 2
  
  I would thought you would have to be logged in to shill, I mean how else will you get paid per post.
  
  Ohh BTW the search term "Windows phone 8 sucks" gets me 14 million hits. Obviously going by hits in Google isn't that great of a bench mark is it?
12. Re:If this was Apple... by Anonymous Coward · 2013-10-01 06:51 · Score: 3, Interesting
  
  Samsung denies also, i guess it's all fine then.
  Refutation and denial are not synonymous terms. While the G5 case has decent responses, Samsung's responses have been less than honest.
  With the Galaxy S4 GPU, their rationale was that other apps (not benchmarking tools) *can* enjoy the higher speed, yet to avoid heat issues this will be strictly limited in availability and time. Portable computing always has to strike this balance, so that's fine. But why then would benchmarking applications get carte blanche to run at higher speeds if the apps (including games) that people are actually using on the platform will either see no boost or a very time limited one?
  To use a car analogy, Samsung have secretly inserted a "new car smell" dispenser in their cars, and set it to be triggered when the car is driven by motoring journalists. Hooray, the amazingly persistent new car smell is touted in reviews, and then in reality most owners find the smell gone within a week. Samsung's defence? The new car smell is not just a trick they put in to fox the reviewers. The smell is available to all owners, but in order to conserve the fluid used, it'll only run at unbridled full power if the driver is named Jeremy Clarkson.
13. Re:If this was Apple... by nomorecwrd · 2013-10-01 07:20 · Score: 3, Informative
  
  So... thats about every user of Windows Phone complaining.
14. Re:If this was Apple... by jeffmflanagan · 2013-10-01 07:23 · Score: 2
  
  Anand is reporting that many Android handset makers are cheating, not just Samsung, so HTC and/or Motorola may be cheating as well. They do mention that Nexus line does not cheat.
15. Re:If this was Apple... by Karlt1 · 2013-10-01 08:59 · Score: 2
  
  even if android was sluggish, which it isnt, it still would not be worth the crap that is windows phone.
  http://androidandme.com/2013/07/devices/last-years-nexus-7-may-get-lag-fix-in-4-3-ota/
  http://www.engadget.com/2012/06/27/project-butter-improves-android-4-1s-speed/
  http://www.engadget.com/2012/06/27/project-butter-improves-android-4-1s-speed/
16. Re:If this was Apple... by Anonymous Coward · 2013-10-01 09:07 · Score: 2, Funny
  
  one million times each
17. Re: If this was Apple... by ElderKorean · 2013-10-01 14:42 · Score: 2
  
  But in introducing the feature, apple even said that 50% of users don't create a passcode, even the 4 digits one, because people find it inconvenient. The fingerprint sensor is for THOSE users. To make "good enough" security convenient.
  I like the locking feature, but would like it to be GPS based, or even phone tower / wifi name. When I'm at home I don't want (or need) a lock on my phone, but when I'm away from home I do. Work could be an options for people too, but I work in a school - there's no way I'm leaving that unlocked, even on my desk.
Should have done a battery benchmark by jandrese · 2013-10-01 05:39 · Score: 4, Interesting

They should have opened up a benchmarking app on both phones (the G2 and the Note3) and then did a battery life test on both phones with them "idle".

--

I read the internet for the articles.
1. Re:Should have done a battery benchmark by jovius · 2013-10-01 05:59 · Score: 2, Insightful
  
  The benchmarking itself seems to be flawed. Samsung wants to benchmark the devices at their full capacity, to see what they are capable of (the higher setting is reached in normal use of some apps anyway). The testers would probably like to do real world comparison tests (and not rely just on numbers). I don't see Samsung doing anything wrong here, even though the benchmarking apps are specifically chosen.
2. Re:Should have done a battery benchmark by jandrese · 2013-10-01 07:10 · Score: 3, Insightful
  
  They're changing the operating profile of their phone to inflate the benchmark results. In real life you would never be able to achieve numbers like that on a phone with a halfway decent battery life, even on applications that behave similarly to the benchmark. No normal use would achieve numbers found in the benchmark, they can't because they involve running the phone super hot and burning through the battery in no time, even when idle.
  
  If the hack were to go to full power when plugged into the wall, then I could maybe see a case for this being legitimate, since it would mean you could theoretically achieve the same results by simply playing your game or whatnot while plugged in, but because they're only switching on full power mode for a handful of specific benchmark applications there is just no excuse.
  
  --
  
  I read the internet for the articles.
Re:Humans will be Humans by 93+Escort+Wagon · 2013-10-01 05:41 · Score: 4, Insightful

If someone is surprised Humans are willing to cheat, rip off, etc to get ahead... well you haven't really been paying attention.
Fixed that for you.
I wouldn't want people to unfairly categorize you as a racist moron.

--
#DeleteChrome
Does not computer by squiggleslash · 2013-10-01 05:41 · Score: 4, Interesting

Even more damning was that the Note 3 was still faster than the G2 when run using 'stealth' (basically renamed) versions of the benchmarking apps which did not get the boost."

Not sure how this is "damning". I'd have thought it would prove the principle that the optimizations aren't app specific.
What am I missing?

--
You are not alone. This is not normal. None of this is normal.
1. Re:Does not computer by Space+cowboy · 2013-10-01 05:59 · Score: 3, Informative
  
  There is a file containing a list of all the common benchmarking apps, and everything in the list is a benchmarking app - nothing else. When one of those packages is run, the phone locks the frequency of all cores to fMax and also seems to fiddle with the GPU.
  The result is a battery-nightmare, but a boost of 20% to *only* benchmark apps. This is despicable - plain and simple.
  See http://arstechnica.com/gadgets/2013/10/galaxy-note-3s-benchmarking-adjustments-inflate-scores-by-up-to-20/
  Simon.
  
  --
  Physicists get Hadrons!
2. Re:Does not computer by Bill+Dimm · 2013-10-01 06:01 · Score: 4, Informative
  
  Here is what the article actually says:
  
  The ironic thing is that even with the benchmark booster disabled, the Note 3 still comes out faster than the G2 in this test. If the intent behind the boosting was simply to ensure that the Note 3 came out ahead in the benchmark race, it doesn't appear to have been necessary in the first place.
  Apparently the "damning" part was completely fabricated by the submitter.
3. Re:Does not computer by prisoner-of-enigma · 2013-10-01 06:21 · Score: 3, Insightful
  
  Not sure how this is "damning". I'd have thought it would prove the principle that the optimizations aren't app specific.
  What am I missing?
  
  It's not app-specific, it's app *name* specific. It's analogous to the Quake/Quack benchmark scandal years (OMG, more than a decade...time flies) ago. Samsung wrote this boosting protocol to enable itself when running benchmarks and *only* when running benchmarks. There is no legitimate way to invoke it, so no user will ever see the benefit of it when running any app *other than* the benchmark itself.
  For the inevitable car analogy: you take a Samsung car for a test drive, and when you floor it you feel 200hp worth of acceleration. Since the car is identical in almost every other aspect to competing HTC cars and Motorola cars (same price, similar trim, same engine) but they only make you feel about 150hp worth of acceleration, you opt for the Samsung car. Only when you drive it off the lot, you only feel 150hp worth of acceleration. You take it back to the dealer thinking something's wrong only to be informed the car will only give you 200hp when in "test drive by prospective customer" mode, and now that you've bought the car you're no longer in that category and cannot invoke it.
  
  --
  In the end they will lay their freedom at our feet and say to us, Make us your slaves, but feed us. - Fyodor Dostoyevsky
4. Re:Does not computer by icebike · 2013-10-01 08:53 · Score: 2
  
  Why did you feed us a link to the exact same Article as the Summary does?
  Did you think that would trick us into reading the full article?
  This is Slashdot. We are wise to those ploys.
  
  --
  Sig Battery depleted. Reverting to safe mode.
Yawn by sunking2 · 2013-10-01 05:41 · Score: 3, Insightful

And 95% of consumers could care less as long as the screen looks nice and the battery lasts more than 2 hours.
Samsung != Android by Freshly+Exhumed · 2013-10-01 05:42 · Score: 2

Take it out on Samsung for doing evil, or at the very least getting caught at it.

--
I deny that I have not avoided attaining the opposite of that which I do not want.
1. Re:Samsung != Android by oodaloop · 2013-10-01 06:15 · Score: 2
  
  And who exactly is blaming Android? Samsung is called out in the title and almost every sentence in the fantastic summary. And the word Android is only used once to refer to the benchamarking apps.
  
  --
  Tic-Tac-Toe, Global Thermonuclear War, and relationships all have the same winning move.
No shock ... by gstoddart · 2013-10-01 05:43 · Score: 4, Insightful

There's lies, damned lies, statistics, and vendor performance numbers.
I'm a little disappointed that there isn't actually any penalties for fudging your benchmarks -- it's blatantly lying to consumers about your product.
And to me, that seems like it's bordering on fraud.

--
Lost at C:>. Found at C.
Huh? by dgatwood · 2013-10-01 05:44 · Score: 3, Insightful

Even more damning was that the Note 3 was still faster than the G2 when run using 'stealth' (basically renamed) versions of the benchmarking apps which did not get the boost.
Wait, what? How is that more damning? It sounds like that means the benchmark is faster even without cheating, which means that they've changed the kernel scheduler/idle timers/clock stepping in a way that, at least for the sorts of tests performed in the benchmark, improves performance—presumably because their case design and/or battery capacity is better, allowing them to get away with less processor throttling. That sounds like it is almost inarguably a good thing. And that's coming from somebody who has dealt with several of Samsung's products and hated almost all of them. What's with the hate?
Unless, of course, they're being too aggressive about keeping the clock speed high, in which case you might argue that their battery life isn't what it should be... but that's pretty subjective.

--
Check out my sci-fi/humor trilogy at PatriotsBooks.
1. Re:Huh? by Anonymous Coward · 2013-10-01 06:13 · Score: 2, Informative
  
  I was perplexed too. After reading the article, I figured it out:
  It's damning in the sense that it's meant to skew comparisons with other Android devices, not to make it look better than their own previous offerings.
Re:Humans will be Humans by Space+cowboy · 2013-10-01 05:55 · Score: 3, Insightful

That's fair comment on the original post, but let's narrow it down a bit...
"If someone is surprised that a manufacturer with a track-record of fudging benchmarks is willing to cheat, rip off, etc to get ahead... well you haven't really been paying attention"
Not all humans are morally and ethically bankrupt. Samsung (as a corporate entity) is though.
Simon

--
Physicists get Hadrons!
No Implication by SuperKendall · 2013-10-01 06:15 · Score: 4, Insightful

Are you implying that us android users wouldn't be "up in arms"?
No implication is needed, we can see quite plainly there is very little outcry over this, just as there wasn't before. Android users simply accept this is the way things are, in a way they do not with any Apple problem whatsoever.

--
"There is more worth loving than we have strength to love." - Brian Jay Stanley
1. Re:No Implication by schlachter · 2013-10-01 09:20 · Score: 2
  
  Apple is held to a higher standard among consumers and industry for their behavior than Android makers are. This is just fact. Good news is that for the most part, they meet that standard.
  
  --
  My God can beat up your God. Just kidding...don't take offense. I know there's no God.
Re:Humans will be Humans by operagost · 2013-10-01 06:44 · Score: 2

Yeah, we can only unfairly categorize 'mericans here.

--

Gamingmuseum.com: Give your 3D accelerator a rest.
8 years ago? by mveloso · 2013-10-01 07:08 · Score: 4, Insightful

It's hard to find info on it, and it was at least 8 years ago. So you're saying that Samsung's benchmark juicing today is like Apple choosing the Intel compiler with extra options back in the day?
This is what Samsung does, in pseudocode:
if app.name == benchmark speed up
This is what Apple did on its benchmarks:
# for G5
cc test.c -altivec
# for x86
gcc test.c
If you can't tell the difference between the two, you're either stupid, or Samsung.
Re:Failed to note by jeffmflanagan · 2013-10-01 07:43 · Score: 3, Informative

You're wrong. The optimizations were found to only run with the particular executables identified by name. Ars renamed the file, and performance plummeted. No app maker can modify the hard-coding in the OS that locks the CPU into high-speed mode. Samsung, other cheating manufacturers, and I suppose ROM modders are the only ones that can access that functionality.
Here is your citation. by Ecuador · 2013-10-01 10:30 · Score: 5, Informative

Ok, I remember reading the Apple benchmarks myself (in utter disbelief - even for Apple it seemed too much), and this article you linked to does not agree with my memory. So let's go directly to the source. Read that benchmark paper yourself on archive.org : http://web.archive.org/web/20030727103031/http://veritest.com/clients/reports/apple/apple_performance.pdf
I gave it a quick look to refresh my memory and here are some highlights:
- They DISABLE hyper-threading on the SPEC rate test, which is the multi-processor test. Then, they ENABLE hyper-threading on the SPEC base, which is the single-processor test!!! They defend this by saying something like "hyper-threading is slower some times". Well, they sure know that, since they only enable it when it will slow down the Pentium! I would have given them the benefit of doubt if they had disabled (or enabled) it for both tests, but selectively enabling/disabling it means you know what you are doing.
- They use -O3 -fast -ffast when compiling for Apple, which uses fast math non-IEEE optimizations. Of course they had the Intel CPU run accurate/IEEE spec code - there is no equivalent -ffast-math used.
- They go on making some other "crazy" optimizations on the G5 like "modify CPU registers to enable memory Read By-pass", or installing a special malloc library that optimizes for speed by sacrificing memory just for the single-threaded benchmark. This is not how you benchmark for comparison purposes, especially if your optimizations for the competing platform are "turning off update" and "turning off hard drive sleep" (they obviously put that stuff just to pretend they "optimized" there as well).
And I am sure there are other things as well, this was from a quick read. And of course let's not mention that they compare the G5 with an Intel P4 CPU, when, at the time, AMD's Athlons/Opterons (64bit versions were just out as well) were destroying Intel (in performance, not sales - but that is another story).
In general, that paper is so ridiculous that I can't believe Apple had kept promoting it after they had been outed. But then again, given Apple's target audience, the explanation is simple. What was even more ridiculous is that when Apple started selling the Intel-based Mac they had kept for a while the section of their website that showed how much faster the G5 Mac was compared to Intel and then on the Intel Mac pages they had comparisons which showed how the Intel Mac is faster than the G5 Mac. No shame!

--
Violence is the last refuge of the incompetent. Polar Scope Align for iOS
Re:All of this and the benchmarks are still subpar by Ecuador · 2013-10-01 10:49 · Score: 3, Insightful

Battery life still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58409.png

You are comparing a phone with a 4 inch screen, with a "phone" that has a 5.7 inch screen. You can't compare battery life when the screen is what uses up most of the power. If you want a huge screen you have to compromise on battery life (and many other things - seriously, the note is ridiculously big to use as an every-day phone).

Browser speed still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58440.png

I don't suppose Samsung can do much about that. It is quite possible that with the same CPU, an Android would still be slower than an iOS device. Sure, Google has made a fast Java VM, but it still is a Java VM, right? For example, I had a Nokia N9 running Meego/Maemo. It could run circles around Android phones with the same CPU.

Graphics performance still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58425.png

Ehm, this result (to which you cleverly linked directly - hiding the context) is ran in native resolution. The Note has almost 3x the iphone's resolution, so it would be pretty strange to come on top in fps. But in all the other GPU benchmarks which are ran at 1080p it does come on top of the iphone.
But in any case I personally prefer a phone that has a good battery life, it can fit in my hand and lets me do whatever I want with it. So that rules out the note and the iphone ;)

--
Violence is the last refuge of the incompetent. Polar Scope Align for iOS