Slashdot Mirror
Samsung Fudging Benchmarks Again On Galaxy Note 3
tlhIngan writes "A few months ago, Samsung was caught gaming benchmarks on the Galaxy S4 (International version). They would lock the GPU at a higher-than-normal frequency when certain applications were run, including many popular Android benchmarking programs. These had the expected result of boosting the performance numbers. This time, the Galaxy Note 3 was caught doing the same thing, boosting CPU scores by 20% over the otherwise identical LG G2 (which uses the same SoC at the same clock). Samsung defends these claims by saying the other apps make use of such functionality, but Ars reversed-engineered the relevant code and discovered it applied only to benchmark applications. Even more damning was that the Note 3 was still faster than the G2 when run using 'stealth' (basically renamed) versions of the benchmarking apps which did not get the boost."
Are you implying that us android users wouldn't be "up in arms"? It affects us more. While apple sells two models of the iPhone, we have a multitude of android phones to choose from. Samsung messing with the benchmark has the potential to cause a customer to chose samsung over HTC, Google Nexus, or Motorola phone.
These comments are my own and do not necessarily reflect the views or opinions of my employer or colleagues...
They should have opened up a benchmarking app on both phones (the G2 and the Note3) and then did a battery life test on both phones with them "idle".
I read the internet for the articles.
If someone is surprised Humans are willing to cheat, rip off, etc to get ahead... well you haven't really been paying attention.
Fixed that for you.
I wouldn't want people to unfairly categorize you as a racist moron.
#DeleteChrome
Not sure how this is "damning". I'd have thought it would prove the principle that the optimizations aren't app specific.
What am I missing?
You are not alone. This is not normal. None of this is normal.
There's lies, damned lies, statistics, and vendor performance numbers.
I'm a little disappointed that there isn't actually any penalties for fudging your benchmarks -- it's blatantly lying to consumers about your product.
And to me, that seems like it's bordering on fraud.
Lost at C:>. Found at C.
> If Apple did this, people would be up in arms!
If Apple did this, they would be suing Samsung for copying.
I'll see your senator, and I'll raise you two judges.
Wrong, they DID care about benchmarking once, while Macs were still Power PC based, then they got caught fudging the benchmarks approximately like Samsung did. They suddenly decided that benchmarks didn't matter soon thereafter. They ought to sue Samsung again, because business methods are patentable...
Just because you're paranoid doesn't mean they aren't out to get you
Citation required, because all I can find is: http://www.theregister.co.uk/2003/06/25/apple_denies_fiddling_g5_xeon/ ... which seems to be refuting the claim...
Simon.
Physicists get Hadrons!
My you are full of unsupported assertions today. "Most every tech company has been caught". Really? Any evidence? "Samsung ... the good guys" You must be kidding. They copy and they clone. Apple does gold so Samsung does gold. Apple sells a 64-bit phone with a 64 bit operating system and conversion tools to take advantage of it. Samsung announces that they'll be building 64 bit phones too, one day. Of course unless Android is converted to 64 bit that will be pointless and there is nothing from Google indicating that is going to happen any time soon. Chrome OS seems to be more important to them these days anyway. And finally, "their hardware is still clearly the best". Evidence or just your opinion based on your limited experience? I've tried Apple, Nokia, HTC and Samsung and liked Samsung the least hardware wise. Consumer Reports and other customer satisfaction survey's I've seen don't rate Samsung all that highly. Apple leads the pack in every survey I've seen.
Very often, people confuse simple with simplistic. The nuance is lost on most. - Clement Mok
Not so much Samsung in the Nexus dept.
Nexus One was by HTC
Nexus 4 was by LG
Nexus 7 is by Asus
Nexus 10 is by Asus
Anyway, it doesn't matter to me who makes the phone, I look at the features, OS and apps. Samsung has done a good job of marketing the Galaxy series. Some people buy because of good marketing. I still hate it when the manufacturer or the telecoms giant mess with the interface and applications... it's usually not an improvement.
I don't read your sig. Why are you reading mine?
Are you implying that us android users wouldn't be "up in arms"?
No implication is needed, we can see quite plainly there is very little outcry over this, just as there wasn't before. Android users simply accept this is the way things are, in a way they do not with any Apple problem whatsoever.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
It's hard to find info on it, and it was at least 8 years ago. So you're saying that Samsung's benchmark juicing today is like Apple choosing the Intel compiler with extra options back in the day?
This is what Samsung does, in pseudocode:
if app.name == benchmark speed up
This is what Apple did on its benchmarks:
# for G5
cc test.c -altivec
# for x86
gcc test.c
If you can't tell the difference between the two, you're either stupid, or Samsung.
Ok, I remember reading the Apple benchmarks myself (in utter disbelief - even for Apple it seemed too much), and this article you linked to does not agree with my memory. So let's go directly to the source. Read that benchmark paper yourself on archive.org : http://web.archive.org/web/20030727103031/http://veritest.com/clients/reports/apple/apple_performance.pdf
I gave it a quick look to refresh my memory and here are some highlights:
- They DISABLE hyper-threading on the SPEC rate test, which is the multi-processor test. Then, they ENABLE hyper-threading on the SPEC base, which is the single-processor test!!! They defend this by saying something like "hyper-threading is slower some times". Well, they sure know that, since they only enable it when it will slow down the Pentium! I would have given them the benefit of doubt if they had disabled (or enabled) it for both tests, but selectively enabling/disabling it means you know what you are doing.
- They use -O3 -fast -ffast when compiling for Apple, which uses fast math non-IEEE optimizations. Of course they had the Intel CPU run accurate/IEEE spec code - there is no equivalent -ffast-math used.
- They go on making some other "crazy" optimizations on the G5 like "modify CPU registers to enable memory Read By-pass", or installing a special malloc library that optimizes for speed by sacrificing memory just for the single-threaded benchmark. This is not how you benchmark for comparison purposes, especially if your optimizations for the competing platform are "turning off update" and "turning off hard drive sleep" (they obviously put that stuff just to pretend they "optimized" there as well).
And I am sure there are other things as well, this was from a quick read. And of course let's not mention that they compare the G5 with an Intel P4 CPU, when, at the time, AMD's Athlons/Opterons (64bit versions were just out as well) were destroying Intel (in performance, not sales - but that is another story).
In general, that paper is so ridiculous that I can't believe Apple had kept promoting it after they had been outed. But then again, given Apple's target audience, the explanation is simple. What was even more ridiculous is that when Apple started selling the Intel-based Mac they had kept for a while the section of their website that showed how much faster the G5 Mac was compared to Intel and then on the Intel Mac pages they had comparisons which showed how the Intel Mac is faster than the G5 Mac. No shame!
Violence is the last refuge of the incompetent. Polar Scope Align for iOS