Slashdot Mirror
Samsung Fudging Benchmarks Again On Galaxy Note 3
tlhIngan writes "A few months ago, Samsung was caught gaming benchmarks on the Galaxy S4 (International version). They would lock the GPU at a higher-than-normal frequency when certain applications were run, including many popular Android benchmarking programs. These had the expected result of boosting the performance numbers. This time, the Galaxy Note 3 was caught doing the same thing, boosting CPU scores by 20% over the otherwise identical LG G2 (which uses the same SoC at the same clock). Samsung defends these claims by saying the other apps make use of such functionality, but Ars reversed-engineered the relevant code and discovered it applied only to benchmark applications. Even more damning was that the Note 3 was still faster than the G2 when run using 'stealth' (basically renamed) versions of the benchmarking apps which did not get the boost."
Are you implying that us android users wouldn't be "up in arms"? It affects us more. While apple sells two models of the iPhone, we have a multitude of android phones to choose from. Samsung messing with the benchmark has the potential to cause a customer to chose samsung over HTC, Google Nexus, or Motorola phone.
These comments are my own and do not necessarily reflect the views or opinions of my employer or colleagues...
Why would you even consider synthetic benchmarks when choosing a phone? Planning on trying some bitcoin mining on the go?
They should have opened up a benchmarking app on both phones (the G2 and the Note3) and then did a battery life test on both phones with them "idle".
I read the internet for the articles.
If someone is surprised Humans are willing to cheat, rip off, etc to get ahead... well you haven't really been paying attention.
Fixed that for you.
I wouldn't want people to unfairly categorize you as a racist moron.
#DeleteChrome
Not sure how this is "damning". I'd have thought it would prove the principle that the optimizations aren't app specific.
What am I missing?
You are not alone. This is not normal. None of this is normal.
And 95% of consumers could care less as long as the screen looks nice and the battery lasts more than 2 hours.
There's lies, damned lies, statistics, and vendor performance numbers.
I'm a little disappointed that there isn't actually any penalties for fudging your benchmarks -- it's blatantly lying to consumers about your product.
And to me, that seems like it's bordering on fraud.
Lost at C:>. Found at C.
Wait, what? How is that more damning? It sounds like that means the benchmark is faster even without cheating, which means that they've changed the kernel scheduler/idle timers/clock stepping in a way that, at least for the sorts of tests performed in the benchmark, improves performance—presumably because their case design and/or battery capacity is better, allowing them to get away with less processor throttling. That sounds like it is almost inarguably a good thing. And that's coming from somebody who has dealt with several of Samsung's products and hated almost all of them. What's with the hate?
Unless, of course, they're being too aggressive about keeping the clock speed high, in which case you might argue that their battery life isn't what it should be... but that's pretty subjective.
Check out my sci-fi/humor trilogy at PatriotsBooks.
You've obviously never used their Blu-Ray players.
Check out my sci-fi/humor trilogy at PatriotsBooks.
> If Apple did this, people would be up in arms!
If Apple did this, they would be suing Samsung for copying.
I'll see your senator, and I'll raise you two judges.
Wrong, they DID care about benchmarking once, while Macs were still Power PC based, then they got caught fudging the benchmarks approximately like Samsung did. They suddenly decided that benchmarks didn't matter soon thereafter. They ought to sue Samsung again, because business methods are patentable...
Just because you're paranoid doesn't mean they aren't out to get you
Citation required, because all I can find is: http://www.theregister.co.uk/2003/06/25/apple_denies_fiddling_g5_xeon/ ... which seems to be refuting the claim...
Simon.
Physicists get Hadrons!
My you are full of unsupported assertions today. "Most every tech company has been caught". Really? Any evidence? "Samsung ... the good guys" You must be kidding. They copy and they clone. Apple does gold so Samsung does gold. Apple sells a 64-bit phone with a 64 bit operating system and conversion tools to take advantage of it. Samsung announces that they'll be building 64 bit phones too, one day. Of course unless Android is converted to 64 bit that will be pointless and there is nothing from Google indicating that is going to happen any time soon. Chrome OS seems to be more important to them these days anyway. And finally, "their hardware is still clearly the best". Evidence or just your opinion based on your limited experience? I've tried Apple, Nokia, HTC and Samsung and liked Samsung the least hardware wise. Consumer Reports and other customer satisfaction survey's I've seen don't rate Samsung all that highly. Apple leads the pack in every survey I've seen.
Very often, people confuse simple with simplistic. The nuance is lost on most. - Clement Mok
That's fair comment on the original post, but let's narrow it down a bit...
"If someone is surprised that a manufacturer with a track-record of fudging benchmarks is willing to cheat, rip off, etc to get ahead... well you haven't really been paying attention"
Not all humans are morally and ethically bankrupt. Samsung (as a corporate entity) is though.
Simon
Physicists get Hadrons!
Not so much Samsung in the Nexus dept.
Nexus One was by HTC
Nexus 4 was by LG
Nexus 7 is by Asus
Nexus 10 is by Asus
Anyway, it doesn't matter to me who makes the phone, I look at the features, OS and apps. Samsung has done a good job of marketing the Galaxy series. Some people buy because of good marketing. I still hate it when the manufacturer or the telecoms giant mess with the interface and applications... it's usually not an improvement.
I don't read your sig. Why are you reading mine?
Are you implying that us android users wouldn't be "up in arms"?
No implication is needed, we can see quite plainly there is very little outcry over this, just as there wasn't before. Android users simply accept this is the way things are, in a way they do not with any Apple problem whatsoever.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Samsung denies also, i guess it's all fine then.
Refutation and denial are not synonymous terms. While the G5 case has decent responses, Samsung's responses have been less than honest.
With the Galaxy S4 GPU, their rationale was that other apps (not benchmarking tools) *can* enjoy the higher speed, yet to avoid heat issues this will be strictly limited in availability and time. Portable computing always has to strike this balance, so that's fine. But why then would benchmarking applications get carte blanche to run at higher speeds if the apps (including games) that people are actually using on the platform will either see no boost or a very time limited one?
To use a car analogy, Samsung have secretly inserted a "new car smell" dispenser in their cars, and set it to be triggered when the car is driven by motoring journalists. Hooray, the amazingly persistent new car smell is touted in reviews, and then in reality most owners find the smell gone within a week. Samsung's defence? The new car smell is not just a trick they put in to fox the reviewers. The smell is available to all owners, but in order to conserve the fluid used, it'll only run at unbridled full power if the driver is named Jeremy Clarkson.
It's hard to find info on it, and it was at least 8 years ago. So you're saying that Samsung's benchmark juicing today is like Apple choosing the Intel compiler with extra options back in the day?
This is what Samsung does, in pseudocode:
if app.name == benchmark speed up
This is what Apple did on its benchmarks:
# for G5
cc test.c -altivec
# for x86
gcc test.c
If you can't tell the difference between the two, you're either stupid, or Samsung.
So... thats about every user of Windows Phone complaining.
You're wrong. The optimizations were found to only run with the particular executables identified by name. Ars renamed the file, and performance plummeted. No app maker can modify the hard-coding in the OS that locks the CPU into high-speed mode. Samsung, other cheating manufacturers, and I suppose ROM modders are the only ones that can access that functionality.
Ok, I remember reading the Apple benchmarks myself (in utter disbelief - even for Apple it seemed too much), and this article you linked to does not agree with my memory. So let's go directly to the source. Read that benchmark paper yourself on archive.org : http://web.archive.org/web/20030727103031/http://veritest.com/clients/reports/apple/apple_performance.pdf
I gave it a quick look to refresh my memory and here are some highlights:
- They DISABLE hyper-threading on the SPEC rate test, which is the multi-processor test. Then, they ENABLE hyper-threading on the SPEC base, which is the single-processor test!!! They defend this by saying something like "hyper-threading is slower some times". Well, they sure know that, since they only enable it when it will slow down the Pentium! I would have given them the benefit of doubt if they had disabled (or enabled) it for both tests, but selectively enabling/disabling it means you know what you are doing.
- They use -O3 -fast -ffast when compiling for Apple, which uses fast math non-IEEE optimizations. Of course they had the Intel CPU run accurate/IEEE spec code - there is no equivalent -ffast-math used.
- They go on making some other "crazy" optimizations on the G5 like "modify CPU registers to enable memory Read By-pass", or installing a special malloc library that optimizes for speed by sacrificing memory just for the single-threaded benchmark. This is not how you benchmark for comparison purposes, especially if your optimizations for the competing platform are "turning off update" and "turning off hard drive sleep" (they obviously put that stuff just to pretend they "optimized" there as well).
And I am sure there are other things as well, this was from a quick read. And of course let's not mention that they compare the G5 with an Intel P4 CPU, when, at the time, AMD's Athlons/Opterons (64bit versions were just out as well) were destroying Intel (in performance, not sales - but that is another story).
In general, that paper is so ridiculous that I can't believe Apple had kept promoting it after they had been outed. But then again, given Apple's target audience, the explanation is simple. What was even more ridiculous is that when Apple started selling the Intel-based Mac they had kept for a while the section of their website that showed how much faster the G5 Mac was compared to Intel and then on the Intel Mac pages they had comparisons which showed how the Intel Mac is faster than the G5 Mac. No shame!
Violence is the last refuge of the incompetent. Polar Scope Align for iOS
Battery life still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58409.png
You are comparing a phone with a 4 inch screen, with a "phone" that has a 5.7 inch screen. You can't compare battery life when the screen is what uses up most of the power. If you want a huge screen you have to compromise on battery life (and many other things - seriously, the note is ridiculously big to use as an every-day phone).
Browser speed still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58440.png
I don't suppose Samsung can do much about that. It is quite possible that with the same CPU, an Android would still be slower than an iOS device. Sure, Google has made a fast Java VM, but it still is a Java VM, right? For example, I had a Nokia N9 running Meego/Maemo. It could run circles around Android phones with the same CPU.
Graphics performance still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58425.png
Ehm, this result (to which you cleverly linked directly - hiding the context) is ran in native resolution. The Note has almost 3x the iphone's resolution, so it would be pretty strange to come on top in fps. But in all the other GPU benchmarks which are ran at 1080p it does come on top of the iphone.
But in any case I personally prefer a phone that has a good battery life, it can fit in my hand and lets me do whatever I want with it. So that rules out the note and the iphone ;)
Violence is the last refuge of the incompetent. Polar Scope Align for iOS