Samsung Fudging Benchmarks Again On Galaxy Note 3
tlhIngan writes "A few months ago, Samsung was caught gaming benchmarks on the Galaxy S4 (International version). They would lock the GPU at a higher-than-normal frequency when certain applications were run, including many popular Android benchmarking programs. These had the expected result of boosting the performance numbers. This time, the Galaxy Note 3 was caught doing the same thing, boosting CPU scores by 20% over the otherwise identical LG G2 (which uses the same SoC at the same clock). Samsung defends these claims by saying the other apps make use of such functionality, but Ars reversed-engineered the relevant code and discovered it applied only to benchmark applications. Even more damning was that the Note 3 was still faster than the G2 when run using 'stealth' (basically renamed) versions of the benchmarking apps which did not get the boost."
If Apple did this, people would be up in arms!
The real Sig captains the Northwestern. This one captains
They probably need the OC to overcome all the fuck-awful samsung bloatware they load on their devices. We don't want your shitty services Samsung. Just give us a clean device.
If someone is surprised Koreans are willing to cheat, rip off, etc to get ahead... well you haven't really been paying attention.
They should have opened up a benchmarking app on both phones (the G2 and the Note3) and then did a battery life test on both phones with them "idle".
I read the internet for the articles.
If someone is surprised Humans are willing to cheat, rip off, etc to get ahead... well you haven't really been paying attention.
Fixed that for you.
I wouldn't want people to unfairly categorize you as a racist moron.
#DeleteChrome
Not sure how this is "damning". I'd have thought it would prove the principle that the optimizations aren't app specific.
What am I missing?
You are not alone. This is not normal. None of this is normal.
And 95% of consumers could care less as long as the screen looks nice and the battery lasts more than 2 hours.
Take it out on Samsung for doing evil, or at the very least getting caught at it.
I deny that I have not avoided attaining the opposite of that which I do not want.
There's lies, damned lies, statistics, and vendor performance numbers.
I'm a little disappointed that there isn't actually any penalties for fudging your benchmarks -- it's blatantly lying to consumers about your product.
And to me, that seems like it's bordering on fraud.
Lost at C:>. Found at C.
Wait, what? How is that more damning? It sounds like that means the benchmark is faster even without cheating, which means that they've changed the kernel scheduler/idle timers/clock stepping in a way that, at least for the sorts of tests performed in the benchmark, improves performance—presumably because their case design and/or battery capacity is better, allowing them to get away with less processor throttling. That sounds like it is almost inarguably a good thing. And that's coming from somebody who has dealt with several of Samsung's products and hated almost all of them. What's with the hate?
Unless, of course, they're being too aggressive about keeping the clock speed high, in which case you might argue that their battery life isn't what it should be... but that's pretty subjective.
Check out my sci-fi/humor trilogy at PatriotsBooks.
That's fair comment on the original post, but let's narrow it down a bit...
"If someone is surprised that a manufacturer with a track-record of fudging benchmarks is willing to cheat, rip off, etc to get ahead... well you haven't really been paying attention"
Not all humans are morally and ethically bankrupt. Samsung (as a corporate entity) is though.
Simon
Physicists get Hadrons!
They fudged 3D Mark benchmarks by optimizing specifically for one of the demos. When someone tried debug mode in which you could freely rotate the camera, the scene was incorrectly rendered. The problem would disappear when renaming the executable. Those were the days of FX 5XXX cards which were somewhat inferior to R3XX chips, especially because of inability to use FP24 in shaders (only 32 or 16 bit precision). This wasn't the only incident, though. ATI had it's share too. At some point Catalyst CC even introduced an option to turn game detection (and optimizations) on and off.
Are you implying that us android users wouldn't be "up in arms"?
No implication is needed, we can see quite plainly there is very little outcry over this, just as there wasn't before. Android users simply accept this is the way things are, in a way they do not with any Apple problem whatsoever.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
It's how Bill Gates built his fortune.
If someone is surprised Humans are willing to cheat, rip off, etc to get ahead... well you haven't really been paying attention.
Fixed that for you.
I wouldn't want people to unfairly categorize you as a racist moron.
Samsung represents a very big chunk of Korean electronics industry and they have the responsibility to choose what kind of image they want to give about the practices of that particular industry.
On individual level, I believe there are many honest Korean people too.
Your precious gizmag article didn't benchmark the S4 and 5s (and surprisingly did not provide battery-life statistics for the S4, which has been out for a while now). If they did, they'd find the 5s is twice as fast as the S4--basically as fast as the Note 3. All this speed is achieved with fewer cores and at lower GHz, because the 5s is a smart, custom design that even includes a motion co-processor. The S4 can not be reasonably used one-handed either, unless you play in the NBA.
Yeah, we can only unfairly categorize 'mericans here.
Gamingmuseum.com: Give your 3D accelerator a rest.
Battery life still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58409.png
Browser speed still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58440.png
Graphics performance still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58425.png
For a device this much larger and heavier, you would expect they could overclock the chipset and get more performance than this.
Benchmarks are problematic by their very nature in that they are typically predictable and a manufacturer can simply say they have tuned their product for a given application. Let me give a good example of this from a product that isn't made for consumer use just to make my point:
http://www.anandtech.com/show/7165/ocz-announces-zdxl-pcie-sql-accelerator-ssd-solution
This is a PCIE SSD product designed to boot server performance explicitly for Microsoft MS SQL Server. This product has been explicitly designed to make a given product that is quite expensive work faster.
Is this product cheating because it has been tuned just to give better results in one application? What if OCZ released a version of the product that was tuned to detect if it was working on MySQL, MS SQL Server and Oracle and optimize performance accordingly? What happens when someone tries to use it on another SQL database and get's less than stellar performance?
For the typical Slashdot car analogy think of Ferrari, they have been accused of cheating on car tests for many years by setting up their cars for the exact track they were about to be tested on. What would happen if Ferrari (or another manufacturer) tied their systems into the car's GPS?
Think of the Top Gear episode where the Nissan GTR knew it had just been driven onto a racetrack and turned off it's speed limiter. You could just as easily program the car to have optimal settings for that exact track and then populate in as many race tracks as you could. Is that cheating or is the manufacturer anticipating the potential needs of their customers and tuning their product accordingly?
If everyone playing the latest Call of Duty get's the same benefit when they buy their card, than can you really say it's cheating? Through the use of patches you can continuously upgrade the card to be optimized with the latest games.
Really? That's like using Slashdot polls for something important. Non-story.
It's hard to find info on it, and it was at least 8 years ago. So you're saying that Samsung's benchmark juicing today is like Apple choosing the Intel compiler with extra options back in the day?
This is what Samsung does, in pseudocode:
if app.name == benchmark speed up
This is what Apple did on its benchmarks:
# for G5
cc test.c -altivec
# for x86
gcc test.c
If you can't tell the difference between the two, you're either stupid, or Samsung.
You're wrong. The optimizations were found to only run with the particular executables identified by name. Ars renamed the file, and performance plummeted. No app maker can modify the hard-coding in the OS that locks the CPU into high-speed mode. Samsung, other cheating manufacturers, and I suppose ROM modders are the only ones that can access that functionality.
Anything else is just a made up number.
Ok, I remember reading the Apple benchmarks myself (in utter disbelief - even for Apple it seemed too much), and this article you linked to does not agree with my memory. So let's go directly to the source. Read that benchmark paper yourself on archive.org : http://web.archive.org/web/20030727103031/http://veritest.com/clients/reports/apple/apple_performance.pdf
I gave it a quick look to refresh my memory and here are some highlights:
- They DISABLE hyper-threading on the SPEC rate test, which is the multi-processor test. Then, they ENABLE hyper-threading on the SPEC base, which is the single-processor test!!! They defend this by saying something like "hyper-threading is slower some times". Well, they sure know that, since they only enable it when it will slow down the Pentium! I would have given them the benefit of doubt if they had disabled (or enabled) it for both tests, but selectively enabling/disabling it means you know what you are doing.
- They use -O3 -fast -ffast when compiling for Apple, which uses fast math non-IEEE optimizations. Of course they had the Intel CPU run accurate/IEEE spec code - there is no equivalent -ffast-math used.
- They go on making some other "crazy" optimizations on the G5 like "modify CPU registers to enable memory Read By-pass", or installing a special malloc library that optimizes for speed by sacrificing memory just for the single-threaded benchmark. This is not how you benchmark for comparison purposes, especially if your optimizations for the competing platform are "turning off update" and "turning off hard drive sleep" (they obviously put that stuff just to pretend they "optimized" there as well).
And I am sure there are other things as well, this was from a quick read. And of course let's not mention that they compare the G5 with an Intel P4 CPU, when, at the time, AMD's Athlons/Opterons (64bit versions were just out as well) were destroying Intel (in performance, not sales - but that is another story).
In general, that paper is so ridiculous that I can't believe Apple had kept promoting it after they had been outed. But then again, given Apple's target audience, the explanation is simple. What was even more ridiculous is that when Apple started selling the Intel-based Mac they had kept for a while the section of their website that showed how much faster the G5 Mac was compared to Intel and then on the Intel Mac pages they had comparisons which showed how the Intel Mac is faster than the G5 Mac. No shame!
Violence is the last refuge of the incompetent. Polar Scope Align for iOS
If someone is surprised Humans are willing to cheat, rip off, etc to get ahead... well you haven't really been paying attention.
Fixed that for you.
I wouldn't want people to unfairly categorize you as a racist moron.
Korean is an ethnicity and a culture. It is not a separate "race". I don't think anyone mentioned a race anywhere. Is it not fair to comment on attitudes of certain cultures when it comes to honesty? Sure, it is stereotyping but, again, it is about a "culture" rather than a "race".
Jesus was a compassionate social conservative who called individuals to sin no more.
Why not Santa Clause or the chupacabra while we're talking about things that don't exist? All kinds of people like to bitch about "Apple Fanboys" on Slashdot, but can never point to any examples when pressed.
Samsung has been known to be unusually corrupt, even for Korea, for quite a while. Here are some links to discussions about it
http://lanle.wordpress.com/2010/04/28/book-on-samsung-divides-korea/
http://www.theverge.com/2012/11/30/3709688/samsung-25-years-lee-kun-hee
http://www.engadget.com/2013/09/26/mobilebench-benchmarks-samsung/ - "MobileBench group aims to improve mobile benchmarking, recruits Samsung but lacks Qualcomm, NVIDIA "
Of course news about a fake are Fake News.
You could even have most people calling Jobs an asshole, like when they pulled IDG books from Apple stores in a fit of pique over "iCon", but you'd still have the finger waggers come in saying "Now if this were Microsoft..."
This is like car manufacturers taping up all of the air intakes and door seams when they do the fuel economy runs. Everybody knows the numbers are cooked. Nobody expects your 3L V8 to actually get 30mpg, ever.
Finally had enough. Come see us over at https://soylentnews.org/
I use Samsung mobile phone S4, really i liked the features of this smart phone. i think, it is better than Galaxy Note.
So what we need is a small "App Launcher" app that does one-touch symlinking of an arbitrary game to the benchmark app name. :)
Maybe multiple benchmark app names, although who would run two demanding apps at once.
Until that exists, can just do it manually I 'spose.
-- perl -e'print pack"H*","6e656d6f406d38792e6f7267"'