Slashdot Mirror
Samsung Fudging Benchmarks Again On Galaxy Note 3
tlhIngan writes "A few months ago, Samsung was caught gaming benchmarks on the Galaxy S4 (International version). They would lock the GPU at a higher-than-normal frequency when certain applications were run, including many popular Android benchmarking programs. These had the expected result of boosting the performance numbers. This time, the Galaxy Note 3 was caught doing the same thing, boosting CPU scores by 20% over the otherwise identical LG G2 (which uses the same SoC at the same clock). Samsung defends these claims by saying the other apps make use of such functionality, but Ars reversed-engineered the relevant code and discovered it applied only to benchmark applications. Even more damning was that the Note 3 was still faster than the G2 when run using 'stealth' (basically renamed) versions of the benchmarking apps which did not get the boost."
If Apple did this, people would be up in arms!
The real Sig captains the Northwestern. This one captains
They probably need the OC to overcome all the fuck-awful samsung bloatware they load on their devices. We don't want your shitty services Samsung. Just give us a clean device.
If someone is surprised Koreans are willing to cheat, rip off, etc to get ahead... well you haven't really been paying attention.
They should have opened up a benchmarking app on both phones (the G2 and the Note3) and then did a battery life test on both phones with them "idle".
I read the internet for the articles.
If someone is surprised Humans are willing to cheat, rip off, etc to get ahead... well you haven't really been paying attention.
Fixed that for you.
I wouldn't want people to unfairly categorize you as a racist moron.
#DeleteChrome
Not sure how this is "damning". I'd have thought it would prove the principle that the optimizations aren't app specific.
What am I missing?
You are not alone. This is not normal. None of this is normal.
And 95% of consumers could care less as long as the screen looks nice and the battery lasts more than 2 hours.
Take it out on Samsung for doing evil, or at the very least getting caught at it.
I deny that I have not avoided attaining the opposite of that which I do not want.
There's lies, damned lies, statistics, and vendor performance numbers.
I'm a little disappointed that there isn't actually any penalties for fudging your benchmarks -- it's blatantly lying to consumers about your product.
And to me, that seems like it's bordering on fraud.
Lost at C:>. Found at C.
Wait, what? How is that more damning? It sounds like that means the benchmark is faster even without cheating, which means that they've changed the kernel scheduler/idle timers/clock stepping in a way that, at least for the sorts of tests performed in the benchmark, improves performance—presumably because their case design and/or battery capacity is better, allowing them to get away with less processor throttling. That sounds like it is almost inarguably a good thing. And that's coming from somebody who has dealt with several of Samsung's products and hated almost all of them. What's with the hate?
Unless, of course, they're being too aggressive about keeping the clock speed high, in which case you might argue that their battery life isn't what it should be... but that's pretty subjective.
Check out my sci-fi/humor trilogy at PatriotsBooks.
That's fair comment on the original post, but let's narrow it down a bit...
"If someone is surprised that a manufacturer with a track-record of fudging benchmarks is willing to cheat, rip off, etc to get ahead... well you haven't really been paying attention"
Not all humans are morally and ethically bankrupt. Samsung (as a corporate entity) is though.
Simon
Physicists get Hadrons!
They fudged 3D Mark benchmarks by optimizing specifically for one of the demos. When someone tried debug mode in which you could freely rotate the camera, the scene was incorrectly rendered. The problem would disappear when renaming the executable. Those were the days of FX 5XXX cards which were somewhat inferior to R3XX chips, especially because of inability to use FP24 in shaders (only 32 or 16 bit precision). This wasn't the only incident, though. ATI had it's share too. At some point Catalyst CC even introduced an option to turn game detection (and optimizations) on and off.
If you think all cultures do this equally you're a fool. Try doing business in east asia, you'll see very different experiences between China, Japan, Taiwan and Korea....
I'm assuming it's the same as the last time this came up, so I may be wrong, but...
The article failed to note that the increased optimizations are not only available for benchmark programs, but also for other programs made by run-of-the-mill developers. The phone and tablets default to the lower clock frequency, because most apps don't need an intense amount of resources. However, particularly in the video game arena, the higher clock rates are available and will be used when needed.
This helps with: battery life, system stability, system longevity, and heating. What's essentially happening is what happens in a typical PC fan. When the device below the fan is cool, it's running at a slower rate. When that device gets hot, the fan cranks up to a higher speed. Same exact concept. It just happens to be the case that many benchmarking tools stress the system in such a way that it triggers the higher settings. This doesn't happen on PCs, but it does happen in the Galaxy S4 and the new Note devices.
If you look in the article(I know...rtfa...imagine that!) They are running a 4 function calculator, which has 3/4 cores turned off and the other core @ 300 MHz. On the other hand, they have a benchmark program, which reaches the max performance of the processor...all cores are on and running @ 2.3(2.27) GHz. It is completely within my expectation that a 4 function calculator takes fewer resources than a benchmarking tool. In fact, that's the point of benchmarking tools(to stress the system to it's maximum potential).
Or do we really need our 4 function calculator app to run at system max settings?
Are you implying that us android users wouldn't be "up in arms"?
No implication is needed, we can see quite plainly there is very little outcry over this, just as there wasn't before. Android users simply accept this is the way things are, in a way they do not with any Apple problem whatsoever.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
It's how Bill Gates built his fortune.
Why would you switch from Samsung when it's "even more damning" [sic] that the Note 3 was faster without resorting to shenanigans? Doesn't this all show how truly innovative Samsung is?
If someone is surprised Humans are willing to cheat, rip off, etc to get ahead... well you haven't really been paying attention.
Fixed that for you.
I wouldn't want people to unfairly categorize you as a racist moron.
Samsung represents a very big chunk of Korean electronics industry and they have the responsibility to choose what kind of image they want to give about the practices of that particular industry.
On individual level, I believe there are many honest Korean people too.
Your precious gizmag article didn't benchmark the S4 and 5s (and surprisingly did not provide battery-life statistics for the S4, which has been out for a while now). If they did, they'd find the 5s is twice as fast as the S4--basically as fast as the Note 3. All this speed is achieved with fewer cores and at lower GHz, because the 5s is a smart, custom design that even includes a motion co-processor. The S4 can not be reasonably used one-handed either, unless you play in the NBA.
Yeah, we can only unfairly categorize 'mericans here.
Gamingmuseum.com: Give your 3D accelerator a rest.
Battery life still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58409.png
Browser speed still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58440.png
Graphics performance still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58425.png
For a device this much larger and heavier, you would expect they could overclock the chipset and get more performance than this.
These last few days I coincidentally have been looking for a replacement for my 4-year old mobile phone. I won't buy Sony due to their shitty treatment of consumers over the years, and I won't buy a shiny self-fleecing machine from Apple. That left the recent HTC and Samsungs, but now I don't need to choose.
Benchmarks are problematic by their very nature in that they are typically predictable and a manufacturer can simply say they have tuned their product for a given application. Let me give a good example of this from a product that isn't made for consumer use just to make my point:
http://www.anandtech.com/show/7165/ocz-announces-zdxl-pcie-sql-accelerator-ssd-solution
This is a PCIE SSD product designed to boot server performance explicitly for Microsoft MS SQL Server. This product has been explicitly designed to make a given product that is quite expensive work faster.
Is this product cheating because it has been tuned just to give better results in one application? What if OCZ released a version of the product that was tuned to detect if it was working on MySQL, MS SQL Server and Oracle and optimize performance accordingly? What happens when someone tries to use it on another SQL database and get's less than stellar performance?
For the typical Slashdot car analogy think of Ferrari, they have been accused of cheating on car tests for many years by setting up their cars for the exact track they were about to be tested on. What would happen if Ferrari (or another manufacturer) tied their systems into the car's GPS?
Think of the Top Gear episode where the Nissan GTR knew it had just been driven onto a racetrack and turned off it's speed limiter. You could just as easily program the car to have optimal settings for that exact track and then populate in as many race tracks as you could. Is that cheating or is the manufacturer anticipating the potential needs of their customers and tuning their product accordingly?
If everyone playing the latest Call of Duty get's the same benefit when they buy their card, than can you really say it's cheating? Through the use of patches you can continuously upgrade the card to be optimized with the latest games.
Really? That's like using Slashdot polls for something important. Non-story.
It's hard to find info on it, and it was at least 8 years ago. So you're saying that Samsung's benchmark juicing today is like Apple choosing the Intel compiler with extra options back in the day?
This is what Samsung does, in pseudocode:
if app.name == benchmark speed up
This is what Apple did on its benchmarks:
# for G5
cc test.c -altivec
# for x86
gcc test.c
If you can't tell the difference between the two, you're either stupid, or Samsung.
The funny thing about the G2 is that when i was playing with it at ATT last week, it was noticably slower/laggier/lower-framerate than both the Optimus G and Optimus G Pro even though they claimed it was like a billion times faster than my Optimus G. It does not surprise me that any of the manufactures are having to lie, now.
Seriously, what do you expect from Samsung? The chairman was convicted for fraud and corruption 2 times and still has his job.
He's also a long time member of the IOC, and the committee didn't think his convictions are a problem. And guess who was the sponsor of the London Olympics?
Anything else is just a made up number.
Hey look it's one of those "all cultures are equal" fucktards.
There could actually be a legitimate reason for this. They're only boosting the benchmark apps (which they don't control) with this mod but it's quite possible that they put this boost code into the apps that they DO control. So for instance the camera app or the browser app could have a similar boost but it's built into the app itself and they didn't have to do this "detect the app to boost" mod. Since they want a realistic representation of the experience on their specially designed apps... they boosted the benchmarks to reflect what their samsung apps internally do.
The same sort of thing is done on the HTC One for the camera app. The moment the app starts up, the app boosts the CPUs as high as they will go for a few seconds to do all that startup stuff. If they really wanted the benchmarks to reflect what that app does they might be tempted to externally boost the benchmarks as well.
I'm an Apple guy but just thought I'd throw this out there...
Let the flames begin!!
If someone is surprised Humans are willing to cheat, rip off, etc to get ahead... well you haven't really been paying attention.
Fixed that for you.
I wouldn't want people to unfairly categorize you as a racist moron.
Since when did Korean become a race?
Ok, I remember reading the Apple benchmarks myself (in utter disbelief - even for Apple it seemed too much), and this article you linked to does not agree with my memory. So let's go directly to the source. Read that benchmark paper yourself on archive.org : http://web.archive.org/web/20030727103031/http://veritest.com/clients/reports/apple/apple_performance.pdf
I gave it a quick look to refresh my memory and here are some highlights:
- They DISABLE hyper-threading on the SPEC rate test, which is the multi-processor test. Then, they ENABLE hyper-threading on the SPEC base, which is the single-processor test!!! They defend this by saying something like "hyper-threading is slower some times". Well, they sure know that, since they only enable it when it will slow down the Pentium! I would have given them the benefit of doubt if they had disabled (or enabled) it for both tests, but selectively enabling/disabling it means you know what you are doing.
- They use -O3 -fast -ffast when compiling for Apple, which uses fast math non-IEEE optimizations. Of course they had the Intel CPU run accurate/IEEE spec code - there is no equivalent -ffast-math used.
- They go on making some other "crazy" optimizations on the G5 like "modify CPU registers to enable memory Read By-pass", or installing a special malloc library that optimizes for speed by sacrificing memory just for the single-threaded benchmark. This is not how you benchmark for comparison purposes, especially if your optimizations for the competing platform are "turning off update" and "turning off hard drive sleep" (they obviously put that stuff just to pretend they "optimized" there as well).
And I am sure there are other things as well, this was from a quick read. And of course let's not mention that they compare the G5 with an Intel P4 CPU, when, at the time, AMD's Athlons/Opterons (64bit versions were just out as well) were destroying Intel (in performance, not sales - but that is another story).
In general, that paper is so ridiculous that I can't believe Apple had kept promoting it after they had been outed. But then again, given Apple's target audience, the explanation is simple. What was even more ridiculous is that when Apple started selling the Intel-based Mac they had kept for a while the section of their website that showed how much faster the G5 Mac was compared to Intel and then on the Intel Mac pages they had comparisons which showed how the Intel Mac is faster than the G5 Mac. No shame!
Violence is the last refuge of the incompetent. Polar Scope Align for iOS
I've worked on these kinds of platforms before, we get given KPIs that we need to hit for a selection of benchmarks. Quite often you need to individually tweak these adjusting memory usage and specific settings to massage them a bit. Bumping up the GPU frequency is a bit of a cheat, but if the hardware is capable of them then is it really a massive issue? It could just be that the test doesn't make best use of the hardware by default.
If someone is surprised Humans are willing to cheat, rip off, etc to get ahead... well you haven't really been paying attention.
Fixed that for you.
I wouldn't want people to unfairly categorize you as a racist moron.
Korean is an ethnicity and a culture. It is not a separate "race". I don't think anyone mentioned a race anywhere. Is it not fair to comment on attitudes of certain cultures when it comes to honesty? Sure, it is stereotyping but, again, it is about a "culture" rather than a "race".
Jesus was a compassionate social conservative who called individuals to sin no more.
Why not Santa Clause or the chupacabra while we're talking about things that don't exist? All kinds of people like to bitch about "Apple Fanboys" on Slashdot, but can never point to any examples when pressed.
Samsung has been known to be unusually corrupt, even for Korea, for quite a while. Here are some links to discussions about it
http://lanle.wordpress.com/2010/04/28/book-on-samsung-divides-korea/
http://www.theverge.com/2012/11/30/3709688/samsung-25-years-lee-kun-hee
http://www.engadget.com/2013/09/26/mobilebench-benchmarks-samsung/ - "MobileBench group aims to improve mobile benchmarking, recruits Samsung but lacks Qualcomm, NVIDIA "
Of course news about a fake are Fake News.
You could even have most people calling Jobs an asshole, like when they pulled IDG books from Apple stores in a fit of pique over "iCon", but you'd still have the finger waggers come in saying "Now if this were Microsoft..."
Samsung isn't merely a corporate entity. It's also a group of people running the corporate entity. They don't get to fob off all their responsibility, even if the law says they do (while still retaining all of the rights, somehow).
If Americans were categorized as Apple, then we'd make things look pretty, ship all the manufacturing overseas, claim it was always the best even if it's obviously not, if you don't agree: it's our way or the highway, as you are holding it wrong, claim everyone copied them and compete with lawyers rather than competition. ... that's kinda depressing as too much of it is too close to the truth.
I can guaranty that if Apple was in this scenario 90% of the commenters in this thread would have a very different reaction.
This is like car manufacturers taping up all of the air intakes and door seams when they do the fuel economy runs. Everybody knows the numbers are cooked. Nobody expects your 3L V8 to actually get 30mpg, ever.
Finally had enough. Come see us over at https://soylentnews.org/
I use Samsung mobile phone S4, really i liked the features of this smart phone. i think, it is better than Galaxy Note.