Samsung Caught Boosting Galaxy S4 Benchmarks

A recent forum post at Beyond3D made an interesting claim: that the Samsung Galaxy S4's GPU ran at 532 MHz for certain whitelisted benchmark applications, and at 480 MHz for everything else. The folks at AnandTech decided to investigate and found out that the phone does indeed let its GPU run at a higher frequency when particular benchmark software is running. They found a similar oddity with the CPU — it wasn't restricted for other apps, but it was forced to run at max speed during benchmarks. Then they decided to look for direct evidence that this was intentional. "Poking around I came across the application changing the DVFS behavior to allow these frequency changes – TwDVFSApp.apk. Opening the file in a hex editor and looking at strings inside (or just running strings on the .odex file) pointed at what appeared to be hard coded profiles/exceptions for certain applications. The string 'BenchmarkBooster' is a particularly telling one. ... Quadrant standard, advanced, and professional, linpack (free, not paid), Benchmark Pi, and AnTuTu are all called out specifically. Nothing for GLBenchmark 2.5.1 though, despite its similar behavior."

Comments

Government Regulation

[ElementOfDestruction]

When every sixth topic on Slashdot is about the evils and perils of Government Regulation, why are we constantly seeing examples of companies misleading, blatantly lying, to their customers? We need more teeth on consumer regulation. I bought my Samsung Galaxy S4 on certain assumptions of power. Remember Hyundai blatantly lying about their fuel numbers for half a decade? They were doled out a punishment, but the boost in sales due to in part by their chain-wide efficiency offset any net losses.

Slashdot readers will remember this, and probably choose an S4 when faced with so few choices. Samsung sees no benefit to not skewing numbers in the future.

Re:Government Regulation

[Bornhuetter]

Arguments about whether "more regulation" is good or bad are in my experience almost always misguided. We need better regulation, but that is much harder to define, and doesn't make for as good a soundbite.

Re: Government Regulation

[Anonymous Coward]

And in my house:

Sony = Pony
Panasonic = Wankasonic
Apple = Crapple
Nokia = Cock-ia

I live alone.

Re:Government Regulation

[__aaltlg1547]

You bought the phone on certain assumptions of power. OK, that's fine. Would a difference of at most 10.8% difference in the benchmark results have changed your choice? Or would you still have bought the Galaxy S4 for its features? Do your applications run acceptably fast? Has the slow speed made you wish you had bought a different phone?

This is not at all the same thing that Hyundai did. They actually lied about their test results from standard tests and were caught when the EPA did the same test and got much lower numbers. If you run the benchmarks on your phone, you will apparently get the same numbers as Samsung published, even if nothing else on your phone is allowed to run that fast.

I'm not saying what Samsung did was right. I'm just saying you're exaggerating the significance.

Re: Government Regulation

[serviscope_minor]

Sony = Pony

What's wrong with ponies? :(

Re: Government Regulation

[Sockatume]

Pony and trap, crap. AC is a cockney.

Re:Government Regulation

[ElementOfDestruction]

It's about getting lied to. It fucking sucks. If the operates 10% slower than it performs on tests, I would expect - at the least - an explanation or a discount off the outrageous selling price. Yes, I really don't care if it operates at peak performance, the way I use my phone I suspect I wouldn't see any performance differential.

But I wouldn't want to buy a Delorean advertised to be capable of going 95 mph, only to find out that it can go 95 mph when it's being timed on a closed course; when normally used, it can only physically run at 86mph. I need 88 mph in a mall parking lot, otherwise the mother fucking Libyans will get me.

Re:Government Regulation

[MachineShedFred]

And Samsung still wouldn't care, evidenced by past behavior (otherwise known as the best predictor of future behavior):

Samsung could face 15B Euro fine
Samsung, LG fined for LCD price fixing
Tax evasion, bribery, and price fixing: how Samsung became the giant that ate Korea
Samsung agrees to plead guilty to DRAM price fixing, pay $300M fine
6 Samsung executives headed to jail for price fixing
Samsung, LG fined for mobile price fixing scheme

Everyone is holding these guys up to be some kind of saints in their battle against the evil Apple Empire, when they are thrice-convicted price fixers that screw their customers over at every opportunity, legal or otherwise; and try to screw the competition by suing over standards-essential patents that they don't license for FRAND terms (allegedly).

Samsung is not a friendly company, but I'll likely be modded down for saying so. Whatever, I've got the karma to burn.

Re:Government Regulation

[phantomfive]

When every sixth topic on Slashdot is about the evils and perils of Government Regulation,

The argument over 'more regulation' vs 'less regulation' is about the stupidest argument out there. It's so unnuanced, and the arguments are based primarily on campaign soundbytes, that I just hang my head and cry everytime I hear someone get in that argument.

The argument is so simple that both sides are right:
1) YES, we need more regulation, good regulation that improves the world. Also,
2) YES, we need less regulation, less bad regulation that hurts the economy for no real benefit.

Every regulation has a cost and a benefit. You need to look at the specific regulation and decide whether the cost is worth the benefit, or whether there's a better way to achieve the same goal. You can't just say, "it's regulation so it's bad!" or "It's regulation so it's good!" but I guess that's too hard for most people.

Then they feel outraged when bad things happen.

Re:Government Regulation

[Anonymous Coward]

I'm modding you down for being one of those dicks that says "I'll likely be modded down for saying so. Whatever, I've got the karma to burn.".
Say what you have to say and let it stand instead of meta commenting.

Re:Government Regulation

[immaterial]

No, because the auto manufacturer is not using that as a way to boost the apparent top speed of the car. Also, you clearly have no idea how a speedometer works or what it's intended accuracy is: about 3% leeway is average, almost exclusively leaning toward the high side (imagine the legal/safety issues if your car duped you into speeding). The speedometer ultimately estimates your speed based on the rpm of your tires against the road - your tire size, the amount of wear on the tire, and the inflation pressure all have an effect on this rpm (they all affect the effective diameter of the tire). There is no practical way to get a 100% accurate speedometer with these constantly varying conditions, so manufacturers tend to err on the side of caution. This well-known information and not some kind of conspiracy to inflate top speeds (which are tested on a track using external equipment, not the car's speedometer!).

Re: And you think they're the only one why?

[Anonymous Coward]

Source, proof, evidence or STFU.

Re:And you think they're the only one why?

[beltsbear]

No, they don't. There is a difference between optimizing a system and overclocking just for specific benchmark apps. Samsung could get fraud charges on this one if they advertised or published the benchmarked speed. It is less obvious if they did not do the publishing themselves.

Sony, for example

[ElementOfDestruction]

Sony has betrayed consumers more than almost any Tech company can name. They're universally hated across all spectrum of Slashdot users.

Yet they're largely poised to win the next "Console War" and they're still one of the premier names in the home entertainment business.

Companies have NOTHING to fear from consumer retaliation. Consumers are by and large stupid, with an extremely short term memory.

Re:Sony, for example

[Andy Dodd]

Part of that is that unlike his predecessor (Stringer) - Hirai realizes that treating your customers like shit is bad for business.

Sony under Hirai is very different from Sony under Stringer - this is most obvious if you look at Sony Mobile, who are one of the largest contributors to the Android Open Source Project (AOSP) and the only manufacturer that actively maintains AOSP builds for some of their devices. There are numerous signs that, rather than squash the anomalous behavior of the former Sony-Ericsson, the rest of Sony is starting to adopt Mobile's ways.

Re:Sony, for example

[AmiMoJo]

Consumers aren't stupid, they just don't care. How many do you think have even heard of the Sony rootkit, or realized what the feature-disabling PS3 update was? The most high profile case was when PSN was hacked, but consumers see them as victims because the only time they are ever taught about computer security is when they go to the movies and see some elite hacker easily breaking into a bank while getting a blowjob. There is nothing in the news or media to offset the terrible John Travolta movies.

I've seen this before

[earthforce_1]

I remember old articles where ATI and Nvidia were both caught out gaming benchmarks, in one case by embedding particular benchmark game strings in their driver, and short cutting a few algorithms to boost their score.
 

Re:And you think they're the only one why?

[Joce640k]

No different than how Samsung made tons of commercials poking fun of iPhone users. If you make a better product just show the product. If you make an inferior product then take cheap shots at the competition.

Yep. Apple would never make adverts poking fun at the competition...

Official answer from Samsung

[pieleric]

There seems to be an official answer from Samsung here: http://samsungtomorrow.com/4676

It's in Korean, but here is the translation, provided by sammobile.com:
"Under ordinary conditions, the Galaxy S4 has been designed to allow a maximum GPU frequency of 533MHz. However, the maximum GPU frequency is lowered to 480MHz for certain gaming apps that may cause an overload, when they are used for a prolonged period of time in full-screen mode. Meanwhile, a maximum GPU frequency of 533MHz is applicable for running apps that are usually used in full-screen mode, such as the S Browser, Gallery, Camera, Video Player, and certain benchmarking apps, which also demand substantial performance.

The maximum GPU frequencies for the Galaxy S4 have been varied to provide optimal user experience for our customers, and were not intended to improve certain benchmark results.

We remain committed to providing our customers with the best possible user experience."

Re:Official answer from Samsung

[Sockatume]

That doesn't tally with the information extracted from the S4 code: it lists several benchmark apps, which when detected activate a "boost" feature that changes the CPU clock.

Re:Official answer from Samsung

[ernest.cunningham]

Except that all testing thus far shows this to not be true, including the discovery of the benchmark booster....

Re:Official answer from Samsung

[Xest]

That's because the app has to tell the phone to stop saving battery and start performing at it's most optimal speed and the danger is that if the benchmark apps aren't built to do this then the benchmark apps will only give a benchmark for the phones power saving mode rather than at it's optimal performance.

There's no overclocking going on, the GPU is rated for 533mhz so running at 532mhz in that configuration isn't any kind of fudge but a genuine representation of how the phone can perform at peak.

Re:Official answer from Samsung

[Xest]

Maybe you should learn to use Google. There's plenty of quotes of 533mhz for the GPU from long before this article and Samsung's response.

One example in the comments section of this article from March which a very quick simple search dug up:

http://www.sammobile.com/2013/03/03/galaxy-s-ivs-specifications-leak-confirm-exynos-octa-powervr-sgx-544mp/

Or are you just being a fanboy and not actually interested in the facts? Because the comment you just posted is a complete lie which suggests that maybe that's the real problem you have.

Re:Official answer from Samsung

[Sockatume]

If "saving battery" is the phone's state whenever it is not running a benchmarking application, it is the phone's normal state.

Re:Official answer from Samsung

[Xest]

"How much does Samsung pay you to spread this misinformation?"

Most likely the same as Apple pays you, nothing, because I'm not a shill and presumably neither are you. Unless you are of course, in which case then they still pay me nothing.

"This thread is littered with multiple posts from you spreading the same misinformation that is clearly wrong to anyone who's read the article - the 533MHZ speed is ONLY available to benchmark apps"

What is it about Samsung's official response that confuses you so much? I think it's quite easy to understand from their simple response that they do this for more than just benchmark apps and also do it for a number of other every day apps:

"Meanwhile, a maximum GPU frequency of 533MHz is applicable for running apps that are usually used in full-screen mode, such as the S Browser, Gallery, Camera, Video Player, and certain benchmarking apps, which also demand substantial performance."

"What is the going rate for spreading misinformation in a Slashdot discussion?"

Now I've explained that I'm not a shill and that I've explained why you're wrong I have a question for you instead - what is the going rate for attacking Samsung despite being wrong? Nothing? I thought so, so why exactly do you insist on doing it?

If you really want to see someone spamming this thread go look at your fan Sockatume, who has posted many more posts backing up your incorrect position than I have correcting the both of you. I don't think he's a shill though, I just think he refuses to back down when he's clearly and demonstrably wrong and instead just likes to witter on even more as if if he posts enough he'll somehow become right, even though that wont happen because you can't change reality.

If you can prove to me that it's all a lie, and that Samsung's other apps listed above don't run at 533mhz, and that no other apps can utilise this feature other than benchmarking apps, and that all the quotes of the GPU in the S4 running at 533mhz are lies and that this was never the case and it's simply being overclocked for benchmarking then I'll gladly concede defeat and agree that Samsung are in the wrong.

Good luck with that though, but kudos if you can in fact somehow uncover such a massive deception campaign dating back to articles all the way from March.

Protip: Just because you disagree with someone doesn't make that other person a shill.

Re:Official answer from Samsung

[gnasher719]

I'm trying to understand what's wrong in making a device to run unrestrained when making the benchmarks. The very idea is to test what the device is capable of.

The idea is not to test how fast a device can run benchmarks, but to use benchmarks to be able to draw conclusions about how fast other apps would run. And this kind of manipulation means the conclusion will be wrong.

Example: I want to know how fast my far can go - but I want to know how fast it can go while still lasting a reasonable time. The manufacturer has a switch that creates 50 more horse powers but makes the engine break down after 20,000 miles. The top speed with the special switch turned on is of no relevance to me, and publishing it without a big caveat is misleading the customer.

Re:Official answer from Samsung

[cheesybagel]

You come back to me with a GPU clock speed quote that comes from actual Samsung literature and not a benchmark app or a source-less web page, and then we'll talk some more.

Sure. Here is the Samsung Exynos 5 Octa flyer.

It's there on page 2. PowerVR SGX544 MP3 533 MHz GPU.

Re: And you think they're the only one why?

[Geoffrey.landis]

Here you go

http://news.cnet.com/8301-1001_3-57593426-92/debate-sparked-about-benchmark-for-intel-arm-chips/

Meh. That link basically says that there are different results from different benchmarks. It says that it's a "not uncommon assertion" that companies "have attempted in the past to "manipulate" benchmarks", but that's not the same as finding code that overclocks the chip specifically when it's running benchmark programs

Still way behind even after stacking the deck

[the computer guy nex]

Shows how far behind Samsung is in terms of hardware engineering. They stack the deck and still can't touch a 9 month old phone. Both browser performance and gaming performance, the 2 most stressful use cases on a smartphone, are way behind Apple.

http://images.anandtech.com/graphs/graph6914/54294.png
http://images.anandtech.com/graphs/graph6914/54296.png
http://images.anandtech.com/graphs/graph6914/54300.png
http://images.anandtech.com/graphs/graph6914/54298.png
http://images.anandtech.com/graphs/graph6914/54305.png

Re:Still way behind even after stacking the deck

[the computer guy nex]

Shows how far behind Samsung is in terms of hardware engineering. They stack the deck and still can't touch a 9 month old phone. Both browser performance and gaming performance, the 2 most stressful use cases on a smartphone, are way behind Apple. http://images.anandtech.com/graphs/graph6914/54305.png

Look at your link. It shows the S4 beating the iP5. Also Sunspider is kind of weird. I think that current Windows Phones with underpowered SoCs post the best scores in more recent comparisons, and that doesn't make a lot of sense.

Regarding your other links, yes, the iP5 has oddly good GPU performance.

The S4 beats the iPhone 5 while in a freezer. It has heat dissipation issues due to poor built quality.

Re:And you think they're the only one why?

[dugancent]

That's nice. It's still unethical and should be treated as such.

I think we should end this

[TheSkepticalOptimist]

I don't think we need to celebrate benchmarking phones, period. This was one of those flamebait trolling things that happened in the PC era where people boasted how superficially fast their beloved shoebox was by putting $10k worth of equipment into and liquid cooling it just to get some high number result in 3D Mark or some other meaningless program.

We don't need this for phones.

Yes phones play games, yes phones are getting faster, but realize that phones and tablets are a HUGE step back from the PC era in terms of performance so benchmarking them means you may as well drag out your dusty Pentium era PC and start boasting about good its benchmark numbers are.

Also when 80% of the apps on the Android platform are unstable POS then I don't care about how fast they crash. Even Chrome quits unexpectedly repeatedly and this is by the company that makes the Android platform on their own Nexus brand devices.

Re:Wake up, fandroids!

[robmv]

Samsung copy everything Apple do, so it is Apple's fault :P

Re:And you think they're the only one why?

[Xest]

Except they're not overclocking anything because the GPU is rated for 533mhz.

They're just making sure that even if the benchmark apps don't tell it to work in it's most high performance profile that it does, because the whole point in benchmarks is to give a benchmark of the optimal performance of a device.

The danger is that if they don't do this then the benchmark programs will give a misleading view of the performance capabilities of the device because they'll only be running it in the more power saving oriented default mode.

What's the alternative, they don't do this and shitty benchmark apps that take no advantage of the optimisation options for the device suggest it's not as powerful as it really is and so they get slated for it being underpowered even if that's not true?

Re:And you think they're the only one why?

[Anonymous Coward]

Wow. Did you even read the article. Or even the summary? They aren't doing this to ensure the device isn't running in power saving mode. The enhanced frequency is _ONLY_ available to benchmark tests. The code even refers to it as BenchmarkBooster. What do you possibly think BenchmarkBooster does?

Seriously, you are the personification of "fandroid" right now.

Re:And you think they're the only one why?

[hattig]

But the device doesn't do 533MHz for the GPU in any other use case, the top clock for the GPU is 480MHz.

This isn't "forcing the system into optimal mode for benchmarks so that power saving, etc, doesn't futz the result".

This is forcing the GPU into a state that never can be attained by any other software on the system.

Of course I'm not ignoring the fact that another Samsung device runs at 533MHz and this was a bad cut and paste job onto the new device!

Re:And you think they're the only one why?

[Sockatume]

Does it?

Re:And you think they're the only one why?

[whisper_jeff]

It doesn't. Other applications can run at 533mhz...

No. They can't. Please try reading the article.

http://www.anandtech.com/show/7187/looking-at-cpugpu-benchmark-optimizations-galaxy-s-4

...all other apps/games were limited to 480MHz.

Other applications can NOT run at 533. The only applications that have access to that speed boost are benchmark apps.

The ONLY apps.

Please read the article before you continue spreading your misinformation.

Re:And you think they're the only one why?

[Sockatume]

Your unstated major premise is "what Samsung has told me is accurate". This is a mistake. Samsung's explanation is a rival hypothesis to Anandtech's. At the moment you have to compare the two hypotheses with the presented data. That data tends to favour Anandtech's explanation.

Re:And you think they're the only one why?

[beltsbear]

If no apps other then the benchmarker run at 533Mhz, then overclocking is a fair word even if the CPU is rated for 533Mhz. If even 20% of the apps were allowed to run at 533Mhz then I would say otherwise, but NONE of the other apps are allowed to run at 533Mhz, just the benchmarker.

It is fraud.

Re:And you think they're the only one why?

[gutnor]

The alternative is to do a benchmark of the performance that the user will really get. What's the point knowing the potential of the phone when at the end of the day it is configured in such a way that you will never reach them.

Re:And you think they're the only one why?

[Sockatume]

Which ones?

You can't handle full power!

[mveloso]

It's not overclocking, it's just that Samsung underclocks their phones to save battery and to stay within the specified thermal envelope.

Only the benchmarking apps run at full speed, because they're the only apps that need the full power of the phone at all times.

Other apps can't handle the full power of the Samsung ecosystem, thus Samsung protects them from the overwhelmingly high power coolness that is the Samsung platform.

So really, everything we do is in the best interest of our customer. We protect our customers from experiencing the full power of our phones to preserve their mental cohesiveness. Anything less would open a wormhole in the fabric of reality, and we wouldn't want that.

Re: And you think they're the only one why?

[Penguinisto]

Maybe they just need to rename the benchmarking binary (I vote quack.exe)?

I jest, but only partly. You see, there is truly nothing new under the sun.

Re:And you think they're the only one why?

[mwvdlee]

They're just making sure that even if the benchmark apps don't tell it to work in it's most high performance profile that it does

If I run a normal app (say, a game), does the GPU get this "most high performance profile" as well?

because the whole point in benchmarks is to give a benchmark of the optimal performance of a device.

The point of a benchmark is to give a benchmark of REALISTIC performance on a device, as a user would get under normal daily usage.

Re:And you think they're the only one why?

[Xest]

All the ones Samsung listed for a start:

S Browser, Gallery, Camera, Video Player

Re: And you think they're the only one why?

[Russ1642]

Steroids in sports actually improve your performance. Speeding up benchmarks only affects the benchmark, not actual device performance.

Re:And you think they're the only one why?

[Holi]

Yes it does.

Re:And you think they're the only one why?

[Holi]

An article cannot be wrong?

S browser, gallery, camera, video player all run at 533mhz
Others can as well if they need to. It's just how many developers out there are going to optimize for one phone?

Just because you don't want to do any research your self because you lazy and would prefer to insult people does not mean that anyone else lied. It just means that you took someones word without verifying it.

Now apologize the the GP for calling them a liar.

Re:And you think they're the only one why?

[Holi]

Because they are not going to hand you evidence of fraud.

Re: And you think they're the only one why?

[pipatron]

Seems to me that it works well. After some up and down while moderators battle about what would be the best option, it has come out as somewhat insightful.

If you've been on slashdot as long as you claim, you must also have seen all those newbies whining as soon as their comment is down-modded, not realizing that there are actually more than two moderators reading, and that the score will vary during the course of a day or two. Don't be one of them.

Re:And you think they're the only one why?

[arekin]

You are a liar who's spamming the hell out of this thread with your lies. The original article clearly states that 533MHz is not available for any other apps nor games - it's only available for benchmark tests.

Stop spreading lies.

http://www.telegraph.co.uk/technology/samsung/10213672/Samsung-deny-performance-boosting-hardware-in-Galaxy-S4.html The original article is wrong.

Re:And you think they're the only one why?

[Savage-Rabbit]

You know this how? There's no way to tell if Apple does this or not since their OS is closed source..

C'mon don't be so hard on poor Samsung, they copied Apple's devices, they copied the look of Apple's mobile OS and now they are seeking to copy Apple's weaselyness.

Re:regulation gulag

[phantomfive]

Oh look, another person who sees the world in black and white.