Go Back   Android Forums > Android Discussion > Android Lounge
Android Lounge A place for general Android discussion and questions.

Get excited for the Samsung Galaxy S5! Find everything you need and discuss it in our Galaxy S5 Forum!

Like Tree15Likes
  • 4 Post By Hadron
  • 1 Post By SiempreTuna
  • 2 Post By AntimonyER
  • 1 Post By boathead
  • 1 Post By funkylogik
  • 1 Post By SiempreTuna
  • 1 Post By boathead
  • 1 Post By SiempreTuna
  • 1 Post By Slug
  • 1 Post By AntimonyER
  • 1 Post By Hadron

test: Reply
 
LinkBack Thread Tools
Old July 30th, 2013, 03:28 PM   #1 (permalink)
AF Contributor
Thread Author (OP)
 
Hadron's Avatar
 
Join Date: Aug 2010
Location: Dimension Jumping
Posts: 11,533
 
Device(s): HTC One (S-Off), HTC Desire (retired)
Carrier: Orange UK

Thanks: 2,150
Thanked 4,771 Times in 3,466 Posts
Default S4 hard-coded to boost benchmark app scores

Well we all know that benchmarks are a useful marketing tool but have little relevance to real world usage. If what Anandtech have dug out is true, it seems that's even more true with the Galaxy S4, and especially the Exynos version:

AnandTech | Looking at CPU/GPU Benchmark Optimizations in Galaxy S 4

The main points are that:

* the GPU in the Exynos version overclocks when running certain benchmark apps to a higher frequency than will be reached when running anything else, no matter how intensive

* the CPU governor behaves similarly: although it doesn't overclock, it does switch to the A15s with a fixed minimum frequency the moment you start these apps, no matter how low the load/

* the Qualcomm variants are similar: load the same apps and they run all 4 cores at max clock, even if you aren't actively running the benchmark (e.g. are just sitting looking at the benchmark app's menu)

* there's software on the phones controlling this behaviour with a hard-coded list of benchmark apps triggering it.

So it seems that at least one manufacturer has recognised the importance of benchmark scores to some customers and hard-coded to maximise performance in some popular benchmarks. You can argue about the CPU governor trickery if you like (that cpu governor setting doesn't reflect real-world behaviour, but it's not actually exceeding the maximum the system could produce if loaded hard enough), but overclocking the GPU above what you can ever get outside of the benchmark app smells to me.

Of course we don't know whether Sammy are the only ones to have thought of this wheeze, just that they seem to have been caught. But as Anand and Brian say, it would be better if the work that went into this had been devoted to optimising real world performance.

Moral: take benchmark scores with an even larger pinch of salt than ever .

__________________
Forum Rules & Guidelines - Android Forums FAQ
If a post helps you, use the Thanks! button.
Spam or offensive? Don't respond, report it /!\
Hadron is online now  
Last edited by Hadron; July 30th, 2013 at 03:35 PM.
Reply With Quote
The Following 4 Users Say Thank You to Hadron For This Useful Post:
Davdi (August 3rd, 2013), El Presidente (July 30th, 2013), funkylogik (July 30th, 2013), ocnbrze (August 1st, 2013)
sponsored links
Old July 30th, 2013, 05:17 PM   #2 (permalink)
share the love peeps ;)
 
funkylogik's Avatar
 
Join Date: Sep 2011
Location: Paisley, Scotland, Western Europe
Gender: Male
Posts: 15,276
 
Device(s): Galaxy S3 (I9300) Carbon nightlies (4.4.2) Nexus 7 (2012) Validus by Ground Zero Roms (4.4.2), Fr
Carrier: 3 (three) UK £15 p/m unltd data/txt

Thanks: 10,955
Thanked 3,878 Times in 3,111 Posts
Default

Thanks man. i always suspected this. Bet AnTuTu is one of the benchmarks since a lot of reviewers use it?
I prefer Geekbench 2 but i do take them with a pinch of salt.
I think theyre only useful when comparing your own phone with itself. Like when you change the kernel or its settings
__________________
http://giffgaff.com/orders/affiliate/funkylogik
For a UK prepay sim with unlimited data, unlimited text, 250mins for £12p/m, no contract, no catch (runs on o2 network)
funkylogik is online now  
Reply With Quote
Old July 31st, 2013, 05:48 AM   #3 (permalink)
Senior Member
 
Join Date: Jan 2013
Posts: 3,444
 
Device(s):
Carrier: Not Provided

Thanks: 45
Thanked 828 Times in 744 Posts
Default Samsung CHEATING in Benchmarks?

Seems that Samsung have been caught manipulating benchmark results. This soon after the Intel benchmark manipulation fuss.

It's the 90s redub!

Basically, Samsung's S4 has a built-in check to identify when it is running common benchmarking tools. When it spots these, it cranks up the GPU and CPU speeds beyond those they normally run at (at least, that's my understanding). Obviously, this means the benchmarks are measuring speeds that the phone is capable of running at, but doesn't actually run at - for me, that kinda counts as cheating, but it's a matter of opinion, I guess.

Thought this might be of interest as, unlike most phone buyers, I would guess there people on here who actually look at benchmarks before selecting which phone to buy.
saptech likes this.
SiempreTuna is offline  
Last edited by SiempreTuna; July 31st, 2013 at 05:52 AM.
Reply With Quote
The Following User Says Thank You to SiempreTuna For This Useful Post:
funkylogik (July 31st, 2013)
Old July 31st, 2013, 05:58 AM   #4 (permalink)
I ain't nobody!
 
saptech's Avatar
 
Join Date: Dec 2011
Location: Third Stone from the Sun
Posts: 3,544
 
Device(s): Motorola Moto G, Samsung Stratosphere, Galaxy Tab 2 SE.
Carrier: T-Mobile

Thanks: 572
Thanked 768 Times in 612 Posts
Default

Just remember, the bigger they are the worser they get.
__________________
Like it is...Yusef Lateef
saptech is offline  
Reply With Quote
Old July 31st, 2013, 07:34 AM   #5 (permalink)
AF Addict
 
AntimonyER's Avatar
 
Join Date: Jun 2010
Location: Statesboro, GA
Posts: 13,546
 
Device(s): Droid DNA, Nexus 7 16GB
Carrier: Verizon

Thanks: 5,016
Thanked 9,309 Times in 4,999 Posts
Default

Thread merged with one already in existence

Just one more reason to not make a phone purchase decision based on benchmark scores IMO.
ocnbrze and Hadron like this.
__________________
Site Rules & Guidelines
Got some help? Hit Thanks!
See something you like? Hit Like!
See something you hate? Hit Report!
AntimonyER is online now  
Reply With Quote
The Following User Says Thank You to AntimonyER For This Useful Post:
ocnbrze (August 1st, 2013)
Old July 31st, 2013, 07:57 AM   #6 (permalink)
share the love peeps ;)
 
funkylogik's Avatar
 
Join Date: Sep 2011
Location: Paisley, Scotland, Western Europe
Gender: Male
Posts: 15,276
 
Device(s): Galaxy S3 (I9300) Carbon nightlies (4.4.2) Nexus 7 (2012) Validus by Ground Zero Roms (4.4.2), Fr
Carrier: 3 (three) UK £15 p/m unltd data/txt

Thanks: 10,955
Thanked 3,878 Times in 3,111 Posts
Default

Benchmarks will never show how optimised a phone is. You have to experience it for yourself
funkylogik is online now  
Reply With Quote
Old July 31st, 2013, 08:03 AM   #7 (permalink)
Member
 
Join Date: May 2013
Posts: 242
 
Device(s):
Carrier: Not Provided

Thanks: 32
Thanked 69 Times in 56 Posts
Default

some of you may recall my recent foray with a clone phone.....near as i can tell, none of the benchmark apps that i tried got things right. antutu, quadrant, cpu z. they were all fooled in one area or another.
funkylogik likes this.
boathead is online now  
Reply With Quote
Old July 31st, 2013, 08:09 AM   #8 (permalink)
share the love peeps ;)
 
funkylogik's Avatar
 
Join Date: Sep 2011
Location: Paisley, Scotland, Western Europe
Gender: Male
Posts: 15,276
 
Device(s): Galaxy S3 (I9300) Carbon nightlies (4.4.2) Nexus 7 (2012) Validus by Ground Zero Roms (4.4.2), Fr
Carrier: 3 (three) UK £15 p/m unltd data/txt

Thanks: 10,955
Thanked 3,878 Times in 3,111 Posts
Default

I remember are you using the phone or is it a paper weight now?
boathead likes this.
funkylogik is online now  
Reply With Quote
Old July 31st, 2013, 08:25 AM   #9 (permalink)
Senior Member
 
Join Date: Jan 2013
Posts: 3,444
 
Device(s):
Carrier: Not Provided

Thanks: 45
Thanked 828 Times in 744 Posts
Default

Quote:
Originally Posted by funkylogik View Post
I remember are you using the phone or is it a paper weight now?
Put a Chinese clone phone on paper

Those things are basically electronic Molotov cocktails!

Be much safer in the

funkylogik likes this.
SiempreTuna is offline  
Reply With Quote
Old July 31st, 2013, 08:28 AM   #10 (permalink)
Member
 
Join Date: May 2013
Posts: 242
 
Device(s):
Carrier: Not Provided

Thanks: 32
Thanked 69 Times in 56 Posts
Default

i am using it! i got one of those cpu control apps that you recommended. i have things turned waaaay down for most things and the battery is at least now doing a a fair job. and i have the cpu zooming for a few apps. the device actually is pretty snappy if i let the cpu motor along.
funkylogik likes this.
boathead is online now  
Reply With Quote
The Following User Says Thank You to boathead For This Useful Post:
funkylogik (July 31st, 2013)
sponsored links
Old July 31st, 2013, 08:58 AM   #11 (permalink)
share the love peeps ;)
 
funkylogik's Avatar
 
Join Date: Sep 2011
Location: Paisley, Scotland, Western Europe
Gender: Male
Posts: 15,276
 
Device(s): Galaxy S3 (I9300) Carbon nightlies (4.4.2) Nexus 7 (2012) Validus by Ground Zero Roms (4.4.2), Fr
Carrier: 3 (three) UK £15 p/m unltd data/txt

Thanks: 10,955
Thanked 3,878 Times in 3,111 Posts
Default

Lol cool. Worth the money then?
funkylogik is online now  
Reply With Quote
Old July 31st, 2013, 09:07 AM   #12 (permalink)
Senior Member
 
Join Date: Jun 2010
Posts: 2,598
 
Device(s): Galaxy S4, Nexus 7
Carrier: giffgaff

Thanks: 136
Thanked 285 Times in 251 Posts
Default

Doesn't surprise me, some benchmarks make the device very warm. Much more so than just playing a game and the benchmark only lasts a few minutes at most.

Also with CPU throttling it's never at highest clockspeeds on the snapdragon version for very long anyway so I don't think that's an issue, or rather it doesn't really matter. Actually I'm happy they're doing this as it saves me forcing the performance governor when running benchmarks.

I'm just wondering how they're changing the GPU clockspeeds on Adreno 320, it's 450MHz max so is that for benchmarks only as well or is that left alone? Didn't see any mention of that.
Shocky is online now  
Last edited by Shocky; July 31st, 2013 at 09:11 AM.
Reply With Quote
Old July 31st, 2013, 09:31 AM   #13 (permalink)
AF Contributor
Thread Author (OP)
 
Hadron's Avatar
 
Join Date: Aug 2010
Location: Dimension Jumping
Posts: 11,533
 
Device(s): HTC One (S-Off), HTC Desire (retired)
Carrier: Orange UK

Thanks: 2,150
Thanked 4,771 Times in 3,466 Posts
Default

From the article I got the impression that it was easier to check the clock speed of the Exynos's GPU than most, so I'm not sure that was tested for the Adreno.
Hadron is online now  
Reply With Quote
The Following User Says Thank You to Hadron For This Useful Post:
funkylogik (July 31st, 2013)
Old August 1st, 2013, 01:32 AM   #14 (permalink)
Senior Member
 
Shotgun84's Avatar
 
Join Date: May 2011
Location: Cambridgeshire, England
Posts: 5,243
 
Device(s): Galaxy S4 GT-I9505 ROM: WanamLite, Kernel: KT-SGS4
Carrier: 3UK

Thanks: 1,612
Thanked 3,034 Times in 1,814 Posts
Default

That's exactly why I don't bother with benchmarks. You'll even see some devs set up their kernels purposely to produce bad benchmark results so everyone stops going on about them.
Shotgun84 is offline  
Reply With Quote
Old August 1st, 2013, 04:05 AM   #15 (permalink)
DON'T PANIC!!!!!!!!!
 
ocnbrze's Avatar
 
Join Date: May 2010
Location: los angeles
Gender: Male
Posts: 22,554
 
Device(s): Galaxy Note 3, HTC Evo 4g LTE(no longer have), HTC 3vo(no longer have), HTC EVO (not activated)
Carrier: Sprint

Thanks: 28,912
Thanked 13,990 Times in 8,951 Posts
ocnbrze@gmail.com
Default

what is a benchmark?
__________________
“The Guide is definitive. Reality is frequently inaccurate.”.......Hitchhiker's Guide to the Universe
“I seldom end up where I wanted to go, but almost always end up where I need to be.”.......Douglas Adams
ocnbrze is offline  
Reply With Quote
Old August 1st, 2013, 04:59 AM   #16 (permalink)
Senior Member
 
Join Date: Jun 2010
Posts: 2,598
 
Device(s): Galaxy S4, Nexus 7
Carrier: giffgaff

Thanks: 136
Thanked 285 Times in 251 Posts
Default

Quote:
Originally Posted by Shotgun84 View Post
That's exactly why I don't bother with benchmarks. You'll even see some devs set up their kernels purposely to produce bad benchmark results so everyone stops going on about them.
A agree you shouldn't take benchmark results too seriously, however limiting max GPU clockspeeds only for benchmarks is very underhanded.

If the GPU can do 533/450MHz as advertised then I expect that in every 3d game I play, not just benchmarks.
Shocky is online now  
Reply With Quote
Old August 1st, 2013, 04:59 AM   #17 (permalink)
Senior Member
 
Join Date: Jan 2013
Posts: 3,444
 
Device(s):
Carrier: Not Provided

Thanks: 45
Thanked 828 Times in 744 Posts
Default

Quote:
Originally Posted by ocnbrze View Post
what is a benchmark?
It's a bit of dirt on a bench.

I believe that's the most useful definition ..
ocnbrze likes this.
SiempreTuna is offline  
Reply With Quote
The Following User Says Thank You to SiempreTuna For This Useful Post:
ocnbrze (August 1st, 2013)
Old August 1st, 2013, 07:23 AM   #18 (permalink)
AF Contributor
Thread Author (OP)
 
Hadron's Avatar
 
Join Date: Aug 2010
Location: Dimension Jumping
Posts: 11,533
 
Device(s): HTC One (S-Off), HTC Desire (retired)
Carrier: Orange UK

Thanks: 2,150
Thanked 4,771 Times in 3,466 Posts
Default

Quote:
Originally Posted by Shocky View Post
If the GPU can do 533/450MHz as advertised then I expect that in every 3d game I play, not just benchmarks.
Sammy have issued a response in which (of course) they claim it's about optimising overall experience rather than gaming benchmarks - though they don't address why the code is called "benchmark booster"

The wording is a bit misleading IMO. It implies that they throttle down the gpu for some games, whereas it's actually only unthrottled for a few apps. And if that's the case, why include benchmarks on the short list of privileged apps - irrelevant to user experience and misleading if most apps have lower gpu performance. So nice try, but I'm not convinced.

The best outcome would be if blogosphere reviewers took the hint and stopped reporting synthetic benchmark scores. Unfortunately not very likely.
Hadron is online now  
Reply With Quote
Old August 1st, 2013, 07:51 AM   #19 (permalink)
AF Addict
 
AntimonyER's Avatar
 
Join Date: Jun 2010
Location: Statesboro, GA
Posts: 13,546
 
Device(s): Droid DNA, Nexus 7 16GB
Carrier: Verizon

Thanks: 5,016
Thanked 9,309 Times in 4,999 Posts
Default

Yeah, sorry Samsung, you've been caught. Curious thing will be if they do it again on the S5.
AntimonyER is online now  
Reply With Quote
The Following User Says Thank You to AntimonyER For This Useful Post:
ocnbrze (August 1st, 2013)
Old August 1st, 2013, 08:20 AM   #20 (permalink)
Senior Member
 
Join Date: Jan 2013
Posts: 3,444
 
Device(s):
Carrier: Not Provided

Thanks: 45
Thanked 828 Times in 744 Posts
Default

Quote:
Originally Posted by AntimonyER View Post
Yeah, sorry Samsung, you've been caught. Curious thing will be if they do it again on the S5.
I'm guessing they'd at least take a little more care to hide it .. and maybe not give it such an incriminating name
SiempreTuna is offline  
Reply With Quote
sponsored links
Old August 1st, 2013, 09:24 AM   #21 (permalink)
AF Contributor
Thread Author (OP)
 
Hadron's Avatar
 
Join Date: Aug 2010
Location: Dimension Jumping
Posts: 11,533
 
Device(s): HTC One (S-Off), HTC Desire (retired)
Carrier: Orange UK

Thanks: 2,150
Thanked 4,771 Times in 3,466 Posts
Default

OK, am I joining too many dots here?

Another Anandtech article on updates to the Exynos Octa notes that there was a bug in the original version of the SoC (the one used in the Exynos S4):

Quote:
Part of the problem with the design was a broken implementation of the CCI-400 coherent bus interface that connect the two CPU islands to the rest of the SoC. In the case of the 5410, the bus was functional but coherency was broken and manually disabled on the Galaxy S 4. The implications are serious from a power consumption (and performance) standpoint. With all caches being flushed out to main memory upon a switch between CPU islands. Neither ARM nor Samsung LSI will talk about the bug publicly, and Samsung didn't fess up to the problem at first either - leaving end users to discover it on their own.
So that means that there's a significant penalty when switching between the two core clusters. And one of the things this "benchmark booster" does is forces the system to use the A15 cores the moment you start a benchmark app. So one consequence is that for apps on the benchmark booster whitelist this bug will not be visible - part of the motivation, or just a fringe benefit?
Hadron is online now  
Reply With Quote
The Following User Says Thank You to Hadron For This Useful Post:
funkylogik (August 1st, 2013)
Old August 1st, 2013, 10:39 AM   #22 (permalink)
Senior Member
 
Join Date: Jan 2013
Posts: 3,444
 
Device(s):
Carrier: Not Provided

Thanks: 45
Thanked 828 Times in 744 Posts
Default

There's no such thing as coincidence .. except when there is
SiempreTuna is offline  
Reply With Quote
Old August 1st, 2013, 12:27 PM   #23 (permalink)
share the love peeps ;)
 
funkylogik's Avatar
 
Join Date: Sep 2011
Location: Paisley, Scotland, Western Europe
Gender: Male
Posts: 15,276
 
Device(s): Galaxy S3 (I9300) Carbon nightlies (4.4.2) Nexus 7 (2012) Validus by Ground Zero Roms (4.4.2), Fr
Carrier: 3 (three) UK £15 p/m unltd data/txt

Thanks: 10,955
Thanked 3,878 Times in 3,111 Posts
Default

I knew samsung didnt have the octocore working the way Arm demonstrated it but trying to hide that in that way is pretty low. Have they ever admitted the s3 "SDS" bug?
funkylogik is online now  
Reply With Quote
Old August 3rd, 2013, 04:27 AM   #24 (permalink)
Senior Member
 
Join Date: Jun 2010
Posts: 2,598
 
Device(s): Galaxy S4, Nexus 7
Carrier: giffgaff

Thanks: 136
Thanked 285 Times in 251 Posts
Default

Quote:
Originally Posted by Hadron View Post
The best outcome would be if blogosphere reviewers took the hint and stopped reporting synthetic benchmark scores. Unfortunately not very likely.
But how do they test performance if not for benchmarks? you can't just say X is faster because it feels smooth. Nobody would take them seriously if they did that.
Shocky is online now  
Reply With Quote
Old August 3rd, 2013, 05:17 AM   #25 (permalink)
Check six!
 
Slug's Avatar
 
Join Date: Aug 2009
Location: Inverness, UK
Gender: Male
Posts: 17,416
 
Device(s): Sony Xperia Z
Carrier: EE

Thanks: 2,651
Thanked 10,869 Times in 5,525 Posts
Send a message via Skype™ to Slug slugbrem@gmail.com
Default

Quote:
Originally Posted by Shocky View Post
you can't just say X is faster because it feels smooth. Nobody would take them seriously if they did that.
Well we sure can't take them seriously if manufacturers are rigging the results either.
Hadron likes this.
Slug is offline  
Reply With Quote
Old August 3rd, 2013, 05:21 AM   #26 (permalink)
AF Addict
 
AntimonyER's Avatar
 
Join Date: Jun 2010
Location: Statesboro, GA
Posts: 13,546
 
Device(s): Droid DNA, Nexus 7 16GB
Carrier: Verizon

Thanks: 5,016
Thanked 9,309 Times in 4,999 Posts
Default

Quote:
Originally Posted by Shocky View Post
But how do they test performance if not for benchmarks? you can't just say X is faster because it feels smooth. Nobody would take them seriously if they did that.
But how can you take them seriously if they do report them, now that we see at least one manufacturer gaming them. Besides, at the end of the day, I don't care about the benchmark score, but I do care about fast it "feels".

Wow, how slow am I, ninja'd by a Slug.
funkylogik likes this.
AntimonyER is online now  
Last edited by AntimonyER; August 3rd, 2013 at 09:29 AM.
Reply With Quote
Old August 3rd, 2013, 05:41 AM   #27 (permalink)
AF Contributor
 
Join Date: Jul 2012
Location: The Shire
Posts: 810
 
Device(s): HTC Wildfire S x 2, Xperia U, Huawei Y300, N7, Generic Allwinner tab, Bluestacks
Carrier: Not Provided

Thanks: 93
Thanked 206 Times in 154 Posts
Default

'How fas it feels' is about right.
I use 2 phones, A stock, unrooted, Xperia U and a Wildfire S which is rooted, running the WildChild3 rom and O/C to 806MHz. But, the WFS with its single core at 89% of the Sony's dual core speed, still feels much more responsive in general use. Even running stock GB/Sense 2.1 it was less laggy.
Davdi is offline  
Reply With Quote
Old August 3rd, 2013, 09:22 AM   #28 (permalink)
Senior Member
 
Join Date: Jun 2010
Posts: 2,598
 
Device(s): Galaxy S4, Nexus 7
Carrier: giffgaff

Thanks: 136
Thanked 285 Times in 251 Posts
Default

Quote:
Originally Posted by Slug View Post
Well we sure can't take them seriously if manufacturers are rigging the results either.
Maybe its time for game benchmarking, but I can't see reviewers going through the effort to root and record framerates.

Or maybe find a way to make the phone treat the benchmarks like any other app, i'm sure some devs out there can figure out a way round it.

Benchmarks themselves are not the issue.

Also one could argue they never said it would always be at 533MHz in 3d applications, much like the CPU in the snapdragon version is never always at 1.9GHz, it has to clock down eventually or it will overheat, same is true of the GPU so one could call it optimized.


Quote:
Originally Posted by AntimonyER View Post
I don't care about the benchmark score, but I do care about fast it "feels".
But you can't measure how something "feels", we all have our own opinion on what we call smooth.
Shocky is online now  
Last edited by Shocky; August 3rd, 2013 at 09:38 AM.
Reply With Quote
Old August 3rd, 2013, 09:31 AM   #29 (permalink)
AF Contributor
Thread Author (OP)
 
Hadron's Avatar
 
Join Date: Aug 2010
Location: Dimension Jumping
Posts: 11,533
 
Device(s): HTC One (S-Off), HTC Desire (retired)
Carrier: Orange UK

Thanks: 2,150
Thanked 4,771 Times in 3,466 Posts
Default

Well if the manufacturer codes for specific apps it will probably be more work to do anything about that than to root the phone.

The problem is that whatever test you use, you want consistency. So if it's based on specific apps (games or benchmarks) then if the same ones are widely used, or used consistently by some influential reviewers, the same temptation is there.

I guess you'd need a performance test that's broad enough that it's less effort to just optimise performance overall . In other words, a suite of tests that really represent usage without relying on one or four apps. But that's more effort for the reviewer.
funkylogik likes this.
Hadron is online now  
Reply With Quote
Reply


Go Back   Android Forums > Android Discussion > Android Lounge
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -5. The time now is 05:06 PM.
Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.