1. Download our Official Android App: Forums for Android!

S4 hard-coded to boost benchmark app scores

Discussion in 'Android Lounge' started by Hadron, Jul 30, 2013.

  1. Hadron

    Hadron Smoke me a kipper...
    VIP Member Thread Starter

    Well we all know that benchmarks are a useful marketing tool but have little relevance to real world usage. If what Anandtech have dug out is true, it seems that's even more true with the Galaxy S4, and especially the Exynos version:

    AnandTech | Looking at CPU/GPU Benchmark Optimizations in Galaxy S 4

    The main points are that:

    * the GPU in the Exynos version overclocks when running certain benchmark apps to a higher frequency than will be reached when running anything else, no matter how intensive

    * the CPU governor behaves similarly: although it doesn't overclock, it does switch to the A15s with a fixed minimum frequency the moment you start these apps, no matter how low the load/

    * the Qualcomm variants are similar: load the same apps and they run all 4 cores at max clock, even if you aren't actively running the benchmark (e.g. are just sitting looking at the benchmark app's menu)

    * there's software on the phones controlling this behaviour with a hard-coded list of benchmark apps triggering it.

    So it seems that at least one manufacturer has recognised the importance of benchmark scores to some customers and hard-coded to maximise performance in some popular benchmarks. You can argue about the CPU governor trickery if you like (that cpu governor setting doesn't reflect real-world behaviour, but it's not actually exceeding the maximum the system could produce if loaded hard enough), but overclocking the GPU above what you can ever get outside of the benchmark app smells to me.

    Of course we don't know whether Sammy are the only ones to have thought of this wheeze, just that they seem to have been caught. But as Anand and Brian say, it would be better if the work that went into this had been devoted to optimising real world performance.

    Moral: take benchmark scores with an even larger pinch of salt than ever ;).


  2. funkylogik

    funkylogik share the love peeps ;)

    Thanks man. i always suspected this. Bet AnTuTu is one of the benchmarks since a lot of reviewers use it?
    I prefer Geekbench 2 but i do take them with a pinch of salt.
    I think theyre only useful when comparing your own phone with itself. Like when you change the kernel or its settings :thumbup:
  3. SiempreTuna

    SiempreTuna Android Expert

    Seems that Samsung have been caught manipulating benchmark results. This soon after the Intel benchmark manipulation fuss.

    It's the 90s redub!

    Basically, Samsung's S4 has a built-in check to identify when it is running common benchmarking tools. When it spots these, it cranks up the GPU and CPU speeds beyond those they normally run at (at least, that's my understanding). Obviously, this means the benchmarks are measuring speeds that the phone is capable of running at, but doesn't actually run at - for me, that kinda counts as cheating, but it's a matter of opinion, I guess.

    Thought this might be of interest as, unlike most phone buyers, I would guess there people on here who actually look at benchmarks before selecting which phone to buy.
    funkylogik likes this.
  4. saptech

    saptech Android Expert

    Just remember, the bigger they are the worser they get.
  5. AntimonyER


    Thread merged with one already in existence ;)

    Just one more reason to not make a phone purchase decision based on benchmark scores IMO. :p
    ocnbrze likes this.
  6. funkylogik

    funkylogik share the love peeps ;)

    Benchmarks will never show how optimised a phone is. You have to experience it for yourself
  7. boathead

    boathead Android Enthusiast

    some of you may recall my recent foray with a clone phone.....near as i can tell, none of the benchmark apps that i tried got things right. antutu, quadrant, cpu z. they were all fooled in one area or another.
  8. funkylogik

    funkylogik share the love peeps ;)

    I remember :D are you using the phone or is it a paper weight now?
  9. SiempreTuna

    SiempreTuna Android Expert

    Put a Chinese clone phone on paper :eek:

    Those things are basically electronic Molotov cocktails! :secruity:

    Be much safer in the :bath:

  10. boathead

    boathead Android Enthusiast

    i am using it! i got one of those cpu control apps that you recommended. i have things turned waaaay down for most things and the battery is at least now doing a a fair job. and i have the cpu zooming for a few apps. the device actually is pretty snappy if i let the cpu motor along.
    funkylogik likes this.
  11. funkylogik

    funkylogik share the love peeps ;)

    Lol cool. Worth the money then? :beer:
  12. Shocky

    Shocky On Probation

    Doesn't surprise me, some benchmarks make the device very warm. Much more so than just playing a game and the benchmark only lasts a few minutes at most.

    Also with CPU throttling it's never at highest clockspeeds on the snapdragon version for very long anyway so I don't think that's an issue, or rather it doesn't really matter. Actually I'm happy they're doing this as it saves me forcing the performance governor when running benchmarks. :D

    I'm just wondering how they're changing the GPU clockspeeds on Adreno 320, it's 450MHz max so is that for benchmarks only as well or is that left alone? Didn't see any mention of that.
  13. Hadron

    Hadron Smoke me a kipper...
    VIP Member Thread Starter

    From the article I got the impression that it was easier to check the clock speed of the Exynos's GPU than most, so I'm not sure that was tested for the Adreno.
    funkylogik likes this.
  14. Shotgun84

    Shotgun84 Extreme Android User

    That's exactly why I don't bother with benchmarks. You'll even see some devs set up their kernels purposely to produce bad benchmark results so everyone stops going on about them.
  15. ocnbrze

    ocnbrze DON'T PANIC!!!!!!!!!

    what is a benchmark?:p
  16. Shocky

    Shocky On Probation

    A agree you shouldn't take benchmark results too seriously, however limiting max GPU clockspeeds only for benchmarks is very underhanded.

    If the GPU can do 533/450MHz as advertised then I expect that in every 3d game I play, not just benchmarks.
  17. SiempreTuna

    SiempreTuna Android Expert

    It's a bit of dirt on a bench.

    I believe that's the most useful definition .. ;)
    ocnbrze likes this.
  18. Hadron

    Hadron Smoke me a kipper...
    VIP Member Thread Starter

    Sammy have issued a response in which (of course) they claim it's about optimising overall experience rather than gaming benchmarks - though they don't address why the code is called "benchmark booster" ;)

    The wording is a bit misleading IMO. It implies that they throttle down the gpu for some games, whereas it's actually only unthrottled for a few apps. And if that's the case, why include benchmarks on the short list of privileged apps - irrelevant to user experience and misleading if most apps have lower gpu performance. So nice try, but I'm not convinced.

    The best outcome would be if blogosphere reviewers took the hint and stopped reporting synthetic benchmark scores. Unfortunately not very likely.
  19. AntimonyER


    Yeah, sorry Samsung, you've been caught. Curious thing will be if they do it again on the S5.
    ocnbrze likes this.
  20. SiempreTuna

    SiempreTuna Android Expert

    I'm guessing they'd at least take a little more care to hide it .. and maybe not give it such an incriminating name :stupido2:
  21. Hadron

    Hadron Smoke me a kipper...
    VIP Member Thread Starter

    OK, am I joining too many dots here?

    Another Anandtech article on updates to the Exynos Octa notes that there was a bug in the original version of the SoC (the one used in the Exynos S4):

    So that means that there's a significant penalty when switching between the two core clusters. And one of the things this "benchmark booster" does is forces the system to use the A15 cores the moment you start a benchmark app. So one consequence is that for apps on the benchmark booster whitelist this bug will not be visible - part of the motivation, or just a fringe benefit?
    funkylogik likes this.
  22. SiempreTuna

    SiempreTuna Android Expert

    There's no such thing as coincidence .. except when there is ;)
  23. funkylogik

    funkylogik share the love peeps ;)

    I knew samsung didnt have the octocore working the way Arm demonstrated it but trying to hide that in that way is pretty low. Have they ever admitted the s3 "SDS" bug?
  24. Shocky

    Shocky On Probation

    But how do they test performance if not for benchmarks? you can't just say X is faster because it feels smooth. Nobody would take them seriously if they did that.
  25. Slug

    Slug Check six!
    VIP Member

    Well we sure can't take them seriously if manufacturers are rigging the results either. ;)

Share This Page