1. Download our Official Android App: Forums for Android!

Support Marshmallow’s Doze broke push notifications: my fix is preventing Doze with MacroDroid

Discussion in 'Android Help' started by cdl, Aug 10, 2016.

  1. cdl

    cdl Android Enthusiast
    Thread Starter
    43

    Feb 4, 2010
    427
    47
    43
    In summary:
    Push notifications don’t work with Marshmallow’s Doze. Receiving instant email notifications is very important to me. My workaround is to use a task automation tool to do ‘something’ at a regular interval to prevent the phone from entering Doze at all. The resulting battery usage is more than acceptable for me: about 1% / hour, measured over a 7-8 hour period (at night), with k9 mail connected to my private email (imap idle for push) and Good Work connected to my work email.


    In detail:
    Doze stops most apps, preventing them from syncing and accessing the network, after a period of inactivity.
    No app can be fully exempt from Doze (despite many claims to the contrary).

    Android’s official page explains that whitelisting an app (settings – battery – battery optimisation) does not completely exempt it from Doze: https://developer.android.com/train...doze-standby.html#support_for_other_use_cases

    An app that is whitelisted can use the network and hold partial wake locks during Doze and App Standby. However, other restrictions still apply to the whitelisted app, just as they do to other apps. For example, the whitelisted app’s jobs and syncs are deferred, and its regular AlarmManager alarms do not fire.

    Notably, the acceptable use cases https://developer.android.com/training/monitoring-device-state/doze-standby.html#whitelisting-cases do NOT mention email clients!

    As of now, no solution has been found to make imap idle work with Doze.

    Many sites and discussions mention that you can disable Doze for certain apps, but that’s incorrect
    ; in fact, there are many reports of push email not working with Doze; at the time of writing, the developers of k9 mail https://github.com/k9mail/k-9/issues/857 and of Aqua Mail https://www.aqua-mail.com/forum/index.php?topic=4858.msg28029#msg28029 have not found a solution. That’s a shame, because these two clients use imap idle for push, which I’m a big fun of: no server implementation required for push; I’ve been using it for 6 years on a number of Android devices without ever experiencing the battery drain which is sometimes mentioned as a drawback; additionally, they are among the few apps which keep your data private (http://androidforums.com/threads/email-which-apps-keep-it-private.935578/ ).


    Push email not based on imap idle doesn’t work with Doze, either:

    As Android’s site explains, receiving high-priority GCM (Google Cloud Messaging) notifications would cause the device from exiting Doze mode; however, there are two issues:

    1) Client-only solutions (like k9 mail with imap idle) don’t work with Doze. You need a GCM-based solution, which requires a server implementation, which means you have to trust someone with the password to your email. Even if you’re willing to pay for this service and to trust the provider for your private email, no sensible IT security manager would: Marshmallow has effectively killed proper push notifications for work email. A company might be willing to invest in its own GCM-based solution, but I have no idea how complicated it would be, if data would have to be shared with Google’s server ( a big no-no in most cases), etc.

    2) Leaving aside cost and trust, as of now I have not found a single email app which can send high priority GCM notifications. Cloudmagic and others rely on GCM, but do not send high-priority notifications; Cloudmagic has just confirmed this to me in an email, and the Aquamail discussion linked earlier says the same about other apps. This may change in the future, of course.


    My solution with a task automation tool
    My solution is to use a task automation tool (I use MacroDroid, which is easier to use than Tasker, but I suppose any similar app would do) to get the phone to do something at regular intervals, so as to prevent the phone from entering Doze at all. I have set up this MacroDroid macro:

    • Trigger: regular interval 45 minutes. Using Gsam battery monitor, it seemed my phone entered Doze after about 1 – 1.5 hours of inactivity. The time may be different for your device. You can leave it stationary on a desk, then come back after a few hours, and use Gsam to check when Doze kicked in.
    • Actions:
      • Keep device awake: screen on for 1 minute
      • Launch Good Work (this might not be necessary, to be honest)
    • Constraints (ie the macro only runs if): screen off and phone not in call

    With these settings, and with k9 email connected to my private email via imap idle for push, and Good Work connected to my work email, I left the phone on a desk overnight (7.5 hours), connected to my home wifi, with keep wi-fi on during sleep set to always, and the battery usage was 1% / hour (as measured by the Gsam battery monitor app). When I woke up, I noticed the led light was flashing, and when I turned the screen on I saw notifications for emails (both k9 and good) received overnight. I checked with gsam (charts -> other) and verified the phone never went into Doze mode. According to Better Battery Stats, the phone was in deep sleep 85% of the time; I would have expected this % to be higher, but I’m not too bothered because 1% / hour is more than acceptable for me, plus I never leave the phone on for 8 hours without touching it (I typically switch it off at night, or leave it in airplane mode if I need an alarm to wake me up). The main thing I was interested in was making sure the phone doesn’t enter doze mode and breaks push emails if, say, I leave the phone on the desk for an hour during a meeting. With this method, I can be sure it doesn’t happen, yet I still have a good battery usage.


    How about root?
    I understand Doze can be tweaked on rooted devices, but I haven’t looked into this because I need to keep my phone non-rooted, otherwise Good Work, which I use to connect to my work email, won’t work.


    Other settings I have (probably not relevant):


    Next steps:

    • I hope the 1%/hour battery usage will stay roughly constant but I’ll keep monitoring the situation.
    • I will also try to fine-tune the macro; maybe I can set it to wake the CPU without waking up the screen (MacroDroid has such an option but I haven’t tried it yet).
    • I don’t know if MacroDroid or Tasker can be used to exit Doze; some people may be interested in exiting Doze after, say, 2 hours, but I don’t want my phone to enter Doze at all.
    • I’ll keep looking for a GCM-based email solution for my private imap email account that can send high priority notifications.
    • I’ll keep chasing Good and my employer’s IT team to see if they have a better solution, e.g. if Good can be fully exempted from Doze or send these high-priority notifications
    • I want to understand if Whatsapp messages are received when in Doze mode or not.


    The day before, the phone entered Doze mode overnight and, when I woke up, neither k9 nor good had received emails. I had to manually refresh both. Oddly, though, the battery usage when in Doze was higher (2% / hour vs 1% / hour) over a comparable 7.5 hour period.


    Final comments:
    I find all of this insane and frustrating. Android went from one extreme to the other: from a flawed OS, which caused battery drain because too many apps were running in the background when they were not supposed to, and with users unable to prevent it (see my discussion: http://androidforums.com/threads/wh...nning-in-the-background-android-flaw.1053919/ ) to a flawed OS which, in its misguided attempt at improving battery life, doesn’t let users have the apps they want running in the background when they want! At first I thought it was a perverse way for Google to force users to reply on its GCM system; however, Marshmallow has been around for almost one year, yet no solution exists to get push email to work properly with Doze. If I had wanted a closed eco-system with a dictatorial “my-way-or-the-highway” approach, I would have bought an iPhone! Doze can make sense for tablets, or at least for how I use my tablet (I am fine with not having push on my tablet if it extends battery life, but I need push on my phone), but breaking push without even giving users the option to fully disable Doze is insane!

    Thoughts? Suggestions?
     

    Advertisement

  2. AZgl1500

    AZgl1500 Android Expert
    618

    Feb 3, 2011
    6,367
    3,120
    618
    Male
    Retired and loving it.
    Oklahoma grasslands
    Very interesting and detailed observations.

    I learned a bit here.
    I am at the other end of the Notifications now, I am retired, and have everything turned OFF, period.
    I have the phone set to be dead as a dog from 00:01 thru 05:00 as I value my sleep a hell of a lot more than someone's insane efforts at waking me up. I really get pissed when the bank issues a SMS alert that a debit transaction was done at 02:01 AM.... argh!!!

    I also, turn the Internet OFF when the screen is locked. DATA is always OFF, and MacroDroid turns off the WiFi when the screen locks. I use a 10 minute screen stay awake so that the screen is not locking on me while I am using the phone casually.

    I am now able to not fool with Airplane Mode, ever..... I also don't have to mess with any settings on the phone. My battery usage is comparable to yours, the WiFi wakes up when the screen is unlocked.

    Aqual Mail is set up to never poll for anything, unless I manually draw down on the screen and tell it to get new messages. And, it only downloads the 1st sentence. Just enough for me to figure out if the message is even worth opening before I delete it.

    MacroDroid is the most valuable tool on my phones... I have it on every phone we own.

    I am more a geek than my wife and daughter, I have 13 or more macros running all the time. Each does a little tiny chore that makes my life easier.

    Yes, Tasker is a great tool, but it requires you learn how to use it. I bought it, messed with it for an hour, and never got past 1st base with it. Wasted money for me. Found MacroDroid and that ended my search for a tool to work when I want it, and not before.
     
    Dannydet likes this.
  3. cdl

    cdl Android Enthusiast
    Thread Starter
    43

    Feb 4, 2010
    427
    47
    43
    Thanks.
    Another comment: this proves to me that the old motto: if it ain’t broken, don’t fix it is always valid!

    I didn’t want Marshmallow because I wanted Sony’s Stamina mode, removed from Marshmallow because it did what Doze does, but better (http://www.xda-developers.com/sonys-stamina-mode-did-it-first/ ). However, I would have never ever ever thought that the new OS could break push notifications, without even offering an alternative. That’s something no sane person could ever conceive.

    Key learnings:
    • don’t upgrade, ever, unless you have very compelling reasons to do so
    • never trust developers not to mess up and turn fully functioning software into unusable mess with the latest upgrade
    • don’t think you can ever anticipate what a new upgrade may break; I could imagine some apps being incompatible, but killing push notifications altogether? Totally crazy
    • no amount of research can prepare you. I moved to Marshmallow some 10 months after it was introduced; I had read about some apps being incompatible, but I didn’t know about Doze killing push notifications
     
    ToniKH likes this.
  4. Jonny M

    Jonny M Lurker
    6

    Sep 22, 2016
    3
    3
    6
    Thanks for the heads up and excellent explaination. Couple of questions.

    So, email push notifications stop when Android N device goes into Doze mode. So this is fairly quick right, like a few minutes after the screen is turned off? Beause I hear in Android N, devices go into Doze mode even when they are in pocket.

    What about apps like Nine Exchange email client. This effecs only IMAP IDLE or does it also effect Exchange ActiveSync clients?
     
  5. cdl

    cdl Android Enthusiast
    Thread Starter
    43

    Feb 4, 2010
    427
    47
    43
    I have Marshamllows, not Nougat, so I don't really know how Doze works in N. Most of the stuff I had read about Doze in Marshamllows was incorrect, e.g. there are countless posts on how you can disable Doze for certain apps (in fact, you can't, not fully), so I would take everything I read about N with a truckload of salt.

    Like I said in the post, the one and only way to wake the phone from Doze is to send it a high priority GCM (google cloud messaging) notification; this means that any app which doesn't rely on high priority GCM notifications will NOT wake the phone from Doze. So, no, your client will not work when the phone is in Doze. Sorry.
     
    ToniKH likes this.
  6. Slug

    Slug Check six!
    Moderator
    2,043

    Aug 1, 2009
    20,552
    16,990
    2,043
    Male
    Mobile phone retail
    Inverness, UK
    Whatsapp (and Gmail) notifications arrive instantly on my Z3+ running 6.0.1.
     
  7. cdl

    cdl Android Enthusiast
    Thread Starter
    43

    Feb 4, 2010
    427
    47
    43
    Even when the phone is in Doze? On my Sony Z5 running Marshmallows, Whatsapp messages wake it up from Doze. As for the Gmail app, I have no idea as I don't use it.
     
  8. Slug

    Slug Check six!
    Moderator
    2,043

    Aug 1, 2009
    20,552
    16,990
    2,043
    Male
    Mobile phone retail
    Inverness, UK
    I assume so. My phone sits in my pocket for most of the day when I'm at work, but Whatsapp notifications appear on my smartwatch within seconds of being posted.
     
  9. chanchan05

    chanchan05 The Doctor
    1,108

    Jun 30, 2011
    14,781
    4,522
    1,108
    Male
    Yeah. Mine too. No problems. I have aggressive doze and doze on the go from greenify, and I still get notifications from the apps I whitelisted almost as soon as I receive them. On usual doze at night, I wake up to find several notifications as well.
     
  10. ToniKH

    ToniKH Lurker
    5

    Sep 24, 2016
    4
    0
    5
    Male
    Then your phone has NOT entered Doze mode. In Android MARSHMALLOW your phone needs to be STILL in order to enter Doze. This is not the case when it's in your pocket.

    Also, having notifications when you wake up does not prove that you've had INSTANT notifications. Doze mode has periodic maintenance windows during which those notifications can get through.

    If you really want to test your phone, then let it lie STILL and UNUSED for couple of hours. Then try sending an email to yourself from your computer and see what happens.

    And for those who are interested (cdl?), there is now an app in the Google Play store which can Disable Doze. Root is not required. App is named Disable Doze.
     
    #10 ToniKH, Sep 24, 2016
    Last edited: Sep 24, 2016
  11. Jonny M

    Jonny M Lurker
    6

    Sep 22, 2016
    3
    3
    6
    Ok, I tested out Android N and it is very aggressive in going into Doze. It has pretty much turned my Nexus 6p into an iPhone. That is to say a few minutes after the screen is turned off, it goes into doze mode and I get no email notifications; not from IMAP IDLE, not from ActiveSync and not even from google Inbox. Only Hangouts and SMS get pushed even in doze. Of course if you get a GCM message then it kicks the phone out of doze and other queued up messages all arrive at once. This is *exactly* the same behavior of iPhone and that is why I swore never to use an iPhone ever. Lack of true push email is for me a giant leap backwards. Push email is what makes a phone a true communication device. Watching media, playing music/games, browsing etc. is of secondary importance to me.

    The irony of it is, what Google is proposing with GCM is nothing but a BIS/BES of old. Google and Apple killed off BlackBerry with the promise of open and standards based push notification support. Go back and watch people talk about how BIS and BES were a bad idea because of single point of failure and we don't need it because you can get push email over internet and so forth. And now we are back a full circle and instead of BIS we have GCM. Sad to say, but I think this is end of the road for me with Android. The choice has boiled down to getting a Google iPhone or an Apple iPhone.

    To be fair, I did find one app that somehow dodges Doze and delivers IMAP IDLE mails (both on Android and iOS), but who knows if they got lucky and slipped through the cracks or what. I can't rely on a system with that kind of uncertainity when the companies in question expect me to make a heavy investment in their ecosystem over time but can't promise my emails will be delivered instantaneously. My primary reason for owning a smartphone is to get push email.

    Luckily Ubuntu Touch has not implemented a notification subsystem yet. I can only keep my fingers crossed and hope they will support some kind of a clean push email technology without all the power saving bs.
     
    #11 Jonny M, Sep 24, 2016
    Last edited: Sep 25, 2016
    higedamc and ToniKH like this.
  12. chanchan05

    chanchan05 The Doctor
    1,108

    Jun 30, 2011
    14,781
    4,522
    1,108
    Male
    Oh I misunderstood. You meant instant notifications? So far I haven't had problems despite Aggressive Doze activated via Greenify (the phone going into Doze after a few minutes instead of an hour or so, no root needed). Although Greenify does have a whitelist that seems to work.
     
  13. ToniKH

    ToniKH Lurker
    5

    Sep 24, 2016
    4
    0
    5
    Male
    Jonny M, I agree with you completely. At least Google should have provided an official way to switch Doze completely off it the user so desires. This forced doze with its delayed notifications is absolutely crazy idea. Even this GCM method is not a real solution, even if it worked correctly. Only high priority notifications are able to break through doze and email notifications are rarely high priority. Now it's up to phone manufacturers and users to find a way to disable doze themselves.

    As far as I know, OnePlus is so far the only phone manufacturer that has recently added an official developer option that can switch doze completely off without any problems. I wish all manufacturer would follow this example. Actually, this should have been done from the start.

    There are also couple of other ways to disable doze, apart from cdl:s way to use Macrodroid to periodically turn the screen on. In the Oneplus forum they found a way to use slightly modified scriptrunner to disable doze. This needs some ADB commands the first time used, but it actually works. I have tested it myself. Only downside is, that the script has to be run manually at every boot. Tasker might be used to automatize this, but I haven't tried that yet. You can find that thread if you google "Oneplus Push notification issues". This method is described in pages 107-109.

    Now there is also this app, Disable Doze, that also disables doze. This app should be able to run automatically at every boot, but unfortunately that doesn't work yet. At least not with my Sony Xperia. This app works fine when, again, user runs it manually at every boot. Or uses Macrodroid for that. But this is still a new app, I hope the developer can work out the start up bug yet.

    Can you test any of these two methods with Nexus and Nougat? Do they still work? So far they have been tested only with Marshmallow.
     
  14. Slug

    Slug Check six!
    Moderator
    2,043

    Aug 1, 2009
    20,552
    16,990
    2,043
    Male
    Mobile phone retail
    Inverness, UK
    Thanks for telling me how my phone behaves in my own pocket. However I think you'll find that Android 6.x is a bit smarter than that. ;)

    What part of "notifications appear on my smartwatch within seconds of being posted" is unclear? :thinking:

    See above. I've been using Android since 1.5 so I kind of know what I'm seeing. In my pocket or sitting on my desk, Whatsapp or Gmail notifications are instant. I can tell that simply by comparing the timestamps.
     
  15. Jonny M

    Jonny M Lurker
    6

    Sep 22, 2016
    3
    3
    6
    @ToniKH those do work on the Nexus 6p, but I don't get why I have to jump through hoops for something as simple as disabling power saving features. And that is all Doze really is. If they don't want the average consumer to bother about it, put the option to disable Doze in developer options. Besides rooting is not an option because Google disables Android Pay if the device is rooted. I've never really been interested in other Android phones other than Nexus because I like the stock Android experience and want OS updates as soon as possible.

    It is quite clear Google is now changing its focus to making Android a consumer OS. As I see it, its only going to get worse going forward. More difficult to root, more difficult to turn off "security" features etc. And that is fine. If that is Google's decision then Android is not the phone for me going forward. Thanks for pointing me to OP3. Looks like they have all the power user features figured out, even though it looks like they are not as quick at getting new versions of Android out to their devices.

    At any rate, it looks like Nexus is dead anyway and the new Pixel or whatever will not be stock Android anymore. Its a real shame when there is so little real difference between the only two viable smartphone platforms.

    I hope one of the other platforms is able to figure out a way change the status quo. I'll sure be rooting for them.
     
    ToniKH likes this.
  16. MLSS

    MLSS Android Expert
    413

    May 20, 2013
    2,245
    1,481
    413
    Male
    Director of a non profit
    Nelsonville, Ohio
    You can choose to optimize an App or not optimize it with Doze.
    I've yet to have an issue with missing a notification that I didn't want to miss because of doze.
     
    kct1975 likes this.
  17. ToniKH

    ToniKH Lurker
    5

    Sep 24, 2016
    4
    0
    5
    Male
    MLSS; optimizing an app does not change the priority of the notifications. Thus, normal priority notifications (for example email push notifications) can't get through doze instantly even if the app involved is optimized.

    I would suggest everyone to read the first post in this thread. Cdl describes there how doze works and what kind of things are a problem. Most of the people don't seem to quite understand how doze works and what kind of problems are happening because of doze. Some people don't even recognize the problem, since they don't know when the phone has entered doze and when it has not. With Marshmallow it's actually quite rare that the phone actually enters doze. With Nougat it will be a completely different thing.

    On the other hand, some people aren't bothered about delayed notifications. This doesn't mean that the problems are not there. Instant notifications are important to some people.

    BTW, Disable Doze has now been updated and now it works automatically at every boot. This pretty much solves the problem for me. Although, I tend to agree with Jonny M that Android is probably going to go in the wrong direction in every update from now on. With old phones you can always choose not to update, but I'm afraid it's going to be a pain to find a decently working phone couple of years from now. And by decent I mean a phone with features that the user himself can control. I'm afraid this disaster with email push notifications is just a start.
     
  18. cdl

    cdl Android Enthusiast
    Thread Starter
    43

    Feb 4, 2010
    427
    47
    43
    Do you know how this app works? Its google play page has no details on how it disables doze on non-rooted phones.
     
  19. ToniKH

    ToniKH Lurker
    5

    Sep 24, 2016
    4
    0
    5
    Male
    I have no idea how it works. I only know, that there has been no problems with doze or push notifications after I installed Disable Doze app. I have periodically checked my phone stats and it has never entered doze after installing this app. Maybe you could ask the details from the developer, he answered pretty quickly to my bug report when I first started using the app.

    For me this notification problem is now solved, I had already basically forgotten the whole thing. Battery life has not been affected by this app, at least there has not been any noticeable changes.
     
    #19 ToniKH, Dec 27, 2016
    Last edited: Dec 27, 2016

Share This Page

Loading...