No notifications for new mails

[costales]

Hi!

In the BQ E4.5 Ubuntu Edition, I'm not getting notificactions for new mails.

Testing (by example)
1. Kill Dekko.
2. Send you an email.
3. Open Dekko.
4. Change quickly to another app before Dekko starts.
5. Bug: No notification
6. Go to Dekko: You'll see your new mail.

FYI as improvement, for me, in Dekko is not clear if Dekko checks the mail after X minutes or is a push notification.

Thanks in advance!

[Dan Chapman  (dpniel)]

Thanks for reporting this,

This is an ongoing known issue. We need support from the platform to be able to get notification support. Currently when the app goes to the background we get suspended and there's nothing we can do from there. We need a system hook that allows us to run in the background every X minutes to handle any new mails and deliver them to the post office. We have had talks previously with the platform team but they wern't going to do anything before RTM/OTA-1 :-(

The only other option we have is to have a cloud service that watches imap mailboxes and sends push notifications from there, but tbh I don't want to go that route as dekko would have to become a paid for app / maybe even subscription service as I personally do not have the finances to run a service like that for free, and I would like dekko to remain a free app in the store.

Dekko checks for mail at various points there is no set interval for checks, apart from when we are in an IDLE state it will send a NOOP to the imap server after 5 minutes to see if there is an update. Checks will also be made like on app start or when re-connecting after waking from suspension. This is still a tricky area to get right with the app suspension as IMAP is designed so that you can stay connected with very little overhead, but the app confinement security model breaks this when continuously waking and suspending our app & connections.

It's going to take some time to iron all this out, but it will get there.

[costales (costales)]

Aweome explanation Dan! :) Thanks a lot!
Just FYI: Dekko is one of my favorites apps.
Cheers!

[Frantisek Sklenar (frantisek-sklenar)]

I agree, very good explanation! I got my Ubuntu BQ phone 2 days ago and I use IMAP for my work email. But smartphone without notifying me for new incoming emails is something strange :( I like this phone, but without this BASIC feature is little bit useless for me. As Dan wrote, maybe this is issue on Canonical.

[Hector (hsanjuan)]

I was thinking... would there be a workaround using the cron daemon ? I can imagine apps are not allowed to set up crontabs and need SO hooks support to come back alive... but maybe users can manually run set up a cron job that wakes the application and fetches emails. It would be nice if it then would notify them in the notification bar...

[JOnathanJOnes (jonathanjones)]

What about using the account system of ubuntu itself.

Under accounts in ubuntu settings it is possible to create for example an account called: "Third Party Email Accounts" or something else.

When this is done, The app could read the database behind the system. Maybe there is an automatic functionality of sync within this system of accounts.

What do you think about this?

[Alan Pope 🍺🐧🐱 🦄 (popey)]

We're looking to get some help from chipaca when he's back from vacation to set up the necessary infrastructure for this. So this will be pushed back beyond OTA-6 (and possibly OTA-7)

[fgores (fgores-2)]

Hi, I have a Meizu ubuntu version and use dekko. But have a few questions/comments:

. I have never received any notifications for new mail, is that correct?
. Messages in the inbox can be/will be deleted only one by one, is that correct?
. When I reply to a message the is no send....I am not able to reply to any mails.....can you imagine. this cannot be correct,right?

OS Ubuntu 15.04(r3), last update 20/07/2015

[Niklas Wenzel (nikwen)]

Thought I'd let you know I've been working on this for some time now. The current version is far from ready but it can already display notifications for incoming emails.
You can follow my progress on Google Plus if you want: https://plus.google.com/u/0/101390620047921378812/posts

[Dan Chapman  (dpniel)]

Hey Niklas.

Awesome work on this so far, you have made great progress!

I have a couple of questions/suggestions:

firstly for the created url that is sent through url-dispatcher. It would be great if it could be and rfc 5092 [1] compliant IMAP url. Particularly because this is the standard for referencing objects on a remote IMAP server but it also includes some useful info like the UIDVALIDITY which ideally a client should be checking, to be sure that the UIDs have not changed between the poll session and the new session created from the dispatched url.

Also in terms of how Dekko works it would be really helpful if the message partid was included as well, so that we would have a direct pointer to the exact message part you created the notification from. Which we can directly fetch and not have to parse the mime structure again to find the interesting part.

Secondly, I read you blog post but it wasn't entirely clear to me how a user will select the client to consume the notifications. IIRC url-dispatcher was recently made clever and if more than one app is registered to consume the same protocol/scheme then the user is prompted to select which app to use. Would it make sense to just utilize that? This is the current state for the mailto: protocol so it would make sense to keep it consistent. What's your thoughts?

Keep up the great work!

[1] https://tools.ietf.org/html/rfc5092

[Niklas Wenzel (nikwen)]

Hey Dan,

I didn't know about that rfc yet! Let me dig into it a big more to check if we can support that. What I can definitely say is that we can add the message partid to the URI. If that is helpful, this shouldn't be a problem. ;)

The application for which the notification is being displayed is the one which the user authorizes the online account for. What I need to check myself is whether the URI from the notification is then sent directly to the application or if other apps can "compete" for it as well. If the former is true, the situation is clear. If the second one is what happens, then the new UI you mentioned should be used automatically by the URI dispatcher. To be honest, I'd prefer the first one as in that case the user could simply give another application access to the account if they wanted it to receive email notifications.

The reason why I wanted to put the app id in there was to enable applications to filter for "their" notifications in case they want to use different email clients for different accounts (e.g. the Gmail app for a Gmail one and Dekko for their Yahoo account). This would only be needed in the second case outlined above though.

I'll look into the questions brought up here. Thanks!

[Niklas Wenzel (nikwen)]

As a reference for all of those who didn't see the blog post: http://nikwen.github.io/ubuntu/2015/09/17/my-work-on-email-notifications.html

[Alberto Mardegan (mardy)]

Thanks Niklas for your excellent work! I've read your blog post, and quickly had a look at the account-polld code. I have a few remarks, which I'm presenting in a strictly random order :-)

- You don't need to create an account plugin for GMail. GMail uses the google account, so for GMail account-polld can simply ship a .service file containing the needed imap parameters
- GMail can use OAuth: https://developers.google.com/gmail/oauth_overview (you can have a look at Dekko's code for details)
- Stressing the above point :-) I believe that OAuth is almost a must, if you want to support 2-factor authentication (I know there are ways around it, but they are not convenient for the user, and OAuth offers better security)
- But yes, we can have a generic IMAP plugin for other account types
- Online Accounts offers an account plugin for SASL. It's not packaged to Ubuntu, but only because no application has used it so far. We can add it, the code is here: https://gitlab.com/accounts-sso/signon-plugin-sasl/tree/master (I'm not sure whether this will make your life easier or harder, since this plugin, unlike all the others, is interactive: the client should invoke it every time a challenge is received from the server -- but I thought I'd mention it anyways)
- I see a lot of data files in your branch, probably related to character encoding; I wonder if you could use a library instead? Maybe GLib offers the functionality you need?

[Niklas Wenzel (nikwen)]

Wow, Alberto! I didn't expect anyone to have a look at the code that fast!

I agree that OAuth authentication for Gmail in connection with using the Gmail accounts plugin would be great. Having a quick look at the Gmail docs, I also think that our IMAP library should be compatible with it. However, as we already have working Gmail notifications using OAuth in the images (although not open to third-party developers at the moment), I'd like to postpone this after we have released the normal password authentication IMAP notifications into the wild as they are supported by basically all IMAP servers.

Furthermore, thank you for the link to the SASL plugin. As that part is being dealt with by our IMAP library though, I don't think it will be too useful here.

The data files you found are indeed from libraries for character encoding. I noticed a few days ago that we can import some of the libraries via Debian packages that are already in the Ubuntu archives so that should reduce that part of the code.

Thanks again! :)

[Niklas Wenzel (nikwen)]

Ok, I spent some time on replacing the golang.org/x/text package with the packaged one from the archives today and discovered that the package is only available for wily at the moment. Thus, I'll revert those changes and keep its source code in place.

[Dan Chapman  (dpniel)]

Quick update. The generic email account plugin for sharing accounts between polld and dekko is located here lp:accounts-plugin-email

It is mostly complete now and going to start getting this moved into lp:account-plugins asap. The plugin includes auto configuration of server details and validates authentication with the mail server using the given credentials. It also makes use of signon-plugin-sasl which means we will be able to support more authentication mechanisms in Dekko than the current PLAIN & LOGIN.

If anyone wishes to give it a test on their desktop, instructions can be found in the README file.

[EAB (adair-boder)]

Btw, I started a request here for Google to send push notifications to Ubuntu. More voices the better ;)

https://productforums.google.com/forum/#!topic/tag-manager/sxMMa2oKKiM;context-place=forum/tag-manager

[Michael Mess (michael-michaelmess)]

With version 0.6.11 (Download: see bug 1501912 ), dekko sometimes seems to receive an incoming email automatically, sometimes it needs manual refresh.

But I would expect that dekko could play a sound and/or display a notification message in the Notifications slider of Ubuntu Phone when a new message gets available.

That should be configurable.

[Dan Chapman  (dpniel)]

Dekko will have POP3 support from the next update. So we will need to provide notification support for that as well.

I'm still not convinced that a polld plugin is the correct approach for this, it doesn't feel like it's going to scale well. I plan to add exchange ews & jmap protocols to Dekko in the future so that would mean creating more polld plugins that duplicates code and becomes more to maintain.

@Pat, @popey from the next update Dekko actually has a separate messaging server that can be run as a daemon, which Dekko communicates with over a local socket. Currently I'm just running it as another process on app launch. Is there some possibility that we could make this server blessed and run in the background somehow or at set intervals and allow it to do what we are trying to do with polld?

It already has most of the logic for notifications as it notifies any connected clients about new mail so it would be trivial to extend that to passing messages to the push service.

Or could we possibly turn it into a proper system service and restrict access via apparmor?

In reality I just don't believe that the majority of email server's out there will implement ubuntu push notifications without it becoming an rfc standard. The majority don't even do it for android/iOS as the IDLE and NOTIFY extensions provide the same thing for IMAP and for services that don't support those extensions are even less likely to support our push notifications.

Unfortunately polling has become so engrained in the email space that I don't think we can force that to change any time soon.

[Alberto Mardegan (mardy)]

Hi Dan, could you please bring this up in the ubuntu-phone mailing list?

I think it would be good to let click packages extend the functionality of account-polld, but the thing is delicate, we must make sure that this possibility does not come to detriment of the battery life. But maybe an idea is to let account-polld run click-provided plugins as separate processes, which get only a handful of seconds of CPU time and then get killed or suspended till the next period.

It would be sad to see a dekko-specific solution here: it would basically mean that our application model is a failure, if we are to give special treatment to every app who needs to do some background processing.

[Sam Bull (dreamsorcerer)]

It sounds like there could be support for the W3C push API in the future. This would have a much better chance of getting email servers to support push notifications. This would allow push notifications to be supported cross platform, including both mobile devices and desktop browsers.
https://lists.launchpad.net/ubuntu-phone/msg18205.html

It looks like Fastmail already support the W3C API, as I am able to subscribe to notifications in Firefox.

[Dan Chapman  (dpniel)]

@mardy, sorry i totally missed your comment. Sure, i'll bring this up on the ubuntu-phone list today.

---

@Sam yeah i agree the W3C push API is the best chance their is at the moment. I'm just not sure how quick adoption would be for email.

I think Fastmail is an exception to this as they do some pretty awesome and useful stuff. IIRC I think their push notifications is part of their jmap spec which the fastmail web ui is already using? I see they also support apples XAPPLEPUSH extension which is rare to see, so they could possibly be willing to adopt the same approach for Ubuntu if demand is there.

[Robin Heroldich (robinheroldich)]

Hi!

Any progress with this? It was reported more than a year. :(

[Alberto Mardegan (mardy)]

After some e-mail discussion, we agreed that I will take care of this bug, and of account-polld in general.
Niklas did some great work on it to support Dekko and IMAP accounts, but as Dan wrote in comment 18, with POP3 support coming to Dekko it becomes apparent that we cannot keep replicating every protocol implementation in account-polld. So, the long term solution is to have account-polld support out-of-process plugins, which can be written in any language and which can possibly reuse some of the application code.
I'll write more on that later, to the mailing list.

As for the immediate fix, I'll add support for Dekko's GMail accounts only (because they use OAuth, and don't require major changes in account-polld). For notifications for generic IMAP and POP3 accounts we'll have to wait, unfortunately.

[Dan Chapman  (dpniel)]

Hi Alberto,

Thanks for the update Alberto.

How is it going to work with gmail notifications and figuring out which app should receive the notification? Is it a case of seeing if dekko has been granted access to the account and falling back to the gmail webapp if not?

For the record i'll just post here my answer to your question on IRC earlier.

Dekko's notification action url should be in the format:

dekko://notify/{OnlineAccountId}/{mailboxName}/{MessageUID}

MessageUID is optional. If the gmail rest api provides the servers immutable message UID then you can include it and Dekko will try to fetch and display it. Dekko uses the mailboxname to trigger a sync on that mailbox so just set that to INBOX

[Niklas Wenzel (nikwen)]

Hi all,

It took me ages to reply to this thread due to very important personal issues. Sadly, I can't afford spending time on account-polld right now as it is too big of a project to handle for me at the moment. Sorry!

However, I have to say that I really like the long-term solution Alberto proposed here. I'm looking forward to it being implemented some time. :)

Cheers,
Niklas

[Alberto Mardegan (mardy)]

Hi all! I made some progress on this. :-)

You can test the account-polld package from my personal phablet PPA: https://launchpad.net/~mardy/+archive/ubuntu/phablet/+packages

I can see that now account-polld is emitting the notifications, however there must be something wrong with Dekko's notification client because the notification is not delivered to the user, and the log files from the notification client grow huge, containing just the line "STILL RUNNING" repeated thousand of times:

phablet@ubuntu-phablet:~$ ls -l ~/.cache/upstart/untrusted-helper-push-helper\:146918*
-rw-r----- 1 phablet phablet 523785 lug 22 13:52 /home/phablet/.cache/upstart/untrusted-helper-push-helper:1469184737566095:dekko.dekkoproject_dekko-notify_0.6.20.log
-rw-r----- 1 phablet phablet 538605 lug 22 14:26 /home/phablet/.cache/upstart/untrusted-helper-push-helper:1469186801149248:dekko.dekkoproject_dekko-notify_0.6.20.log
-rw-r----- 1 phablet phablet 527760 lug 22 14:28 /home/phablet/.cache/upstart/untrusted-helper-push-helper:1469186906427380:dekko.dekkoproject_dekko-notify_0.6.20.log
-rw-r----- 1 phablet phablet 517635 lug 22 14:39 /home/phablet/.cache/upstart/untrusted-helper-push-helper:1469187535926510:dekko.dekkoproject_dekko-notify_0.6.20.log

Here's an extract of the D-Bus traffic, taken with dbus-monitor:
http://paste.ubuntu.com/20442458/

I believe that the problem is with Dekko's push client, because if I replace it with the simple script used by the GMail webapp, I get the notification.

[Robin Heroldich (robinheroldich)]

Will this land in OTA-13?

[John M. (jhnmlkvch9)]

I'm using Dekko 0.6.205 with OTA-13 and doesn't look like it is still working.

[John M. (jhnmlkvch9)]

So will IMAP IDLE work when this bug is fixed or is there no hope of it working for Ubuntu Touch given the lifecycle policy.

[Robin Heroldich (robinheroldich)]

Any update on this? As I know the work is ready on Ubuntu's side, no we only need a new Dekko release, don't we?

[Dan Chapman  (dpniel)]

Hey Robin. Not quite :-(

Looks like the work in accounts-polld has only landed in yakkety. Is that right @mardy?

Also for this to work in Dekko all the accounts need to be in online accounts. Some of the work for
that is done. Mainly in Dekko's messaging daemon. But it still requires the logic for the UI/Setup wizard and also work out how the user is going to be able to edit their account settings after the online account is created. which currently isn't possibly in online accounts. Also the polld plugin needs implementing so I need to find out if there is any docs for that.

If anyone wants to help out with this please give me a ping.

[Alberto Mardegan (mardy)]

Yes, the account-polld work hasn't landed yet. I'll try to get things moving :-)

[Gregory Opera (gregoryopera)]

Ubuntu 16.04 LTS ("Xenial Xerus") here, and I too have no notifications for new e-mails... "Pulling" to refresh does not seem to work, and closing then re-opening usually doesn't work either.