Recurring events not displayed

[Probot]

[Impact]

 * The EWS plugin doesn't display recurring/repeated events in the calender. Only single time events are displayed.
Outlook Online displays all the events correctly.

The EWS plugin doesn't seem to thrown any errors, so people using it may not notice some events are missing.

This actually is a bug on the server side, that the evolution developers have worked around.

[Test Case]

 * You need a EWS server, the best instructions are from https://social.technet.microsoft.com/Forums/office/en-US/9952d9ea-6040-46b8-93d7-f163c09acd70/bug-in-ews-invalid-ical-format-if-recurrent-event-modified?forum=exchangesvrdevelopment

Basically:

    1. Create a recurring event with no end date. E.g. every Monday at 10am on OWA/Outlook Online (I used as start date 29th of August 2016)
    2. Query the calendar item through EWS and check the MimeContent field. It's OK, it contains a END:VCALENDAR
    3. Change a single instance by moving it to Tuesday 10am (in OWA), I moved the one from 19th September to 20st September 2016
    4. Now query the recurrent calendar item again and voilà: the "END:VCALENDAR" is now missing

[Regression Potential]

 * EWS plugin breaks in other ways.

[Other Info]

 * This is fixed in Zesty.
 * No testers have come forward about 14.04 and the code has diverged a bit since then. (I don't have a test environment for this, so any fix would rely on other testers)

Versions:
Ubuntu Gnome: 15.10
evolution-ews: 3.18.0-3~wily1
MS Exchange Server: 2016

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[KML (zemoh)]

Exactly the same problem here (on Debian 8.4, Jessie, so not only a Ubuntu problem).

evolution - 3.12.9~git20141130.241663-1+b1
evolution-ews - 3.12.9~git20141130.278fe7-1+b1
libevolution - 3.12.9~git20141130.241663-1+b1

[KML (zemoh)]

Oh, and i'm connecting to Outlook365
(additional info to my earlier comment #2).

[Probot (probot)]

In the meantime I updated to the following versions, but the problem still persists:

Ubuntu 16.04
evolution 3.18.5.2-0ubuntu1
evolution-common 3.18.5.2-0ubuntu1
evolution-data-server 3.18.5-1ubuntu1
evolution-data-server-common 3.18.5-1ubuntu1
evolution-ews 3.18.5-1
libevolution 3.18.5.2-0ubuntu

[Matthew Gregg (mcg)]

Also using O365, events seem to repeat for a while then stop. Tried removing accounts, removing .cache/evolution/calendar, but doesn't fix anything.

[Probot (probot)]

I updated to 3.20 and the problem is still there.
It may be related to the version of Exchange.

I have two accounts set up:
One with Exchange 2016 where the recurring events don't show up. (Outlook365 also uses Exchange 2016).

One with Exchange 2010 where everything works fine

So it may be related to EWS + Exchange 2016

Can someone confirm this?

[Probot (probot)]

I found out what's causing this issue.
Actually it's a bug in MS Exchange 2016.

Thankfully the evolution-ews developers created a workaround to solve this issue.
See here for further details:
https://bugzilla.gnome.org/show_bug.cgi?id=771370

The recurring events are not displayed correctly if one of the instances is detached, i.e., is moved to another day.

You need to recompile the source yourself, including the patch or wait for the next release.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Matthew Gregg (mcg)]

FWIW: evolution-ews version 3.22.3-1 from zesty appears to fix this.

[Bryan Quigley (bryanquigley)]

Have the fix in a test PPA for xenial - https://launchpad.net/~bryanquigley/+archive/ubuntu/1540461/

Let me know if that works.

Btw, Is anyone experiencing this issue on 14.04?

[Bryan Quigley (bryanquigley)]

This is already fixed in Zesty and appears that evolution-ews 3.22.3 was also backported to yakkety-updates.

[Bryan Quigley (bryanquigley)]

Can anyone confirm if this is fixed in 16.10 ews?

[Matthew Gregg (mcg)]

Where is the 3.22.3 yakkety backport?

[Bryan Quigley (bryanquigley)]

Sorry, looks like I was looking at evolution (which does have a 3.22.3 in 16.10 updates) and not ews.

[Bryan Quigley (bryanquigley)]

Confirmed, the new code is not in 16.10. Patch above.

[Jeremy Bícha (jbicha)]

Bryan, thanks for working on this.

What do you think about updating yakkety to 3.22.4?

https://git.gnome.org/browse/evolution-ews/tree/NEWS/?h=gnome-3-22

[Bryan Quigley (bryanquigley)]

@Jeremy, that would certainly work too. I haven't gotten any testers for 16.10 so far for this bug so not sure how many are following on 16.10 with evolution-ews.

If there is 3.22.4 in flight somewhere for yakkety my patch should definitely be dropped.

[Matthew Gregg (mcg)]

I am using EWS 3.22.3-1 on Yakkety that I pulled in from Zesty(yes I know), it works great :-) I'd be glad to test.

[Jeremy Bícha (jbicha)]

Bryan, no, there isn't any other yakkety evolution-ews SRU in progress.

Are you interested in preparing a 3.22.4 update for yakkety instead?

[Bryan Quigley (bryanquigley)]

>Are you interested in preparing a 3.22.4 update for yakkety instead?

I can give it a try, but I haven't really done multiple patches like that before. Can I just have the whole update as one debian/patches/full-3.22.4-update ?

[Probot (probot)]

Hey Bryan, thanks for the fix!

Unfortunately I've Evolution 3.20 installed on Xenial, thus I can't test the version from your ppa.
Is it possible that you create a .deb for 3.20.5?

thanks!

[Jeremy Bícha (jbicha)]

Bryan, it wouldn't be multiple patches.

Unless something else is needed, the debdiff would just be the debian/changelog entry. (It wouldn't need the evolution-2016-broken-mime-type.patch).

I mean if it's too confusing, I can do the yakkety update for you.

[Bryan Quigley (bryanquigley)]

@probot I don't see evolution existing for 3.20. In fact, AFAICT no Ubuntu releases include 3.20.

[Bryan Quigley (bryanquigley)]

debdiff of the new 3.22.3 package for yakkety

[Bryan Quigley (bryanquigley)]

Not sure if these are needed for this kind of fix, so adding them here anyway:
evolution-ews_3.22.3-0ubuntu1.debian.tar.xz
evolution-ews_3.22.3-0ubuntu1.dsc
evolution-ews_3.22.3-0ubuntu1_source.build
evolution-ews_3.22.3-0ubuntu1_source.changes

[Probot (probot)]

@bryanquigley I used ppa:gnome3-team/gnome3-staging to update gnome and all deps to 3.20, thus I currently have 3.20 installed. There is no update to 3.22 for 16.04 and I would like to avoid going back to 3.18

Unfortunately I don't know how to exactly integrate your patch in 3.20, so I would be very happy if you can point me in the right direction

[Bryan Quigley (bryanquigley)]

@probot Either of the patches for 1604 or 1610 really should just work with some version number changes. But getting it into that PPA would require someone with access to it.

[Jeremy Bícha (jbicha)]

Bryan, thanks for working on this. Here's a few changes I made before I uploaded to the SRU queue:

- I ran update-maintainer (from ubuntu-dev-tools). My build tools don't like to build a package with a -ubuntu version number without the maintainer being set to an ubuntu.com email

- For 16.10, I opened bug 1663863 as a tracking bug for the new version. I made sure both that bug and this bug were in the changelog. And I mentioned Exchange 2016 in the changelogs.

- For 16.10, I bumped the evolution and evolution-data-server version to 3.22.3 since that's what the new version needs according to configure.ac

- By the way, you can use "xenial" instead of "xenial-proposed". Launchpad will handle them the same way. I didn't change that in your package since -proposed works fine too.

- Personally, for sponsoring, the only part I needed was the diff of the debian/ directory.

I am unsubscribing ubuntu-sponsors now. If anything else needs sponsoring, feel free to resubscribe.

[Jeremy Bícha (jbicha)]

Probot, I uploaded the fix to the GNOME3 Staging PPA for Ubuntu 16.04 LTS. Give it an hour or so to build and publish. If you want GNOME 3.20, I recommend you upgrade to Ubuntu 16.10. Ubuntu 16.10 fixes more bugs and is better supported than the GNOME3 Staging PPA. Note that you'll need to upgrade about every six months until Ubuntu 18.04 LTS since the non-LTS releases aren't supported for very long.

[Jeremy Bícha (jbicha)]

For Ubuntu 16.04 and 16.10, a member of the Stable Release Update team will need to manually approve this update for the packages to be built for the -proposed repository. There will be a new post to this bug when that happens.

[Probot (probot)]

I can confirm that your code changes fix the bug. Thank you very much!
After a year when I reported it, I can finally see all my calendar entries again. Microsoft did still not yet fix the problem on their side :(

[Bryan Quigley (bryanquigley)]

@Jeremey,

Thanks! I appreciate the detailed post, very helpful for next time.

[Bryan Quigley (bryanquigley)]

Sorry, @Jeremy

[Łukasz Zemczak (sil2100)]

Hello Probot, or anyone else affected,

Accepted evolution-ews into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/evolution-ews/3.18.5-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Hello Probot, or anyone else affected,

Accepted evolution-ews into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/evolution-ews/3.22.3-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Probot (probot)]

This resolves the issue.

Tested on Ubunutu 16.04 with ppa:gnome3-team/gnome3-staging and gnome 3.20.

Tested evolution-ews version: 3.20.5-1~ubuntu16.04.2

Events show up again. You first need to clear the evolution cache `rm -rf ~/.cache/evolution/calendar`

[Probot (probot)]

At least the modified event now shows up in the calendar.

The not so important problem is now that the "old" event also still shows up.

I.e.:

- Create recurring events every Friday 10am
- Move one event (e.g. Friday, 24. Februar) to another day (e.g. Thursday, 23. Feb)
- Evolution now shows two events, the old one (Friday, 24FEB) and the new one (Thursday, 23FEB)

[Bryan Quigley (bryanquigley)]

@Probot, Thanks for testing but this verification process only applies to the evolution-ews in the archive.

If the archive -proposed version also has this problem then I'd be curious if it only affects 16.04's 3.18 version and not the full yakkety 3.22.3 backport.

[Jeremy Bícha (jbicha)]

Probot, I looked upstream to see if I could find a similar bug. I found
https://bugzilla.gnome.org/770434
which suggests that this bug might be fixed in Ubuntu 16.10 ("yakkety") which includes this commit:

https://git.gnome.org/browse/evolution-data-server/commit/?id=5c7579d

Since neither Bryan nor I have an Exchange 2016 account, we're dependent on testers like you to verify the fixes for this bug.

Like Bryan said, we don't need to verify the GNOME3 PPA, we just need to verify the proposed versions from comment #37 and comment #38 for Ubuntu 16.10 and 16.04 LTS. And according to Ubuntu's Stable Release policy, the fix needs to be confirmed in 16.10 for the update to be fully released to 16.04 LTS.

[Matthew Gregg (mcg)]

I've switched to evolution-ews 3.22.3-0ubuntu1 from yakkety-proposed, cleared my calendar cache and calendar entries appear to be recurring correctly. Seems to be a good fix.

[Jeremy Bícha (jbicha)]

@Matthew, thanks! Can you confirm whether you can reproduce the bug mentioned in comment #40 ?

[Matthew Gregg (mcg)]

@Jeremy I can confirm the bug from #40 :-(

[Jeremy Bícha (jbicha)]

Ok, while that bug is important, I don't think it should block fixing the original bug reported here so I'm going this mark bug as verified for yakkety. We still need someone to verify the proposed xenial package.

Matthew, could you report that new bug to GNOME? We'll also need a separate Launchpad bug for that issue. You might find some helpful information at
https://wiki.ubuntu.com/Bugs/Upstream/GNOME

[Matthew Gregg (mcg)]

Sadly, I'm not gonna have time for reporting this upstream.

[Bryan Quigley (bryanquigley)]

I've got a confirmation this is fixed in 16.04. As for the duplicate event part, I'm not sure it's trivial to tell what event we should be displaying given the MS bug, so really we can work around it this bit (and display some invalid events), but then they really need to fix it.

[Robie Basak (racb)]

Releasing, but please note that the packaging version numbering scheme used for this update is prone to collisions and unwanted ordering interactions, if you're not already aware of this. It just happens to not collide this time. See https://wiki.ubuntu.com/SecurityTeam/UpdatePreparation#Update_the_packaging for a scheme that works better (IMHO).

[Launchpad Janitor (janitor)]

This bug was fixed in the package evolution-ews - 3.18.5-1ubuntu1

---------------
evolution-ews (3.18.5-1ubuntu1) xenial-proposed; urgency=medium

  * Fix recurring events not displayed with Exchange 2016 (LP: #1540461)

 -- Bryan Quigley <email address hidden> Thu, 02 Feb 2017 11:25:57 -0500

[Robie Basak (racb)]

The verification of the Stable Release Update for evolution-ews has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package evolution-ews - 3.22.3-0ubuntu1

---------------
evolution-ews (3.22.3-0ubuntu1) yakkety-proposed; urgency=medium

  * New upstream release (LP: #1663863)
    - This fixes recurring events not display with Exchange 2016 (LP: #1540461)
  * Bump minimum evolution and evolution-data-server versions to 3.22.3

 -- Bryan Quigley <email address hidden> Thu, 09 Feb 2017 12:19:36 -0500

[Jeremy Bícha (jbicha)]

Are any of you still using Ubuntu 16.10 and are interested in verifying an update of evolution-ews to 3.22.6? The current version in 16.10 is 3.22.3.

https://git.gnome.org/browse/evolution-ews/tree/NEWS/?h=gnome-3-22