Dovecot panics when sieve filter outputs much data

Bug #1633220 reported by M97N on 2016-10-13
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
dovecot (Ubuntu)
High
Christian Ehrhardt 
Xenial
High
Christian Ehrhardt 
Yakkety
High
Christian Ehrhardt 

Bug Description

[Impact]

 * a crash of sieve when filtering large mails

 * The fix is done via a backport of upstream
   https://github.com/dovecot/pigeonhole/commit/a95b0579 which solves
   it by destroying the output stream before the fd is closed.
   Patch is nice and small, just moving two lines and applies as-is.

[Test Case]

 * Set up dovecot with sieve filtering on top:
   - https://help.ubuntu.com/lts/serverguide/dovecot-server.html
   - http://wiki2.dovecot.org/Pigeonhole/Sieve/Examples
   - https://easyengine.io/tutorials/mail/server/sieve-filtering/
   Then push a bunch of larger mails through, >60k seems to be the area to
   trigger it.

 * It can be complex to set up, but fortunately the reporter is very
   active and willing to verify Xenial and Yakkety in their setup.

[Regression Potential]

 * The change itself is rather minimal and from upstream.

 * given the change any regression we might overlook should be contained
   to sieve which is not default enabled without a user configuring it.

 * Did build and autopkgtest prior to suggest the upload all working
   as expected (known to fail in Xenial, I fixed that back in Yakkety).

[Other Info]

 * The change as done upstream is active for quite a while upstream and
   not changed anymore later on. It was even moved into dovecot-core to be
   reused for more use-cases and none of them triggered the need to
   rewrite the code, see:
   https://github.com/dovecot/core/blob/master/src/lib-program-client/program-client.c#L94

----

When using dovecot with vnd.dovecot.filter, where the filter returns at least ~60KB of data back to dovecot, dovecot panics (see dovecot log in attached apport report).

The error's origin is the pigeonhole plugin. The described error is already fixed in the github repo for sieve-pigeonhole [1].

We successfully managed to apply the patch locally for ubuntu's current version of pigeonhole (0.4.13 (7b14904)) by simply cherry picking the mentioned commit.

System information:
  Description: Ubuntu 16.04.1 LTS
  Release: 16.04
  dovecot-core:
    Installed: 1:2.2.22-1ubuntu2.1
  dovecot-sieve:
    Installed: 1:2.2.22-1ubuntu2.1

  $ doveconf | head -n2
  # 2.2.22 (fe789d2): /etc/dovecot/dovecot.conf
  # Pigeonhole version 0.4.13 (7b14904)

[1] https://github.com/dovecot/pigeonhole/commit/a95b0579

M97N (exioreed) wrote :
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in dovecot (Ubuntu):
status: New → Confirmed
Nish Aravamudan (nacc) wrote :

Thank you for filing this bug report! We will add it to our queue and look to provide a fix soon.

Changed in dovecot (Ubuntu):
status: Confirmed → Triaged
tags: added: bitesize
tags: added: server-next
Changed in dovecot (Ubuntu):
importance: Undecided → High
Changed in dovecot (Ubuntu Xenial):
assignee: nobody → ChristianEhrhardt (paelzer)
Changed in dovecot (Ubuntu):
assignee: nobody → ChristianEhrhardt (paelzer)
Changed in dovecot (Ubuntu Yakkety):
assignee: nobody → ChristianEhrhardt (paelzer)

Ok, this is broken in Yakkety as well (added a bug task) as it uses the same pigeonhole version.
The latest Debian release upgraded pigeonhole and by that can be considered fixed.
dovecot (1:2.2.25-1) unstable; urgency=medium
  * [cc29a81] Imported Upstream version 2.2.25
  * [d19bcca] Updated pigeonhole patch to 0.4.14
  * [16db179] Merged in some features of the Ubuntu dovecot package.
    + dovecot-core: added lsb-base dependency.
    + dovecot-core: Added apport hook.
    + dovecot-imapd,dovecot-pop3d: Added ufw profiles.
    Thanks to Christian Erhardt <email address hidden>
    (Closes: #828864)

Here an outlined plan of action to follow SRU policy to be available in the Dev release first:

1. merge latest dovecot to fix in Zesty (also a chance to drop more delta)
2. SRU for Xenial
3. SRU for Yakkety?

@M97N - for Yakkety SRU consideration, is that a likely case to happen (more than 60k)?
Also since this requires the special setup - would you be willing to verify this in Xenial and Yakkety once made available?

Changed in dovecot (Ubuntu Xenial):
status: New → Triaged
Changed in dovecot (Ubuntu Yakkety):
status: New → Triaged
importance: Undecided → High
Changed in dovecot (Ubuntu Xenial):
importance: Undecided → High
M97N (exioreed) wrote :

@ChristianEhrhardt - Yes, it is sure that dovecot panics for e-mails of this size.
We encountered the issue with e-mails which are a few kilobytes smaller, but the exact threshold
might vary (e.g. depending on 'server speed').
All in all 60K seems to be the right magnitude.
I would also like to mention that the bug occurs independent of the executed filter.
A simple filter that would read the mail from stdin and pass it to stdout without any modification would be sufficient to trigger the described behavior.

We could verify a fix / a newer version against our setup in Xenial and Yakkety.

FYI - I realized that the latest dovecot FTBFS on Ubuntu.
So some extra work, I bed a pardon this takes a few extra days depending how much I get to it.

The related FTBFS is fixed in 1636781, now waiting for a few things to settle on zesty release that just opened to be able to go on.

It would be great to have some early verification for what I plan to bring to zesty, so I build the same for yakkety and made it available at a ppa: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1633220-dovecot

If with your setup that triggers the issue, you could confirm that this fixes it it would help me to go on with that and then also prep the backport work for the older releases.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dovecot - 1:2.2.25-1ubuntu1

---------------
dovecot (1:2.2.25-1ubuntu1) zesty; urgency=medium

  * Merge with Debian (LP: #1633220); Remaining Changes:
    + Add updated autopkgtest to debian/tests/*.
    + Drop build dependency on libstemmer-dev (universe)
    + Use Snakeoil SSL certificates by default
      - d/control: Depend on ssl-cert
    + Add mail-stack-delivery
      - add package in d/rules, d/control
      - add d/*mail-stack-delivery* maintainer scripts and default conf
      - d/mail-stack-delivery.preinst: Move previously installed backups and
        config files to a new package namespace.
      - d/mail-stack-delivery.README.Debian clarified use of configuration files
    + Disable dovecot-lucene plugin as it had various issues, has universe
      dependencies and is deprecated in favor of solr anyway.
    + handle conffile removal of /etc/init/dovecot.conf (due to dropping
      upstart). Can be removed once no upgrade path from <yakkety is left.
  * Added changes:
    + Fix FTBFS of dovecot 2.2.25 in Ubuntu due to being incompatible with
      -Bsymbolic-functions linker flag (LP: #1636781).
  * Dropped Changes as they got accepted in Debian:
    + add lsb base dependency to ensure debian/dovecot-core.dovecot.init is
      working correctly
    + Add ufw integration:
      - d/dovecot-core.ufw.profile: new ufw profile.
      - d/rules: install profile in dovecot-core.
      - d/control: dovecot-core - suggest ufw.
    + Add apport hook:
      - d/rules, d/source_dovecot.py
    + Remove lintian override for drac

 -- Christian Ehrhardt <email address hidden> Tue, 25 Oct 2016 13:12:40 +0200

Changed in dovecot (Ubuntu):
status: Triaged → Fix Released
description: updated
description: updated
description: updated

Now that this is available in latest Dev Release I prepared debdiffs and SRU template for Xenial and Yakkety.

I added Debdiffs here as reference.

And I Uploaded to unapproved queue for the SRU Team to consider.

@M97N - The SRU Team will post here once it (hopefully) gets to -proposed so you can then verify the fix

Changed in dovecot (Ubuntu Xenial):
status: Triaged → In Progress
Changed in dovecot (Ubuntu Yakkety):
status: Triaged → In Progress

Hello M97N, or anyone else affected,

Accepted dovecot into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dovecot/1:2.2.24-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in dovecot (Ubuntu Yakkety):
status: In Progress → Fix Committed
tags: added: verification-needed

Just saw the Xenial reject based on other changes that went in and I missed - re-basing ...

actually I think it was correct, need to discuss that

Martin Pitt (pitti) wrote :

Hello M97N, or anyone else affected,

Accepted dovecot into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dovecot/2.2.22-1ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in dovecot (Ubuntu Xenial):
status: In Progress → Fix Committed

Hi M97N,
you were very responsive before so this might just have been dropped accidentially.
I wanted to ping you again if you could do a verification on the proposed package as oulined in the former post.
Only after that it will be made available to everybody.

M97N (exioreed) wrote :

@ChristianEhrhardt - Thanks for the ping and sorry for delay. We scheduled the verification of the proposed fix for friday evening (CET) now.

M97N (exioreed) wrote :

We did the verification for xenial and yakkety. Everything works fine.
Find the version information for both systems below.

--- xenial
$ uname -a
  Linux ubuntest 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ doveconf | head -n 2
  # 2.2.22 (fe789d2): /etc/dovecot/dovecot.conf
  # Pigeonhole version 0.4.13 (7b14904)

$ dpkg -s dovecot-core
  Package: dovecot-core
  Version: 1:2.2.22-1ubuntu2.2

--- yakkety
$ uname -a
  Linux yakketest 4.8.0-27-generic #29-Ubuntu SMP Thu Oct 20 21:03:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ doveconf | head -n 2
  # 2.2.24 (a82c823): /etc/dovecot/dovecot.conf
  # Pigeonhole version 0.4.13 (7b14904)
$ dpkg -s dovecot-core
  Package: dovecot-core
  Version: 1:2.2.24-1ubuntu1.1

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dovecot - 1:2.2.22-1ubuntu2.2

---------------
dovecot (1:2.2.22-1ubuntu2.2) xenial; urgency=medium

  * d/p/fix-sieve-pigeonhole-crash-on-huge-mails.patch: Fix sieve-pigeonhole
    crash when filtering too much data (LP: #1633220)

 -- Christian Ehrhardt <email address hidden> Wed, 09 Nov 2016 13:13:08 +0100

Changed in dovecot (Ubuntu Xenial):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for dovecot has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dovecot - 1:2.2.24-1ubuntu1.1

---------------
dovecot (1:2.2.24-1ubuntu1.1) yakkety; urgency=medium

  * d/p/fix-sieve-pigeonhole-crash-on-huge-mails.patch: Fix sieve-pigeonhole
    crash when filtering too much data (LP: #1633220)

 -- Christian Ehrhardt <email address hidden> Wed, 09 Nov 2016 13:13:28 +0100

Changed in dovecot (Ubuntu Yakkety):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers