assertion failure in message_part_finish when searching large folder

[Dave Moskovitz]

[Impact]

Searching emails triggers a assertion failure if malformed emails are
present in the mail box.

[Test Case]
With an INBOX containing a malformed email, perform a server-side search, such as:

  # doveadm index
  # doveadm search -u ubuntu mailbox inbox body blah

or

  # doveadm search -u ubuntu mailbox inbox body JPMorgan

In an affected situation, this triggers an assertion failure in message-parser.c.
In an unaffected case, these searches will return one or more hash ids of email messages.

[Where Problems Could Occur]
This patch alters code involving iteration over a C linked list in
relation to an assertion, so problems to look for would involve
assertions, crashes or memory handling issues particularly associated
with message handling functionality. Further, as this section of code
involves message processing, issues related to searching, organizizing,
or otherwise iterating through email messages may be of relevance.

Mitigating the risk is that the patch's actual change is simply to add a
pre-condition to a section of logic, meaning that any issues would
probably be limited to situations where preparsed messages are being
handled.

[Original Report]
I'm experiencing identical behaviour to Debian bug #970386 (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970386) ... Can we please have this fix applied to dovecot-core 1:2.3.7.2-1ubuntu3? Searching is fairly broken atm.

Thanks!

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

This is the upstream patch: https://github.com/dovecot/core/commit/a668d767a710ca18ab6e7177d8e8be22a6b024fb

[Bryce Harrington (bryce)]

Verified code changes are already present in hirsute's dovecot-gu.
Should probably also check if affects bionic. There are sample emails in the debian bug report https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970386, at comment #34.

[Bryce Harrington (bryce)]

The code for dovecot 2.2.33.2 currently in bionic appears to be significantly different from what's in focal. The code that this patch fixes doesn't seem to exist in the bionic codebase, at least not in the same form. Still worth checking, but it'll have to be done using a test case.

[Bryce Harrington (bryce)]

I've packaged this fix in PPA here:

    https://launchpad.net/~bryce/+archive/ubuntu/dovecot-sru-lp1912118

However, I'm not clear exactly how to reproduce the crash using the emails. I've downloaded them and stuck them in a user INBOX, but the debian bug report doesn't list the exact commands being used to execute the server-side search, but I gather a simple `doveadm index` isn't sufficient to trigger it?

[Bryce Harrington (bryce)]

Hi Dave,

Ubuntu focal is up to version 1:2.3.7.2-1ubuntu3.3 now, which includes fixes for malformed MIME patches, however does not yet have the fix from deb #970386. However, I'm also not able to reproduce the reported issue when downgrading to 1:2.3.7.2-1ubuntu3:

dovecot-core:
  Installed: 1:2.3.7.2-1ubuntu3
  Candidate: 1:2.3.7.2-1ubuntu3.3
  Version table:
     1:2.3.7.2-1ubuntu3.3 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
 *** 1:2.3.7.2-1ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
        100 /var/lib/dpkg/status

# doveadm search -u ubuntu BODY JPMorgan
193f6434a25b7f60548c01009e423642 1

I've tested SUBJECT and KEYWORD searches, as well as `doveadm index` but do not trigger the crash so far.

Could you assist us in identifying a simple test case we can use to reproduce the issue and validate the fix?

[Dave Moskovitz (dave-thinktank)]

Here's what I'm getting:

doveadm search mailbox inbox body blah

doveadm(dave): Panic: file message-parser.c: line 174 (message_part_finish): assertion failed: (ctx->nested_parts_count > 0)
doveadm(dave): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xf0b2b) [0x7f2f562cfb2b] -> /usr/lib/dovecot/libdovecot.so.0(+0xf0b67) [0x7f2f562cfb67] -> /usr/lib/dovecot/libdovecot.so.0(+0x54bcf) [0x7f2f56233bcf] -> /usr/lib/dovecot/libdovecot.so.0(+0x51e31) [0x7f2f56230e31] -> /usr/lib/dovecot/libdovecot.so.0(message_parser_parse_next_block+0x104) [0x7f2f562b5304] -> /usr/lib/dovecot/libdovecot.so.0(message_search_msg+0xa0) [0x7f2f562b7700] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xd614e) [0x7f2f5646214e] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_search_args_foreach+0x4d) [0x7f2f563e584d] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xd6e98) [0x7f2f56462e98] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xd822c) [0x7f2f5646422c] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_search_next_nonblock+0x11d) [0x7f2f5646498d] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next_nonblock+0x24) [0x7f2f563ef114] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next+0x47) [0x7f2f563ef187] -> /usr/bin/doveadm(+0x3cf42) [0x56367fc8ff42] -> /usr/bin/doveadm(+0x33595) [0x56367fc86595] -> /usr/bin/doveadm(+0x34166) [0x56367fc87166] -> /usr/bin/doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x225) [0x56367fc87f05] -> /usr/bin/doveadm(doveadm_cmd_run_ver2+0x500) [0x56367fc983d0] -> /usr/bin/doveadm(doveadm_cmd_try_run_ver2+0x3e) [0x56367fc9842e] -> /usr/bin/doveadm(main+0x1d9) [0x56367fc77a69] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f2f55e6b0b3] -> /usr/bin/doveadm(_start+0x2e) [0x56367fc77efe]
Aborted (core dumped)

Short of sending you the inbox in question, is there any other info I can provide that would be helpful?

[Bryce Harrington (bryce)]

Hi Dave, thanks for getting back quickly. Yes there's a couple things that might help this move forward.

Before that, this is what I see on my end:

root@dovecot-sru-lp1912118-focal:/home/ubuntu# apt-cache policy dovecot-core
dovecot-core:
  Installed: 1:2.3.7.2-1ubuntu3
  Candidate: 1:2.3.7.2-1ubuntu3.3
  Version table:
     1:2.3.7.2-1ubuntu3.3 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
 *** 1:2.3.7.2-1ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
        100 /var/lib/dpkg/status
root@dovecot-sru-lp1912118-focal:/home/ubuntu# doveadm search -u ubuntu mailbox inbox body blah
root@dovecot-sru-lp1912118-focal:/home/ubuntu# doveadm search -u ubuntu mailbox inbox body JPMorgan
193f6434a25b7f60548c01009e423642 1
root@dovecot-sru-lp1912118-focal:/home/ubuntu# systemctl status dovecot
● dovecot.service - Dovecot IMAP/POP3 email server
     Loaded: loaded (/lib/systemd/system/dovecot.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2021-05-04 23:01:36 UTC; 18min ago
       Docs: man:dovecot(1)
             http://wiki2.dovecot.org/
   Main PID: 306 (dovecot)
      Tasks: 7 (limit: 77019)
     Memory: 7.2M
     CGroup: /system.slice/dovecot.service
             ├─306 /usr/sbin/dovecot -F
             ├─335 dovecot/anvil
             ├─336 dovecot/log
             ├─338 dovecot/config
             ├─503 dovecot/stats
             ├─526 dovecot/auth
             └─543 dovecot/auth -w

May 04 23:01:36 dovecot-sru-lp1912118-focal systemd[1]: Started Dovecot IMAP/POP3 email server.
May 04 23:01:36 dovecot-sru-lp1912118-focal dovecot[306]: master: Dovecot v2.3.7.2 (3c910f64b) starting up without an>
root@dovecot-sru-lp1912118-focal:/home/ubuntu#

The first option you could do to help is since you *can* reproduce it, would be to install the PPA I mentioned above and verify whether that package solves the problem on your end. Although I much prefer being able to reproduce issues myself, that would at least allow us to move ahead with the SRU to get the fix released publicly.

The second option is if you could send me an inbox that reproduces it for you - not necessarily the one you have now, but maybe download the emails from the debian bug report and add them to your inbox. (I'm wondering if I constructed the inbox incorrectly.)

The third thing is if you have anything non-stock in your dovecot config files, to attach them. Also if you have any 3rd party FTS software, or other dovecot-* components installed, let me know. That will help me ensure I'm more closely reproducing your environment.

Thanks ahead of time.
Bryce (<email address hidden>)

[Dave Moskovitz (dave-thinktank)]

Hi Bryce,

I've installed the PPA on a cloned image of the server in question, and can confirm that it fixes the problem.

There is nothing non-stock in my environment.

Thank you so much for working on this! It's greatly appreciated.

Best regards
Dave

[Bryce Harrington (bryce)]

Hi Dave,

Thanks for confirming the fix. We've had a second confirmation of the issue and fix by Timo, so although I've not been able to reproduce this myself, I'll proceed with rolling the fix out. I'll need to ask help from you and Timo to perform the validation when the SRU team requests it.

[Chris Halse Rogers (raof)]

Hello Dave, or anyone else affected,

Accepted dovecot into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dovecot/1:2.3.7.2-1ubuntu3.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Bryce Harrington (bryce)]

Dave and Timo - can you please install dovecot from focal-proposed (as described above) and verify that the package update solves the search issue you encountered?

Thanks ahead of time!

[Timo (marove2000)]

Hello Bryce,

sorry for the long wait. I've installed dovecot-core with your new build and can confirm: the bug is fixed :).
Thank you very much for your help.

Unfortunately I can not find the option do add "verification-done-focal". Can you perform this operation?

Regards
Timo

[Bryce Harrington (bryce)]

Sure, thanks & done!

[Launchpad Janitor (janitor)]

This bug was fixed in the package dovecot - 1:2.3.7.2-1ubuntu3.5

---------------
dovecot (1:2.3.7.2-1ubuntu3.5) focal; urgency=medium

  * d/p/handle-unbounded-mime.patch: Fix crash during deinit when
    searching mails with non-ending MIME boundaries.
    (LP: #1912118)

 -- Bryce Harrington <email address hidden> Sun, 11 Apr 2021 13:25:41 -0700

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for dovecot has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.