doveadm requires read access to SSL key

[Michael Stilkerich]

[Impact]
When running doveadm on a dovecot configured to use SSL, if the user doesn't have read permissions for ssl_cert or ssl_key, doveadm will fail with a fatal error and non-zero exit code. This impacts cronjobs and tools like Postfixadmin, as is being widely reported in the wild.

[Test Plan]
* Prepare the LXC container:

  $ sudo apt-get update
  $ sudo apt-get -y full-upgrade

* Do a dovecot configuration with SSL certificates, where the SSL key is
  only readable by the root user. These restrictive permissions on the
  SSL key should work per the dovecot configuration, as the server reads
  the files before dropping its privileges.

  https://doc.dovecot.org/configuration_manual/dovecot_ssl_configuration/
  "Dovecot opens both of these files while still running as root, so you
  don’t need to give Dovecot any special permissions to read them (in
  fact: do not give dovecot user any permissions to the key file)."

https://ubuntu.com/server/docs/mail-dovecot

Generate a custom certificate using openssh:
$ sudo openssl req -new -x509 -days 1000 -nodes -out "/etc/dovecot/dovecot.pem" \
    -keyout "/etc/dovecot/private/dovecot.pem"

* Set the SSL settings by editing /etc/dovecot/conf.d/10-ssl.conf and
  make sure the cert and key config looks like this:

  ssl_cert = </etc/dovecot/private/dovecot.pem
  ssl_key = </etc/dovecot/private/dovecot.key

  $ sudo systemctl stop dovecot.service
  $ sudo systemctl start dovecot.service

* Run doveadm as root, it should output the help text.

  $ sudo doveadm
  usage: doveadm [-Dv] [-f <formatter>] <command> [<args>]

    altmove [-u <user>|-A] [-S <socket_path>] [-r] <search query>
  ...

* Run doveadm as another user, e.g. the one set for mail_uid in the
  dovecot configuration (for me, this is the user vmail) - doveadm will
  not show the help text but die with an error like:

  $ doveadm
  doveconf: Fatal: Error in configuration file /etc/dovecot/conf.d/10-ssl.conf line 12: ssl_key: Can't open file <...>: Permission denied

That error will be gone if you apply the above commits to the 2.3.16
source (tried that). Whether there are any regressions I cannot tell
though.

Due to the quantity of lines changed and the types of changes being made, that it would be worth longer than usual testing in -proposed before rolling out to all users.

User testing in production can be of value, since the principle impact is the side effects from hitting the fatal error, rather than the error itself.

[Where Problems Could Occur]
The fix itself is straightforward, but it builds on several other refactoring patches. Since that's a large number of lines affected it's infeasible to visually ascertain the correctness of the changes. While in theory these could be re-implemented to minimize the number of code lines changed, doing so would introduce some unknown level of risk of adding errors via typos or other mistakes. For that reason, the new proposed package carries the set of patches unmodified from upstream.

The refactoring changes include changes to internal APIs (i.e. new and changed functions and structs) and behavioral changes when SSL is enabled. Dovecot is a service rather than a library, so the API changes shouldn't affect other software shipped in Ubuntu, however there exists the potential that there may be internal callers to the changed APIs that were missed in the refactoring.

Due to the patches changing behavior with SSL, it would be also worth watching for odd behavioral effects when enabling SSL, and issues related to configuration.

[Original Report]
Since upgrading from focal to jammy, I have issues with my cronjobs running doveadm as the vmail user (e.g., to train bayes filter). doveadm dies as it tries to read the SSL private key, although it does not need it.

This is a known bug in dovecot that was fixed with 2.3.17. I believe this is critical since granting the vmail user read permissions to the private SSL key is not desirable from a security perspective.

The corresponding entry in the Dovecot 2.3.17 changelog:
doveadm: v2.3.11 regression: Commands failed if ssl_cert or ssl_key files weren't readable by the user running doveadm, even though doveadm didn't actually use these settings

Answers to the requested information:

Description: Ubuntu 22.04.1 LTS
Release: 22.04

dovecot-core:
  Installed: 1:2.3.16+dfsg1-3ubuntu2.1
  Candidate: 1:2.3.16+dfsg1-3ubuntu2.1
  Version table:
 *** 1:2.3.16+dfsg1-3ubuntu2.1 500
        500 http://de.archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages
        500 http://de.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1:2.3.16+dfsg1-3ubuntu2 500
        500 http://de.archive.ubuntu.com/ubuntu jammy/main amd64 Packages

[Lucas Kanashiro (lucaskanashiro)]

Thanks for taking the time to report this bug and trying to make Ubuntu better.

This was indeed fixed in version 2.3.17 per upstream NEWS file:

https://github.com/dovecot/core/blob/2.3.17/NEWS#L40-L42

However, going through the commits I was not able to easily spot the needed commit(s) to be backported to Jammy. I am adding this bug to our backlog.

[Michael Stilkerich (mstilkerich)]

Hi,

  how about this commit range:

https://github.com/dovecot/core/compare/77dd654...368a967

[Athos Ribeiro (athos-ribeiro)]

Hi Michael,

Thanks for the pointers. That range does look promising. Next steps here should be to evaluate those patches (are they all needed for the fix?) and push a backport of those to a PPA so we can ensure they indeed fix the issue (and introduce no regressions).

In the meanwhile, would you be able to provide a simple/minimal reproducer so we can test a potential fix?

[Michael Stilkerich (mstilkerich)]

Hi Athos,

  I think the needed commit is 36ff43f. The ones before do some code refactorings without that this commit will not apply. The one commit after 36ff43f that I included in the list does some cleanup work (removing unneeded stuff) and should not strictly be needed.

Reproducing the issue is fairly simply:

- Do a dovecot configuration with SSL certificates, where the SSL key is only readable by the root user. These restrictive permissions on the SSL key should work per the dovecot configuration, as the server reads the files before dropping its privileges [1].

- For this, set the SSL settings like this:

ssl_cert = </some/path/fullchain.cer
ssl_key = </some/path/example.com.key

- Run doveadm as root, it should output the help text.

- Run doveadm as another user, e.g. the one set for mail_uid in the dovecot configuration (for me, this is the user vmail) - doveadm will not show the help text but die with an error like:

doveconf: Fatal: Error in configuration file /etc/dovecot/conf.d/10-ssl.conf line 12: ssl_key: Can't open file /some/path/example.com.key: Permission denied

That error will be gone if you apply the above commits to the 2.3.16 source (tried that). Whether there are any regressions I cannot tell though.

[1] https://doc.dovecot.org/configuration_manual/dovecot_ssl_configuration/: "Dovecot opens both of these files while still running as root, so you don’t need to give Dovecot any special permissions to read them (in fact: do not give dovecot user any permissions to the key file)."

[Bryce Harrington (bryce)]

Commit 36ff43f by itself does not apply cleanly, so either some of the refactorings will need included, or 36ff43f will need modification to be backportable.

In any case, from a cursory read of the patch, this changes some behavior that may need to be thought through to make sure it doesn't accidentally introduce a regression for existing users.

[Bryce Harrington (bryce)]

Hi,

I've backported the patch, applying the specific refactorings needed. This will need doublechecked to make sure no regression is introduced.

I've packaged the fix into a PPA here for folks to test:

   https://launchpad.net/~bryce/+archive/ubuntu/dovecot-sru-lp1991564/

Please install and verify it does indeed fix the original issue:

   $ sudo add-apt-repository -yus ppa:bryce/dovecot-sru-lp1991564
   $ sudo apt-get install dovecot-core
   <steps as described in comment #4>

[Bryce Harrington (bryce)]

Using the test case suggested in comment #4, I installed the archive version of dovecot, then as a non-root user:

  dovecot-sru-lp1991564-jammy+22.04: ~$ doveadm
  doveconf: Fatal: Error in configuration file /etc/dovecot/conf.d/10-ssl.conf line 12: ssl_cert: Can't open file /etc/dovecot/private/dovecot.pem: Permission denied

The backport of just commit 36ff43f proved insufficient to fix the issue, as the same error prints out.

I've updated the PPA with a new package version that includes the full range of commits mentioned in comment #2, and that does indeed appear to resolve that error:

dovecot-sru-lp1991564-jammy+22.04: ~$ doveadm
Error: net_connect_unix(/run/dovecot/stats-writer) failed: Permission denied
usage: doveadm [-Dv] [-f <formatter>] <command> [<args>]

  altmove [-u <user>|-A] [-S <socket_path>] [-r] <search query>
  auth cache|login|lookup|test
  backup [-u <user>|-A] [-S <socket_path>] [-fPRU] [-l <secs>] [-r <rawlog path>] [-m <mailbox>] [-g <mailbox_guid>] [-n <namespace> | -N] [-x <exclude>] [-s <state>] [-t <start date>] -d|<dest>
  batch [-u <user>|-A] [-S <socket_path>] <sep> <cmd1> [<sep> <cmd2> [..]]
  config [doveconf parameters]
  [...]

I'm not sure about that initial Error line, but at least now it prints usage as per expected from comment #4.

Before proceeding with an SRU, I'd like to see broader testing of this PPA by several Dovecot users affected by this problem, both to verify the fix is worthwhile and that the package doesn't introduce any other obvious regressions. The reason is that the patchset to fix this is on the larger side for what we generally look for in SRUs (e.g. we can't prove the changes are error-free just by visual inspection). But if we know this fixes issues impacting multiple users, that may help justify the SRU as worth the risk. Meanwhile hopefully the PPA will be useful for you to resolve the issue locally.

[Sean (strobilomyces)]

I tested Bryce's PPA above (thank you), which of course got rid of the SSL error.

You can fix the stats-writer error by changing the mode for stats-reader and stats-writer in /etc/dovecot/conf.d/10-master.conf to 0666.

This is my working block:

service stats {
  unix_listener stats-reader {
    user = vmail
    group = vmail
    mode = 0666
  }

  unix_listener stats-writer {
    user = vmail
    group = vmail
    mode = 0666
  }
}

On restart, doveadm runs as expected.

As far as whether it's worthwhile patching Dovecot, for me this was a blocking error in Postfixadmin. In recent versions, using Dovecot 2.3.16, you get warnings that interrupt the setup process and, more critically, you can't log into the Postfixadmin panel. In searching for a fix for this, I saw quite a few people--both on the PFA repo and sites like Server Fault--who weren't able to resolve this without building Dovecot from source.

Distributor ID: Ubuntu
Description: Ubuntu 22.04.1 LTS
Release: 22.04
Codename: jammy

[Bryce Harrington (bryce)]

I've uploaded the SRU for this to the jammy SRU review queue, however jammy is in a release freeze so this may not roll out for a couple weeks. Which is perfectly fine -- this change would benefit from extra testing by users to make sure it isn't introducing any regressions.

This PPA now has the same fix that is being proposed for jammy-updates, so please install the package from here and report to this bug report over the coming week or two if you observe any problems, or if the package is working smoothly. The more feedback we can gather, the more assurance the SRU team will have that this is safe to deploy.

PPA: https://launchpad.net/~bryce/+archive/ubuntu/dovecot-sru-lp1991564

$ sudo add-apt-repository ppa:bryce/dovecot-sru-lp1991564
$ sudo apt update
$ sudo apt dist-upgrade
$ apt-cache policy dovecot-*

[Sean (strobilomyces)]

So far so good for me. Thanks again.

[Robie Basak (racb)]

I took a look at this for the SRU team. I appreciate the challenge here. I see that 2.3.17 is only "one away" from 2.3.16 that we have in Jammy, so maybe that is safer than cherry-picking extensive changes? But there is also 2.3.17.1 which fixes some OpenSSL issues. Are any of these related to the commits being cherry-picked in this upload?

I think maybe the only thing we can do is a very deep review of the necessary changes. That's going to take a while :-/

[Bryce Harrington (bryce)]

The fixes included in 2.3.17.1 are given by this range, and summarized below:

    https://github.com/dovecot/core/compare/2.3.17...2.3.17.1

1. lib-ssl-iostream: Fix assert-crash when OpenSSL returns syscall error
  -> Fixes regression caused by f41874b

2. dsync: Add back missing -T parameter
  -> Fixes regression caused by 5676c51

3. lib-master: Add MASTER_SERVICE_FLAG_DISABLE_SSL_SET
  master: Use MASTER_SERVICE_FLAG_DISABLE_SSL_SET
  -> Fixes regression caused by 36ff43f (***)

4. dsync: Add back various missing parameters
  -> Fixes regression caused by 5676c51

5. man: Add missing parameters to doveadm-sync

6. dsync: Fix -I max-size to actually be the max-size rather than min-size

For reference, here are the commits for the changes this branch introduces:

    https://github.com/dovecot/core/compare/77dd654...368a967

79a210c1f7e94a1863f17db0b9f14b6d3c89e15f
c6cea575772189495a612d4e55f207a2d5009db2
a7fb3cce6a6048e52ce657f9085d54b24bc8a28d
36ff43f1a9aff8594d08f791e77ea13390fd569f (***)
368a96783e8a64a62d07da4adcf2f3b1b525380b

This indicates that at least the two commits from item (3) in 2.3.17.1 relates to the changes included in this branch. Note that the affected commit, 36ff43f, is not one of the refactoring changes but is rather the fix itself that this branch seeks to integrate.

I did not find a bug report for this fix, however the related discussion appears to be this thread:

    https://www.mail-archive.com/search?<email address hidden>&q=subject:%22Re%5C%3A+2.3.17+broken+on+CentOS8+%5C%2F+bug%22&o=newest&f=1

The commit includes a comment that describes the problem being solved by the added option:

        /* Don't read any SSL settings. This is mainly needed to prevent master
    process from trying to pass through huge list of SSL CA certificates
    through environment for ssl_ca setting, which could fail. Although
    the same problem can still happen with standalone doveadm if it
    reads settings via doveconf instead of config socket. */

[Alex Dehnert (adehnert)]

What's the current status here? It sounds like package from ppa:bryce/dovecot-sru-lp1991564 turns out to have a regression? (I haven't noticed an issue, but I'm running it on server that's just a replication target, and only for a couple days so far.)

[Bryce Harrington (bryce)]

Hi Alex, there isn't a regression, this SRU is waiting for the SRU administration team to finish reviewing. I'm not sure the reason for the delay, but assuming it's due to the large line-length of the patches included.

Thank you for running the PPA with the fix and sharing your experience. Would you say the system reproduces the issue without the PPA, and is resolved (with no side-effects) with the PPA installed?

[Steve Langasek (vorlon)]

Well, that is some rather extensive refactoring of the code, but I've reviewed it and it all makes sense in context (though I think the goal could have been accomplished without the refactoring).

[Steve Langasek (vorlon)]

Hello Michael, or anyone else affected,

Accepted dovecot into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dovecot/1:2.3.16+dfsg1-3ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Alex Dehnert (adehnert)]

I'm not very good about noticing updates, sorry...

Re Bryce (#14): I think so -- my recollection is I seemed to be seeing this bug before upgrading from the PPA, and then it stopped. I downgraded back to stock jammy(-updates) and confirmed breakage:
ssh -ldovecot-vmail $host doveadm dsync-server -u$user
doveconf: Fatal: Error in configuration file /etc/dovecot/conf.d/10-ssl.conf line 12: ssl_cert: Can't open file /etc/dovecot/dovecot.pem: Permission denied

and "doveadm replicator replicate $name" followed by "doveadm replicator status '*'" reported it failed to replicate.

While running with the PPA, I hadn't noticed any obvious issues with replication or anything else.

So yes, I repro without the PPA and the PPA resolves the issue with no noticed side-effects.

Re Steve (#16): I just upgraded from jammy-proposed. The replication command no longer errors, and triggering replication results in a successful sync status (same commands as above). I haven't noticed any issues yet, but I've been running this build for about five minutes so that's not *especially* strong evidence.

Package versions:
# dpkg -l dovecot-* | grep ^ii
ii dovecot-core 1:2.3.16+dfsg1-3ubuntu2.2 arm64 secure POP3/IMAP server - core files
ii dovecot-gssapi 1:2.3.16+dfsg1-3ubuntu2.2 arm64 secure POP3/IMAP server - GSSAPI support
ii dovecot-imapd 1:2.3.16+dfsg1-3ubuntu2.2 arm64 secure POP3/IMAP server - IMAP daemon
ii dovecot-lmtpd 1:2.3.16+dfsg1-3ubuntu2.2 arm64 secure POP3/IMAP server - LMTP server
ii dovecot-sieve 1:2.3.16+dfsg1-3ubuntu2.2 arm64 secure POP3/IMAP server - Sieve filters support

[Alex Dehnert (adehnert)]

The jammy-proposed version has now been running for about two days. dovecot handles IMAP access as expected and replication seems to work. There's some replication lag and a couple temporary failures according to my monitoring, but I have no particular reason to believe those are new (or not new, admittedly, I don't usually pay close attention).

[Przemyslaw Kwiatkowski (micha-micha)]

Steve (#16): As far as I can see - there is no amd64 version available in -proposed. (Why?)

The PPA version seems to work very well. I would lovely switch to -proposed and test more. :-)

[Robie Basak (racb)]

This FTBFS on amd64 on jammy-proposed so certainly cannot have been verified. @Bryce, please could you take a look? It might also be worth reconsidering the testing approach here given this was mistakenly considered verified.

[Robie Basak (racb)]

> It might also be worth reconsidering the testing approach here given this was mistakenly considered verified.

My mistake. It was verified on arm64, looking at the comments. So no problem with that verification, but we do need the amd64 side investigated and fixed before we can release.

[Przemyslaw Kwiatkowski (micha-micha)]

#21:
> but we do need the amd64 side investigated and fixed before we can release.

I can test it as soon as amd64 build is available.

[Bryce Harrington (bryce)]

I investigated the amd64 build issue. It was simply a test case that hit some sort of CPU time limit, which presumably was due to autopkgtest infrastructure flakiness. A trivial rebuild took care of it. Shame someone didn't already try that, although honestly this is something autopkgtest itself probably should do automatically since it's so trivial.

https://launchpad.net/ubuntu/+source/dovecot/1:2.3.16+dfsg1-3ubuntu2.2
* Jammy:
  - [FULLYBUILT] amd64 (Accepted)
  - [FULLYBUILT] arm64
  - [FULLYBUILT] armhf
  - [FULLYBUILT] ppc64el
  - [FULLYBUILT] riscv64
  - [FULLYBUILT] s390x

[Przemyslaw Kwiatkowski (micha-micha)]

Just installed amd64. Seems to work. :-)
Will watch for problems... :-)

[Bryce Harrington (bryce)]

triage-jammy+22.04: ~$ sudo apt-get install dovecot-core=1:2.3.16+dfsg1-3ubuntu2
...
The following additional packages will be installed:
  ssl-cert
Suggested packages:
  dovecot-gssapi dovecot-imapd dovecot-ldap dovecot-lmtpd dovecot-lucene dovecot-managesieved dovecot-mysql
  dovecot-pgsql dovecot-pop3d dovecot-sieve dovecot-solr dovecot-sqlite dovecot-submissiond ntp
The following NEW packages will be installed:
  dovecot-core ssl-cert
...
Created symlink /etc/systemd/system/multi-user.target.wants/dovecot.service → /lib/systemd/system/dovecot.service.
dovecot.socket is a disabled or a static unit not running, not starting it.
Processing triggers for man-db (2.10.2-1) ...
triage-jammy+22.04: ~$ sudo openssl req -new -x509 -days 1000 -nodes -out "/etc/dovecot/dovecot.pem" \
> -keyout "/etc/dovecot/private/dovecot.pem"
............+.........+...............+............+
...
triage-jammy+22.04: ~$ ls -l /etc/dovecot/private/dovecot.pem
ls: cannot access '/etc/dovecot/private/dovecot.pem': Permission denied
triage-jammy+22.04: ~$ ls -l /etc/dovecot/dovecot.pem
-rw-r--r-- 1 root root 1224 May 25 18:50 /etc/dovecot/dovecot.pem
triage-jammy+22.04: ~$ grep /etc/dovecot/private /etc/dovecot/conf.d/10-ssl.conf
ssl_cert = </etc/dovecot/private/dovecot.pem
ssl_key = </etc/dovecot/private/dovecot.key

triage-jammy+22.04: ~$ sudo systemctl stop dovecot.service
triage-jammy+22.04: ~$ sudo systemctl start dovecot.service
triage-jammy+22.04: ~$ sudo doveadm
usage: doveadm [-Dv] [-f <formatter>] <command> [<args>]

  altmove [-u <user>|-A] [-S <socket_path>] [-r] <search query>
...
triage-jammy+22.04: ~$ doveadm
doveconf: Fatal: Error in configuration file /etc/dovecot/conf.d/10-ssl.conf line 12: ssl_cert: Can't open file /etc/dovecot/private/dovecot.pem: Permission denied
triage-jammy+22.04: ~$

Then testing fixed version after enabling -proposed:

triage-jammy+22.04: ~$ sudo apt-get -y full-upgrade
triage-jammy+22.04: ~$ sudo apt-get install dovecot-core
triage-jammy+22.04: ~$ sudo apt-cache policy dovecot-core
dovecot-core:
  Installed: 1:2.3.16+dfsg1-3ubuntu2.2
  Candidate: 1:2.3.16+dfsg1-3ubuntu2.2
  Version table:
 *** 1:2.3.16+dfsg1-3ubuntu2.2 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1:2.3.16+dfsg1-3ubuntu2.1 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     1:2.3.16+dfsg1-3ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
triage-jammy+22.04: ~$ groups
bryce root sudo staff
triage-jammy+22.04: ~$ sudo systemctl stop dovecot.service
triage-jammy+22.04: ~$ sudo systemctl start dovecot.service
triage-jammy+22.04: ~$ doveadm
usage: doveadm [-Dv] [-f <formatter>] <command> [<args>]

  altmove [-u <user>|-A] [-S <socket_path>] [-r] <search query>
...

[Przemyslaw Kwiatkowski (micha-micha)]

It works 3 days, no issues.

[Launchpad Janitor (janitor)]

This bug was fixed in the package dovecot - 1:2.3.16+dfsg1-3ubuntu2.2

---------------
dovecot (1:2.3.16+dfsg1-3ubuntu2.2) jammy; urgency=medium

  * Backport fix + refactoring for handling ssl settings
    (LP: #1991564)
    - d/p/split-master_service_ssl_settings_to_iostream_set-to-client-server-functions.patch
    - d/p/split-off-master_service_ssl_server_settings.patch
    - d/p/remove-unused-master_service_is_ssl_module_loaded.patch
    - d/p/use-ssl-server-settings-only-when-necessary.patch
    - d/p/remove-unnecessary-master_service_flag_use_ssl_settings.patch

 -- Bryce Harrington <email address hidden> Wed, 11 Jan 2023 06:39:32 -0800

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for dovecot has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.