SSL issue upgrading postfix

[tom]

[Impact]

 * On upgrade depending on config postfix might mess with said config as
   it mis-detects being an upgrade from postfix-2.

 * This was fixes in Debian later on, this cherry picks the fix

[Test Case]

 * have a config with
    submission inet n - - - - smtpd
   And then upgrade (reinstall), example:
   - apt install postfix
   - change the line to the above in /etc/postfix/master.cf
     (that was essentially removing a comment and y -> -)
   - apt install --reinstall postfix
   This will have changed the config:
     - on upgrade you see
       setting explicit chroot on /etc/postfix:submission/inet/chroot
     - the value now is
       submission inet n - y - - smtpd

[Regression Potential]

 * Upgrades behave slightly different, but in the way we want. We can
   break down into two things:
   1. only if coming from the right former versions (wanted)
   2. if the upgrade is done it is done in a safer way (wanted)

[Other Info]

 * In a perfect world the version compare would be against the first
   ubuntu 3.x version, but taking the cherry pick as-is appears much
   cleaner for review and also there was never anything released (nor
   seems it remotely likely to be) that would fall in the gap between
   the first Debian / Ubuntu postfix 3 version.

----

issue upgrading posfix on Ubuntu 16.04.3 LTS

in file master.cf, line submission=

the upgrade procedure has changed the field chroot to 'y', but it was '-'. 'y' is wrong

#pre upgrade it was
submission inet n - - - - smtpd

#then it become
submission inet n - y - - smtpd

the 'y' is wrong

http://www.postfix.org/master.5.html

sais "Chroot (default: Postfix >= 3.0: n, Postfix <3.0: y)"

with 'y' the server rejects all SSL smtp mails

[Christian Ehrhardt  (paelzer)]

Hi Tom,
on the version originally release in Xenial I see:

3.1.0-3:
#submission inet n - y - - smtpd

3.1.0-3ubuntu0.1
#submission inet n - y - - smtpd

After the upgrade it still is the same, and it is a comment in both.
There is another similar line related, which is:

3.1.0-3:
smtp inet n - y - - smtpd

3.1.0-3ubuntu0.1
smtp inet n - y - - smtpd

The switch form "-" to "y" means swicth from built-in default to an explicit yes, the column is for using a chroot. The doc you are referring is exactly documenting that:
  Chroot (default: Postfix >= 3.0: n, Postfix <3.0: y)
That does not mean "set y / n in that file" but instead it means if you have set "-" then depending on the version this is what you get.

Since on my upgrade I didn't see the change I checked for a potential Trusty->Xenial upgrade.
There I found the config change (as upstream changed defaults from 2->3).

On trusty it had:
2.11.0-1ubuntu1:
smtp inet n - - - - smtpd
[...]
#submission inet n - - - - smtpd

So on 2.x it had "-" which was implying the default (which was yes), but on 3.x it is "y" (as the default is no now) to have no change in behavior upon upgrade.

Since "submission" is a comment anyway I looked at what happens on upgrade.
Note: This is no normal conffile, it is handled by the *inst scripts.

So I took a trusty system, modified the conf to have the line uncommented.
Then I upgraded.
There the file is as it was before and the admin has to fix it u on upgrade.
But I think this was not forgotten, instead it is just too complex to convert the old config to a surely working new one.

So yes going from Trusty to Xenial if you had "-" set (which was default yes) it now changes its meaning to default "no". The defaults of a new install are good, and I found the following from the changelog:
postfix (3.1.0-1) unstable; urgency=medium

  [LaMont Jones]

  * Explicitly chroot services that we want chrooted in master.cf on fresh
    installs.
  * Convert defaults as needed for 3.0+ on upgrade to minimize compatibility
    warnings.

Not sure if there is better handling that I overlook.
Subscribing LaMont for an opinion.

[tom (tmassimi)]

Hi,

thanks for the investigation, I can give you some other details:

I made a system update with a lot of packages, including this one:
postfix:amd64 3.1.0-3 => 3.1.0-3ubuntu0.1
which I presumed is the cause the problem.

Looking at the package diff I cannot see where the problem is

http://launchpadlibrarian.net/332320988/postfix_3.1.0-3_3.1.0-3ubuntu0.1.diff.gz

but I can tell you that update has changed the file main.cf in this way:

 Index: etc/postfix/master.cf
===================================================================
--- etc/postfix/master.cf (revision 3249)
+++ etc/postfix/master.cf (working copy)
@@ -14,11 +14,6 @@
 #tlsproxy unix - - - - 0 tlsproxy

 # 3 feb 2017: open port 587
-submission inet n - - - - smtpd
- -o syslog_name=postfix/submission
- -o smtpd_tls_security_level=encrypt
- -o smtpd_sasl_auth_enable=yes
- -o smtpd_client_restrictions=permit_sasl_authenticated,reject
 # -o milter_macro_daemon_name=ORIGINATING
 #smtps inet n - - - - smtpd
 # -o syslog_name=postfix/smtps
@@ -27,6 +22,11 @@
 # -o smtpd_client_restrictions=permit_sasl_authenticated,reject
 # -o milter_macro_daemon_name=ORIGINATING
 #628 inet n - - - - qmqpd
+submission inet n - y - - smtpd
+ -o syslog_name=postfix/submission
+ -o smtpd_tls_security_level=encrypt
+ -o smtpd_sasl_auth_enable=yes
+ -o smtpd_client_restrictions=permit_sasl_authenticated,reject
 smtp inet n - y - 10 smtpd
 pickup fifo n - y 60 1 pickup
 cleanup unix n - y - 0 cleanup

(Yes, I have the whole /etc saved in svn on a second machine,
each night the second machine rsyncs its '/etc working copy' from the main server,
then cron sends me an email with the svn diff)

the problem is that with that 'y' the server rejects some email,
reverting it to '-' all is fine as it was before that update
(I mean, the "submission inet" line was not commented before the update)

this is the full list of the updated packages in that day,
maybe some other package was involved?

 --------------------- dpkg status changes Begin ------------------------

 Upgraded:
    apache2-bin:amd64 2.4.18-2ubuntu3.4 => 2.4.18-2ubuntu3.5
    apache2-data:all 2.4.18-2ubuntu3.4 => 2.4.18-2ubuntu3.5
    apache2-utils:amd64 2.4.18-2ubuntu3.4 => 2.4.18-2ubuntu3.5
    apache2:amd64 2.4.18-2ubuntu3.4 => 2.4.18-2ubuntu3.5
    bind9-host:amd64 1:9.10.3.dfsg.P4-8ubuntu1.7 => 1:9.10.3.dfsg.P4-8ubuntu1.8
    binutils:amd64 2.26.1-1ubuntu1~16.04.4 => 2.26.1-1ubuntu1~16.04.5
    ca-certificates:all 20160104ubuntu1 => 20170717~16.04.1
    cpp-5:amd64 5.4.0-6ubuntu1~16.04.4 => 5.4.0-6ubuntu1~16.04.5
    cryptsetup-bin:amd64 2:1.6.6-5ubuntu2 => 2:1.6.6-5ubuntu2.1
    cryptsetup:amd64 2:1.6.6-5ubuntu2 => 2:1.6.6-5ubuntu2.1
    curl:amd64 7.47.0-1ubuntu2.2 => 7.47.0-1ubuntu2.4
    dnsutils:amd64 1:9.10.3.dfsg.P4-8ubuntu1.7 => 1:9.10.3.dfsg.P4-8ubuntu1.8
    g++-5:amd64 5.4.0-6ubuntu1~16.04.4 => 5.4.0-6ubuntu1~16.04.5
    gcc-5-base:a...

[Christian Ehrhardt  (paelzer)]

Hi,
wise to track all of /etc.
I can't reproduce to have this update changing my postfix conf.

The only change that would touch this in a meaningful way should be form a 2.x to a 3.x installation.

Hmm, the only thing that would remotely make sense is that the fix to "- debian/postfix.postinst: Provide main/master.cf.proto for multi-inst." somehow affects you.

The change is small enough to post it here:
+# Make sure we have main/master.cf.proto for multi-inst (#838528)
+if [ ! -f /etc/postfix/main.cf.proto ]; then
+ cp /usr/share/postfix/main.cf.dist /etc/postfix/main.cf.proto
+fi
+if [ ! -f /etc/postfix/master.cf.proto ]; then
+ cp /usr/share/postfix/master.cf.dist /etc/postfix/master.cf.proto
+fi
+

That doesn't directly overrride the .cf that changed for you, but in case you run multi-inst afterwards it might change it.

Hmm, maybe I'm just too inexperienced in postfix setups.
Can you re-create this in a VM guest or such to have a few clear steps like
1. install postfix as version X
2. do magic I miss
3. upgrade to version Y
4. see the change

I enabled the submission line(s) in a test system, but while there was some buzz about the config it didn't change (http://paste.ubuntu.com/25871780/).
I tried but it did only change the file when going 2.x to 3.x for me.

Maybe LaMont has a better idea what might be going on :-/

[tom (tmassimi)]

Hello, it happened again.

this time I have the log of the installation procedure:

========
Setting up postfix (3.1.0-3ubuntu0.2) ...
Installing new version of config file /etc/init.d/postfix ...
initctl: Unable to connect to Upstart: Failed to connect to socket
/com/ubuntu/upstart: Connection refused
insserv: warning: script 'screen-cleanup' missing LSB tags and overrides
insserv: Default-Start undefined, assuming empty start runlevel(s) for
script `screen-cleanup'
insserv: Default-Stop undefined, assuming empty stop runlevel(s) for
script `screen-cleanup'
setting explicit chroot on /etc/postfix:submission/inet/chroot

Postfix is now set up with the changes above. If you need to make changes, edit
/etc/postfix/main.cf (and others) as needed. To view Postfix configuration
values, see postconf(1).

After modifying main.cf, be sure to run '/etc/init.d/postfix reload'.

Running newaliases
Setting up procmail (3.22-25ubuntu0.16.04.1) ...
========

this line is strange, I don't think my postfix is running in chroot

setting explicit chroot on /etc/postfix:submission/inet/chroot

[Christian Ehrhardt  (paelzer)]

The line you wonder essentially does:
  set_postconf -F -c "/etc/postfix" submission/inet/chroot=y

And yes this is setting the "y" in your case I'd think.

As I outlined before this was (I guess) meant to carry the old default to stay the same on an upgrade.
IMHO this is fine on a certain upgrade from 2.x to 3.x to carry on old behavior, but not later on.
To do so it is in a guard like this:
  if `dpkg --compare-versions "$2" lt 3.1.0-1~`; then

It also is only done if the user selects "no configuration" in the config prompt.

So when coming FROM a upgrade pre Xenial this would be done once.
Which is kind of ok (not want to go into too much details)
- I'd have wanted a check if it was "-" before doing so
- also the check should be "lt-nl" to not trigger on a fresh install

But those details aside, on your system this should only trigger if you had been upgrading from 2.x to 3.x.
And if it did the former "-" was a "yes" and is now an explcit "y" just for this reason.
See [1] about the issue it silences instead of keeping "-"

If you hit this on an upgrade from 3.x -> 3.x that would be wrong - did you?.

[Christian Ehrhardt  (paelzer)]

[1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=816172

[tom (tmassimi)]

> If you hit this on an upgrade from 3.x -> 3.x that would be wrong - did you?.

yes, the log says it was an upgrade from
3.1.0-3ubuntu0.1
to
3.1.0-3ubuntu0.2

/var/log# grep post dpkg.log.1
2017-12-19 12:56:30 upgrade postfix:amd64 3.1.0-3ubuntu0.1 3.1.0-3ubuntu0.2
2017-12-19 12:56:30 status half-configured postfix:amd64 3.1.0-3ubuntu0.1
2017-12-19 12:56:32 status unpacked postfix:amd64 3.1.0-3ubuntu0.1
2017-12-19 12:56:32 status half-installed postfix:amd64 3.1.0-3ubuntu0.1
2017-12-19 12:56:33 status half-installed postfix:amd64 3.1.0-3ubuntu0.1
2017-12-19 12:56:33 status unpacked postfix:amd64 3.1.0-3ubuntu0.2
2017-12-19 12:56:33 status unpacked postfix:amd64 3.1.0-3ubuntu0.2
2017-12-19 12:57:55 configure postfix:amd64 3.1.0-3ubuntu0.2 <none>
2017-12-19 12:57:55 status unpacked postfix:amd64 3.1.0-3ubuntu0.2
2017-12-19 12:57:55 status unpacked postfix:amd64 3.1.0-3ubuntu0.2
2017-12-19 12:57:55 status unpacked postfix:amd64 3.1.0-3ubuntu0.2
2017-12-19 12:57:55 status unpacked postfix:amd64 3.1.0-3ubuntu0.2
2017-12-19 12:57:55 status unpacked postfix:amd64 3.1.0-3ubuntu0.2
2017-12-19 12:57:55 status unpacked postfix:amd64 3.1.0-3ubuntu0.2
2017-12-19 12:57:55 status unpacked postfix:amd64 3.1.0-3ubuntu0.2
2017-12-19 12:57:55 status unpacked postfix:amd64 3.1.0-3ubuntu0.2
2017-12-19 12:57:55 status unpacked postfix:amd64 3.1.0-3ubuntu0.2
2017-12-19 12:57:55 status unpacked postfix:amd64 3.1.0-3ubuntu0.2
2017-12-19 12:57:55 status unpacked postfix:amd64 3.1.0-3ubuntu0.2
2017-12-19 12:57:55 status unpacked postfix:amd64 3.1.0-3ubuntu0.2
2017-12-19 12:57:55 status unpacked postfix:amd64 3.1.0-3ubuntu0.2
2017-12-19 12:57:55 status half-configured postfix:amd64 3.1.0-3ubuntu0.2
2017-12-19 12:57:57 status installed postfix:amd64 3.1.0-3ubuntu0.2

> To do so it is in a guard like this:
> if `dpkg --compare-versions "$2" lt 3.1.0-1~`; then

I guess here "$2" is the new version, this is what I can see on my machine:

# dpkg --compare-versions 3.1.0-3ubuntu0.0 lt 3.1.0-3ubuntu0.1~; echo $?
0
# dpkg --compare-versions 3.1.0-3ubuntu0.1 lt 3.1.0-3ubuntu0.1~; echo $?
1
# dpkg --compare-versions 3.1.0-3ubuntu0.2 lt 3.1.0-3ubuntu0.1~; echo $?
1
# dpkg --compare-versions 3.1.0-3ubuntu0.3 lt 3.1.0-3ubuntu0.1~; echo $?
1

it returns true at each update, but i don't know what you have in the 'then' part

[Christian Ehrhardt  (paelzer)]

I think I found it, the check to:
  if `dpkg --compare-versions "$2" lt 3.1.0-1~`; then
has no else clause that we could hit.

But by checking more in Detail I realized that this version check is not in all releases.
It was added in 3.1.3-1 and thereby is in all releases.
It is in Zesty and later but missing in Xenial and that is what hits you.

So this comes down to essentially https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=833103

[Christian Ehrhardt  (paelzer)]

https://git.launchpad.net/postfix/commit/?id=37b37e93b5b58ae8b01cdd68b2059922e891d0aa

[Christian Ehrhardt  (paelzer)]

Based on the insight that this was added to 3.x but post Xenial being released I provided a fix to be tried in ppa [1].
An associated merge proposal for the packaging is up for review at [2]

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3095
[2]: https://code.launchpad.net/~paelzer/ubuntu/+source/postfix/+git/postfix/+merge/335622

[Christian Ehrhardt  (paelzer)]

I updated the bug to include the SRU teamplate, but I wonder as I don't see my upload in the SRU queue.
Uploading again postfix_3.1.0-3ubuntu0.3_source.changes ... - this time it appears in the queue.
The upload is the same, so if there are anywhere two of them just kill one of them.

[Chris J Arges (arges)]

Hello tom, or anyone else affected,

Accepted postfix into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/postfix/3.1.0-3ubuntu0.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Christian Ehrhardt  (paelzer)]

Verified from xenial:
submission inet n - - - - smtpd
becomes
submission inet n - - y - smtpd
With proposed
submission inet n - - - - smtpd
stays
submission inet n - - - - smtpd

As intended for 3.x->3.x upgrades.

[Launchpad Janitor (janitor)]

This bug was fixed in the package postfix - 3.1.0-3ubuntu0.3

---------------
postfix (3.1.0-3ubuntu0.3) xenial; urgency=medium

  * Only run post 3.0 master.cf fixup if upgrading from version before
    the fix. (LP: #1727699)
    This is a pick of 37b37e93 from Debian packaging repo at
    https://git.launchpad.net/postfix

 -- Christian Ehrhardt <email address hidden> Tue, 02 Jan 2018 13:48:25 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for postfix has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.