upgrade from fresh bionic to focal needlessly prompts user

[Seth Arnold]

Upgrading from a fresh 18.04 LTS install to focal unexpectedly prompts for how to handle a change to /etc/ssh/sshd_config

To reproduce the issue:

lxc launch ubuntu:18.04 u18
lxc exec u18 -- bash
# within container
do-release-upgrade -d
# select restart services when prompted

Eventually you'll be prompted to accept changes to /etc/ssh/sshd_config or not because of "local changes".

Thanks

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: openssh-server 1:8.1p1-5
ProcVersionSignature: Ubuntu 4.15.0-62.69-generic 4.15.18
Uname: Linux 4.15.0-62-generic x86_64
ApportVersion: 2.20.11-0ubuntu16
Architecture: amd64
Date: Fri Jan 31 03:37:55 2020
ProcEnviron:
 TERM=rxvt-unicode-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
SourcePackage: openssh
UpgradeStatus: Upgraded to focal on 2020-01-31 (0 days ago)

[Marc Deslauriers (mdeslaur)]

Perhaps the hash is missing in debian/openssh-server.ucf-md5sum?

[Bryce Harrington (bryce)]

I'm able to easily reproduce this in lxc using the steps provided.

[Robie Basak (racb)]

Is this related to bug 1747464 perhaps?

[Bryce Harrington (bryce)]

Marc's comment #3 seems plausible:

stirling:~/ubuntu/Openssh$ lxc exec u18 -- bash
root@u18:~# md5sum /etc/ssh/sshd_config
203e9b92fe3623aeba277ee44297f7dd /etc/ssh/sshd_config
root@u18:~# grep -c 203e9b92fe3623aeba277ee44297f7dd /usr/share/openssh/sshd_config.md5sum
0
root@u18:~#

Looking at sshd_config on a fresh installed 18.04 lxc and a fresh 20.04, the sshd_config files do indeed differ by exactly the diff shown during upgrade (and provided in comment #2).

The md5sum checking was introduced in Debian on Dec 2016 with openssh (1:7.4p1-1)

  * Start handling /etc/ssh/sshd_config using ucf. The immediate motivation
    for this is to deal with deprecations of options related to protocol 1,
    but something like this has been needed for a long time (closes:
    #419574, #848089):
    - sshd_config is now a slightly-patched version of upstream's, and only
      contains non-default settings (closes: #147201).
    - I've included as many historical md5sums of default versions of
      sshd_config as I could reconstruct from version control, but I'm sure
      I've missed some.
    - Explicitly synchronise the debconf database with the current
      configuration file state in openssh-server.config, to ensure that the
      PermitRootLogin setting is properly preserved.
    - UsePrivilegeSeparation now defaults to the stronger "sandbox" rather
      than "yes", per upstream.

It's implemented in openssh-server.postinst:

        ...
        ... sed statements to customize $new_config from upstream for debian
        ...
        mkdir -p /etc/ssh
        ucf --three-way --debconf-ok \
                --sum-file /usr/share/openssh/sshd_config.md5sum \
                "$new_config" /etc/ssh/sshd_config
        ucfr openssh-server /etc/ssh/sshd_config

AFAICT the /usr/share/openssh/sshd_config.md5sum is identical on freshly lxc'd 18.04 and 20.04.

Running the ucf command on a focal lxc container with the 18.04 and 20.04 sshd_config files captured from fresh lxc installs reproduces the same debconf prompt about the changed config, and then issues this output:

stirling:~/ubuntu/Openssh/fix-apt-misprompt$ sudo ucf --no-action --three-way --debconf-ok --sum-file /usr/share/openssh/sshd_config.md5sum ./sshd_config.20.04 ./sshd_config.18.04
[sudo] password for bryce:
Replacing config file /home/bryce/ubuntu/Openssh/fix-apt-misprompt/sshd_config.18.04 with new version
cp -pf /home/bryce/ubuntu/Openssh/fix-apt-misprompt/sshd_config.18.04 /home/bryce/ubuntu/Openssh/fix-apt-misprompt/sshd_config.18.04.ucf-old
cp -pf /home/bryce/ubuntu/Openssh/fix-apt-misprompt/sshd_config.20.04 /home/bryce/ubuntu/Openssh/fix-apt-misprompt/sshd_config.18.04
cp -pf /var/lib/ucf/hashfile.6 /var/lib/ucf/hashfile.7
cp -pf /var/lib/ucf/hashfile.5 /var/lib/ucf/hashfile.6
cp -pf /var/lib/ucf/hashfile.4 /var/lib/ucf/hashfile.5
cp -pf /var/lib/ucf/hashfile.3 /var/lib/ucf/hashfile.4
cp -pf /var/lib/ucf/hashfile.2 /var/lib/ucf/hashfile.3
cp -pf /var/lib/ucf/hashfile.1 /var/lib/ucf/hashfile.2
cp -pf /var/lib/ucf/hashfile.0 /var/lib/ucf/hashfile.1
cp -pf /var/lib/ucf/hashfile /var/lib/ucf/hashfile.0
(egrep -v "[[:space:]]\/home\/bryce\/ubuntu\/Openssh\/fix\-apt\-misprompt\/ssh...

[Bryce Harrington (bryce)]

So, the trivial fix is to simply append 203e9b92fe3623aeba277ee44297f7dd to openssh-server.ucf-md5sum, as Marc had suggested above.

I can proceed with that as the fix.

---

But this suggests a few direct questions/thoughts:

0. Does the installer use the openssh-server.ucf-md5sum from the new package, or the previously installed one? If the latter, then the md5sum will need added via SRU.

1. Where in the process did the md5sum get out of sync? I'm not spotting conf changes from the CVE patches by our security team, so looks like this got to us via debian?

2. Our merge review processes need to account for conf file changes with ucf packages. Although, in this case openssh presumably got sync'd so Ubuntu-side process changes would not have caught it.

3. There have been other reports of similar misbehavior with wrongly detected conf file changes (Robie's LP #1747464 mentioned above may be one, there's likely others). Is ucf also being used in these cases, and are those problems likewise caused by missing md5sums in their packages?

4. Is this failure mode something that can be caught in autopkgtests? If so, then per-package checks seem warranted to prevent this in the future.

5. Even better than #3 would be a distro-wide CI check for all packages using ucf, to ensure all distro-default installed conf files (from all pockets) have their conf file md5sums registered.

In addition, some broader scoped / philosophical / "dumb" questions:

1. Are md5sums the best way to identify config file changes? E.g. if the change is just a timestamp and commented out line (such as in this case) that shouldn't count as a "change". What about like filtering out commented lines, and checksumming that?

2. Why are commented out lines included in distro-provided conf files? Is it just for documentation, in which case those would be better kept elsewhere and just referenced? (Yes, this is more a debian/upstream policy question which we probably don't have say on...)

3. Is upgrade the right time to be prompting users about config file changes, even if they have legitimate local config changes? With cloud instances, unattended-upgrades, etc. it's not so safe to assume a human is babysitting the dist-upgrade, and breakages during dist-upgrades can be pretty catastrophic for users. It's also a frequently seen pattern in our own bug triaging workloads. Are there any other ways to solve this need?

(Yes, much of the above is better fodder for blogs, and no need to discuss it in depth here...)

[T Jeske (t-jeske)]

Another problem is that while showing this prompt, you can't log in via SSH. If you leave your upgrade unattended and lose the connection to your server, you won't be able to log in via SSH again to complete/continue the upgrade.

[Christian Ehrhardt  (paelzer)]

Fixed in 1:8.2p1-1 and
 openssh | 1:8.2p1-4 | focal | source
 openssh | 1:8.2p1-4ubuntu0.1 | focal-updates | source
 openssh | 1:8.3p1-1 | groovy | source

So this should be fix released, btu wasn't updated.