conversion from sshd service to socket is too bumpy

[Jonathan Kamens]

During upgrade from Jammy to Kinetic, I get asked what to do because my sshd_config has been modified. I say to do a 3-way merge. It says 3-way merge fails. I shrug, figure I'll just restore my customizations with Ansible after the upgrade like I always do, and tell it to use the vendor version of the file. This removes my custom Port settings, so they are not migrated over to the ssh.socket settings like https://discourse.ubuntu.com/t/sshd-now-uses-socket-based-activation-ubuntu-22-10-and-later/30189 says they would be. I subsequently run my Ansible which restores the customizations and enables the ssh service, but now ssh.service and ssh.socket are enabled at the same time, sshd isn't listening on my specified ports, and everything is a mess. I've never used socket-based activation before and have no idea how to configure it so now I have to go reading man pages, Googling all over the place, and generally struggle to figure out what the heck is going wrong.

I don't know what the right answer is here, but I really feel like some effort needs to be put into figuring out a smoother transition for people who are upgrading to Kinetic.

ProblemType: Bug
DistroRelease: Ubuntu 22.10
Package: openssh-server 1:9.0p1-1ubuntu6
ProcVersionSignature: Ubuntu 5.19.0-15.15-generic 5.19.0
Uname: Linux 5.19.0-15-generic x86_64
ApportVersion: 2.23.0-0ubuntu2
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
Date: Mon Sep 26 11:41:58 2022
InstallationDate: Installed on 2019-08-16 (1136 days ago)
InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416)
SourcePackage: openssh
UpgradeStatus: Upgraded to kinetic on 2022-09-24 (1 days ago)

[Lucas Kanashiro (lucaskanashiro)]

Thanks for taking the time to report this bug and trying to make Ubuntu better.

Most relevant information can be found in the discourse post you mentioned. Could you please share your customization of the configuration file? So we can understand better the upgrade path you are going through.

I am setting the status to Incomplete until you provide more information, once you do that please set it back to New and we will take a look again.

[Jonathan Kamens (jik)]

Hello,

I am unclear about why you are asking for the contents of my sshd_config file, given that the problem I described here isn't the merge conflict, but rather what happens in terms of the migration from ssh service to socket when the user opts to install the vendor version of sshd_config as a result of the merge conflict. You're not going to be able to entirely avoid merge conflicts, so you need to come up with a better way to handle this conversion, e.g., by migrating the address and port settings from sshd_config to listen.conf in a preinst script that runs before dpkg tries to install the new sshd_config.

In any case, I can't tell you exactly what was in my sshd_config before the problem occurred because, as I indicated, I've upgraded my machine to Kinetic and in the process replaced sshd_config. However, I believe it looked approximately like this:

# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options override the
# default value.

Include /etc/ssh/sshd_config.d/*.conf

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin prohibit-password
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#PubkeyAuthentication yes

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
KbdInteractiveAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the KbdInteractiveAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via KbdInteractiveAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the...

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Feedback on this is appreciated as we do want to make the transition as smooth as possible.

I would warn though that just overwriting configuration files with "known good" ones following a release upgrade, as you're doing with Ansible, is not advisable in the general case, and cannot be a supportable path in packaging. Because a release upgrade means major version bumps by definition, and configuration file syntaxes change. The assumption of "known good" therefore cannot be made in release upgrades.

Therefore, I'm not sure the upgrade path can really be improved here. You can't assume that you can overwrite configuration with what you had in a previous release without looking into the details of how your previous change may no longer apply correctly.

What we can do is improve documentation on this, but the documentation has yet to be written because Kinetic hasn't been released yet. The documentation is expected to appear in the release notes at https://discourse.ubuntu.com/t/kinetic-kudu-release-notes/27976, which is the usual place for us to note upgrade path edge cases.

I wonder if there's some conflict here between the conffile (or ucf?) prompt and the upgrade path handling, so I subscribed Steve to take a look.

[Jonathan Kamens (jik)]

1) Nobody reads the release notes.

2) I am not "overwriting configuration files with 'known good' ones," I am making specific changes to the config files with ansible plays. Specifically, before fixing my playbook for Kinetic, it looked like this:

    - name: enable root ssh public key authentication
      lineinfile:
        dest: /etc/ssh/sshd_config
        line: PermitRootLogin without-password
      register: rootsshyeskey
    - name: disable root ssh non-key authentication
      replace:
        dest: /etc/ssh/sshd_config
        regexp: ^\s*PermitRootLogin(?!.*without-password).*\n?
      register: rootsshnopassword
    - name: add default sshd port
      lineinfile:
        dest: /etc/ssh/sshd_config
        line: 'Port 22'
      when: extra_ssh_port is defined
    - name: add additional sshd port
      lineinfile:
        dest: /etc/ssh/sshd_config
        line: 'Port {{extra_ssh_port}}'
      when: extra_ssh_port is defined
      register: sshdport

[Robie Basak (racb)]

On Wed, Sep 28, 2022 at 01:19:35PM -0000, Jonathan Kamens wrote:
> 1) Nobody reads the release notes.

That's where we document edge case issues that can't be handled
automatically. Maybe we can handle this automatically, which is why I
invited Steve to take a look. But if it turns out that we can't, then
how else do you propose that we document this?

> 2) I am not "overwriting configuration files with 'known good' ones," I
> am making specific changes to the config files with ansible plays.

Either way, it's necessary for you to 1) discover that you need to
update your customisation, and then 2) update your customisation
yourself. I don't see any automated way for the upgrade to Kinetic to
automatically update your Ansible playbooks for you.

You seem to be proposing that we should change Ubuntu in some
way to accommodate your case, but it's not obvious to me what we can do,
or what you expect us to do.

Robie

[Jonathan Kamens (jik)]

Two ideas, one of which I've already mentioned above:

1) You could migrate the address and port settings from sshd_config to listen.conf _before_ installing the new sshd_config, so that they will be preserved even if the config gets replaced with the vendor version because of a merge conflict like it was in my case.

2) You could have the preinst script detect if the service is currently enabled and going to be switched to the socket, and if so, pop up an interactive warning explaining the change and telling the user where to look for more info, like many other packages do, e.g., the Docker restart warning. If you'd like you can only display the warning if there are address or port settings specified in the user's old sshd_config.

Finally, as I said in my original report, "I don't know what the right answer is here." What I "expect [you] to do" when someone points out an issue like this is to put some thought into how to make it better rather than seeming to imply that the person who reported the issue is being petulant. Putting it another way: neither of the ideas above is rocket science, and I'm not sure why I should have to be the one to think of them, rather than the people who are, you know, being paid to do that.

[Daniel Tang (daniel-z-tg)]

> what happens in terms of the migration from ssh service to socket when the user opts to install the vendor version of sshd_config as a result of the merge conflict

I upgraded today 2022-09-29 and installing the vendor version is not intended to break the port settings because there's a migration script. I commented out all Port= lines and both the first ssh connection and at least the second connection work.

> listen.conf

I don't know of this `listen.conf` you are talking about.

> You could migrate the address and port settings from sshd_config

There is supposed to be an automatic migration. Unfortunately it generates garbage and needs two lines to be manually added: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/1991199

[Steve Langasek (vorlon)]

I don't have a complete solution to this, but I do have several points.

- debconf prompts on upgrade are not, in general, good UX. It disrupts the flow of the upgrade. Sometimes it's necessary, but if all it's doing is telling the user they *may* have a broken config after upgrading, well, if every package for which that's true did that upgrades would be very slow indeed. And for users upgrading a large number of systems, that becomes one more nuisance. So no, I don't think we should add a prompt on upgrade for this. (There's also the practical problem that if we introduced such a prompt at this point in the release cycle, it would not realistically get translated, reducing accessibility for our users vs communicating this in other ways that could be localized out-of-band.)

- As Robie pointed out in comment #4, there is no guarantee that ansible playbooks work consistently across releases. Regardless of whether we made changes that would have allowed the migration of settings in your case, if you had had to reinstall kinetic instead of upgrading from jammy, those changes in the openssh-server maintainer scripts would not have taken effect. Your ansible playbook is therefore buggy wrt kinetic, and should be fixed, which is out of scope for Ubuntu and the bug tracker. But the following contents in a file named /etc/systemd/system/ssh.socket.d/addresses.conf should set you on the right path:
  [Socket]
  ListenStream=
  ListenStream=$portnum

- I have long been displeased with ucf's three-way-merge support. In particular, when identical content exists both in the user's version and in the new version but not in the base version, ucf will treat this as a merge conflict. This is awful, and specifically caused problems for upgrades from all cloud images when the user had not modified the sshd config at all (LP: #1990863). I've applied a workaround for this in openssh 1ubuntu7 (currently in the unapproved queue) that's specific to the cloud image case and ensures clean upgrades without prompting for users that have not modified sshd_config. I could generalize this to all users with modified configs, resulting in two prompts on upgrade but a better chance of successful three-way merging. Do you think that would be an improvement over the status quo?

- Finally, there is some code I'm evaluating landing that would add a systemd generator for the listenstream settings. This would only take effect at boot, but would make it possible for users to continue managing their port/listenaddress settings in sshd.conf as before. However, we would not land this in time for the kinetic release, but would instead consider it for the next release to improve our LTS-to-LTS compatibility.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssh - 1:9.0p1-1ubuntu7

---------------
openssh (1:9.0p1-1ubuntu7) kinetic; urgency=medium

  * Update list of stock sshd_config checksums to include those from
    jammy and kinetic.
  * Add a workaround for LP: #1990863 (now fixed in livecd-rootfs) to
    avoid spurious ucf prompts on upgrade.
  * Move /run/sshd creation out of the systemd unit to a tmpfile config
    so that sshd can be run manually if necessary without having to create
    this directory by hand. LP: #1991283.

  [ Nick Rosbrook ]
  * debian/openssh-server.postinst: Fix addresses.conf generation when only
    non-default Port is used in /etc/ssh/sshd_config (LP: #1991199).

 -- Steve Langasek <email address hidden> Mon, 26 Sep 2022 21:55:14 +0000