openssh-server cannot listen or bind to anything other than :::22 after upgrading to 22.10 from 22.04

[Christophe M.]

This is a bug report to separate the second issue that was reported in this bug report:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/1993478

There's an issue after upgrading to 22.10 from 22.04 that prevents opensshd from listening to anything other than :::22. I already commented in the bug report I linked, so I'll just copy/paste and add some details. I guess.

The issue is that after upgrading, sshd doesn't use the Listen port or ListenAddress config from the sshd_config file or any custom config file that was in the sshd_config.d drop in folder anymore.

Other drop in settings from sshd.config.d seem to be applied normally, the issue seem to be only for IP binding and custom ports.

If I change Accept=no by Accept=yes in ssh.socket and reloads the socket unit, I can start sshd on a different port and I can also bind the IP to something else than ::

There's an issue still, an instance of sshd is still listening to :::22 that is not started by SSHD but by init.

root@ubuntulocal:~# netstat -antp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 568/vsftpd
tcp 0 0 0.0.0.0:622 0.0.0.0:* LISTEN 571/sshd: /usr/sbin
tcp 0 272 192.168.1.225:622 192.168.1.220:2473 ESTABLISHED 1027/sshd: root@pts
tcp6 0 0 :::22 :::* LISTEN 1/init

If I reboot after changing this no to yes in ssh.socket does not survive a reboot and fails to load sshd with a "Failed to queue service startup job" error.
Oct 21 15:41:56 ubuntulocal systemd[1]: ssh.socket: Failed to queue service startup job (Maybe the service file is missing or not a template unit?): Invalid argument
Oct 21 15:41:56 ubuntulocal systemd[1]: ssh.socket: Failed with result 'resources'.

I had to mask/stop the sshd.socket unit and create a custom sshd service in /etc/systemd/system to be able start sshd on a custom port and IP.

chris@ubuntulocal:~$ systemctl status ssh.socket
● ssh.socket - OpenBSD Secure Shell server socket
     Loaded: loaded (/lib/systemd/system/ssh.socket; enabled; preset: enabled)
     Active: active (running) since Fri 2022-10-21 23:08:09 UTC; 1min 24s ago
      Until: Fri 2022-10-21 23:08:09 UTC; 1min 24s ago
   Triggers: ● ssh.service
     Listen: [::]:22 (Stream)
      Tasks: 0 (limit: 18899)
     Memory: 4.0K
        CPU: 418us
     CGroup: /system.slice/ssh.socket

[Christophe M. (chrism34)]

I suppose this wont get fixed any time soon. Apparently, UX config prime over manual configuration now, for a configuration file that's mostly used by server admin. Time to learn yum I suppose.

source:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/1991592

[Steve Langasek (vorlon)]

The error in your (Machine-1) log is:

> Setting up openssh-server (1:9.0p1-1ubuntu7) ...
> Replacing config file /etc/ssh/sshd_config with new version
> Synchronizing state of ssh.service with SysV service script with /lib/systemd/systemd-sysv-install.
> Executing: /lib/systemd/systemd-sysv-install disable ssh
> Removed "/etc/systemd/system/sshd.service".
> Removed "/etc/systemd/system/multi-user.target.wants/ssh.service".
> Created symlink /etc/systemd/system/sockets.target.wants/ssh.socket → /lib/systemd/system/ssh.socket.
> rescue-ssh.target is a disabled or a static unit not running, not starting it.
> Failed to restart ssh.socket: Unit ssh.socket has a bad unit file setting.
> See system logs and 'systemctl status ssh.socket' for details.
> Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
> dpkg: error processing package openssh-server (--configure):
> installed openssh-server package post-installation script subprocess returned error exit status 1

I will try to reproduce this here, but please also attach the contents of /etc/systemd/system/ssh.socket.d from the affected system.

[Chris M. (mchris34)]

[Unit]
After=ssh.socket
Requires=ssh.socket

[Chris M. (mchris34)]

The whatever caused this "> Failed to restart ssh.socket: Unit ssh.socket has a bad unit file setting." is related to the 22.04 install. The openSSH-server on machine-1 only has customs ports in the sshd_config and the rest of the config in the sshd_config.d drop in folder. Everything related to systemd was never touched.

I joined the sshd config from machine 1.

My local machine is the one that refuses to boot after I change accept=no to yes in ssh.socket, systemctl daemon-reload and reboot. I tried to change no to yes to reflect 22.04 settings as I missed the news and I didn't Ubuntu uses socket to save some 9MB of memory(...) when openssh-server isn't used. I didn't try on my live server but I'm pretty sure if I tried the result would be the same.

[Chris M. (mchris34)]

"My local machine is the one that refuses to boot" -> refused to start openssh-server at boot.

I believe it happened because I didn't know ssh.service was disabled when I restarted.

[Steve Langasek (vorlon)]

On Tue, Oct 25, 2022 at 01:55:53AM -0000, Chris M. wrote:
> [Unit]
> After=ssh.socket
> Requires=ssh.socket
>
> ** Attachment added: "00-socket.conf"

This appears to be the contents of
/etc/systemd/system/ssh.service.d/00-socket.conf. I am looking for the
contents of the /etc/systemd/system/ssh.socket.d directory. There should be
a file named /etc/systemd/system/ssh.socket.d/addresses.conf there, which is
what I'm looking for (along with anything else that might be there).

I have done a jammy->kinetic upgrade using the Machine-1 sshd config that
you provided, and I do NOT see the errors that were shown in your apt term
log. I get an addresses.conf as follows:

# cat /etc/systemd/system/ssh.socket.d/addresses.conf
[Socket]
ListenStream=
ListenStream=0.0.0.0:623
#

And the ssh socket activation works as expected.

[Chris M. (mchris34)]

The machine that returned an openssh-server error and left me with a partially installed openssh-server package (dpkg IF flag) at the end of the upgrade process didn't have anything in ssh.socket.d, nor it had a ssh.socket.d folder in /etc/systemd/system/.

The local machine I tested the upgrade on before trying on my live server had a file, override.conf, if I remember correctly, in /etc/systemd/system/ssh.socket.d.

>I have done a jammy->kinetic upgrade using the Machine-1 sshd config that
>you provided, and I do NOT see the errors that were shown in your apt term
>log.

Everything related to systemd was from the 22.04 install. The first and only time I ever touched anything related to openssh-server systemd service and socket units was two days ago after the 22.10 upgrade process returned that openssh-server error.

I'll try to do a 22.04 -> 22.10 upgrade on my local machine later today.

[Steve Langasek (vorlon)]

On Tue, Oct 25, 2022 at 03:12:55AM -0000, Chris M. wrote:
> The machine that returned an openssh-server error and left me with a
> partially installed openssh-server package (dpkg IF flag) at the end of
> the upgrade process didn't have anything in ssh.socket.d, nor it had a
> ssh.socket.d folder in /etc/systemd/system/.

Then I do not have an explanation for the upgrade failing with the error
message in the log.

> Failed to restart ssh.socket: Unit ssh.socket has a bad unit file setting.

The ssh.socket unit as shipped in the package does not have syntax errors.
The maintainer script does, on upgrade, create a config snippet under
/etc/systemd/system/ssh.socket.d if there are Port or ListenAddress settings
in sshd_config, which could, if we have bugs, have syntax errors. And the
sshd_config you provided does specify both Port and ListenAddress, and with
that config installed locally, an upgrade does create
/etc/systemd/system/ssh.socket.d/addresses.conf for me (without syntax
errors). But you assert that this file does not exist after upgrade. So I
can neither explain why you don't have
/etc/systemd/system/ssh.socket.d/addresses.conf after upgrade, nor why you
get a configuration error on upgrade in the absence of this file.

> I'll try to do a 22.04 -> 22.10 upgrade on my local machine later today.

Thanks. At the moment this is unreproducible and there is no further action
for us to take.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer https://www.debian.org/
<email address hidden> <email address hidden>

[Chris M. (mchris34)]

You can close this. I did an upgrade from ubuntu 18.04 since I initially upgraded to 22.04 from this version, and openssh-server's sockets didn't break the install.

If the error happened it's either related to the Ubuntu 18.04 install/image that was provided by my host or some freak corruption. If I had touched anything related to systemd's openssh-server but forgot about it, it would have been the service and a .d folder in /etc/systemd/system -> https://i.imgur.com/eTRBEmc.png

I joined the dist-upgrade folder with the two ssh.service.d and ssh.socket.d folder and override files that were created during the upgrade.

[Lukas Märdian (slyon)]

Thanks for your confirmation, closing.