openssh-server-1:9.2p1-2ubuntu1 cannot be installed from active ssh session

Bug #2020474 reported by scottfk
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
openssh (Ubuntu)
Fix Released
Medium
Nick Rosbrook

Bug Description

Installation seems to fail on restarting ssh.socket via systemctl

Setting up openssh-server (1:9.2p1-2ubuntu1) ...
rescue-ssh.target is a disabled or a static unit not running, not starting it.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
dpkg: error processing package openssh-server (--configure):
 installed openssh-server package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 openssh-server
E: Sub-process /usr/bin/dpkg returned an error code (1)

$ systemctl status ssh.socket
× ssh.socket - OpenBSD Secure Shell server socket
     Loaded: loaded (/lib/systemd/system/ssh.socket; enabled; preset: enabled)
     Active: failed (Result: resources) since Tue 2023-05-23 15:01:41 CEST; 48s ago
   Duration: 3h 6min 36.071s
   Triggers: ● ssh.service
     Listen: [::]:22 (Stream)
        CPU: 2ms

May 23 11:55:05 venus2 systemd[1]: Listening on ssh.socket - OpenBSD Secure Shell server socket.
May 23 15:01:41 venus2 systemd[1]: ssh.socket: Deactivated successfully.
May 23 15:01:41 venus2 systemd[1]: Closed ssh.socket - OpenBSD Secure Shell server socket.
May 23 15:01:41 venus2 systemd[1]: Stopping ssh.socket - OpenBSD Secure Shell server socket...
May 23 15:01:41 venus2 systemd[2631]: ssh.socket: Failed to create listening socket ([::]:22): Address already in use
May 23 15:01:41 venus2 systemd[1]: ssh.socket: Failed to receive listening socket ([::]:22): Input/output error
May 23 15:01:41 venus2 systemd[1]: ssh.socket: Failed to listen on sockets: Input/output error
May 23 15:01:41 venus2 systemd[1]: ssh.socket: Failed with result 'resources'.
May 23 15:01:41 venus2 systemd[1]: Failed to listen on ssh.socket - OpenBSD Secure Shell server socket.

At this point, sshd is no longer listening for new connections. A manual systemctl restart of ssh.socket fails with the same error. I am ssh-ed into this box, so I *think* the failure is because my session is already sitting on port 22, maybe? The only way I can be sure I will be able to ssh to this box again is to reboot it (so that ssh.socket can start cleanly).

$ lsb_release -rd
No LSB modules are available.
Description: Ubuntu Mantic Minotaur (development branch)
Release: 23.10

$ apt policy openssh-server
openssh-server:
  Installed: 1:9.2p1-2ubuntu1
  Candidate: 1:9.2p1-2ubuntu1
  Version table:
 *** 1:9.2p1-2ubuntu1 500
        500 http://ch.ports.ubuntu.com/ubuntu-ports mantic-proposed/main riscv64 Packages
        100 /var/lib/dpkg/status
     1:9.0p1-1ubuntu8.1 500
        500 http://ch.ports.ubuntu.com/ubuntu-ports mantic/main riscv64 Packages

ProblemType: Bug
DistroRelease: Ubuntu 23.10
Package: openssh-server 1:9.2p1-2ubuntu1
ProcVersionSignature: Ubuntu 6.2.0-19.19.1-generic 6.2.6
Uname: Linux 6.2.0-19-generic riscv64
ApportVersion: 2.26.1-0ubuntu3
Architecture: riscv64
CasperMD5CheckResult: unknown
CloudArchitecture: riscv64
CloudBuildName: server
CloudID: nocloud
CloudName: unknown
CloudPlatform: nocloud
CloudSerial: 20230413.1
CloudSubPlatform: seed-dir (/var/lib/cloud/seed/nocloud-net)
Date: Tue May 23 14:58:35 2023
SSHDConfig: Error: command ['/usr/sbin/sshd', '-T'] failed with exit code 1: /etc/ssh/sshd_config.d/50-cloud-init.conf: Permission denied
SourcePackage: openssh
UpgradeStatus: Upgraded to mantic on 2023-05-12 (11 days ago)

Revision history for this message
scottfk (scottfk) wrote :
Revision history for this message
Nick Rosbrook (enr0n) wrote :

How are you configuring sshd to run? It seems like you may have partially-configured socket-activated sshd. The postinst script should only try to resart ssh.socket if it was already enabled, which should not be the case if e.g. you reverted the socket-activation change. So, did you try and disable socket-activated sshd in the past?

The issue is not your active ssh session, because that is not the process that is listening on port 22.

Changed in openssh (Ubuntu):
status: New → Incomplete
Revision history for this message
scottfk (scottfk) wrote :

This is just the out-of-the-box sshd config from the preinstalled Ubuntu 23.04 Server image (https://ubuntu.com/download/risc-v). I didn't make any changes to it.

Interestingly... on my amd64 desktop, I only see a ssh.socket systemd service. On an rpi and my VisionFive2 I see both ssh.socket *and* ssh.service.

Revision history for this message
Nick Rosbrook (enr0n) wrote :

Er, is there something else running on port 22 then? What does lsof -i:22 show?

Revision history for this message
scottfk (scottfk) wrote :

# lsof -i :22
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 140u IPv6 22354 0t0 TCP *:ssh (LISTEN)
sshd 904 root 3u IPv6 22354 0t0 TCP *:ssh (LISTEN)
sshd 905 root 3u IPv6 22354 0t0 TCP *:ssh (LISTEN)
sshd 905 root 4u IPv6 18405 0t0 TCP venus2.home:ssh->hydra.home:40926 (ESTABLISHED)
sshd 905 root 5u IPv6 18405 0t0 TCP venus2.home:ssh->hydra.home:40926 (ESTABLISHED)
sshd 997 scottfk 3u IPv6 22354 0t0 TCP *:ssh (LISTEN)
sshd 997 scottfk 4u IPv6 18405 0t0 TCP venus2.home:ssh->hydra.home:40926 (ESTABLISHED)
sshd 997 scottfk 5u IPv6 18405 0t0 TCP venus2.home:ssh->hydra.home:40926 (ESTABLISHED)

Revision history for this message
Nick Rosbrook (enr0n) wrote :

Yeah that does look a bit odd. What do the following show?

# systemctl status 904
# systemctl status 905
# systemctl status 997

Revision history for this message
scottfk (scottfk) wrote :
Download full text (8.1 KiB)

Before retriggering the post-installation:

root@venus2:/home/scottfk# systemctl status 904
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled)
    Drop-In: /etc/systemd/system/ssh.service.d
             └─00-socket.conf
     Active: active (running) since Tue 2023-05-23 15:17:36 CEST; 4h 40min ago
TriggeredBy: ● ssh.socket
       Docs: man:sshd(8)
             man:sshd_config(5)
   Main PID: 904 (sshd)
      Tasks: 1 (limit: 9378)
     Memory: 5.1M
        CPU: 468ms
     CGroup: /system.slice/ssh.service
             └─904 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"

May 23 15:17:35 venus2 systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
May 23 15:17:36 venus2 sshd[904]: Server listening on :: port 22.
May 23 15:17:36 venus2 systemd[1]: Started ssh.service - OpenBSD Secure Shell server.
May 23 15:17:36 venus2 sshd[905]: Accepted publickey for scottfk from 2a02:1210:2a23:5600:9e82:2fb7:9772:3bf6 port 4092>
May 23 15:17:36 venus2 sshd[905]: pam_unix(sshd:session): session opened for user scottfk(uid=1000) by (uid=0)
root@venus2:/home/scottfk# systemctl status 905
● session-2.scope - Session 2 of User scottfk
     Loaded: loaded (/run/systemd/transient/session-2.scope; transient)
  Transient: yes
     Active: active (running) since Tue 2023-05-23 15:17:37 CEST; 4h 41min ago
      Tasks: 8
     Memory: 18.2M
        CPU: 4.330s
     CGroup: /user.slice/user-1000.slice/session-2.scope
             ├─ 905 "sshd: scottfk [priv]"
             ├─ 997 "sshd: scottfk@pts/0"
             ├─ 998 -bash
             ├─1113 sudo bash -o vi
             ├─1114 sudo bash -o vi
             ├─1115 bash -o vi
             ├─1301 systemctl status 905
             └─1302 less

May 23 15:17:37 venus2 systemd[1]: Started session-2.scope - Session 2 of User scottfk.
May 23 15:17:41 venus2 sshd[905]: pam_env(sshd:session): deprecated reading of user environment enabled
May 23 17:41:33 venus2 sudo[1108]: scottfk : TTY=pts/0 ; PWD=/home/scottfk ; USER=root ; COMMAND=/usr/bin/lsof -i :22
May 23 17:41:33 venus2 sudo[1108]: pam_unix(sudo:session): session opened for user root(uid=0) by scottfk(uid=1000)
May 23 17:41:33 venus2 sudo[1108]: pam_unix(sudo:session): session closed for user root
May 23 17:42:20 venus2 sudo[1113]: scottfk : TTY=pts/0 ; PWD=/home/scottfk ; USER=root ; COMMAND=/usr/bin/bash -o vi
May 23 17:42:20 venus2 sudo[1113]: pam_unix(sudo:session): session opened for user root(uid=0) by scottfk(uid=1000)
root@venus2:/home/scottfk# systemctl status 997
● session-2.scope - Session 2 of User scottfk
     Loaded: loaded (/run/systemd/transient/session-2.scope; transient)
  Transient: yes
     Active: active (running) since Tue 2023-05-23 15:17:37 CEST; 4h 41min ago
      Tasks: 8
     Memory: 18.3M
        CPU: 4.376s
     CGroup: /user.slice/user-1000.slice/session-2.scope
             ├─ 905 "sshd: scottfk [priv]"
             ├─ 997 "sshd: scottfk@pts/0"
             ├─ 998 -bash
             ├─1113 sudo bash -o vi
             ├─1114 sudo bash -o vi
             ├─1115 bash -o vi
             ├─1304 systemctl status 997
             └─1305 less

May 23 ...

Read more...

Revision history for this message
scottfk (scottfk) wrote :

So I tried this in an amd64 VM. Installing this version of openssh-server (from inside a GNOME session) worked just fine.

ssh-ing to localhost and attempting a reinstall ended up in the exact same FAIL.

So it seems the upgrade of openssh-server 1:9.2p1-2ubuntu1 can't be done from an ssh session (regardless of processor architecture). I'll do more poking around to see if I can't figure out why systemd won't/can't.

Revision history for this message
scottfk (scottfk) wrote :

root@venus2:/home/scottfk# systemctl restart ssh.socket
Job failed. See "journalctl -xe" for details.

May 23 20:36:21 venus2 systemd[2014]: ssh.socket: Failed to create listening socket ([::]:22): Address already in use
May 23 20:36:21 venus2 systemd[1]: ssh.socket: Failed to receive listening socket ([::]:22): Input/output error
May 23 20:36:21 venus2 systemd[1]: ssh.socket: Failed to listen on sockets: Input/output error
May 23 20:36:21 venus2 systemd[1]: ssh.socket: Failed with result 'resources'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit ssh.socket has entered the 'failed' state with result 'resources'.
May 23 20:36:21 venus2 systemd[1]: Failed to listen on ssh.socket - OpenBSD Secure Shell server socket.
░░ Subject: A start job for unit ssh.socket has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit ssh.socket has finished with a failure.
░░
░░ The job identifier is 1542 and the job result is failed.

Revision history for this message
scottfk (scottfk) wrote :

root@venus2:/etc/systemd/system/ssh.service.d# /usr/sbin/sshd -d
debug1: sshd version OpenSSH_9.2, OpenSSL 3.0.8 7 Feb 2023
debug1: private host key #0: ssh-rsa SHA256:jBRj9thNiN3QA5SVICY9H4/b2f2tvIqyKv2krB+q1+E
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:+AoeQEDs9PNiD54EnDi/YTJksSDND59lJqCe5BAgdz0
debug1: private host key #2: ssh-ed25519 SHA256:fJj+sS6kyWxB/3Z36XR7DE7J1w9TYvR8GLRT0Kr+DNU
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 22 on 0.0.0.0.
Bind to port 22 on 0.0.0.0 failed: Address already in use.
debug1: Bind to port 22 on ::.
Bind to port 22 on :: failed: Address already in use.
Cannot bind any address.

Revision history for this message
Nick Rosbrook (enr0n) wrote :

You're right, I was able to reproduce this just now in the same way. I also tried in Lunar, but Lunar does not appear to be affected.

summary: - openssh-server-1:9.2p1-2ubuntu1 fails post-installation
+ openssh-server-1:9.2p1-2ubuntu1 cannot be installed from active ssh
+ session
Changed in openssh (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Nick Rosbrook (enr0n) wrote :

This appears to have to do with the patch addressing bug 2011458.

Nick Rosbrook (enr0n)
tags: added: rls-mm-incoming
Revision history for this message
Nick Rosbrook (enr0n) wrote :

Workaround:

For me, when I reboot, the ssh.socket unit comes up again and I can establish a session. To avoid continuously hitting the error in the postint, run e.g. dpkg --configure -a from a non-ssh session, or script this to happen once on reboot or something.

Changed in openssh (Ubuntu):
importance: Undecided → Medium
Revision history for this message
scottfk (scottfk) wrote :

Still an issue in the latest version:

Setting up openssh-server (1:9.2p1-2ubuntu2) ...
rescue-ssh.target is a disabled or a static unit not running, not starting it.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
dpkg: error processing package openssh-server (--configure):
 installed openssh-server package post-installation script subprocess returned error exit status 1
Setting up sosreport (4.5.3ubuntu2) ...
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for ufw (0.36.2-1) ...
Errors were encountered while processing:
 openssh-server
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)
scottfk@venus2:~$ apt policy openssh-server
openssh-server:
  Installed: 1:9.2p1-2ubuntu2
  Candidate: 1:9.2p1-2ubuntu2
  Version table:
 *** 1:9.2p1-2ubuntu2 500
        500 http://ch.ports.ubuntu.com/ubuntu-ports mantic-proposed/main riscv64 Packages
        100 /var/lib/dpkg/status
     1:9.0p1-1ubuntu8.1 500
        500 http://ch.ports.ubuntu.com/ubuntu-ports mantic/main riscv64 Packages

Re: Workaround, adding this to root's crontab sorted me:
@reboot dpkg --configure -a

Jeremy Bícha (jbicha)
tags: added: update-excuse
Nick Rosbrook (enr0n)
Changed in openssh (Ubuntu):
assignee: nobody → Nick Rosbrook (enr0n)
status: Confirmed → In Progress
Nick Rosbrook (enr0n)
tags: added: foundations-todo
Nick Rosbrook (enr0n)
tags: removed: rls-mm-incoming
Jeremy Bícha (jbicha)
Changed in openssh (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
scottfk (scottfk) wrote :

Seems fixed. Huzzah!

$ apt policy openssh-server
openssh-server:
  Installed: 1:9.2p1-2ubuntu3
  Candidate: 1:9.2p1-2ubuntu3
  Version table:
 *** 1:9.2p1-2ubuntu3 500
        500 http://ch.ports.ubuntu.com/ubuntu-ports mantic-proposed/main riscv64 Packages
        100 /var/lib/dpkg/status
     1:9.0p1-1ubuntu8.1 500
        500 http://ch.ports.ubuntu.com/ubuntu-ports mantic/main riscv64 Packages

Revision history for this message
scottfk (scottfk) wrote :

Seems this has now made its way downstream to openssh-server 9.0 (on a different box):

The following packages will be upgraded:
  openssh-client openssh-server openssh-sftp-server
3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 1295 kB of archives.
After this operation, 65.5 kB disk space will be freed.
Do you want to continue? [Y/n]
Get:1 http://ca.ports.ubuntu.com/ubuntu-ports lunar-proposed/main arm64 openssh-sftp-server arm64 1:9.0p1-1ubuntu8.2 [36.0 kB]
Get:2 http://ca.ports.ubuntu.com/ubuntu-ports lunar-proposed/main arm64 openssh-server arm64 1:9.0p1-1ubuntu8.2 [409 kB]
Get:3 http://ca.ports.ubuntu.com/ubuntu-ports lunar-proposed/main arm64 openssh-client arm64 1:9.0p1-1ubuntu8.2 [849 kB]
Fetched 1295 kB in 1s (921 kB/s)
Preconfiguring packages ...
(Reading database ... 104735 files and directories currently installed.)
Preparing to unpack .../openssh-sftp-server_1%3a9.0p1-1ubuntu8.2_arm64.deb ...
Unpacking openssh-sftp-server (1:9.0p1-1ubuntu8.2) over (1:9.0p1-1ubuntu8.1) ...
Preparing to unpack .../openssh-server_1%3a9.0p1-1ubuntu8.2_arm64.deb ...
Unpacking openssh-server (1:9.0p1-1ubuntu8.2) over (1:9.0p1-1ubuntu8.1) ...
Preparing to unpack .../openssh-client_1%3a9.0p1-1ubuntu8.2_arm64.deb ...
Unpacking openssh-client (1:9.0p1-1ubuntu8.2) over (1:9.0p1-1ubuntu8.1) ...
Setting up openssh-client (1:9.0p1-1ubuntu8.2) ...
Setting up openssh-sftp-server (1:9.0p1-1ubuntu8.2) ...
Setting up openssh-server (1:9.0p1-1ubuntu8.2) ...
rescue-ssh.target is a disabled or a static unit not running, not starting it.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
dpkg: error processing package openssh-server (--configure):
 installed openssh-server package post-installation script subprocess returned error exit status 1
Processing triggers for man-db (2.11.2-1) ...
Processing triggers for ufw (0.36.1-4.1) ...
Errors were encountered while processing:
 openssh-server
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

Should I cut a new bug for this? It's a different arch and a different version of Ubuntu, but it's the exact same symptom (and, likely, cause).

Revision history for this message
Nick Rosbrook (enr0n) wrote :

You are correct that this is the same issue. However, the reason you are seeing this in Lunar is that
you are running with -proposed enabled, which is where packages are staged before being released to -release, or -updates for stable release updates. So, this means that you installed 1:9.0p1-1ubuntu8.1 from lunar-proposed, which contains the bug that causes the problem on upgrade. However, thanks to this bug report from you, we have uploaded 1:9.0p1-1ubuntu8.2 to lunar-proposed to fix the underlying issue *before* openssh is released to lunar-updates. This means that users who upgrade to 1:9.0p1-1ubuntu8.2 from 1:9.0p1-1ubuntu8 (i.e. they never installed 1:9.0p1-1ubuntu8.1) will not see this bug. Since a buggy version of openssh was not yet released to lunar-updates, we did not apply the extra patches that we did in mantic to work around the bug on upgrade (for mantic, a buggy version was released to mantic-release, so the situation was a bit different).

So, anyone running -proposed on kinetic or lunar will hit this bug once, but those users should be able to reboot, run dpkg --configure -a, and be sorted.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (16.6 KiB)

This bug was fixed in the package openssh - 1:9.2p1-2ubuntu3

---------------
openssh (1:9.2p1-2ubuntu3) mantic; urgency=medium

  * Fix upgrade of openssh-server with active ssh session (LP: #2020474)
    - debian/patches/systemd-socket-activation.patch:
      + Do force closing of listen sockets in child process
      + Set rexec_flag = 0 when sshd is socket-activated so that child process
        does not re-exec
    - debian/openssh-server.postint:
      + When upgrading from affected versions of openssh, do not try to
        restart systemd units, and instead indicate that a reboot is required
    - debian/tests/systemd-socket-activation:
      + Reboot the testbed before starting the test
    - debian/rules:
      + Do not stop ssh.socket on upgrade
  * d/p/test-set-UsePAM-no-on-some-tests.patch: set UsePAM=no for some tests

openssh (1:9.2p1-2ubuntu2) mantic; urgency=medium

  * debian/README.Debian: Fix path of addresses.conf drop-in

openssh (1:9.2p1-2ubuntu1) mantic; urgency=medium

  * Merge with Debian unstable (LP: #2018094). Remaining changes:
    - debian/rules: modify dh_installsystemd invocations for
      socket-activated sshd
    - debian/openssh-server.postinst: handle migration of sshd_config options
      to systemd socket options on upgrade.
    - debian/README.Debian: document systemd socket activation.
    - debian/patches/socket-activation-documentation.patch: Document in
      sshd_config(5) that ListenAddress and Port no longer work.
    - debian/openssh-server.templates: include debconf prompt explaining
      when migration cannot happen due to multiple ListenAddress values
    - debian/.gitignore: drop file
    - debian/openssh-server.postrm: remove systemd drop-ins for
      socket-activated sshd on purge
    - debian/openssh-server.ucf-md5sum: Update list of stock sshd_config
      checksums to include those from jammy and kinetic.
    - debian/openssh-server.tmpfile,debian/systemd/ssh.service: Move
      /run/sshd creation out of the systemd unit to a tmpfile config so
      that sshd can be run manually if necessary without having to create
      this directory by hand.
    - debian/patches/systemd-socket-activation.patch: Fix sshd
      re-execution behavior when socket activation is used
    - debian/tests/systemd-socket-activation: Add autopkgtest for systemd socket
      activation functionality.
  * Dropped changes, included in Debian:
    - debian/patches/systemd-socket-activation.patch: Initial implementation
  * New changes:
    - debian/README.Debian: mention drop-in configurations in instructions
      for disabling sshd socket activation (LP: #2017434).
    - debian/openssh-server.ucf-md5sum: update for Ubuntu delta

openssh (1:9.2p1-2) unstable; urgency=medium

  * Fix mistakenly-unreleased entry for 1:9.2p1-1 in debian/NEWS.

openssh (1:9.2p1-1) unstable; urgency=medium

  * Set "UsePAM yes" when running regression tests, to match our default
    sshd configuration.
  * Ignore Lintian error about depending on lsb-base for now, to avoid
    problems with partial upgrades on non-default init systems.
  * New upstream release (https://www.openssh.com/releasenotes.html#9.2p1):
    - [SECURITY] sshd...

Changed in openssh (Ubuntu):
status: Fix Committed → Fix Released
Benjamin Drung (bdrung)
tags: removed: foundations-todo
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.