Comment 14 for bug 2004551

I'm starting to think that there is something wrong with the horsea machine indeed: still not able to upgrade to lunar. Starting from jammy or from kinetic failed.

#Jammy

Deploying again jammy and doing do-release-upgrade again failed with ssh, with differents things after rebooting and trying the do the release upgrade (I have outputs, but I don't remember exactly at what point I did the reboot of the machine, i.e: "Missing privilege separation directory: /run/sshd" could happen after reboot, but not sure).

#Kinetic
So, ok... let's think is something wrong with the jammy image that is being deployed (a remote possibility, I think, but ...).... trying the upgrading from Kinetic: failed. I checked the system status before starting the upgrade and it was not degraded.

It was stucked at 97% on:

Setting up snapd (2.58+22.10.1) ...
Installing new version of config file /etc/apt/apt.conf.d/20snapd.conf ...
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
snapd.failure.service is a disabled or a static unit not running, not starting it.
snapd.snap-repair.service is a disabled or a static unit not running, not starting it.
Failed to restart snapd.mounts-pre.target: Operation refused, unit snapd.mounts-pre.target may be requested by dependenc
y only (it is configured to refuse manual start/stop).
See system logs and 'systemctl status snapd.mounts-pre.target' for details.

system status was:

           │ └─unattended-upgrades.service
           │ ├─ 1227 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
           │ └─11628 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
           └─user.slice
             └─user-1000.slice
               └─session-1.scope
                 ├─ 1856 "sshd: ubuntu [priv]"
                 ├─ 1937 "sshd: ubuntu@pts/0"
                 ├─ 1941 -bash
                 ├─ 2302 sudo apt upgrade -y
                 ├─ 2303 sudo apt upgrade -y
                 ├─ 2304 apt upgrade -y
                 ├─ 5750 /usr/bin/dpkg --status-fd 39 --configure --pending
                 ├─ 5751 sh -c "(test -x /usr/lib/needrestart/dpkg-status && /usr/lib/needrestart/dpkg-status || cat > /dev/null>
                 ├─ 5752 sh -c "(test -x /usr/lib/needrestart/dpkg-status && /usr/lib/needrestart/dpkg-status || cat > /dev/null>
                 ├─ 5753 /bin/sh /usr/lib/needrestart/dpkg-status
                 ├─11336 /bin/sh /var/lib/dpkg/info/snapd.postinst configure 2.57.5+22.10ubuntu0.1
                 ├─11501 /usr/bin/perl /usr/bin/deb-systemd-invoke restart snapd.aa-prompt-listener.service snapd.apparmor.servi>
                 ├─11563 systemctl --quiet --system restart snapd.aa-prompt-listener.service snapd.apparmor.service snapd.autoim>
                 └─11564 /bin/systemd-tty-ask-password-agent --watch

After that, it failed with:

Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
Setting up openssh-sftp-server (1:9.0p1-1ubuntu7.3) ...
Setting up needrestart (3.6-1ubuntu0.1) ...
Setting up openssh-server (1:9.0p1-1ubuntu7.3) ...
rescue-ssh.target is a disabled or a static unit not running, not starting it.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
dpkg: error processing package openssh-server (--configure):
 installed openssh-server package post-installation script subprocess returned error exit status 1
Setting up bind9-dnsutils (1:9.18.12-0ubuntu0.22.10.2) ...
Processing triggers for libc-bin (2.36-0ubuntu4) ...
Processing triggers for rsyslog (8.2208.0-1ubuntu2) ...
Processing triggers for ufw (0.36.1-4build1) ...
Processing triggers for man-db (2.10.2-2) ...
Processing triggers for dbus (1.14.0-2ubuntu3) ...
Processing triggers for initramfs-tools (0.140ubuntu17) ...
update-initramfs: Generating /boot/initrd.img-5.19.0-46-generic
Processing triggers for ca-certificates (20230311ubuntu0.22.10.1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Errors were encountered while processing:
 openssh-server
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

System status after that was:

ubuntu@node-horsea:~$ systemctl status -l
Failed to read server status: Connection timed out

ubuntu@node-horsea:~$ systemctl status -l

● node-horsea
    State: degraded
    Units: 425 loaded (incl. loaded aliases)
     Jobs: 1 queued
   Failed: 7 units
    Since: Thu 2023-07-27 10:48:27 UTC; 18min ago
  systemd: 251.4-1ubuntu7.3
   CGroup: /
           ├─init.scope
           │ └─1 /lib/systemd/systemd --- --system --deserialize 22
           ├─system.slice
           │ ├─cron.service
           │ │ └─1138 /usr/sbin/cron -f -P
           │ ├─dbus.service
           │ │ ├─1139 @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
           │ │ └─6714 @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
           │ ├─irqbalance.service
           │ │ ├─1145 /usr/sbin/irqbalance --foreground
           │ │ └─6719 /usr/sbin/irqbalance --foreground
           │ ├─multipathd.service
           │ │ └─6654 /sbin/multipathd -d -s
           │ ├─packagekit.service
           │ │ └─2204 /usr/libexec/packagekitd
           │ ├─polkit.service
           │ │ └─2208 /usr/libexec/polkitd --no-debug
           │ ├─rsyslog.service
           │ │ └─12184 /usr/sbin/rsyslogd -n -iNONE
           │ ├─snapd.service
           │ │ ├─ 1150 /usr/lib/snapd/snapd
           │ │ └─23845 /usr/lib/snapd/snapd
           │ ├─ssh.service
           │ │ ├─ 1849 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"
           │ │ ├─11762 "sshd: ubuntu [priv]"
           │ │ ├─11841 "sshd: ubuntu@pts/3"
           │ │ ├─11842 -bash
           │ │ ├─23869 systemctl status -l
           │ │ └─23870 less
           │ ├─system-getty.slice
           │ │ └─<email address hidden>
           │ │ └─6840 /sbin/agetty -o "-p -- \\u" --noclear - linux
           │ ├─system-serial\x2dgetty.slice
           │ │ └─<email address hidden>
           │ │ └─6834 /sbin/agetty -o "-p -- \\u" --keep-baud 115200,57600,38400,9600 - vt220
           │ ├─systemd-journald.service
           │ │ └─3834 /lib/systemd/systemd-journald
           │ ├─systemd-logind.service
           │ │ └─11626 /lib/systemd/systemd-logind
           │ ├─systemd-networkd.service
           │ │ ├─1086 /lib/systemd/systemd-networkd
           │ │ └─4522 /lib/systemd/systemd-networkd
           │ ├─systemd-resolved.service
           │ │ └─6942 /lib/systemd/systemd-resolved
           │ ├─systemd-timesyncd.service
           │ │ └─5877 /lib/systemd/systemd-timesyncd
           │ ├─systemd-udevd.service
           │ │ └─udev
           │ │ └─5921 /lib/systemd/systemd-udevd
           │ └─unattended-upgrades.service
           │ ├─ 1227 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
           │ └─11628 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
           └─user.slice
             └─user-1000.slice
               └─session-1.scope
                 ├─ 1856 "sshd: ubuntu [priv]"
                 ├─ 1937 "sshd: ubuntu@pts/0"
                 ├─ 1941 -bash
                 ├─23835 sudo su
                 ├─23836 sudo su
                 ├─23837 su
                 └─23838 bash