After updating systemd packages to 245.4-4ubuntu3.13 and libvirt packages to 6.0.0-0ubuntu8.14, it is no longer possible to start kvm virtual machines

[Gerd Jakobovitsch]

# systemctl status systemd-machined.service
● systemd-machined.service - Virtual Machine and Container Registration Service
     Loaded: loaded (/lib/systemd/system/systemd-machined.service; static; vendor preset: enabled)
     Active: inactive (dead)
       Docs: man:systemd-machined.service(8)
             https://www.freedesktop.org/wiki/Software/systemd/machined

Sep 17 15:40:42 MGL1-SRV-0006 systemd[1]: Dependency failed for Virtual Machine and Container Registration Service.
Sep 17 15:40:42 MGL1-SRV-0006 systemd[1]: systemd-machined.service: Job systemd-machined.service/start failed with result 'dependency'.
Sep 17 15:42:26 MGL1-SRV-0006 systemd[1]: Dependency failed for Virtual Machine and Container Registration Service.
Sep 17 15:42:26 MGL1-SRV-0006 systemd[1]: systemd-machined.service: Job systemd-machined.service/start failed with result 'dependency'.
Sep 17 16:02:55 MGL1-SRV-0006 systemd[1]: Dependency failed for Virtual Machine and Container Registration Service.
Sep 17 16:02:55 MGL1-SRV-0006 systemd[1]: systemd-machined.service: Job systemd-machined.service/start failed with result 'dependency'.
Sep 17 16:04:39 MGL1-SRV-0006 systemd[1]: Dependency failed for Virtual Machine and Container Registration Service.
Sep 17 16:04:39 MGL1-SRV-0006 systemd[1]: systemd-machined.service: Job systemd-machined.service/start failed with result 'dependency'.

Trying to start a virtual machine from openstack, it defines the machine, but is not able to execute it.

I used to have libvirt apparmor errors, but disabling libvirt in apparmor didn't solve the problem:
[ 27.785713] audit: type=1400 audit(1631888214.441:67): apparmor="DENIED" operation="capable" profile="libvirtd" pid=4817 comm="libvirtd" capability=17 capname="sys_rawio"

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: systemd-container 245.4-4ubuntu3.11
ProcVersionSignature: Ubuntu 5.11.0-34.36~20.04.1-generic 5.11.22
Uname: Linux 5.11.0-34-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.20
Architecture: amd64
CasperMD5CheckResult: skip
Date: Fri Sep 17 16:23:39 2021
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)

[Gerd Jakobovitsch (gerd-jakobovitsch)]

I tried to downgrade the packages, but the problem persisted. For libvirt, the packages for 6.0.0-0ubuntu8.13 were no longer available, so I downgraded for 6.0.0-0ubuntu8.3.

[Dan Streetman (ddstreet)]

you state you have systemd version 245.4-4ubuntu3.13 installed, but your description shows otherwise:
Package: systemd-container 245.4-4ubuntu3.11

[Gerd Jakobovitsch (gerd-jakobovitsch)]

As I stated in the comment, I downgraded the systemd packages to 3.11 to try to get to a working state, but the error persisted after the downgrade.

To get the full picture, these were the updated packages after which kvm stopped to work:

root@MGL1-SRV-0006:~# apt list --upgradable
Listing... Done
libnss-mymachines/focal-updates 245.4-4ubuntu3.13 amd64 [upgradable from: 245.4-4ubuntu3.11]
libnss-systemd/focal-updates 245.4-4ubuntu3.13 amd64 [upgradable from: 245.4-4ubuntu3.11]
libpam-systemd/focal-updates 245.4-4ubuntu3.13 amd64 [upgradable from: 245.4-4ubuntu3.11]
libsystemd0/focal-updates 245.4-4ubuntu3.13 amd64 [upgradable from: 245.4-4ubuntu3.11]
libudev1/focal-updates 245.4-4ubuntu3.13 amd64 [upgradable from: 245.4-4ubuntu3.11]
libudisks2-0/focal-updates 2.8.4-1ubuntu2 amd64 [upgradable from: 2.8.4-1ubuntu1]
libvirt-clients/focal-updates 6.0.0-0ubuntu8.14 amd64 [upgradable from: 6.0.0-0ubuntu8.13]
libvirt-daemon-driver-qemu/focal-updates 6.0.0-0ubuntu8.14 amd64 [upgradable from: 6.0.0-0ubuntu8.13]
libvirt-daemon-driver-storage-rbd/focal-updates 6.0.0-0ubuntu8.14 amd64 [upgradable from: 6.0.0-0ubuntu8.13]
libvirt-daemon-system-systemd/focal-updates 6.0.0-0ubuntu8.14 amd64 [upgradable from: 6.0.0-0ubuntu8.13]
libvirt-daemon-system/focal-updates 6.0.0-0ubuntu8.14 amd64 [upgradable from: 6.0.0-0ubuntu8.13]
libvirt-daemon/focal-updates 6.0.0-0ubuntu8.14 amd64 [upgradable from: 6.0.0-0ubuntu8.13]
libvirt0/focal-updates 6.0.0-0ubuntu8.14 amd64 [upgradable from: 6.0.0-0ubuntu8.13]
python3-os-vif/focal-updates 2.0.0-0ubuntu2 all [upgradable from: 2.0.0-0ubuntu1]
python3-pyroute2/focal-updates 0.5.9-0ubuntu2 all [upgradable from: 0.5.9-0ubuntu1]
systemd-container/focal-updates 245.4-4ubuntu3.13 amd64 [upgradable from: 245.4-4ubuntu3.11]
systemd-sysv/focal-updates 245.4-4ubuntu3.13 amd64 [upgradable from: 245.4-4ubuntu3.11]
systemd/focal-updates 245.4-4ubuntu3.13 amd64 [upgradable from: 245.4-4ubuntu3.11]
udev/focal-updates 245.4-4ubuntu3.13 amd64 [upgradable from: 245.4-4ubuntu3.11]
udisks2/focal-updates 2.8.4-1ubuntu2 amd64 [upgradable from: 2.8.4-1ubuntu1]

[Gerd Jakobovitsch (gerd-jakobovitsch)]

The full downgrade requests were:

# apt install libvirt0=6.0.0-0ubuntu8.3 libvirt-daemon=6.0.0-0ubuntu8.3 libvirt-clients=6.0.0-0ubuntu8.3 libvirt-daemon-driver-qemu=6.0.0-0ubuntu8.3 libvirt-daemon-driver-storage-rbd=6.0.0-0ubuntu8.3 libvirt-daemon-system=6.0.0-0ubuntu8.3
# apt install systemd=245.4-4ubuntu3.11 udev=245.4-4ubuntu3.11 systemd-container=245.4-4ubuntu3.11 systemd-sysv=245.4-4ubuntu3.11 libudev1=245.4-4ubuntu3.11 libsystemd0=245.4-4ubuntu3.11 libnss-mymachines=245.4-4ubuntu3.11 libnss-systemd=245.4-4ubuntu3.11 libpam-systemd=245.4-4ubuntu3.11

I tried just downgrading libvirt, then upgraded it again and downgraded systemd, then both sets. None option worked.

[Gerd Jakobovitsch (gerd-jakobovitsch)]

Some additional info:

The servers where I had problem were running a hyper convergent solution, with ceph installed via ceph-ansible, including at least a docker container (running node_exporter) - it is failing as well. These servers were initially installed with general kernel (5.4.0-81-generic) and a hwe kernel (5.8.0-34) was installed afterwards.

I succeeded in installing a new server with hwe kernel and updated libraries, as well as upgrade a system without ceph nor docker; at the upgrade, some errors concerned me, but didn't cause any bigger trouble:

Setting up libvirt-daemon-driver-qemu (6.0.0-0ubuntu8.14) ...
Setting up libvirt-daemon (6.0.0-0ubuntu8.14) ...
Setting up libvirt-daemon-driver-storage-rbd (6.0.0-0ubuntu8.14) ...
Setting up libvirt-daemon-system (6.0.0-0ubuntu8.14) ...
Job failed. See "journalctl -xe" for details.
virtlockd.service is a disabled or a static unit, not starting it.
virtlogd.service is a disabled or a static unit, not starting it.
Setting up libvirt-daemon dnsmasq configuration.

-- A start job for unit libvirtd.service has begun execution.
--
-- The job identifier is 369924.
Sep 17 18:18:25 MGL1-SRV-0018 systemd[1]: Started Virtualization daemon.
-- Subject: A start job for unit libvirtd.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit libvirtd.service has finished successfully.
--
-- The job identifier is 369924.
Sep 17 18:18:25 MGL1-SRV-0018 nova-compute[3163250]: 2021-09-17 18:18:25.703 3163250 ERROR nova.scheduler.client.report [req-b80a57e5-3e9d-4dec-a861-d04363077b0c - - - - -] [req-38886c27-53ea-41b9-bf14-f68080cac535] Failed to update traits to [COMPUTE_TRUSTED_CERTS,COMPUTE_IMAGE_TYPE_AMI,HW_CPU_X86_VMX,HW_CPU_X86_MPX,COMPUTE_VOLUME_ATTACH_WITH_TAG,HW_CPU_X86_AVX512DQ,COMPUTE_GRAPHICS_MODEL_QXL,COMPUTE_NET_VIF_MODEL_E1000,COMPUTE_NET_VIF_MODEL_E1000E,HW_CPU_X86_AESNI,COMPUTE_NET_ATTACH_INTERFACE_WITH_TAG,COMPUTE_NET_ATTACH_INTERFACE,COMPUTE_DEVICE_TAGGING,HW_CPU_X86_SSE2,HW_CPU_X86_AVX2,COMPUTE_STORAGE_BUS_VIRTIO,HW_CPU_X86_SSSE3,COMPUTE_ACCELERATORS,COMPUTE_NET_VIF_MODEL_PCNET,HW_CPU_X86_AVX512CD,COMPUTE_GRAPHICS_MODEL_CIRRUS,HW_CPU_X86_FMA3,HW_CPU_X86_SSE41,HW_CPU_X86_AVX512F,COMPUTE_IMAGE_TYPE_QCOW2,COMPUTE_STORAGE_BUS_SATA,COMPUTE_GRAPHICS_MODEL_VGA,HW_CPU_X86_BMI,COMPUTE_STORAGE_BUS_IDE,COMPUTE_IMAGE_TYPE_ARI,COMPUTE_STATUS_DISABLED,COMPUTE_NET_VIF_MODEL_SPAPR_VLAN,HW_CPU_X86_MMX,COMPUTE_NET_VIF_MODEL_RTL8139,COMPUTE_IMAGE_TYPE_RAW,COMPUTE_STORAGE_BUS_FDC,COMPUTE_GRAPHICS_MODEL_NONE,HW_CPU_X86_F16C,HW_CPU_X86_SSE,COMPUTE_RESCUE_BFV,HW_CPU_X86_AVX,COMPUTE_NET_VIF_MODEL_NE2K_PCI,COMPUTE_IMAGE_TYPE_AKI,COMPUTE_STORAGE_BUS_SCSI,HW_CPU_X86_BMI2,COMPUTE_IMAGE_TYPE_ISO,COMPUTE_VOLUME_MULTI_ATTACH,COMPUTE_VOLUME_EXTEND,HW_CPU_HYPERTHREADING,HW_CPU_X86_SSE42,HW_CPU_X86_AVX512VL,HW_CPU_X86_ABM,COMPUTE_NET_VIF_MODEL_VIRTIO,HW_CPU_X86_CLMUL,HW_CPU_X86_AVX512BW,HW_CPU_X86_AVX512VNNI,COMPUTE_GRAPHICS_MODEL_VMVGA,COMPUTE_GRAPHICS_MODEL_VIRTIO,COMPUTE_NODE,COMPUTE_STORAGE_BUS_USB] for resource provider with UUID f16fc1ab-3b42-431f-9d20-96a27224b90e. Got 409: {"errors": [{"status": 409, "title": "Conflict", "detail": "There was...

[Launchpad Janitor (janitor)]

[Expired for systemd (Ubuntu) because there has been no activity for 60 days.]