lxd-agent-setup cannot be started after a distro upgrade to 24.04

[Heinrich Schuchardt]

Starting the lxd-agent.service service fails with kernel 6.5.0-9-generic.

# /lib/systemd/lxd-agent-setup[ 1238.980879] 9pnet_virtio: no channels available for device config

[ 1239.456813] 9pnet_virtio: no channels available for device config
Couldn't mount virtiofs or 9p, failing.

The "9pnet_virtio: no channels available for device config" messages occur frequently in the kernel log and did not exist in kernels prior to 6.5.

Removing the 9pnet_virtio module does not resolve the issue.

Best regards

Heinrich

ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: linux-image-6.5.0-9-generic 6.5.0-9.9.1
ProcVersionSignature: Ubuntu 6.5.0-9.9.1-generic 6.5.3
Uname: Linux 6.5.0-9-generic riscv64
ApportVersion: 2.28.0-0ubuntu1
Architecture: riscv64
CasperMD5json:
 {
   "result": "skip"
 }
Date: Fri Feb 23 12:28:11 2024
InstallationDate: Installed on 2023-12-04 (81 days ago)
InstallationMedia: Ubuntu-Server 24.04 "Noble Numbat" - Daily riscv64 (20231204)
ProcEnviron:
 LANG=en_US.UTF-8
 PATH=(custom, no user)
 SHELL=/bin/bash
 TERM=vt220
SourcePackage: linux-riscv
UpgradeStatus: No upgrade log present (probably fresh install)

[Heinrich Schuchardt (xypron)]

The installed snap version is 5.20-f3dd836.

[Simon Déziel (sdeziel)]

The kernel config looks sane:

$ wget http://ports.ubuntu.com/pool/main/l/linux-riscv/linux-modules-6.5.0-9-generic_6.5.0-9.9.1_riscv64.deb
$ ar -x linux-modules-6.5.0-9-generic_6.5.0-9.9.1_riscv64.deb
$ tar -xf data.tar ./boot/config-6.5.0-9-generic
$ grep 9P boot/config-6.5.0-9-generic
CONFIG_NET_9P=m
CONFIG_NET_9P_FD=m
CONFIG_NET_9P_VIRTIO=m
CONFIG_NET_9P_RDMA=m
# CONFIG_NET_9P_DEBUG is not set
CONFIG_VIDEO_MT9P031=m
CONFIG_9P_FS=m
CONFIG_9P_FSCACHE=y
CONFIG_9P_FS_POSIX_ACL=y
CONFIG_9P_FS_SECURITY=y
$ grep VIRTIO_FS boot/config-6.5.0-9-generic
CONFIG_VIRTIO_FS=m

For the virtiofs part, we know it currently doesn't work on non-amd64 but the 9p fallback should have worked. So it seems the 9pnet_virtio module loaded fine but somehow couldn't find the config drive. I'll try to dig that a bit deeper, thanks for reporting.

[Heinrich Schuchardt (xypron)]

I have these files on my hardware created in 2021

/etc/systemd/system/multi-user.target.wants/lxd-agent.service
/etc/systemd/system/multi-user.target.wants/lxd-agent-9p.service

which don't exist on fresh installs.

Simon suggested to let the package delete them.

[Simon Déziel (sdeziel)]

I got the initial assessment wrong, thanks Heinrich for clearing up that confusion :)

Here is what happened here:

On Jammy, lxd-agent-loader (0.5) is configured to start lxd-agent.service when the multi-user.target was reached coupled with a ConditionPathExists=/dev/virtio-ports/org.linuxcontainers.lxd
This means it would do nothing if that virtio-ports device was missing.

On Noble, lxd-agent-loader (0.6) was changed to have udev start lxd-agent.service only when it sees a virtio-ports LXD knows. Along with that change, the ConditionPathExists was removed as no longer relevant.

Heinrich did a distro-upgrade moving lxd-agent-loader from 0.5 -> 0.6 but this left behind the following symlinks:

```
/etc/systemd/system/multi-user.target.wants/lxd-agent.service
/etc/systemd/system/multi-user.target.wants/lxd-agent-9p.service
```

Causing lxd-agent.service to always want to start, even if the virtio-ports is not there, thus causing this bug/failure to start lxd-agent.service.

The fix will be to have the package cleanup the old symlinks left in the multi-user.target.wants directory.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dan Bungert (dbungert)]

adjusting title, as it doesn't appear to be arch or kernel ver specific.
Thanks for the suggested workaround of cleaning up the symlinks, that does seem to help.

[Simon Déziel (sdeziel)]

Thanks Dan, that prompted me to also call out that's something that manifests only on distro upgrade.

[Heinrich Schuchardt (xypron)]

Uninstalling lxd-agent-loader with 'sudo apt-get purge lxd-agent-loader' does not remove the symbolic link

/etc/systemd/system/multi-user.target.wants/lxd-agent.service -> /lib/systemd/system/lxd-agent.service

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxd-agent-loader - 0.7

---------------
lxd-agent-loader (0.7) noble; urgency=medium

  * d/lxd-agent-setup.postinst: Cleanup former unit files (LP: #2054814)
  * lxd-agent-setup: Bump /run/lxd_agent size to 50M (LP: #2058302)
  * lxd-agent-setup: Attempt to restore SELinux labels (LP: #2058303)

 -- Simon Deziel <email address hidden> Tue, 13 Feb 2024 16:48:15 -0500