virt-manager does not notice libvirt-daemon-driver-lxc being installed until reboot

[hanshenrik]

If you install libvirt-daemon-driver-lxc while virt-manager is running, it will not notice that libvirt-daemon-driver-lxc has been installed until a system reboot, it seems.

restarting virt-manager does nothing. killall virt-manager; does nothing.
it insist the lxc-driver is not available, until you reboot the system, then it notice.

Noticed on ubuntu 24.04 virt-manager 4.1.0 libvirtd (libvirt) 10.0.0 libvirt-daemon-driver-lxc 10.0.0-2ubuntu8.5

[Christian Ehrhardt (paelzer)]

Hi,
since it is essentially split feature of libvirt I'd assume it is only detected when libvirt restarts.
virt-manager only asked libvirt what it can provide, and that hasn't been changed.
Could you check if - instead of a hard reboot - a restart of libvirtd would re-detect it as well?

If so we might need to balance the risk of restarting that service vs the comfort of any sich split out features to be detected right on install. I say balance as these packages could restart libvirtd on install to trigger it being detected.

For now, I'd appreciate if you could check if

$ systemctl restart libvirtd.service

Would make it be detected in your case

P.S. An alternative is to check with upstream if there is any intent for runtime detection of new features. Now that we split out more (split daemons in 25.04) this might be even more worth to sort out.

P.P.S. and maybe mention which release you are on, as even if it helps and we make the change it likely will only change (as behavior changes) in new releases and for old ones (to avoid regression) might be better as documentation entry.

[hanshenrik (hanshenrik)]

@paelzer can confirm, manually running

$ systemctl restart libvirtd.service

fix the problem without needing a full system restart.

I also tried "reload" in place of "restart", but the reload did not resolve the problem, only restart did.

>might need to balance the risk

i suggest asking the user if apt is running as interactive, and default to no when apt is running noninteractive.

>An alternative is to check with upstream if there is any intent for runtime detection of new features

I agree, that would be best. Hope someone else can look at it though, I am switching to Incus.

>and maybe mention which release you are on

ubuntu 24.04 virt-manager 4.1.0 libvirtd (libvirt) 10.0.0 libvirt-daemon-driver-lxc 10.0.0-2ubuntu8.5

[Christian Ehrhardt (paelzer)]

Interesting and thanks for checking.

BTW - checking if lxc is available but without virt-manager or such (which keeps a connection up btw and therefore will never have it restart) is better done via: ` virsh --connect lxc:/// list`

If lxc support is loaded you will get:

$ virsh --connect lxc:/// list
 Id Name State
--------------------

If nothing is available to the daemon you'll see:

$ virsh --connect lxc:/// list
error: failed to connect to the hypervisor
error: no connection driver available for lxc://

[Christian Ehrhardt (paelzer)]

Equipped with that we can make some tests...

On one hand libvirt, if there is no API or domain load will shutdown after 120 seconds anyway.
I checked and just waiting will make it work.

I see the socket fall back to:
● libvirtd.socket - libvirt legacy monolithic daemon socket
...
     Active: active (listening)

And then from this state it immediately works:

$ sudo apt install libvirt-daemon-driver-lxc
$ virsh --connect lxc:/// list
error: failed to connect to the hypervisor
error: no connection driver available for lxc:///

But surely if a host is rather active (domains and api calls up) or has an open connection (like virt-manager) it will never recycle. And I can see that users would expect it to work after install (if possible).

---

The mentioned reload defined as
  "On receipt of SIGHUP libvirtd will reload its configuration."
Will NOT recycle the process.
I've seen e.g. my:
  560007 /usr/sbin/libvirtd --timeout 120
stay as is, but get new entries to read the conf:
...
Jan 07 10:34:51 Keschdeichel dnsmasq[2696]: read /etc/hosts - 13 names
Jan 07 10:34:51 Keschdeichel dnsmasq[2696]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 names
Jan 07 10:34:51 Keschdeichel dnsmasq-dhcp[2696]: read /var/lib/libvirt/dnsmasq/default.hostsfile

Therefore as you said "systemctl reload" isn't enough to make it realize there is something new.
Yet on the other hand restarting it can be heavy.
We are doing so on any package upgrade of libvirt-daemon-system itself, the question now is if there is a way in between from reload and restarting that is better in this case.

[Christian Ehrhardt (paelzer)]

It seems to be relative save, as it is already built to hand listening to the socket and back.
I've had an infinite loop of "virsh list" run and it never ran into an error when restarting many times in the background. The times changed, as you'd expect with usually
  real 0m0,043s
and inbetween on my restarts like
  real 0m0,678s
  real 0m0,598s
But never broken.
I even made it struggle to shut down which still worked but then was
  real 0m30,789s

So overall it seems restarting is rather safe nowadays.
And installing/upgrading such packages not a high frequency task.

The only thing that comes to mind is that we might want to delay and unify these if many libvirt-daemon-driver-* packages install or upgrade (they all build from the same source, so they'd usually all appear in one go). With the goal of only restarting it once.

For upgrades we could even maybe achieve that by ordering any libvirt-daemon-driver-* before the respective libvirt-daemon-system-systemd - not 100% sure if that is cleaner as complex dependencies can be just that - complex :-/. And since we need postinst restart for individual "installs" anyway we might want to leave that complexity out.

[Christian Ehrhardt (paelzer)]

So much for completeness, Sergio wanted to reach out to upstream folks to ensure if there is a better way we'd know about it. Assigning to him as a reminder to report what came out of that.

[Sergio Durigan Junior (sergiodj)]

Thanks for the investigation, Christian.

I'll open a discussion with the Debian maintainer to see what he thinks.

[Sergio Durigan Junior (sergiodj)]

FTR, we've discussed this case with the Debian maintainer and he agrees with our proposal to have libvirtd.service restart whenever a new libvirt module is installed.

In fact, there had already been a Merge Proposal on salsa to do just that, but it fell through the cracks. They resurrected it here:

https://salsa.debian.org/libvirt-team/libvirt/-/merge_requests/252

It should be uploaded to unstable very soon, and will be picked up in the next merge (or sooner, depending on the timeframe).

[Christian Ehrhardt (paelzer)]

11.0 in Debian has the changes that Sergio and I participated together with the other Debian maintainers.
A move 10.10 -> 11.0 should be small and 11.0 is the target for plucky anyway - so that is the next step.

[Christian Ehrhardt (paelzer)]

The change is good, but quite invasive to the handling of services and packaging.
I'd consider the risk of breaking people quit high when pushing that back as an SRU to noble :-/

We know that a restart of the service will resolve the situation, so people have a manual workaround.

Furthermore any reboot of the system as well as any update of libvirt (quite common) will resolve the situation.

And as we have see even just waiting 120 seconds without further interaction or open connection would stop the service, fall back to socket activation and then resolve the problem.

I'm tempted to say that therefore the risk/gain is more towards risk which makes me consider marking already active releases as won't fix.

I feel bad for leaving some potential question marks and discomfort, like the one that made Hans file this bug - but I'd need an idea how it could/should be solved less intensively in old releases. Maybe without any trigger magic and just putting the restart into all the modules with .so files? Not sure about it ...

[Christian Ehrhardt (paelzer)]

MR up: https://code.launchpad.net/~paelzer/ubuntu/+source/libvirt/+git/libvirt/+merge/480845
PPA: https://launchpad.net/~paelzer/+archive/ubuntu/lp-2092350-merge-11.0-plucky

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 11.0.0-2ubuntu1

---------------
libvirt (11.0.0-2ubuntu1) plucky; urgency=medium

  * Merge with Debian unstable (LP: #2092350). Remaining changes:
    - Disable libssh2 support (universe dependency)
    - d/control: add libzfslinux-dev to build-deps
    - d/control: drop libvirt-lxc, vbox and xen drivers to suggest
    - debian/patches/ubuntu/ovmf_paths.patch: adjust paths to secboot.fd UEFI
      Secure Boot enabled variants of the OVMF firmware and variable store for
      the paths where we ship these files in Ubuntu.
    - Set qemu-group to kvm (for compat with older ubuntu)
    - Additional apport package-hook
    - Autostart default bridged network (As upstream does, but not Debian).
      In addition to just enabling it our solution provides:
      + do not autostart if subnet is already taken (e.g. in guests).
      + iterate some alternative subnets before giving up
    - d/p/ubuntu/Allow-libvirt-group-to-access-the-socket.patch: This is
      the group based access to libvirt functions as it was used in Ubuntu
      for quite a long time.
      + d/p/ubuntu/daemon-augeas-fix-expected.patch fix some related tests
        due to the group access change.
      + d/libvirt-daemon-driver-qemu.postinst*: add users in sudo to the libvirt
        group.
    - Update README.Debian with Ubuntu changes
    - d/p/ubuntu/ubuntu_machine_type.patch: accept ubuntu types as pci440fx
    - fix autopkgtests (LP 1899180)
      + d/t/control, d/t/smoke-qemu-session: fixup smoke-qemu-session by making
        vmlinuz available and accessible (Debian bug 848314)
      + d/t/control: fix smoke-qemu-session by ensuring the service will run
        installing libvirt-daemon-system
      + d/t/smoke-lxc: fix smoke-lxc by ignoring potential issues on destroy as
        long as the following undefine succeeds
      + d/t/smoke-lxc: use systemd instead of sysV to restart the service
      + d/t/control, d/t/smoke-lxc: retry service restart and skip test if
        failing; This was flaky on some release/architectures
      + d/t/smoke-lxc: retry check_domain being flaky on arm64
    - dnsmasq related enhancements
      + run dnsmasq as libvirt-dnsmasq (LP 1743718)
      + d/libvirt-daemon-driver-qemu.postinst*: add libvirt-dnsmasq user and group
      + d/libvirt-daemon-driver-qemu.postrm*: remove libvirt-dnsmasq user and group
        on purge
      + d/p/ubuntu/dnsmasq-as-priv-user: write dnsmasq config with user
        libvirt-dnsmasq and adapt the self tests to expect that config
      + d/libvirt-daemon-system.postinst: fix old libvirt-dnsmasq users group
      + Add dnsmasq configuration to work with system wide dnsmasq-base
    - d/p/ubuntu/set-default-machine-to-ubuntu.patch: to select default
      machine type correctly with newer qemu/libvirt
    - d/p/ubuntu/lp-1861125-ubuntu-models: recognize Ubuntu models for
      (LP 1861125) fixups
    - d/p/ubuntu/wait-for-qemu-kvm.patch - avoid hangs on startup (LP 1887592)
    - d/libvirt-daemon-common.libvirt-guests.default: shut guests down
      in parallel
    - Apparmor Delta that is Ubuntu specific or yet to be upstreamed
      split into logical pieces. Fi...