live migration fails because of missing cpu features vmx-*

[Guillaume Boutry]

[ Impact ]

libvirt users who attempt to migrate a VM to a Noble host will experience failures due to missing CPU features, which ultimately prevents the migration from concluding successfully. This is related to bug #2051754.

[ Test Plan ]

The reporter has a setup ready to reproduce the problem. In theory, all that is needed to trigger the error is:

1) Create LXD VMs.

2) Install libvirt inside each one, create a VM inside one, and then try to migrate it to the other.

There is also an extensive testsuite covering many migration scenarios that is used whenever a new libvirt/QEMU is prepared to be uploaded. I will be running the testsuite against the new libvirt to make sure it passes all tests.

[ Where problems could occur ]

Fortunately the fix is somewhat simple and contained: it's just one patch (that's already been accepted upstream), which changes a very specific code path in the virCPUx86UpdateLive function (inside src/cpu/cpu_x86.c). The problem happens in a very specific scenario where you're trying to migrate a VM between hosts using a similar libvirt version; in this case, the first libvirt (where the VM is created) correctly detects that there are some CPU features missing, but the second libvirt (where the VM will be migrated to) will fail when it notices that those same CPU features are not present.

Patches modifying code in this area have the possibility of messing up some (obscure?) migration path. We hope to prevent such a regression from happening by running our migration testsuite (as described in the Test Plan); it has already caught failures affecting corner cases in the past. If this proves insufficient, we can revert the changes temporarily while working with upstream (who has been responsive about this issue) to improve the existing solution.

[ Original Description ]

Trying to live migrate between two libvirt daemons 24.04 hosted on fresh LXD VMs is failing

virsh migrate --verbose --live demo-server-$host --copy-storage-all qemu+ssh://$other_host/system
error: operation failed: guest CPU doesn't match specification: missing features: vmx-apicv-xapic,vmx-apicv-register,vmx-apicv-vid,vmx-vmfunc,vmx-posted-intr,vmx-eptp-switching

Libvirt VMs were created with the following commands:
virt-install -n demo-server-$host --virt-type=kvm --arch=x86_64 --machine=q35 --description "demo-server migration" --os-variant=ubuntu22.04 --ram=2048 --cpu host-model,topology.sockets=1,topology.cores=2,topology.threads=1 --vcpus=2 --graphics none --cdrom /tmp/jammy-server-cloudimg-amd64.img --network bridge:br0

lsb_release -rd
Description: Ubuntu 24.04 LTS
Release: 24.04

apt-cache policy libvirt-daemon
libvirt-daemon:
  Installed: 10.0.0-2ubuntu8.2

ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: libvirt-daemon 10.0.0-2ubuntu8.2
ProcVersionSignature: Ubuntu 6.8.0-36.36-generic 6.8.4
Uname: Linux 6.8.0-36-generic x86_64
ApportVersion: 2.28.1-0ubuntu3
Architecture: amd64
CasperMD5CheckResult: unknown
CloudArchitecture: x86_64
CloudBuildName: server
CloudID: lxd
CloudName: lxd
CloudPlatform: lxd
CloudSerial: 20240702
CloudSubPlatform: LXD socket API v. 1.0 (/dev/lxd/sock)
Date: Wed Jul 3 18:43:50 2024
ProcEnviron:
 LANG=C.UTF-8
 PATH=(custom, no user)
 SHELL=/bin/bash
 TERM=xterm-256color
 XDG_RUNTIME_DIR=<set>
SourcePackage: libvirt
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 10.0.0-2ubuntu9

---------------
libvirt (10.0.0-2ubuntu9) oracular; urgency=medium

  * d/p/u/lp-2071848-fix-migration-with-disabled-vmx-features.patch:
    Fix migration issues with disabled vmx-* CPU features. (LP: #2071848)

 -- Sergio Durigan Junior <email address hidden> Thu, 04 Jul 2024 16:06:28 -0400

[Brian Murray (brian-murray)]

Hello Guillaume, or anyone else affected,

Accepted libvirt into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/10.0.0-2ubuntu8.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Guillaume Boutry (gboutry)]

Hi, I've just tested the proposed packages, and the issue is indeed fixed on my side.

I'll attach the reproduction script and logs to this ticket.

[Sergio Durigan Junior (sergiodj)]

Thank you, Guillaume!

As mentioned in the Test Plan, I also ran our QEMU migration testsuite against the new package, and here are the results:

prep (x86_64) : Pass 25 F/S/N 0/0/0 - RC 0 (29 min 53453 lin)
migrate (x86_64) : Pass 105 F/S/N 0/0/0 - RC 999 (24 min 91793 lin)
cross (x86_64) : Pass 24 F/S/N 0/0/0 - RC 999 (42 min 45610 lin)
misc (x86_64) : Pass 103 F/S/N 0/0/0 - RC 0 (31 min 58738 lin)

The "RC 999" code is not a problem here; it just means that the test script wasn't able to fully shutdown the LXD containers used for testing. What matters is the fact that all tests passed.

Therefore, I consider this bug as verified for Noble. Tagging accordingly.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 10.0.0-2ubuntu8.3

---------------
libvirt (10.0.0-2ubuntu8.3) noble; urgency=medium

  * d/p/u/lp-2071848-fix-migration-with-disabled-vmx-features.patch:
    Fix migration issues with disabled vmx-* CPU features. (LP: #2071848)

 -- Sergio Durigan Junior <email address hidden> Fri, 05 Jul 2024 14:48:04 -0400

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for libvirt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.