evmc not supported on AMD processors but is enabled by default for "windows" instances

[Forrest Fuqua]

on AMD Processors it is known that QEMU does not not support hv-evmc
but as of notes/update-libvirt-enlightenments-for-windows-23abea98cc1db667.yaml it was enabled

There is no way that I can see to single out this flag and nova should check if the remote CPU is AMD and disable this enablement.

You can test this on any AMD machine with "qemu-system-x86_64 --nographic -cpu host,hv_relaxed,hv_vapic,hv_spinlocks=8191,hv_vpindex,hv_runtime,hv_synic,hv_stimer,hv_reset,hv_frequencies,hv_time,hv_evmcs --enable-kvm" and it will throw a critical error of "Hyper-V enlightened VMCS (hv-evmcs) is not supported by kernel"

Here are example logs that are thrown by nova when trying to start the instance, the instance was set to os_type: windows to have it throw this. This was tested on Zed with a AMD EYPC 7702

2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [None req-2172765f-0769-4b82-88fa-3cd68144ee42 ff259b03efc14bee83c881bc4920a12d a69aa169729a4dbea0d9b9683072260d - - default default] [instance: d416180c-cd56-4830-9787-3bb4953d9b02] Failed to build and run instance: libvirt.libvirt]
2023-03-04T20:57:46.694416Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.tsc-scale [bit 4]
2023-03-04T20:57:46.694441Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.vmcb-clean [bit 5]
2023-03-04T20:57:46.694465Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.pause-filter [bit 10]
2023-03-04T20:57:46.694489Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.pfthreshold [bit 12]
2023-03-04T20:57:46.694513Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.svme-addr-chk [bit 28]
2023-03-04T20:57:46.694549Z qemu-kvm: Hyper-V enlightened VMCS (hv-evmcs) is not supported by kernel
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] Traceback (most recent call last):
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/compute/manager.py", line 2517, in _build_and_run_instance
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] self.driver.spawn(context, instance, image_meta,
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 4366, in spawn
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] self._create_guest_with_network(
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7724, in _create_guest_with_network
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] self._cleanup(
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] self.force_reraise()
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] raise self.value
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7701, in _create_guest_with_network
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] guest = self._create_guest(
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7640, in _create_guest
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] guest.launch(pause=pause)
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/guest.py", line 168, in launch
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] LOG.exception('Error launching a defined domain with XML: %s',
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] self.force_reraise()
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] raise self.value
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/guest.py", line 165, in launch
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] return self._domain.createWithFlags(flags)
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 193, in doit
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] result = proxy_call(self._autowrap, f, *args, **kwargs)
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 151, in proxy_call
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] rv = execute(f, *args, **kwargs)
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] six.reraise(c, e, tb)
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/six.py", line 719, in reraise
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] raise value
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] rv = meth(*args, **kwargs)
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] File "/usr/lib64/python3.9/site-packages/libvirt.py", line 1398, in createWithFlags
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] raise libvirtError('virDomainCreateWithFlags() failed')
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2023-03-04T20:57:46.694375Z qemu-kvm: warning: This feature depends on other features that were not request]
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] 2023-03-04T20:57:46.694416Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.tsc-scale [bit 4]
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] 2023-03-04T20:57:46.694441Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.vmcb-clean [bit 5]
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] 2023-03-04T20:57:46.694465Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.pause-filter [bit 10]
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] 2023-03-04T20:57:46.694489Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.pfthreshold [bit 12]
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] 2023-03-04T20:57:46.694513Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.svme-addr-chk [bit 28]
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] 2023-03-04T20:57:46.694549Z qemu-kvm: Hyper-V enlightened VMCS (hv-evmcs) is not supported by kernel
2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]
2023-03-04 15:57:47.726 7 INFO os_vif [None req-2172765f-0769-4b82-88fa-3cd68144ee42 ff259b03efc14bee83c881bc4920a12d a69aa169729a4dbea0d9b9683072260d - - default default] Successfully unplugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:23:e5:4f,bridge_name='br-int',has_traffic)
2023-03-04 15:57:47.981 7 INFO nova.compute.manager [None req-2172765f-0769-4b82-88fa-3cd68144ee42 ff259b03efc14bee83c881bc4920a12d a69aa169729a4dbea0d9b9683072260d - - default default] [instance: d416180c-cd56-4830-9787-3bb4953d9b02] Took 0.26 seconds to deallocate network for instan.
2023-03-04 15:57:48.060 7 INFO nova.scheduler.client.report [None req-2172765f-0769-4b82-88fa-3cd68144ee42 ff259b03efc14bee83c881bc4920a12d a69aa169729a4dbea0d9b9683072260d - - default default] Deleted allocations for instance d416180c-cd56-4830-9787-3bb4953d9b02
2023-03-04 15:57:48.079 7 INFO nova.compute.manager [-] [instance: d416180c-cd56-4830-9787-3bb4953d9b02] During sync_power_state the instance has a pending task (block_device_mapping). Skip.

[sean mooney (sean-k-mooney)]

there are no references to this being a intel exclucive featur in the libvirt docs

https://libvirt.org/formatdomain.html#hypervisor-features

but it would be relitivly simple to check for amd here

https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L6196

unless you have seen this in the hyper-v virt driver this should just have the libvirt tag

[sean mooney (sean-k-mooney)]

triaging as medium as this will only affect a subset of deplyments where amd host are used in combination with images that decalre the os_type as windows.

this can be worked around by not setting OS_TYPE=windows which will disable all enlightementes.

[sean mooney (sean-k-mooney)]

i also dont see it documented here

https://www.qemu.org/docs/master/system/i386/hyperv.html

that hv-evmc so i dont think that its widely known that its not supproted on amd.

can you link to where this is documented?

[Forrest Fuqua (fffics)]

It does show in the docs, its just not pointed out very well,

https://www.qemu.org/docs/master/system/i386/hyperv.html

hv-evmcs

    The enlightenment is nested specific, it targets Hyper-V on KVM guests. When enabled, it provides Enlightened VMCS version 1 feature to the guest. The feature implements paravirtualized protocol between L0 (KVM) and L1 (Hyper-V) hypervisors making L2 exits to the hypervisor faster. *The feature is Intel-only.*

    Note: some virtualization features (e.g. Posted Interrupts) are disabled when hv-evmcs is enabled. It may make sense to measure your nested workload with and without the feature to find out if enabling it is beneficial.

    Requires: hv-vapic

[sean mooney (sean-k-mooney)]

ack there are two ways to fix this.

the simple one would be to just stop enabling it always.

the more complex one is to advertise this capability as a trait and recorded if vms use it or not then schulde based on the trait and its usage.

I'm leaning toward the first option now as a backpackable fix and the second approach as a wishlist feature to do in the future.

[Sylvain Bauza (sylvain-bauza)]

Putting the bug importance to High as it's preventing a nova boot on a windows-based image on AMD processors (if the image property is set). This is quite a large impact and we would like to fix it sooner than later.

[Ionuț Bîru (ionut-3)]

for me is important to keep os_type=windows in place and for running instances, on compute nodes with amd I had to modify nova/virt/libvirt/driver.py line 6170 to set hv.evmcs = False.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/899776

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/900629

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/904183

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/899776
Committed: https://opendev.org/openstack/nova/commit/86a35e97d286cbb6e23f8cc7bec5a05f022da0cb
Submitter: "Zuul (22348)"
Branch: master

commit 86a35e97d286cbb6e23f8cc7bec5a05f022da0cb
Author: Artom Lifshitz <email address hidden>
Date: Tue Oct 31 22:52:50 2023 -0400

    libvirt: Stop unconditionally enabling evmcs

    In I008841988547573878c4e06e82f0fa55084e51b5 we started enabling a
    bunch of libvirt enlightenments for Windows unconditionally. Turns
    out, the `evmcs` enlightenment only works on Intel hosts, and we broke
    the ability to run Windows guests on AMD machines. Until we become
    smarter about conditionally enabling evmcs (with something like traits
    for host CPU features), just stop enabling it at all.

    Change-Id: I2ff4fdecd9dc69de283f0e52e07df1aeaf0a9048
    Closes-bug: 2009280

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/nova/+/909082

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/904183
Committed: https://opendev.org/openstack/nova/commit/e618e78edc6293d248a5fa2eb63b3fa636250fca
Submitter: "Zuul (22348)"
Branch: master

commit e618e78edc6293d248a5fa2eb63b3fa636250fca
Author: songjie <email address hidden>
Date: Mon Dec 25 16:59:36 2023 +0800

    libvirt: stop enabling hyperv feature reenlightenment

    The 'reenlightenment' hyperv enlightenment will cause
    instances live-migration to fail (KVM currently doesn’t
    fully support reenlightenment notifications, see
    www.qemu.org/docs/master/system/i386/hyperv.html),
    so don't enable it now.

    Change-Id: I6821819450bc96e4304125ea3b76a0e462e6e33f
    Closes-Bug: #2046549
    Related-Bug: #2009280

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/nova/+/909086

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/nova/+/909098

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/nova/+/909228

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/nova/+/909229

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/909082
Committed: https://opendev.org/openstack/nova/commit/0b7a59ad2812c3501332155211b18f224cd55bca
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 0b7a59ad2812c3501332155211b18f224cd55bca
Author: Artom Lifshitz <email address hidden>
Date: Tue Oct 31 22:52:50 2023 -0400

    libvirt: Stop unconditionally enabling evmcs

    In I008841988547573878c4e06e82f0fa55084e51b5 we started enabling a
    bunch of libvirt enlightenments for Windows unconditionally. Turns
    out, the `evmcs` enlightenment only works on Intel hosts, and we broke
    the ability to run Windows guests on AMD machines. Until we become
    smarter about conditionally enabling evmcs (with something like traits
    for host CPU features), just stop enabling it at all.

    Change-Id: I2ff4fdecd9dc69de283f0e52e07df1aeaf0a9048
    Closes-bug: 2009280
    (cherry picked from commit 86a35e97d286cbb6e23f8cc7bec5a05f022da0cb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/909086
Committed: https://opendev.org/openstack/nova/commit/436e525a970bb991e147a0b4c3ce389a815252c2
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 436e525a970bb991e147a0b4c3ce389a815252c2
Author: songjie <email address hidden>
Date: Mon Dec 25 16:59:36 2023 +0800

    libvirt: stop enabling hyperv feature reenlightenment

    The 'reenlightenment' hyperv enlightenment will cause
    instances live-migration to fail (KVM currently doesn’t
    fully support reenlightenment notifications, see
    www.qemu.org/docs/master/system/i386/hyperv.html),
    so don't enable it now.

    Change-Id: I6821819450bc96e4304125ea3b76a0e462e6e33f
    Closes-Bug: #2046549
    Related-Bug: #2009280
    (cherry picked from commit e618e78edc6293d248a5fa2eb63b3fa636250fca)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/909229
Committed: https://opendev.org/openstack/nova/commit/47271bb705de445ca5e17ba5e3d503cade2a0959
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 47271bb705de445ca5e17ba5e3d503cade2a0959
Author: songjie <email address hidden>
Date: Mon Dec 25 16:59:36 2023 +0800

    libvirt: stop enabling hyperv feature reenlightenment

    The 'reenlightenment' hyperv enlightenment will cause
    instances live-migration to fail (KVM currently doesn’t
    fully support reenlightenment notifications, see
    www.qemu.org/docs/master/system/i386/hyperv.html),
    so don't enable it now.

    Change-Id: I6821819450bc96e4304125ea3b76a0e462e6e33f
    Closes-Bug: #2046549
    Related-Bug: #2009280
    (cherry picked from commit e618e78edc6293d248a5fa2eb63b3fa636250fca)
    (cherry picked from commit 436e525a970bb991e147a0b4c3ce389a815252c2)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/909228
Committed: https://opendev.org/openstack/nova/commit/2bffc53ce8fb5239cb9cbd59f3bbd51577da0c4a
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 2bffc53ce8fb5239cb9cbd59f3bbd51577da0c4a
Author: Artom Lifshitz <email address hidden>
Date: Tue Oct 31 22:52:50 2023 -0400

    libvirt: Stop unconditionally enabling evmcs

    In I008841988547573878c4e06e82f0fa55084e51b5 we started enabling a
    bunch of libvirt enlightenments for Windows unconditionally. Turns
    out, the `evmcs` enlightenment only works on Intel hosts, and we broke
    the ability to run Windows guests on AMD machines. Until we become
    smarter about conditionally enabling evmcs (with something like traits
    for host CPU features), just stop enabling it at all.

    Change-Id: I2ff4fdecd9dc69de283f0e52e07df1aeaf0a9048
    Closes-bug: 2009280
    (cherry picked from commit 86a35e97d286cbb6e23f8cc7bec5a05f022da0cb)
    (cherry picked from commit 0b7a59ad2812c3501332155211b18f224cd55bca)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/nova/+/909541

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/909541
Committed: https://opendev.org/openstack/nova/commit/3ef339a383e7e23085a582260badb347d96d8935
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 3ef339a383e7e23085a582260badb347d96d8935
Author: Artom Lifshitz <email address hidden>
Date: Tue Oct 31 22:52:50 2023 -0400

    libvirt: Stop unconditionally enabling evmcs

    In I008841988547573878c4e06e82f0fa55084e51b5 we started enabling a
    bunch of libvirt enlightenments for Windows unconditionally. Turns
    out, the `evmcs` enlightenment only works on Intel hosts, and we broke
    the ability to run Windows guests on AMD machines. Until we become
    smarter about conditionally enabling evmcs (with something like traits
    for host CPU features), just stop enabling it at all.

    Change-Id: I2ff4fdecd9dc69de283f0e52e07df1aeaf0a9048
    Closes-bug: 2009280
    (cherry picked from commit 86a35e97d286cbb6e23f8cc7bec5a05f022da0cb)
    (cherry picked from commit 0b7a59ad2812c3501332155211b18f224cd55bca)
    (cherry picked from commit 2bffc53ce8fb5239cb9cbd59f3bbd51577da0c4a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/909098
Committed: https://opendev.org/openstack/nova/commit/201bfa5bd21e3436b5882880530d5937b7dca61c
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 201bfa5bd21e3436b5882880530d5937b7dca61c
Author: songjie <email address hidden>
Date: Mon Dec 25 16:59:36 2023 +0800

    libvirt: stop enabling hyperv feature reenlightenment

    The 'reenlightenment' hyperv enlightenment will cause
    instances live-migration to fail (KVM currently doesn’t
    fully support reenlightenment notifications, see
    www.qemu.org/docs/master/system/i386/hyperv.html),
    so don't enable it now.

    Change-Id: I6821819450bc96e4304125ea3b76a0e462e6e33f
    Closes-Bug: #2046549
    Related-Bug: #2009280
    (cherry picked from commit e618e78edc6293d248a5fa2eb63b3fa636250fca)
    (cherry picked from commit 436e525a970bb991e147a0b4c3ce389a815252c2)
    (cherry picked from commit 47271bb705de445ca5e17ba5e3d503cade2a0959)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 29.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 26.3.0 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "sean mooney <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/nova/+/900629
Reason: fixed as part of https://review.opendev.org/c/openstack/nova/+/904183

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 27.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 28.1.0 release.