With ZED nova enabled some new hyperv settings for instances with
os_type="windows" property (https://opendev.org/openstack/nova/commit/57ab45323cf5617ebd2decd757e708673d949a8f).
Enabling the reenlightenment setting can lead to unmigrateable VMs.
QEMU currently does not support full re-enlightenment notifications (see https://www.qemu.org/docs/master/system/i386/hyperv.html).
Windows VMs which have hyperv or WSL enabled make use of that feature,
but qemu can only migrate them if a tsc-frequency is set. The frequency
setting however is not applied and when set would restrict the target hv
to support the exact frequency.
Trying to migrate instances with reenlightenment enabled will fail
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/eventlet/hubs/hub.py", line 476, in fire_timers
timer() File "/usr/local/lib/python3.10/dist-packages/eventlet/hubs/timer.py", line 59, in __call__
cb(*args, **kw)
File "/usr/local/lib/python3.10/dist-packages/eventlet/event.py", line 175, in _do_send
waiter.switch(result)
File "/usr/local/lib/python3.10/dist-packages/eventlet/greenthread.py", line 221, in main
result = function(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/nova/utils.py", line 656, in context_wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/nova/virt/libvirt/driver.py", line 10214, in _live_migration_operation
with excutils.save_and_reraise_exception():
File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
self.force_reraise()
File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
raise self.value
File "/usr/local/lib/python3.10/dist-packages/nova/virt/libvirt/driver.py", line 10203, in _live_migration_operation
guest.migrate(self._live_migration_uri(dest),
File "/usr/local/lib/python3.10/dist-packages/nova/virt/libvirt/guest.py", line 642, in migrate
self._domain.migrateToURI3(
File "/usr/local/lib/python3.10/dist-packages/eventlet/tpool.py", line 193, in doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/eventlet/tpool.py", line 151, in proxy_call rv = execute(f, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/eventlet/tpool.py", line 132, in execute
six.reraise(c, e, tb)
File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
raise value
File "/usr/local/lib/python3.10/dist-packages/eventlet/tpool.py", line 86, in tworker
rv = meth(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/libvirt.py", line 2157, in migrateToURI3
raise libvirtError('virDomainMigrateToURI3() failed')
libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor:
2023-12-14T09:05:01.441735Z qemu-system-x86_64: Guest enabled re-enlightenment notifications, 'tsc-frequency=' has to be specified
2023-12-14T09:05:01.441854Z qemu-system-x86_64: error while loading state for instance 0x0 of device 'cpu'
2023-12-14T09:05:01.448766Z qemu-system-x86_64: load of migration failed: Invalid argument
Adding the tsc clock or removing the hypver reenlightenment will make migration work again.
eg.:
<clock>
<timer name='tsc' frequency='2095078000'/>
</clock>
A related bug for windows enlightenments targets evmcs: https://bugs.launchpad.net/nova/+bug/2009280
Fix proposed to branch: master
Review: https:/