Nova fails to live migrate VMs even though source and destination hosts are the same

[birbilakos]

I'm using ansible-openstack 2023.2 installation. Testing out the live migration feature via horizon, it fails to do the migration on identically spec'ed source and destination hosts:

Error: Failed to live migrate instance to host "sjc-lnxserver-136". Details

Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility. 0 Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult (HTTP 400) (Request-ID: req-a5ce90ec-bbd3-4130-b868-592897759b68)

virsh caps is practically the same (expect the reported frequency) across both source and destination.
    <cpu>
      <arch>x86_64</arch>
      <model>Cascadelake-Server-noTSX</model>
      <vendor>Intel</vendor>
      <microcode version='83899908'/>
      <counter name='tsc' frequency='3591521000' scaling='yes'/>
      <topology sockets='1' dies='1' cores='8' threads='2'/>
      <feature name='ds'/>
      <feature name='acpi'/>
      <feature name='ss'/>
      <feature name='ht'/>
      <feature name='tm'/>
      <feature name='pbe'/>
      <feature name='dtes64'/>
      <feature name='monitor'/>
      <feature name='ds_cpl'/>
      <feature name='vmx'/>
      <feature name='smx'/>
      <feature name='est'/>
      <feature name='tm2'/>
      <feature name='xtpr'/>
      <feature name='pdcm'/>
      <feature name='dca'/>
      <feature name='osxsave'/>
      <feature name='tsc_adjust'/>
      <feature name='cmt'/>
      <feature name='intel-pt'/>
      <feature name='pku'/>
      <feature name='ospke'/>
      <feature name='md-clear'/>
      <feature name='stibp'/>
      <feature name='arch-capabilities'/>
      <feature name='xsaves'/>
      <feature name='mbm_total'/>
      <feature name='mbm_local'/>
      <feature name='invtsc'/>
      <feature name='rdctl-no'/>
      <feature name='ibrs-all'/>
      <feature name='skip-l1dfl-vmentry'/>
      <feature name='mds-no'/>
      <feature name='tsx-ctrl'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='2048'/>
      <pages unit='KiB' size='1048576'/>
    </cpu>

Any ideas?

[birbilakos (birbilis)]

Apologies but I'm not sure if the selected project is the correct one to be honest.

[birbilakos (birbilis)]

Logs from the 'destination' server's nova-compute process:

Jan 13 13:35:09 sjc-lnxserver-126 nova-compute[48110]: 2024-01-13 13:35:09.517 48110 ERROR nova.virt.libvirt.driver [None req-7edbc759-2798-430b-a9fc-a76e87a25f67 a15b6525790645ecb5c746903316e42c 87a83841a4e24c68999ca1b87dc05593 - - default default] CPU doesn't have compatibility.

                                                       0

                                                       Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
Jan 13 13:35:09 sjc-lnxserver-126 nova-compute[48110]: 2024-01-13 13:35:09.567 48110 ERROR oslo_messaging.rpc.server [None req-7edbc759-2798-430b-a9fc-a76e87a25f67 a15b6525790645ecb5c746903316e42c 87a83841a4e24c68999ca1b87dc05593 - - default default] Exception during message handling: nova.exception.MigrationPreCheckError: Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility.

                                                       0

                                                       Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
                                                       2024-01-13 13:35:09.567 48110 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
                                                       2024-01-13 13:35:09.567 48110 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.0.0/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9668, in check_can_live_migrate_destination
                                                       2024-01-13 13:35:09.567 48110 ERROR oslo_messaging.rpc.server self._compare_cpu(None, source_cpu_info, instance)
                                                       2024-01-13 13:35:09.567 48110 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.0.0/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 10039, in _compare_cpu
                                                       2024-01-13 13:35:09.567 48110 ERROR oslo_messaging.rpc.server raise exception.InvalidCPUInfo(reason=m % {'ret': ret, 'u': u})
                                                       2024-01-13 13:35:09.567 48110 ERROR oslo_messaging.rpc.server nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility.
                                                       2024-01-13 13:35:09.567 48110 ERROR oslo_messaging.rpc.server
                                                       2024-01-13 13:35:09.567 48110 ERROR oslo_messaging.rpc.server 0
                                                       2024-01-13 13:35:09.567 48110 ERROR oslo_messaging.rpc.server
                                                       2024-01-13 13:35:09.567 48110 ERROR oslo_messaging.rpc.server Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
                                                       2024-01-13 13:35:09.567 ...

[birbilakos (birbilis)]

# virsh cpu-compare virsh-caps-124-full.xml
CPU described in virsh-caps-124-full.xml is identical to host CPU

[keerthivasan (keerthivassan86)]

Starting Antelope, nova is no more using cpu_compare, instead it uses Hypervisor_cpu_compare for pre migration checks.

https://bugs.launchpad.net/nova/+bug/2023035
https://bugs.launchpad.net/nova/+bug/2039803

Please refer to above bug, that can help

https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.skip_cpu_compare_at_startup
and
https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.skip_cpu_compare_on_dest
are related workaroudn options that i was referring too
using cpu_mode=custom and a sepcific cpu_model might also work

[birbilakos (birbilis)]

I can confirm that the following workaround works:

/etc/nova/nova.conf
...
[workarounds]
skip_cpu_compare_at_startup = true
skip_cpu_compare_on_dest = true
...