OpenStack Compute (nova)

Live migration CPU pre check failed

Bug #1910331 reported by puthi on 2021-01-06

This bug report is a duplicate of: Bug #1898715: Live migration fails despite matching CPUs. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	New	Undecided	Unassigned

Bug Description

Description
===========
During live-migration with --block-migrate enable, CPU check always fail even both source host and destination host are identical in term of hardware and software version.

Steps to reproduce
==================
# nova live-migration --block-migrate instance computenode2

After a bit, check on computenode2, I can see the error in the log below

Expected result
===============
instance got moved to computenode2

Actual result
=============
instance is still on its old compute node

Environment
===========
1. Openstack version, OS version and libvirt version
# rpm -qa | egrep "nova|kvm|libvirt" | sort
libvirt-bash-completion-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-client-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-daemon-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-daemon-driver-interface-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-daemon-driver-network-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-daemon-driver-nodedev-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-daemon-driver-nwfilter-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-daemon-driver-qemu-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-daemon-driver-secret-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-daemon-driver-storage-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-daemon-driver-storage-core-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-daemon-driver-storage-disk-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-daemon-driver-storage-gluster-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-daemon-driver-storage-iscsi-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-daemon-driver-storage-iscsi-direct-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-daemon-driver-storage-logical-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-daemon-driver-storage-mpath-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-daemon-driver-storage-rbd-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-daemon-driver-storage-scsi-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-daemon-kvm-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
libvirt-libs-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
openstack-nova-common-20.4.1-1.el8.noarch
openstack-nova-compute-20.4.1-1.el8.noarch
python3-libvirt-6.0.0-1.module_el8.3.0+555+a55c8938.x86_64
python3-nova-20.4.1-1.el8.noarch
python3-novaclient-15.1.1-1.el8.noarch
qemu-kvm-4.2.0-34.module_el8.3.0+555+a55c8938.x86_64
qemu-kvm-block-curl-4.2.0-34.module_el8.3.0+555+a55c8938.x86_64
qemu-kvm-block-gluster-4.2.0-34.module_el8.3.0+555+a55c8938.x86_64
qemu-kvm-block-iscsi-4.2.0-34.module_el8.3.0+555+a55c8938.x86_64
qemu-kvm-block-rbd-4.2.0-34.module_el8.3.0+555+a55c8938.x86_64
qemu-kvm-block-ssh-4.2.0-34.module_el8.3.0+555+a55c8938.x86_64
qemu-kvm-common-4.2.0-34.module_el8.3.0+555+a55c8938.x86_64
qemu-kvm-core-4.2.0-34.module_el8.3.0+555+a55c8938.x86_64

# cat /etc/redhat-release
CentOS Linux release 8.3.2011

# uname -a
Linux computenode2 4.18.0-240.1.1.el8_3.x86_64 #1 SMP Thu Nov 19 17:20:08 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

2. Which hypervisor did you use?
KVM

3. Which networking type did you use?
Neutron with OpenVSwitch

Logs & Configs
==============

- On computende2 I can see this error:

2021-01-06 10:57:45.795 6992 INFO nova.virt.libvirt.driver [req-34f4dc34-dcb1-4632-813b-caf9f1a47439 7584ec4fa0fc45ddb25c2603357a7912 b5608fc845894e68abe34703c801b6e6 - default default] Instance launched has CPU info
: {"arch": "x86_64", "model": "Cascadelake-Server-noTSX", "vendor": "Intel", "topology": {"cells": 2, "sockets": 1, "cores": 22, "threads": 2}, "features": ["lm", "pdpe1gb", "arch-capabilities", "smx", "lahf_lm", "rt
m", "avx", "fma", "est", "ss", "vmx", "avx512dq", "fsgsbase", "ibrs-all", "rdtscp", "taa-no", "mmx", "pclmuldq", "xtpr", "clflushopt", "spec-ctrl", "mca", "dtes64", "mds-no", "apic", "pcid", "fpu", "monitor", "cx16",
"stibp", "avx512f", "adx", "md-clear", "tm2", "pse", "hle", "invpcid", "sse2", "xsaveopt", "xsavec", "cx8", "popcnt", "pse36", "tsc-deadline", "avx512-bf16", "ht", "tm", "arat", "avx512vnni", "de", "pni", "mpx", "ts
x-ctrl", "3dnowprefetch", "f16c", "smap", "sse4.2", "avx512bw", "xsave", "acpi", "pge", "ds_cpl", "bmi1", "ssbd", "pku", "avx512cd", "smep", "tsc_adjust", "aes", "clflush", "pdcm", "cmov", "fxsr", "rdseed", "avx2", "
ds", "pbe", "rdctl-no", "bmi2", "sse", "sse4.1", "abm", "pat", "mce", "pae", "invtsc", "tsc", "x2apic", "xgetbv1", "msr", "clwb", "sep", "skip-l1dfl-vmentry", "vme", "pschange-mc-no", "movbe", "syscall", "ssse3", "nx
", "xsaves", "dca", "mtrr", "avx512vl", "rdrand", "erms", "intel-pt"]}
2021-01-06 10:57:45.797 6992 ERROR nova.virt.libvirt.driver [req-34f4dc34-dcb1-4632-813b-caf9f1a47439 7584ec4fa0fc45ddb25c2603357a7912 b5608fc845894e68abe34703c801b6e6 - default default] CPU doesn't have compatibility.

Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server [req-34f4dc34-dcb1-4632-813b-caf9f1a47439 7584ec4fa0fc45ddb25c2603357a7912 b5608fc845894e68abe34703c801b6e6 - default default] Exception during message handling: nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility.

Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server function_name, call_dict, binary, tb)
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server raise value
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1372, in decorated_function
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 219, in decorated_function
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server raise value
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 207, in decorated_function
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 6789, in check_can_live_migrate_destination
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server block_migration, disk_over_commit)
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8114, in check_can_live_migrate_destination
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server self._compare_cpu(None, source_cpu_info, instance)
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8439, in _compare_cpu
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server raise exception.InvalidCPUInfo(reason=m % {'ret': ret, 'u': u})
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility.
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server 0
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server

- nova config on each compute node:

# egrep -v "^#|^$" /etc/nova/nova.conf
[libvirt]
virt_type = kvm

Note
======
The live-migration will work as expected if I change the nova.conf config to any one of the config below:
1- Config1:
[libvirt]
virt_type = qemu

2- Config2:
[libvirt]
virt_type = kvm
cpu_mode=custom
cpu_models = Cascadelake-Server-noTSX,Skylake-Server-noTSX-IBRS,Broadwell-noTSX-IBRS

Revision history for this message

Balazs Gibizer (balazs-gibizer) wrote on 2021-01-07:

It seems like a duplicate of https://bugs.launchpad.net/nova/+bug/1898715 for which the patch merged about a months ago https://review.opendev.org/c/openstack/nova/+/758761

I'm marking this as duplicate. Please remove the duplication tag if you think this is a separate issue.

Report a bug

This report contains Public information

Everyone can see this information.

Duplicate of bug #1898715 Remove

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.