Live migration between hosts with differents CPU models fails

[Jonathan Raffre]

When trying to migrate an instance from a new compute node to an older compute node (hardware-wise), nova-computes fails saying that the CPU models are incompatible and leaves the instance in Error Status.

I did specify the CPU model in instances with the options :

libvirt_cpu_mode=custom
libvirt_cpu_model=kvm64

While it seems normal to refuse migration when libvirt_cpu_mode is set to host-model or host-passthrough, setting it to custom and using kvm64 in instances should not impact CPU compatibility between hosts.

Migrating the other way around (old to new) works correctly, mainly because of a superseeding set of instructions.

I'm using the Folsom Ubuntu packaged release of nova-compute on Ubuntu 12.10.

Edit : Adding stacktrace

MSG_ID is fe5d9a7c1a634d7f9c0270e35fed083b from (pid=2363) multicall /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:354
[req-0512c53b-1b31-4924-a7d8-1ab57505ca0b ac90b7751f7f4da78f238916f3d01788 0173fc0a9e3d42caa34659f84ee7cb7c] Live migration of instance 97177534-6947-4756-9db5-b92fd4e504df to host os-node1 failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/admin_actions.py", line 282, in _migrate_live
    disk_over_commit, host)
  File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 94, in inner
    return f(self, context, instance, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1960, in live_migrate
    disk_over_commit, instance, host)
  File "/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py", line 96, in live_migration
    dest=dest))
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line 80, in call
    return rpc.call(context, self._get_topic(topic), msg, timeout)
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", line 102, in call
    return _get_impl().call(cfg.CONF, context, topic, msg, timeout)
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 712, in call
    rpc_amqp.get_connection_pool(conf, Connection))
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 368, in call
    rv = list(rv)
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 336, in __iter__
    raise result
RemoteError: Remote error: InvalidCPUInfo_Remote Unacceptable CPU info: CPU doesn't have compatibility.
2012-11-14 22:33:41 TRACE nova.api.openstack.compute.contrib.admin_actions
0
2012-11-14 22:33:41 TRACE nova.api.openstack.compute.contrib.admin_actions
Refer to http://libvirt.org/html/libvirt-libvirt.html#virCPUCompareResult
Traceback (most recent call last):
2012-11-14 22:33:41 TRACE nova.api.openstack.compute.contrib.admin_actions
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 275, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)
2012-11-14 22:33:41 TRACE nova.api.openstack.compute.contrib.admin_actions
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
    return getattr(proxyobj, method)(ctxt, **kwargs)
2012-11-14 22:33:41 TRACE nova.api.openstack.compute.contrib.admin_actions
  File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
    temp_level, payload)
2012-11-14 22:33:41 TRACE nova.api.openstack.compute.contrib.admin_actions
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
2012-11-14 22:33:41 TRACE nova.api.openstack.compute.contrib.admin_actions
  File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
    return f(*args, **kw)
2012-11-14 22:33:41 TRACE nova.api.openstack.compute.contrib.admin_actions
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2076, in check_can_live_migrate_destination
    instance, block_migration, disk_over_commit)
2012-11-14 22:33:41 TRACE nova.api.openstack.compute.contrib.admin_actions
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2238, in check_can_live_migrate_destination
    self._compare_cpu(source_cpu_info)
2012-11-14 22:33:41 TRACE nova.api.openstack.compute.contrib.admin_actions
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2367, in _compare_cpu
    raise exception.InvalidCPUInfo(reason=m % locals())
2012-11-14 22:33:41 TRACE nova.api.openstack.compute.contrib.admin_actions
InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility.
2012-11-14 22:33:41 TRACE nova.api.openstack.compute.contrib.admin_actions
0
2012-11-14 22:33:41 TRACE nova.api.openstack.compute.contrib.admin_actions
Refer to http://libvirt.org/html/libvirt-libvirt.html#virCPUCompareResult
2012-11-14 22:33:41 TRACE nova.api.openstack.compute.contrib.admin_actions

[kdsoo (dongsoo-kim)]

I'm stuck in same situation.
Before start debugging I'm giving a guess on different number of cores between each host.
I'm running 64 core and 48 core hosts and trying to perform live migration.
It fails with exact same reason you are facing.

[Vish Ishaya (vishvananda)]

Daniel: any ideas on this one? Superficially it looks like _compare_cpu might need to be fixed?

[Daniel Berrange (berrange)]

It is worse than you might imagine. The problem here is that there are two CPU models for each host - the model that the host pCPU has, and the model that the guest vCPU uses. The migration checks are comparing the host CPUs from the two machines. So they can compare as incompatible, even if you've explicitly configured the guest CPU to be a model that is a lowest common denominator. The correct approach is to compare the source guest CPU model with the target host CPU model, but I don't think it is easy todo since, since nothing in the Nova compute APIs exposes any data about guest CPU models. Fixing this correctly is quite a significant job, which I was planning on looking at in Havana. Not sure how practical it is to do a fix for Grizzly.

[ChangBo Guo(gcb) (glongwave)]

I met the same issue , and I didn't see other hypervisor drivers check CPU models . What is the problem if we don't check this ?
Is there any way administrator can ignore this check ,such as adding one option to ignore this ?
Here is my stacktrace and CPU models , the CPU models seemed similar.
http://paste.openstack.org/show/37807/

[Stephen (belrik)]

Also experiencing the exact same problem and wishing there was a way to work around it. Set both CPUs to mode=custom and model=kvm64 in nova.conf

Would appreciate any tip on how to disable/skip this check and perform migration as I'd really like to test this feature before Havana comes out. Error is "ERROR nova.openstack.common.rpc.common [req-05352bd7-fc35-4ef3-9d3e-8950c2e2f8fd 52524ad5d7ff489e9032b029dae5a799 535810064d95435fb9b21d98cbe1abf5] Returning exception Unacceptable CPU info: CPU doesn't have compatibility."

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/41386

[Jian Wen (wenjianhn)]

I prefer fixing the incorrect cpu check.
Price of delegating it to libvirt is too heavy.

[Julien (julienitsme)]

I guys,

I have the same error but my two servers have the same CPU mode :

Host 1:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 23
Stepping: 6
CPU MHz: 2992.446
BogoMIPS: 5985.06
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 6144K
NUMA node0 CPU(s): 0-7

Host2 :
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 23
Stepping: 10
CPU MHz: 2992.632
BogoMIPS: 5985.06
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 6144K
NUMA node0 CPU(s): 0-7

Error in nova-api.log :

2013-08-26 23:25:50.012 ERROR nova.api.openstack.compute.contrib.admin_actions [req-5f28e5de-1312-4e07-81e7-893f439d7ccb f235975a22d64ca3a28f4b5dc1cbceb8 00eb7054314244fca82cf79f2caa8ff7] Live migration of instance 32e94646-2a3a-4c53-b169-becd433880eb to host HOST01 failed
2013-08-26 23:25:50.012 9958 TRACE nova.api.openstack.compute.contrib.admin_actions Traceback (most recent call last):
2013-08-26 23:25:50.012 9958 TRACE nova.api.openstack.compute.contrib.admin_actions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/admin_actions.py", line 284, in _migrate_live
2013-08-26 23:25:50.012 9958 TRACE nova.api.openstack.compute.contrib.admin_actions disk_over_commit, host)
2013-08-26 23:25:50.012 9958 TRACE nova.api.openstack.compute.contrib.admin_actions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 134, in inner
2013-08-26 23:25:50.012 9958 TRACE nova.api.openstack.compute.contrib.admin_actions return f(self, context, instance, *args, **kw)
2013-08-26 23:25:50.012 9958 TRACE nova.api.openstack.compute.contrib.admin_actions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2454, in live_migrate
2013-08-26 23:25:50.012 9958 TRACE nova.api.openstack.compute.contrib.admin_actions disk_over_commit, instance, host_name)
2013-08-26 23:25:50.012 9958 TRACE nova.api.openstack.compute.contrib.admin_actions File "/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py", line 110, in live_migration
2013-08-26 23:25:50.012 9958 TRACE nova.api.openstack.compute.contrib.admin_actions dest=dest))
2013-08-26 23:25:50.012 9958 TRACE nova.api.openstack.compute.contrib.admin_actions File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line 80, in call
2013-08-26 23:25:50.012 9958 TRACE nova.api.openstack.compute.contrib.admin_actions return rpc.call(context, self._get_topic(topic), msg, timeout)
2013-08-26 23:25...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/53746

[Antonio Messina (arcimboldo)]

Would it be possible to have a configuration flag to *disable* this check, in case someone actually know what he is doing and still want to migrate?

So far it's a very important feature we are missing, and I don't see any valid reason for it.

I promise I will be very careful and will only disable the check when I am sure that I can actually migrate instances between two machines. and I promise I will not open tickets in case something goes wrong :)

[Yaguang Tang (heut2008)]

@antonio

after applying my patch, it's unnecessary to disable this check, you can config the guest cpu model , if the model is unset, then the guest instance can be migrated to any servers.

[Khanh-Toan TRAN (toan-tran)]

We have the same problem with our Intel Xeon servers. When we try migrating VM from one server to another, the destination compute does not recognize the CPU's compatibility. After inspecting for a while, we believe that the problem is due to some of the VM's CPU features that do not appear in destination host' CPU.

Nova convert CPU_info into XML records and compare the two. If we look at http://libvirt.org/html/libvirt-libvirt.html#virCPUCompareResult as suggested :

enum virCPUCompareResult {
    VIR_CPU_COMPARE_ERROR = -1
    VIR_CPU_COMPARE_INCOMPATIBLE = 0
    VIR_CPU_COMPARE_IDENTICAL = 1
    VIR_CPU_COMPARE_SUPERSET = 2
    VIR_CPU_COMPARE_LAST = 3
}

then if one of the VM'sCPU_info feature is not in host's CPU_info, then the CPU_info's XML is not identical or superset, which leads to INCOMPATIBLE. In our case it's the "ds" feature (see log below).

We failed to understand why the source compute would put such a feature in the VM's CPU_info. By default qemu uses host's CPU_info thus they shoud be identical.

If we omit the check (by forcing returning 2), the migration processes smoothly. So we think that the _compare_cpu function in /nova/nova/virt/libvirt/driver.py should consider essential features and optional features.

-----------------------
VM's CPU (given by source compute):

2013-11-27 13:40:04.942 18644 INFO nova.virt.libvirt.driver [req-e2ee030b-d097-434e-94e0-316c3f9f835d 6ac8163893924c058c98e8595de6bec4 15f206d66a044758af148796928d5efb] Instance launched has CPU info:
{"vendor": "Intel", "model": "SandyBridge", "arch": "x86_64", "features": ["pdpe1gb", "osxsave", "dca", "pcid", "pdcm", "xtpr", "tm2", "est", "smx", "vmx", "ds_cpl", "monitor", "dtes64", "pbe", "tm", "ht", "ss", "acpi", "ds", "vme"], "topology": {"cores": 6, "threads": 2, "sockets": 1}}
2013-11-27 13:40:04.944 18644 DEBUG nova.virt.libvirt.config [req-e2ee030b-d097-434e-94e0-316c3f9f835d 6ac8163893924c058c98e8595de6bec4 15f206d66a044758af148796928d5efb] Generated XML <cpu>
  <arch>x86_64</arch>
  <model>SandyBridge</model>
  <vendor>Intel</vendor>
  <topology sockets="1" cores="6" threads="2"/>
  <feature name="pdpe1gb"/>
  <feature name="osxsave"/>
  <feature name="dca"/>
  <feature name="pcid"/>
  <feature name="pdcm"/>
  <feature name="xtpr"/>
  <feature name="tm2"/>
  <feature name="est"/>
  <feature name="smx"/>
  <feature name="vmx"/>
  <feature name="ds_cpl"/>
  <feature name="monitor"/>
  <feature name="dtes64"/>
  <feature name="pbe"/>
  <feature name="tm"/>
  <feature name="ht"/>
  <feature name="ss"/>
  <feature name="acpi"/>
  <feature name="ds"/>
  <feature name="vme"/>
</cpu>
  from (pid=18644) to_xml /mnt/VMS/devstack/nova/nova/virt/libvirt/config.py:70

------------------------
Destination and Source compute CPU: cat /proc/cpuinfo

processor : 23
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
stepping : 7
microcode : 0x70d
cpu MHz : 2000.081
cache size : 15360 KB
physical id : 1
siblings : 12
core id : 5
cpu cores : 6
apicid : 43
initial apicid : 4...

[Yaguang Tang (heut2008)]

the patch is blocked by a blueprint which has been delayed to June, I am trying to push again.

[lizheming (lizheming-li)]

hi Yaguang:
I have the same problem and have do some reserch.
I use inter cpu as example:
this is inter cpu model:
Conroe<Penryn<Nehalem<Westmere<SandyBridge<Haswell

in openstack, if you migrating vm from Haswell to SandyBridge, it is OK. but if you migrating vm from SandyBridge
to Haswell, it failed. because different cpu have different core cmd. Haswell core cmd include SandyBridge core cmd.
so it can migrate.

if we want to migrate between different cpu mode, we can use the same cpu mode. libvirt support the cpu model lable.
like this:
<cpu>
         <model>SandyBridge</model>
<cpu>

if you want migrate from SandyBridge to Haswell, you can set the lable, model is SandyBridge. because Haswell is also compatible with SandyBridge.

but openstack don't support set the mode cpu, so we can modify the code.

if we disable the check, it will get some unknown error. because the vm run the different core cmd.

thanks

[Yaguang Tang (heut2008)]

lizheming,

OpenStack do support set CPU mode, and kvm will handle the compatibility if you set cpu_mode=none even host have different CPU model. and my patch for this is to let libvirt do the check for CPU compatibility.

 222 cfg.StrOpt('cpu_mode',
 223 help='Set to "host-model" to clone the host CPU feature flags; '
 224 'to "host-passthrough" to use the host CPU model exactly; '
 225 'to "custom" to use a named CPU model; '
 226 'to "none" to not set any CPU model. '
 227 'If virt_type="kvm|qemu", it will default to '
 228 '"host-model", otherwise it will default to "none"',
 229 deprecated_name='libvirt_cpu_mode',
 230 deprecated_group='DEFAULT'),
 231 cfg.StrOpt('cpu_model',
 232 help='Set to a named libvirt CPU model (see names listed '
 233 'in /usr/share/libvirt/cpu_map.xml). Only has effect if '
 234 'cpu_mode="custom" and virt_type="kvm|qemu"',
 235 deprecated_name='libvirt_cpu_model',
 236 deprecated_group='DEFAULT'),

[Facundo Maldonado (facundo-n-maldonado)]

Hi Yaguang, can you tell me which is the blueprint that is blocking the patch?
Thanks.

[Chris Friesen (cbf123)]

Just wondering where we're at with this now...

[Sean Dague (sdague)]

Clearly not in progress any more

[SlOPS (vtabolin)]

I have the same bug in Icehouce. The problem is that the check in method check_can_live_migrate_destination compares physical source CPU from database unlike to real parameters of VM.

The one siple hack is to comment
self._compare_cpu(source_cpu_info)
string from nova/virt/libvirt/driver.py

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/53746
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Davanum Srinivas (DIMS) (dims-v)]

Fix proposed again

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/53746
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=79a0755597f4983367eb0caf4669ffb881b4f720
Submitter: Jenkins
Branch: master

commit 79a0755597f4983367eb0caf4669ffb881b4f720
Author: yunhong jiang <email address hidden>
Date: Fri Feb 6 07:03:45 2015 -0800

    libvirt: fix overly strict CPU model comparison in live migration

    The current libvirt driver migration code checks whether the
    dest host can support the guest, by comparing the source host
    CPU model. This is overly strict, as the guest may well be
    running with a more restrictive CPU model that hides many
    host features. The correct approach is to compare against
    the guest's configured CPU model.

    To achieve the compatibility when migrate instance from old compute
    node which does not provide vcpu_model in instance object, the source
    host's compute info is still provided.

    Co-authored: Daniel P. Berrange <email address hidden>
    Co-authored: Yunhong Jiang <email address hidden>

    Closes-Bug: 1082414

    Change-Id: I65f505fec64c65d2641d2bfd940cde44bcd83a78

[Daniel Schneller (daniel-schneller)]

Any chance of this getting backported to Icehouse?

[jiang, yunhong (yunhong-jiang)]

To backport is a bit tricky because we are using the instance object to achieve it.

[Daniel Schneller (daniel-schneller)]

Ok. So if I wanted to have reliable live migration on my Icehouse cluster, would this be a valid approach?

1) Find lowest common denominator CPU model across all hypervisors using virsh capabilities / virsh cpu-baseline
2) Configure nova-compute.conf to include cpu_mode=custom, cpu_model=Westmere (in my case)
3) Disable the CPU check in nova/virt/libvirt/driver.py (comment out line 4248)
    4246 # Compare CPU
    4247 source_cpu_info = src_compute_info['cpu_info']
    4248 self._compare_cpu(source_cpu_info)

Assuming this cluster is not going to get any older CPU or non-Intel CPU models in the future, this should be safe, right?

[Daniel Schneller (daniel-schneller)]

...and assuming, of course, that all VMs get rebooted once, so they "forget" about the CPU capabilities they might have received from a more modern hypervisor.

[jiang, yunhong (yunhong-jiang)]

Yes, I think it should work.

[Tony Walker (tony-walker-h)]

I'm seeing a regression when cpu_mode is set to 'none'. In my case, check_can_live_migrate_destination() is passed an instance with a vcpu_model attribute. This is then passed onto the call to _compare_cpu(). The problem however is that instance.vcpu_model.model is None. The new logic in _compare_cpu() ultimately causes cpu.to_xml() to result in a cpu model of None which causes the following exception:

TRACE oslo_messaging.rpc.dispatcher libvirtError: invalid argument: no guest CPU model specified

Since there is a specific check for CONF.libvirt.cpu_mode being set to 'none' in _get_guest_cpu_model_config(), can we fallback to the old logic? I've currently got this implemented and working:

- if not instance.vcpu_model:
+ if not instance.vcpu_model or not instance.vcpu_model.model:

Thanks

[Tony Walker (tony-walker-h)]

In fact, unless cpu_mode is set to 'custom' and cpu_model has been explicitly set, I end up with a vcpu_model object that looks like the below which will always raise the exception since model==None:

VirtCPUModel(arch=None,features=[],match='exact',mode='host-model',model=None,topology=VirtCPUTopology,vendor=None)

[jiang, yunhong (yunhong-jiang)]

Hi, Tony, thanks for reporting the regression. I didn't test this scenario. I tested the one reported on the bug report.

I will try to reproduce this tomorrow. BTW, can you please tell me , before this patch, will it work if the target host or source host's cpu_mode is set as None?

Thanks
--jyh

[Tony Walker (tony-walker-h)]

Hi,

Yes, this worked previously when cpu_mode is set to None. I currently have the config set to cpu_mode=None which causes 'mode' to be set to 'host-model' in _get_guest_cpu_model_config(). Regardless, I'm left with a VirtCPUModel instance with a model of 'None' when the check is made.

Thanks,

Tony

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/161944

[Markus Zoeller (markus_z) (mzoeller)]

The issue found by Tony Walker is also discussed in bug 1433933

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/161944
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5653bd291665bcdecad46ed6654a04c49e4b1dda
Submitter: Jenkins
Branch: master

commit 5653bd291665bcdecad46ed6654a04c49e4b1dda
Author: yunhong jiang <email address hidden>
Date: Thu Mar 5 15:09:49 2015 -0800

    Check host cpu_info if no cpu_model for guest

    When no cpu_model specified to guest, we should use host cpu_info to
    check for live migration.

    Change-Id: I410f53f6e54a698a48d8d5566ce192cda0365c98
    Closes-Bug: #1082414