When I live migrate VMs from one node to another(both are the same species), nova service at the destination node throws exception below, my physical compute nodes are both ubuntu 14.04lts and openstack git stable/icehouse
(I notice there are similar bug reports like https://bugs.launchpad.net/nova/+bug/1303536, but the codes throwing exception seem not the same )
/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py
4229 # Compare CPU
4230 source_cpu_info = src_compute_info['cpu_info']
4231 self._compare_cpu(source_cpu_info)
Refer to http://libvirt.org/html/libvirt-libvirt.html#virCPUCompareResult
2014-08-05 14:20:11.027 914 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: XML error: CPU feature `erms' specified more than once
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher payload)
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info())
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 4440, in check_can_live_migrate_destination
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher block_migration, disk_over_commit)
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4231, in check_can_live_migrate_destination
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher self._compare_cpu(source_cpu_info)
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4373, in _compare_cpu
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher LOG.error(m, {'ret': ret, 'u': u})
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4369, in _compare_cpu
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher ret = self._conn.compareCPU(cpu.to_xml(), 0)
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher rv = execute(f,*args,**kwargs)
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher rv = meth(*args,**kwargs)
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/libvirt.py", line 3191, in compareCPU
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher if ret == -1: raise libvirtError ('virConnectCompareCPU() failed', conn=self)
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher libvirtError: XML error: CPU feature `erms' specified more than once
2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher
2014-08-05 14:20:11.031 914 ERROR oslo.messaging._drivers.common [-] Returning exception XML error: CPU feature `erms' specified more than once to caller
2014-08-05 14:20:11.031 914 ERROR oslo.messaging._drivers.common [-] Returning exception XML error: CPU feature `erms' specified more than once to caller
2014-08-05 14:20:11.031 914 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped\n payload)\n', ' File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function\n e, sys.exc_info())\n', ' File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 4440, in check_can_live_migrate_destination\n block_migration, disk_over_commit)\n', ' File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4231, in check_can_live_migrate_destination\n self._compare_cpu(source_cpu_info)\n', ' File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4373, in _compare_cpu\n LOG.error(m, {\'ret\': ret, \'u\': u})\n', ' File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4369, in _compare_cpu\n ret = self._conn.compareCPU(cpu.to_xml(), 0)\n', ' File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit\n result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker\n rv = meth(*args,**kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/libvirt.py", line 3191, in compareCPU\n if ret == -1: raise libvirtError (\'virConnectCompareCPU() failed\', conn=self)\n', "libvirtError: XML error: CPU feature `erms' specified more than once\n"]
2014-08-05 14:20:57.206 914 DEBUG nova.openstack.common.periodic_task [-] Running periodic task ComputeManager._poll_volume_usage run_periodic_tasks /usr/local/lib/python2.7/dist-packages/nova/openstack/common/periodic_task.py:178
2014-08-05 14:20:57.207 914 DEBUG nova.openstack.common.periodic_task [-] Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /usr/local/lib/python2.7/dist-packages/nova/openstack/common/periodic_task.py:178
2014-08-05 14:20:57.207 914 DEBUG nova.openstack.common.periodic_task [-] Running periodic task ComputeManager.update_available_resource run_periodic_tasks /usr/local/lib/python2.7/dist-packages/nova/openstack/common/periodic_task.py:178
my cpu info is :
.....
processor : 39
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
stepping : 4
microcode : 0x416
cpu MHz : 3000.011
cache size : 25600 KB
physical id : 1
siblings : 20
core id : 12
cpu cores : 10
apicid : 57
initial apicid : 57
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips : 6001.50
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
This has been fixed in trunk , now being backport to stable/icehouse https:/