LIve Migration Broken on Icehouse-1 proposed with Trusty

[Ken Schroeder]

Have encountered a live migration bug when testing live-migration on latest COI Icehouse-1 proposed branch on Trusty.
Seems there is a patch committed for Juno and has Icehouse backport potential.

https://bugs.launchpad.net/nova/+bug/1303536

2014-08-08 14:28:19.771 30273 ERROR nova.virt.libvirt.driver [req-19ff82b5-2a13-4967-bf3f-15b68047d272 0a746b53830d4810a5749d2c945e6e5a ce94ec315eb9469ab0e333b0f81fbdac] CPU doesn't have compatibility.

XML error: CPU feature `erms' specified more than once

Refer to http://libvirt.org/html/libvirt-libvirt.html#virCPUCompareResult
2014-08-08 14:28:19.808 30273 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: XML error: CPU feature `erms' specified more than once
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher payload)
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info())
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4438, in check_can_live_migrate_destination
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher block_migration, disk_over_commit)
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4210, in check_can_live_migrate_destination
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher self._compare_cpu(source_cpu_info)
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4352, in _compare_cpu
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher LOG.error(m, {'ret': ret, 'u': u})
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4348, in _compare_cpu
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher ret = self._conn.compareCPU(cpu.to_xml(), 0)
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher rv = execute(f,*args,**kwargs)
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher rv = meth(*args,**kwargs)
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/libvirt.py", line 3191, in compareCPU
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher if ret == -1: raise libvirtError ('virConnectCompareCPU() failed', conn=self)
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher libvirtError: XML error: CPU feature `erms' specified more than once
2014-08-08 14:28:19.808 30273 TRACE oslo.messaging.rpc.dispatcher
2014-08-08 14:28:19.811 30273 ERROR oslo.messaging._drivers.common [-] Returning exception XML error: CPU feature `erms' specified more than once to caller
2014-08-08 14:28:19.811 30273 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped\n payload)\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function\n e, sys.exc_info())\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4438, in check_can_live_migrate_destination\n block_migration, disk_over_commit)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4210, in check_can_live_migrate_destination\n self._compare_cpu(source_cpu_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4352, in _compare_cpu\n LOG.error(m, {\'ret\': ret, \'u\': u})\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4348, in _compare_cpu\n ret = self._conn.compareCPU(cpu.to_xml(), 0)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit\n result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker\n rv = meth(*args,**kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/libvirt.py", line 3191, in compareCPU\n if ret == -1: raise libvirtError (\'virConnectCompareCPU() failed\', conn=self)\n', "libvirtError: XML error: CPU feature `erms' specified more than once\n"]

[Mark T. Voelker (mvoelker)]

Note that there's an icehouse backport in review already here:

https://review.openstack.org/#/c/100632/

[Mark T. Voelker (mvoelker)]

Patch merged upstream this morning, I'll pull it in shortly.

[Ken Schroeder (kschroed)]

This fixed the CPU Feature error and was able to get live-migration with --block-migrate and with cinder bootvol I'm seeing issues with RBD integration now where system does not treat nova created RBD volumes as shared storage.

TRACE oslo.messaging.rpc.dispatcher InvalidSharedStorage: svl4-csm-b-nova1-001 is not on shared storage: Live migration can not be used without shared storage.

2104 TRACE oslo.messaging.rpc.dispatcher InvalidSharedStorage: svl4-csm-b-nova1-001 is not on shared storage: Live migration can not be used without shared storage.
2014-08-25 03:49:00.941 2104 TRACE oslo.messaging.rpc.dispatcher
2014-08-25 03:49:00.943 2104 ERROR oslo.messaging._drivers.common [-] Returning exception svl4-csm-b-nova1-001 is not on shared storage: Live migration can not be used without shared storage. to caller

Current kernel version is at 3.13.0-34

[Mark T. Voelker (mvoelker)]

I'm on the road at the moment, but just off the cuff that sounds like the classic "you need Ceph patches because this doesn't work out of the box until Juno" issue. References:

http://ceph.com/docs/next/rbd/rbd-openstack/#configuring-nova

https://review.openstack.org/#/c/91722/