Cinder volume migration does not work in kilo

[Bjoern]

Cinder volume migration does not work in kilo and it seems the related to https://bugs.launchpad.net/cinder/+bug/1403902 since the cinder scheduler expects the host in special format, including the cinder pool. This information is unfortunately not really exposed unless you query against the cinder api (get_pools). One sample command including a pool:

cinder migrate 8302d939-4ca1-4a83-8ad5-534dd61d4c3b "617569-cinder01_cinder_volumes_container-b9c480ca@lvm#LVM_iSCSI"

Additionally I did see this error inside the nova log which indicates that volume migrations fail while attaching to the nova guest https://bugs.launchpad.net/nova/+bug/1451860

2015-05-24 21:51:17.908 36478 INFO nova.compute.manager [req-50a85c19-a98a-4cea-97f6-6906df1398d1 cd93ded7844d44cc9da5f67104813993 fd2714bd133c4ca7b7d09db35735adb2 - - -] [ins
tance: 646aa03e-561d-4225-81b7-46c9cc5503cd] Attaching volume 8302d939-4ca1-4a83-8ad5-534dd61d4c3b to /dev/vdd
2015-05-24 21:51:32.506 36478 ERROR root [req-1f5b2973-f09c-4a0d-a1b5-1160b6ee1488 cd93ded7844d44cc9da5f67104813993 fd2714bd133c4ca7b7d09db35735adb2 - - -] Original exception
being dropped: ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 340, in decorated_function\n return f
unction(self, context, *args, **kwargs)\n', ' File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 4939, in swap_volume\n context, old_volume_id, in
stance_uuid=instance.uuid)\n', "AttributeError: 'unicode' object has no attribute 'uuid'\n"]
2015-05-24 21:51:32.507 36478 ERROR root [req-1f5b2973-f09c-4a0d-a1b5-1160b6ee1488 cd93ded7844d44cc9da5f67104813993 fd2714bd133c4ca7b7d09db35735adb2 - - -] Original exception
being dropped: ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 295, in decorated_function\n return f
unction(self, context, *args, **kwargs)\n', ' File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 352, in decorated_function\n kwargs[\'instance\']
, e, sys.exc_info())\n', ' File "/usr/local/lib/python2.7/dist-packages/nova/compute/utils.py", line 87, in add_instance_fault_from_exc\n fault_obj.instance_uuid = instanc
e.uuid\n', "AttributeError: 'unicode' object has no attribute 'uuid'\n"]
2015-05-24 21:51:32.508 36478 ERROR oslo_messaging.rpc.dispatcher [req-1f5b2973-f09c-4a0d-a1b5-1160b6ee1488 cd93ded7844d44cc9da5f67104813993 fd2714bd133c4ca7b7d09db35735adb2 -
 - -] Exception during message handling: string indices must be integers
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_
reply
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 6756, in swap_volume
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher new_volume_id)
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher payload)
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 311, in decorated_function
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher instance_uuid = keyed_args['instance']['uuid']
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher TypeError: string indices must be integers
2015-05-24 21:51:32.508 36478 TRACE oslo_messaging.rpc.dispatcher

[Kevin Carter (kevin-carter)]

This looks like an issue with upstream cinder. @Bjoern can you open issue with them and add this issue to the upstream bug?

[Bjoern (bjoern-t)]

I linked our project to both bugs. Since the first one was supposedly committed to 2015.1.0, I'm wondering when we are going to the latest 2015.1.0 tag or beyond once the 2nd bug is fixed. This definitely has documentation impact if we can't include the nova bug

[Darren Birkett (darren-birkett)]

See https://bugs.launchpad.net/openstack-ansible/+bug/1451860 where osad kilo was updated to use the head of all upstream kilo branches in order to consume the bugfixes (we don't need to wait for 2015.1.1).