KVM block migration fails with an error that nova-compute cannot grab the base image from glance because of an authentication issue.
The command that I used was:
nova-manage vm block_migration --ec2_id=i-00000001 --dest=compute1
The python trace reported in nova-compute is:
012-03-07 15:50:13 ERROR nova.rpc.amqp [-] Exception during message handling(nova.rpc.amqp): TRACE: Traceback (most recent call last):(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/rpc/amqp.py", line 252, in _process_data(nova.rpc.amqp): TRACE: rval = node_func(context=ctxt, **node_args)(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/compute/manager.py", line 1930, in pre_live_migration(nova.rpc.amqp): TRACE: disk)(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2016, in pre_block_migration
(nova.rpc.amqp): TRACE: size=info['disk_size'])(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 995, in _cache_image(nova.rpc.amqp): TRACE: call_if_not_exists(base, fn, *args, **kwargs)(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/utils.py", line 858, in inner(nova.rpc.amqp): TRACE: retval = f(*args, **kwargs)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 992, in call_if_not_exists(nova.rpc.amqp): TRACE: fn(target=base, *args, **kwargs)(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/virt/libvirt/utils.py", line 264, in fetch_image(nova.rpc.amqp): TRACE: images.fetch_to_raw(context, image_id, target, user_id, project_id)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/virt/images.py", line 71, in fetch_to_raw(nova.rpc.amqp): TRACE: metadata = fetch(context, image_href, path_tmp, user_id, project_id)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/virt/images.py", line 65, in fetch
(nova.rpc.amqp): TRACE: (path, e.strerror))
(nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
(nova.rpc.amqp): TRACE: self.gen.next()
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/virt/images.py", line 57, in fetch
(nova.rpc.amqp): TRACE: metadata = image_service.get(context, image_id, image_file)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/image/glance.py", line 263, in get
(nova.rpc.amqp): TRACE: _reraise_translated_image_exception(image_id)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/image/glance.py", line 261, in get
(nova.rpc.amqp): TRACE: image_id)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/image/glance.py", line 145, in _call_retry
(nova.rpc.amqp): TRACE: return getattr(client, name)(*args, **kwargs)
(nova.rpc.amqp): TRACE: File "/opt/stack/glance/glance/client.py", line 90, in get_image
(nova.rpc.amqp): TRACE: res = self.do_request("GET", "/images/%s" % image_id)
(nova.rpc.amqp): TRACE: File "/opt/stack/glance/glance/common/client.py", line 58, in wrapped
(nova.rpc.amqp): TRACE: return func(self, *args, **kwargs)
(nova.rpc.amqp): TRACE: File "/opt/stack/glance/glance/common/client.py", line 385, in do_request
(nova.rpc.amqp): TRACE: self._authenticate()
(nova.rpc.amqp): TRACE: File "/opt/stack/glance/glance/common/client.py", line 363, in _authenticate
(nova.rpc.amqp): TRACE: auth_plugin.authenticate()
(nova.rpc.amqp): TRACE: File "/opt/stack/glance/glance/common/auth.py", line 119, in authenticate
(nova.rpc.amqp): TRACE: self.check_auth_params()
(nova.rpc.amqp): TRACE: File "/opt/stack/glance/glance/common/auth.py", line 83, in check_auth_params
(nova.rpc.amqp): TRACE: raise exception.MissingCredentialError(required=required)
(nova.rpc.amqp): TRACE: ImageNotAuthorized: Not authorized for image 9aeafd9b-3f8d-466b-914b-c47a455e01a9.
(nova.rpc.amqp): TRACE:
2012-03-07 15:50:13 ERROR nova.rpc.amqp [-] Returning exception Not authorized for image 9aeafd9b-3f8d-466b-914b-c47a455e01a9. to caller
2012-03-07 15:50:13 ERROR nova.rpc.amqp [-] ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/rpc/amqp.py", line 252, in _process_data\n rval = node_func(context=ctxt, **node_args)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1930, in pre_live_migration\n disk)\n', ' File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2016, in pre_block_migration\n size=info[\'disk_size\'])\n', ' File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 995, in _cache_image\n call_if_not_exists(base, fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 858, in inner\n retval = f(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 992, in call_if_not_exists\n fn(target=base, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/virt/libvirt/utils.py", line 264, in fetch_image\n images.fetch_to_raw(context, image_id, target, user_id, project_id)\n', ' File "/opt/stack/nova/nova/virt/images.py", line 71, in fetch_to_raw\n metadata = fetch(context, image_href, path_tmp, user_id, project_id)\n', ' File "/opt/stack/nova/nova/virt/images.py", line 65, in fetch\n (path, e.strerror))\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/opt/stack/nova/nova/virt/images.py", line 57, in fetch\n metadata = image_service.get(context, image_id, image_file)\n', ' File "/opt/stack/nova/nova/image/glance.py", line 263, in get\n _reraise_translated_image_exception(image_id)\n', ' File "/opt/stack/nova/nova/image/glance.py", line 261, in get\n image_id)\n', ' File "/opt/stack/nova/nova/image/glance.py", line 145, in _call_retry\n return getattr(client, name)(*args, **kwargs)\n', ' File "/opt/stack/glance/glance/client.py", line 90, in get_image\n res = self.do_request("GET", "/images/%s" % image_id)\n', ' File "/opt/stack/glance/glance/common/client.py", line 58, in wrapped\n return func(self, *args, **kwargs)\n', ' File "/opt/stack/glance/glance/common/client.py", line 385, in do_request\n self._authenticate()\n', ' File "/opt/stack/glance/glance/common/client.py", line 363, in _authenticate\n auth_plugin.authenticate()\n', ' File "/opt/stack/glance/glance/common/auth.py", line 119, in authenticate\n self.check_auth_params()\n', ' File "/opt/stack/glance/glance/common/auth.py", line 83, in check_auth_params\n raise exception.MissingCredentialError(required=required)\n', 'ImageNotAuthorized: Not authorized for image 9aeafd9b-3f8d-466b-914b-c47a455e01a9.\n']
The image does exist in glance
breu@controller:~/devstack$ nova image-list
+--------------------------------------+-----------------------------------+--------+--------+
| ID | Name | Status | Server |
+--------------------------------------+-----------------------------------+--------+--------+
| 4611e6f5-f36d-458e-8553-3ec0e86e07de | cirros-0.3.0-x86_64-blank-kernel | ACTIVE | |
| 9aeafd9b-3f8d-466b-914b-c47a455e01a9 | cirros-0.3.0-x86_64-blank | ACTIVE | |
| bfd1f0ac-89a1-4f9f-b060-6265484a04b5 | cirros-0.3.0-x86_64-blank-ramdisk | ACTIVE | |
+--------------------------------------+-----------------------------------+--------+--------+
Ok it looks like this is because you're migrating with nova-manage with a glance that is using keystone. Nova-manage knows nothing about credentials so this is bound to fail.
That said, there are a couple issues with block migration relating to context. We can fix most of them with a little patch which I will submit, but there is one remaining limitation.
Currently, block migration will fail if it is based on a private image. A workaround would be for an administrator to make the image public prior to migrating and set it back private afterwards, but that is pretty ugly. I'm discussing with keystone how we can handle it better.