kvm block_migration fails with ImageNotAuthorized error

Bug #949438 reported by Joe Breu on 2012-03-07
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Vish Ishaya

Bug Description

KVM block migration fails with an error that nova-compute cannot grab the base image from glance because of an authentication issue.

The command that I used was:
nova-manage vm block_migration --ec2_id=i-00000001 --dest=compute1

The python trace reported in nova-compute is:

012-03-07 15:50:13 ERROR nova.rpc.amqp [-] Exception during message handling(nova.rpc.amqp): TRACE: Traceback (most recent call last):(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/rpc/amqp.py", line 252, in _process_data(nova.rpc.amqp): TRACE: rval = node_func(context=ctxt, **node_args)(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/compute/manager.py", line 1930, in pre_live_migration(nova.rpc.amqp): TRACE: disk)(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2016, in pre_block_migration
(nova.rpc.amqp): TRACE: size=info['disk_size'])(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 995, in _cache_image(nova.rpc.amqp): TRACE: call_if_not_exists(base, fn, *args, **kwargs)(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/utils.py", line 858, in inner(nova.rpc.amqp): TRACE: retval = f(*args, **kwargs)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 992, in call_if_not_exists(nova.rpc.amqp): TRACE: fn(target=base, *args, **kwargs)(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/virt/libvirt/utils.py", line 264, in fetch_image(nova.rpc.amqp): TRACE: images.fetch_to_raw(context, image_id, target, user_id, project_id)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/virt/images.py", line 71, in fetch_to_raw(nova.rpc.amqp): TRACE: metadata = fetch(context, image_href, path_tmp, user_id, project_id)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/virt/images.py", line 65, in fetch
(nova.rpc.amqp): TRACE: (path, e.strerror))
(nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
(nova.rpc.amqp): TRACE: self.gen.next()
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/virt/images.py", line 57, in fetch
(nova.rpc.amqp): TRACE: metadata = image_service.get(context, image_id, image_file)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/image/glance.py", line 263, in get
(nova.rpc.amqp): TRACE: _reraise_translated_image_exception(image_id)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/image/glance.py", line 261, in get
(nova.rpc.amqp): TRACE: image_id)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/image/glance.py", line 145, in _call_retry
(nova.rpc.amqp): TRACE: return getattr(client, name)(*args, **kwargs)
(nova.rpc.amqp): TRACE: File "/opt/stack/glance/glance/client.py", line 90, in get_image
(nova.rpc.amqp): TRACE: res = self.do_request("GET", "/images/%s" % image_id)
(nova.rpc.amqp): TRACE: File "/opt/stack/glance/glance/common/client.py", line 58, in wrapped
(nova.rpc.amqp): TRACE: return func(self, *args, **kwargs)
(nova.rpc.amqp): TRACE: File "/opt/stack/glance/glance/common/client.py", line 385, in do_request
(nova.rpc.amqp): TRACE: self._authenticate()
(nova.rpc.amqp): TRACE: File "/opt/stack/glance/glance/common/client.py", line 363, in _authenticate
(nova.rpc.amqp): TRACE: auth_plugin.authenticate()
(nova.rpc.amqp): TRACE: File "/opt/stack/glance/glance/common/auth.py", line 119, in authenticate
(nova.rpc.amqp): TRACE: self.check_auth_params()
(nova.rpc.amqp): TRACE: File "/opt/stack/glance/glance/common/auth.py", line 83, in check_auth_params
(nova.rpc.amqp): TRACE: raise exception.MissingCredentialError(required=required)
(nova.rpc.amqp): TRACE: ImageNotAuthorized: Not authorized for image 9aeafd9b-3f8d-466b-914b-c47a455e01a9.
(nova.rpc.amqp): TRACE:
2012-03-07 15:50:13 ERROR nova.rpc.amqp [-] Returning exception Not authorized for image 9aeafd9b-3f8d-466b-914b-c47a455e01a9. to caller
2012-03-07 15:50:13 ERROR nova.rpc.amqp [-] ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/rpc/amqp.py", line 252, in _process_data\n rval = node_func(context=ctxt, **node_args)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1930, in pre_live_migration\n disk)\n', ' File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2016, in pre_block_migration\n size=info[\'disk_size\'])\n', ' File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 995, in _cache_image\n call_if_not_exists(base, fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 858, in inner\n retval = f(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 992, in call_if_not_exists\n fn(target=base, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/virt/libvirt/utils.py", line 264, in fetch_image\n images.fetch_to_raw(context, image_id, target, user_id, project_id)\n', ' File "/opt/stack/nova/nova/virt/images.py", line 71, in fetch_to_raw\n metadata = fetch(context, image_href, path_tmp, user_id, project_id)\n', ' File "/opt/stack/nova/nova/virt/images.py", line 65, in fetch\n (path, e.strerror))\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/opt/stack/nova/nova/virt/images.py", line 57, in fetch\n metadata = image_service.get(context, image_id, image_file)\n', ' File "/opt/stack/nova/nova/image/glance.py", line 263, in get\n _reraise_translated_image_exception(image_id)\n', ' File "/opt/stack/nova/nova/image/glance.py", line 261, in get\n image_id)\n', ' File "/opt/stack/nova/nova/image/glance.py", line 145, in _call_retry\n return getattr(client, name)(*args, **kwargs)\n', ' File "/opt/stack/glance/glance/client.py", line 90, in get_image\n res = self.do_request("GET", "/images/%s" % image_id)\n', ' File "/opt/stack/glance/glance/common/client.py", line 58, in wrapped\n return func(self, *args, **kwargs)\n', ' File "/opt/stack/glance/glance/common/client.py", line 385, in do_request\n self._authenticate()\n', ' File "/opt/stack/glance/glance/common/client.py", line 363, in _authenticate\n auth_plugin.authenticate()\n', ' File "/opt/stack/glance/glance/common/auth.py", line 119, in authenticate\n self.check_auth_params()\n', ' File "/opt/stack/glance/glance/common/auth.py", line 83, in check_auth_params\n raise exception.MissingCredentialError(required=required)\n', 'ImageNotAuthorized: Not authorized for image 9aeafd9b-3f8d-466b-914b-c47a455e01a9.\n']

The image does exist in glance
breu@controller:~/devstack$ nova image-list
| ID | Name | Status | Server |
| 4611e6f5-f36d-458e-8553-3ec0e86e07de | cirros-0.3.0-x86_64-blank-kernel | ACTIVE | |
| 9aeafd9b-3f8d-466b-914b-c47a455e01a9 | cirros-0.3.0-x86_64-blank | ACTIVE | |
| bfd1f0ac-89a1-4f9f-b060-6265484a04b5 | cirros-0.3.0-x86_64-blank-ramdisk | ACTIVE | |

Vish Ishaya (vishvananda) wrote :

Ok it looks like this is because you're migrating with nova-manage with a glance that is using keystone. Nova-manage knows nothing about credentials so this is bound to fail.

That said, there are a couple issues with block migration relating to context. We can fix most of them with a little patch which I will submit, but there is one remaining limitation.

Currently, block migration will fail if it is based on a private image. A workaround would be for an administrator to make the image public prior to migrating and set it back private afterwards, but that is pretty ugly. I'm discussing with keystone how we can handle it better.

Changed in nova:
milestone: none → essex-rc1
assignee: nobody → Vish Ishaya (vishvananda)
status: New → Triaged
importance: Undecided → Medium

Fix proposed to branch: master
Review: https://review.openstack.org/5053

Changed in nova:
status: Triaged → In Progress

Reviewed: https://review.openstack.org/5053
Committed: http://github.com/openstack/nova/commit/b0f1f1feb59bbca3196d9bba52cd32f46aa84b87
Submitter: Jenkins
Branch: master

commit b0f1f1feb59bbca3196d9bba52cd32f46aa84b87
Author: Vishvananda Ishaya <email address hidden>
Date: Wed Mar 7 15:02:13 2012 -0800

    allow block migration to talk to glance/keystone

     * removes unnecessary get_admin_context()
     * addresses bug 949438

    Change-Id: Iec9352f4c07179a39c796d235f0787ade30d5747

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2012-03-20
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in nova:
milestone: essex-rc1 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers