Cinder volume creation fails if image is used as source: 'ImageNotAuthorized: Not authorized for image <id>'

[Artem Panchenko]

Fuel version info (7.0 build #140): http://paste.openstack.org/show/411183/

OSTF test "Create volume and boot instance from it (failure)" fails, here is a part of 'cinder-volume' logs on cinder node:

<155>Aug 6 12:44:51 node-3 cinder-volume 2015-08-06 12:44:51.709 13388 ERROR cinder.volume.flows.manager.create_volume [req-50cae2ae-a43d-4598-8f95-d7f0085f4ec4 c16a7a58728c44d7a2ff0cf681050bc8 fca7f0b9d93b40
2e8d5d12db7ec44ad6 - - -] Volume 6391a99e-73e7-45e4-8246-43b2447935f2: create failed
<159>Aug 6 12:44:51 node-3 cinder-volume 2015-08-06 12:44:51.710 13388 DEBUG taskflow.engines.action_engine.runner [req-50cae2ae-a43d-4598-8f95-d7f0085f4ec4 c16a7a58728c44d7a2ff0cf681050bc8 fca7f0b9d93b402e8d
5d12db7ec44ad6 - - -] Exiting old state 'SCHEDULING' in response to event 'wait_finished' on_exit /usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/runner.py:201
<159>Aug 6 12:44:51 node-3 cinder-volume 2015-08-06 12:44:51.711 13388 DEBUG taskflow.engines.action_engine.runner [req-50cae2ae-a43d-4598-8f95-d7f0085f4ec4 c16a7a58728c44d7a2ff0cf681050bc8 fca7f0b9d93b402e8d
5d12db7ec44ad6 - - -] Entering new state 'WAITING' in response to event 'wait_finished' on_enter /usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/runner.py:205
<159>Aug 6 12:44:51 node-3 cinder-volume 2015-08-06 12:44:51.711 13388 DEBUG taskflow.engines.action_engine.runner [req-50cae2ae-a43d-4598-8f95-d7f0085f4ec4 c16a7a58728c44d7a2ff0cf681050bc8 fca7f0b9d93b402e8d
5d12db7ec44ad6 - - -] Exiting old state 'WAITING' in response to event 'examine_finished' on_exit /usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/runner.py:201
<159>Aug 6 12:44:51 node-3 cinder-volume 2015-08-06 12:44:51.712 13388 DEBUG taskflow.engines.action_engine.runner [req-50cae2ae-a43d-4598-8f95-d7f0085f4ec4 c16a7a58728c44d7a2ff0cf681050bc8 fca7f0b9d93b402e8d
5d12db7ec44ad6 - - -] Entering new state 'ANALYZING' in response to event 'examine_finished' on_enter /usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/runner.py:205
<156>Aug 6 12:44:51 node-3 cinder-volume 2015-08-06 12:44:51.713 13388 WARNING cinder.volume.manager [req-50cae2ae-a43d-4598-8f95-d7f0085f4ec4 c16a7a58728c44d7a2ff0cf681050bc8 fca7f0b9d93b402e8d5d12db7ec44ad6
 - - -] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (bdeb8bc8-7fa3-4987-913e-f82f91133ea6) transitioned into state 'REVERTED' from state 'REVERTING'
<159>Aug 6 12:44:51 node-3 cinder-volume 2015-08-06 12:44:51.714 13388 DEBUG taskflow.engines.action_engine.runner [req-50cae2ae-a43d-4598-8f95-d7f0085f4ec4 c16a7a58728c44d7a2ff0cf681050bc8 fca7f0b9d93b402e8d
5d12db7ec44ad6 - - -] Exiting old state 'ANALYZING' in response to event 'completed' on_exit /usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/runner.py:201
<159>Aug 6 12:44:51 node-3 cinder-volume 2015-08-06 12:44:51.714 13388 DEBUG taskflow.engines.action_engine.runner [req-50cae2ae-a43d-4598-8f95-d7f0085f4ec4 c16a7a58728c44d7a2ff0cf681050bc8 fca7f0b9d93b402e8d
5d12db7ec44ad6 - - -] Entering new state 'GAME_OVER' in response to event 'completed' on_enter /usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/runner.py:205
<159>Aug 6 12:44:51 node-3 cinder-volume 2015-08-06 12:44:51.715 13388 DEBUG taskflow.engines.action_engine.runner [req-50cae2ae-a43d-4598-8f95-d7f0085f4ec4 c16a7a58728c44d7a2ff0cf681050bc8 fca7f0b9d93b402e8d
5d12db7ec44ad6 - - -] Exiting old state 'GAME_OVER' in response to event 'reverted' on_exit /usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/runner.py:201
<159>Aug 6 12:44:51 node-3 cinder-volume 2015-08-06 12:44:51.716 13388 DEBUG taskflow.engines.action_engine.runner [req-50cae2ae-a43d-4598-8f95-d7f0085f4ec4 c16a7a58728c44d7a2ff0cf681050bc8 fca7f0b9d93b402e8d
5d12db7ec44ad6 - - -] Entering new state 'REVERTED' in response to event 'reverted' on_enter /usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/runner.py:205
<156>Aug 6 12:44:51 node-3 cinder-volume 2015-08-06 12:44:51.717 13388 WARNING cinder.volume.manager [req-50cae2ae-a43d-4598-8f95-d7f0085f4ec4 c16a7a58728c44d7a2ff0cf681050bc8 fca7f0b9d93b402e8d5d12db7ec44ad6
 - - -] Flow 'volume_create_manager' (a997d43b-5510-4f31-a86b-b0afe0b7412c) transitioned into state 'REVERTED' from state 'RUNNING'
<155>Aug 6 12:44:51 node-3 cinder-volume 2015-08-06 12:44:51.746 13388 ERROR oslo_messaging.rpc.dispatcher [req-50cae2ae-a43d-4598-8f95-d7f0085f4ec4 c16a7a58728c44d7a2ff0cf681050bc8 fca7f0b9d93b402e8d5d12db7e
c44ad6 - - -] Exception during message handling: Not authorized for image 7a23fcfa-4202-42d1-bd5b-cf9e366f51de.
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 468, in create_volume
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher _run_flow()
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 456, in _run_flow
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher flow_engine.run()
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 96, in run
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher for _state in self.run_iter():
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 153, in run_iter
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher failure.Failure.reraise_if_any(failures.values())
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/types/failure.py", line 244, in reraise_if_any
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher failures[0].reraise()
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/types/failure.py", line 251, in reraise
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher six.reraise(*self._exc_info)
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 67, in _execute_task
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher result = task.execute(**arguments)
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 285, in execute
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher image_id),
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 257, in get_location
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher _reraise_translated_image_exception(image_id)
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 255, in get_location
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher image_meta = client.call(context, 'get', image_id)
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 176, in call
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher return getattr(client.images, method)(*args, **kwargs)
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/glanceclient/v2/images.py", line 173, in get
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher resp, body = self.http_client.get(url)
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 262, in get
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher return self._request('GET', url, **kwargs)
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 230, in _request
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher raise exc.from_response(resp, resp.text)
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher ImageNotAuthorized: Not authorized for image 7a23fcfa-4202-42d1-bd5b-cf9e366f51de.
2015-08-06 12:44:51.746 13388 TRACE oslo_messaging.rpc.dispatcher

Steps to reproduce:

1. Create environment with NeutronVXLAN
2. Add 1 controller and 1 compute+cinder nodes
3. Deploy environment
4. Run OSTF

Expected result:

- all health checks passed

Actual result:

- test "Create volume and boot instance from it (failure)" failed

Diagnostic snapshot is attached.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/210063

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vasyl Saienko (<email address hidden>) on branch: master
Review: https://review.openstack.org/210063

[Vasyl Saienko (vsaienko)]

This bug is duplicate of https://bugs.launchpad.net/fuel/+bug/1482192

But also it contain comment related to separate bug. In order to avoid confusion, I marked this bug as duplicate and opened new bug:

https://bugs.launchpad.net/fuel/+bug/1482361

[Artem Panchenko (apanchenko-8)]

@Vasyl,

thanks for creating separate bug about the issue described in comments https://bugs.launchpad.net/fuel/+bug/1482239/comments/2

Bug the original description of this bug is NOT a duplicate of bug 1482192.

[Artem Panchenko (apanchenko-8)]

It is configuration issue. After I modified '/etc/cinder/cinder.conf' on compute+cinder nodes cinder-volume service became to work fine:

http://paste.openstack.org/show/412141/

So, looks like puppet should configure correct values for the following parameters:

auth_strategy
auth_uri
identity_uri

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/210184

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/210217

[OpenStack Infra (hudson-openstack)]

Change abandoned by Artem Panchenko (<email address hidden>) on branch: master
Review: https://review.openstack.org/210184
Reason: https://review.openstack.org/#/c/210217/

[Vasyl Saienko (vsaienko)]

auth_uri, identity_uri and auth_strategy were lost for cinder-volume node after merge https://review.openstack.org/#/c/206351/7/deployment/puppet/openstack/manifests/cinder.pp

There is CR that fix them: https://review.openstack.org/#/c/210217/
At the moment it is blocker for Master CI, so I'm rising this bug to Critical.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/210217
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=154bb6a82296a564679d3294f80bac6c1d714b41
Submitter: Jenkins
Branch: master

commit 154bb6a82296a564679d3294f80bac6c1d714b41
Author: vsaienko <email address hidden>
Date: Fri Aug 7 09:21:08 2015 +0300

    update cinder.conf on cinder node

     Add auth_uri, indentity_uri and auth_strategy
     they needed for cinder-volume service

    Closes-Bug: #1482239

    Change-Id: I874bb83e2b42329f1ccc714af6565f963ff7aa68

[oleksii shyman (oshyman)]

Verified on ISO #246