devstack: Failed to create share

[kiran pawar]

Steps to reproduce
===================
1. source openrc admin admin
2. create manila share netwok using private network of "openstack network list"
3. create share using share network from step1

expected
========
Share will be created

actual
======
Share creation fails due to glance policy access error even though its admin user.

[kiran pawar (kpdev)]

Jun 19 13:54:11 kpdev-test-do-not-delete manila-scheduler[524523]: INFO manila.scheduler.weighers.goodness [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] Goodness function result for host kpdev-test-do-not-delete@generic2#GENERIC2: 0.
Jun 19 13:54:11 kpdev-test-do-not-delete manila-scheduler[524523]: DEBUG manila.scheduler.weighers.goodness [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] Goodness: 0 {{(pid=524523) _weigh_object /opt/stack/manila/manila/scheduler/weighers/goodness.py:48}}
Jun 19 13:54:11 kpdev-test-do-not-delete manila-scheduler[524523]: DEBUG manila.scheduler.weighers.goodness [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] Done checking host 'kpdev-test-do-not-delete@generic2#GENERIC2' {{(pid=524523) _weigh_object /opt/stack/manila/manila/scheduler/weighers/goodness.py:49}}
Jun 19 13:54:11 kpdev-test-do-not-delete manila-scheduler[524523]: DEBUG manila.scheduler.drivers.filter [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] Choosing for share: WeighedHost [host: kpdev-test-do-not-delete@generic1#GENERIC1, weight: 0.0] {{(pid=524523) _schedule_share /opt/stack/manila/manila/scheduler/drivers/filter.py:262}}
Jun 19 13:54:11 kpdev-test-do-not-delete manila-share[524472]: DEBUG oslo_concurrency.lockutils [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] Acquiring lock "share_manager_affd74a3-ea59-4360-b659-faccf73c9ebd" by "manila.share.manager.ShareManager._provide_share_server_for_share.<locals>._wrapped_provide_share_server_for_share" {{(pid=524472) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:404}}
Jun 19 13:54:11 kpdev-test-do-not-delete manila-share[524472]: DEBUG oslo_concurrency.lockutils [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] Lock "share_manager_affd74a3-ea59-4360-b659-faccf73c9ebd" acquired by "manila.share.manager.ShareManager._provide_share_server_for_share.<locals>._wrapped_provide_share_server_for_share" :: waited 0.001s {{(pid=524472) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:409}}
Jun 19 13:54:11 kpdev-test-do-not-delete manila-share[524472]: DEBUG manila.share.manager [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] Using share_server 7370038f-88bb-46d8-b0da-25b7800c8fe8 for share instance e206480f-7f4e-4406-a5ea-049908771bae {{(pid=524472) _wrapped_provide_share_server_for_share /opt/stack/manila/manila/share/manager.py:755}}
Jun 19 13:54:11 kpdev-test-do-not-delete manila-share[524472]: DEBUG manila.share.drivers.generic [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] Creating share server '7370038f-88bb-46d8-b0da-25b7800c8fe8'. {{(pid=524472) _setup_server /opt/stack/manila/manila/share/drivers/generic.py:902}}
Jun 19 13:54:11 kpdev-test-do-not-delete manila-share[524472]: DEBUG glanceclient.common.http [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] Request returned failure status 403. {{(pid=524472) _handle_response /usr/local/lib/python3.8/dist-packages/glanceclient/common/http.py:119}}
Jun 19 13:54:12 kpdev-test-do-not-delete manila-share[524472]: DEBUG oslo_concurrency.lockutils [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] L...

[Goutham Pacha Ravi (gouthamr)]

Was this failure on a local/devstack environment?

It looks like this piece of code is in the path where the failure occurred:

https://github.com/openstack/manila/blob/c976f041ddb936af113d0e36f9b6bf2e0c496246/manila/share/drivers/service_instance.py#L538-L540

Can you try to use the glance user's credentials directly and perform the list images call that the driver's doing?

[kiran pawar (kpdev)]

yes, its with devstack.
reproduce with master
does not reproduce with stable/2023.1

[Vida Haririan (vhariria)]

Hello, Thanks for reporting this issue.
Please provide the local.conf, the issue is not reproducible in the CI

[Goutham Pacha Ravi (gouthamr)]

And glance.conf; and manila.conf as well please..

[Vida Haririan (vhariria)]

See additional related discussions at https://meetings.opendev.org/meetings/manila/2023/manila.2023-06-29-15.00.log.html

[kiran pawar (kpdev)]

attaching glance.conf and manila.conf

[kiran pawar (kpdev)]

reproducable on devstack master 14.07.2023

ERROR oslo_messaging.rpc.server [None req-63e7216e-139c-49fb-ba4a-0ae04ceca970 admin None] Exception during message handling: glanceclient.exc.HTTPForbidden: HTTP 403 Forbidden: You are not authorized to complete get_images action.