devstack: Failed to create share

[kiran pawar]

Steps to reproduce
===================
1. source openrc admin admin
2. create manila share netwok using private network of "openstack network list"
3. create share using share network from step1

expected
========
Share will be created

actual
======
Share creation fails due to glance policy access error even though its admin user.

[kiran pawar (kpdev)]

Jun 19 13:54:11 kpdev-test-do-not-delete manila-scheduler[524523]: INFO manila.scheduler.weighers.goodness [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] Goodness function result for host kpdev-test-do-not-delete@generic2#GENERIC2: 0.
Jun 19 13:54:11 kpdev-test-do-not-delete manila-scheduler[524523]: DEBUG manila.scheduler.weighers.goodness [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] Goodness: 0 {{(pid=524523) _weigh_object /opt/stack/manila/manila/scheduler/weighers/goodness.py:48}}
Jun 19 13:54:11 kpdev-test-do-not-delete manila-scheduler[524523]: DEBUG manila.scheduler.weighers.goodness [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] Done checking host 'kpdev-test-do-not-delete@generic2#GENERIC2' {{(pid=524523) _weigh_object /opt/stack/manila/manila/scheduler/weighers/goodness.py:49}}
Jun 19 13:54:11 kpdev-test-do-not-delete manila-scheduler[524523]: DEBUG manila.scheduler.drivers.filter [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] Choosing for share: WeighedHost [host: kpdev-test-do-not-delete@generic1#GENERIC1, weight: 0.0] {{(pid=524523) _schedule_share /opt/stack/manila/manila/scheduler/drivers/filter.py:262}}
Jun 19 13:54:11 kpdev-test-do-not-delete manila-share[524472]: DEBUG oslo_concurrency.lockutils [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] Acquiring lock "share_manager_affd74a3-ea59-4360-b659-faccf73c9ebd" by "manila.share.manager.ShareManager._provide_share_server_for_share.<locals>._wrapped_provide_share_server_for_share" {{(pid=524472) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:404}}
Jun 19 13:54:11 kpdev-test-do-not-delete manila-share[524472]: DEBUG oslo_concurrency.lockutils [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] Lock "share_manager_affd74a3-ea59-4360-b659-faccf73c9ebd" acquired by "manila.share.manager.ShareManager._provide_share_server_for_share.<locals>._wrapped_provide_share_server_for_share" :: waited 0.001s {{(pid=524472) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:409}}
Jun 19 13:54:11 kpdev-test-do-not-delete manila-share[524472]: DEBUG manila.share.manager [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] Using share_server 7370038f-88bb-46d8-b0da-25b7800c8fe8 for share instance e206480f-7f4e-4406-a5ea-049908771bae {{(pid=524472) _wrapped_provide_share_server_for_share /opt/stack/manila/manila/share/manager.py:755}}
Jun 19 13:54:11 kpdev-test-do-not-delete manila-share[524472]: DEBUG manila.share.drivers.generic [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] Creating share server '7370038f-88bb-46d8-b0da-25b7800c8fe8'. {{(pid=524472) _setup_server /opt/stack/manila/manila/share/drivers/generic.py:902}}
Jun 19 13:54:11 kpdev-test-do-not-delete manila-share[524472]: DEBUG glanceclient.common.http [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] Request returned failure status 403. {{(pid=524472) _handle_response /usr/local/lib/python3.8/dist-packages/glanceclient/common/http.py:119}}
Jun 19 13:54:12 kpdev-test-do-not-delete manila-share[524472]: DEBUG oslo_concurrency.lockutils [None req-73db15e2-ee71-462b-a02c-0a0b6e103ee7 admin None] L...

[Goutham Pacha Ravi (gouthamr)]

Was this failure on a local/devstack environment?

It looks like this piece of code is in the path where the failure occurred:

https://github.com/openstack/manila/blob/c976f041ddb936af113d0e36f9b6bf2e0c496246/manila/share/drivers/service_instance.py#L538-L540

Can you try to use the glance user's credentials directly and perform the list images call that the driver's doing?

[kiran pawar (kpdev)]

yes, its with devstack.
reproduce with master
does not reproduce with stable/2023.1

[Vida Haririan (vhariria)]

Hello, Thanks for reporting this issue.
Please provide the local.conf, the issue is not reproducible in the CI

[Goutham Pacha Ravi (gouthamr)]

And glance.conf; and manila.conf as well please..

[Vida Haririan (vhariria)]

See additional related discussions at https://meetings.opendev.org/meetings/manila/2023/manila.2023-06-29-15.00.log.html

[kiran pawar (kpdev)]

attaching glance.conf and manila.conf

[kiran pawar (kpdev)]

reproducable on devstack master 14.07.2023

ERROR oslo_messaging.rpc.server [None req-63e7216e-139c-49fb-ba4a-0ae04ceca970 admin None] Exception during message handling: glanceclient.exc.HTTPForbidden: HTTP 403 Forbidden: You are not authorized to complete get_images action.

[kiran pawar (kpdev)]

this issue still persist. Tried on centos9

================= glance-api.conf ================
[DEFAULT]
use_keystone_limits = True
enforce_secure_rbac = true
worker_self_reference_url = http://127.0.0.1:60999
logging_exception_prefix = ERROR %(name)s %(instance)s
logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s
logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s
logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}}
public_endpoint = http://10.180.1.148/image
show_multiple_locations = False
show_image_direct_url = False
transport_url = rabbit://stackrabbit:secret@10.180.1.148:5672/
image_cache_driver = centralized_db
image_cache_dir = /opt/stack/data/glance/cache/
use_syslog = False
debug = True

[database]
connection = mysql+pymysql://root:secret@127.0.0.1/glance?charset=utf8&plugin=dbcounter

[oslo_concurrency]
lock_path = /opt/stack/data/glance/locks

[paste_deploy]
flavor = keystone+cachemanagement

[keystone_authtoken]
memcached_servers = localhost:11211
cafile = /opt/stack/data/ca-bundle.pem
project_domain_name = Default
project_name = service
user_domain_name = Default
password = secret
username = glance
auth_url = http://10.180.1.148/identity
interface = public
auth_type = password

[oslo_messaging_notifications]
driver = messagingv2

[glance_store]
filesystem_store_datadir = /opt/stack/data/glance/images/

[cors]
allowed_origin = http://10.180.1.148

[oslo_policy]
enforce_new_defaults = true
enforce_scope = true

[oslo_limit]
endpoint_id = a768c52107df4f7c9578515b49770bb2
system_scope = all
auth_url = http://10.180.1.148/identity
auth_type = password
username = glance
password = secret
user_domain_name = Default

====================== manila.conf ==============

[keystone_authtoken]
memcached_servers = localhost:11211
cafile = /opt/stack/data/ca-bundle.pem
project_domain_name = Default
project_name = service
user_domain_name = Default
password = secret
username = manila
auth_url = http://10.180.1.148/identity
interface = public
auth_type = password

[DEFAULT]
data_node_access_ips = 10.2.5.0/24
logging_exception_prefix = ERROR %(name)s %(instance)s
logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s
logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s
logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}}
transport_url = rabbit://stackrabbit:secret@10.180.1.148:5672/
manila_service_keypair_name = manila-service
enabled_share_backends = generic1,generic2
use_scheduler_creating_share_from_snapshot = False
replica_state_update_interval = 300
lvm_share_volume_group = lvm-shares
wsgi_keep_alive = False
enabled_share_protocols = NFS,CIFS
check_hash = True
periodic_deferred_delete_interval = 10
default_share_group_type = default
default_share_type = default
state_path = /opt/stack/data/manila
osapi_share_extensio...

[kiran pawar (kpdev)]

Reproduce on master 01.08.2024

create ubuntun 22.04 image, install devstack.
1. source openrc admin demo
2. create share network
3. create share..

[kiran pawar (kpdev)]

The devstack envirornment created locally set GLANCE_ENFORCE_SCOPE="True" by default causing glance to throw error.

If we set GLANCE_ENFORCE_SCOPE="False" in local.conf everything works fine. Checked zuul config, it does the same https://cf7800dce1b91328e4f8-616b34f6dd94893c803bfdcdd39959b5.ssl.cf5.rackcdn.com/921313/8/check/manila-tempest-plugin-generic/fe7c88b/controller/logs/local_conf.txt