Keystone protection tests are broken

[Lance Bragstad]

The keystone functional protection tests that use tempest are failing during Neutron setup:

+ functions-common:is_service_enabled:1965 : return 1
+ functions-common:run_process:1575 : time_stop run_process
+ functions-common:time_stop:2309 : local name
+ functions-common:time_stop:2310 : local end_time
+ functions-common:time_stop:2311 : local elapsed_time
+ functions-common:time_stop:2312 : local total
+ functions-common:time_stop:2313 : local start_time
+ functions-common:time_stop:2315 : name=run_process
+ functions-common:time_stop:2316 : start_time=1628535788991
+ functions-common:time_stop:2318 : [[ -z 1628535788991 ]]
++ functions-common:time_stop:2321 : date +%s%3N
+ functions-common:time_stop:2321 : end_time=1628535789042
+ functions-common:time_stop:2322 : elapsed_time=51
+ functions-common:time_stop:2323 : total=5353
+ functions-common:time_stop:2325 : _TIME_START[$name]=
+ functions-common:time_stop:2326 : _TIME_TOTAL[$name]=5404
+ ./stack.sh:main:1297 : is_service_enabled q-svc
+ functions-common:is_service_enabled:1965 : return 0
+ ./stack.sh:main:1297 : [[ True == \T\r\u\e ]]
+ ./stack.sh:main:1298 : echo_summary 'Creating initial neutron network elements'
+ ./stack.sh:echo_summary:426 : [[ -t 3 ]]
+ ./stack.sh:echo_summary:432 : echo -e Creating initial neutron network elements
+ ./stack.sh:main:1301 : type -p neutron_plugin_create_initial_networks
+ ./stack.sh:main:1304 : create_neutron_initial_network
+ lib/neutron_plugins/services/l3:create_neutron_initial_network:164 : local project_id
++ lib/neutron_plugins/services/l3:create_neutron_initial_network:165 : oscwrap project list
++ lib/neutron_plugins/services/l3:create_neutron_initial_network:165 : grep ' demo '
++ lib/neutron_plugins/services/l3:create_neutron_initial_network:165 : get_field 1
++ functions-common:get_field:726 : local data field
++ functions-common:get_field:727 : read data
++ functions-common:oscwrap:2349 : return 0
+ lib/neutron_plugins/services/l3:create_neutron_initial_network:165 : project_id=
+ lib/neutron_plugins/services/l3:create_neutron_initial_network:166 : die_if_not_set 166 project_id 'Failure retrieving project_id for demo'
+ functions-common:die_if_not_set:216 : local exitcode=0

and

/usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
You are not authorized to perform the requested action: identity:get_service. (HTTP 403) (Request-ID: req-1fcb0513-4511-48cf-ac80-34c241ddb211)
++functions-common:oscwrap:2349 return 1
+lib/glance:configure_glance_quotas:298 iniset /etc/glance/glance-api.conf oslo_limit endpoint_id
+lib/glance:configure_glance_quotas:302 openstack role add --user glance --user-domain Default --system all reader
/usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
You are not authorized to perform the requested action: identity:list_roles. (HTTP 403) (Request-ID: req-cab7850c-d24b-4bc7-9068-9153f2af0955)
[1191190 Async create_glance_accounts:1239951]: finished create_glance_accounts with result 1 in 23 seconds

There appears to be 403s when using keystone tokens to setup things in devstack, so the tests don't even run.

I was able to reproduce this locally using:

DEVSTACK_PARALLEL=True
KEYSTONE_ENFORCE_SCOPE=True
enable_plugin keystone https://opendev.org/openstack/keystone

[Lance Bragstad (lbragstad)]

I think this is failing because glance recently added support for unified limits and quotas, which required changes in devstack to set limits in keystone [0].

The glance user requires system-level access to read quotas, and giving that assignment to glance needs to be done as a system-administrator. If KEYSTONE_ENFORCE_SCOPE == True the request will fail because devstack is using a project-admin persona to make the request, resulting in a 403 [1].

[0] https://review.opendev.org/q/If58d8474cae95b1be3044bd52010b8288a7f5fcc
[1] https://github.com/openstack/devstack/blob/a5ed116814fa3a435f15231aa7b18d389f917844/lib/glance#L311-L313

[Lance Bragstad (lbragstad)]

Actually - there was a recent change to devstack that pulled the keystone enforce scope logic out of keystone's devstack plugin and into devstack [0] where it's done during keystone's configuration.

I think this is affecting the protection tests described in this because because everything after keystone's configuration step starts failing with 403s. Part of the reason why we implemented keystone's devstack plugin scope configuration in test-config is because devstack doesn't support system scope yet [1]. It can't determine which profile or account to use for creating system-level resources.

There is a revert proposed to devstack until we have time to implement proper system-scope checking in devstack [2].

[0] https://review.opendev.org/c/openstack/devstack/+/778975
[1] https://github.com/openstack/keystone/blob/master/devstack/plugin.sh#L52-L53
[2] https://review.opendev.org/c/openstack/devstack/+/804025

[Ghanshyam Mann (ghanshyammann)]

reverted the devstack change fixed this https://review.opendev.org/c/openstack/devstack/+/804025

[Ghanshyam Mann (ghanshyammann)]

adding a devstack job also to avoid such regression https://review.opendev.org/c/openstack/devstack/+/804263

[David Wilde (dave-wilde)]

This looks to have been fixed with the changes to devstack. Feel free to re-open if you feel this is still an issue that needs fixing in Keystone.