Static creds with system scope cause test falure

[Ilya Popov]

This issue appears for static creds scenario.

Test tempest.scenario.test_unified_limits.ImageQuotaTest demands admin creds with system scope. If you dont provide admin creds with system scope - you will get following error:

setUpClass (tempest.scenario.test_unified_limits.ImageQuotaTest)
----------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/root/venvs/tempest/lib/python3.8/site-packages/tempest/test.py", line 168, in setUpClass
    raise value.with_traceback(trace)

      File "/root/venvs/tempest/lib/python3.8/site-packages/tempest/test.py", line 153, in setUpClass
    cls.setup_credentials()

      File "/root/venvs/tempest/lib/python3.8/site-packages/tempest/test.py", line 372, in setup_credentials
    manager = cls.get_client_manager(

      File "/root/venvs/tempest/lib/python3.8/site-packages/tempest/test.py", line 717, in get_client_manager
    creds = getattr(cred_provider, credentials_method)()

      File "/root/venvs/tempest/lib/python3.8/site-packages/tempest/lib/common/preprov_creds.py", line 331, in get_system_admin_creds
    system_admin = self._get_creds(['admin'], scope='system')

      File "/root/venvs/tempest/lib/python3.8/site-packages/tempest/lib/common/preprov_creds.py", line 268, in _get_creds
    useable_hashes = self._get_match_hash_list(roles, scope)

      File "/root/venvs/tempest/lib/python3.8/site-packages/tempest/lib/common/preprov_creds.py", line 231, in _get_match_hash_list
    raise lib_exc.InvalidCredentials(

    tempest.lib.exceptions.InvalidCredentials: Invalid Credentials
Details: No credentials matching role: admin, scope: system specified in the accounts file

So I added following account to accounts.yml:

- username: 'tempest-adm1'
  password: 'pass'
  roles:
     - 'admin'
  resources:
    network: 'tempest-adm-net1'
  system: 'system'

And setUpClass (tempest.scenario.test_unified_limits.ImageQuotaTest) was successfully passed.

But I faced the issue for another test if it picks tempest-adm1 account:

setUpClass (tempest.api.volume.admin.test_volume_quotas_negative.VolumeQuotasNegativeTestJSON)
----------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/root/venvs/tempest/lib/python3.8/site-packages/tempest/test.py", line 168, in setUpClass
    raise value.with_traceback(trace)

      File "/root/venvs/tempest/lib/python3.8/site-packages/tempest/test.py", line 161, in setUpClass
    cls.resource_setup()

      File "/root/venvs/tempest/lib/python3.8/site-packages/tempest/api/volume/admin/test_volume_quotas_negative.py", line 42, in resource_setup
    original_quota_set = (cls.admin_quotas_client.show_quota_set(

      File "/root/venvs/tempest/lib/python3.8/site-packages/tempest/lib/services/volume/v3/quotas_client.py", line 43, in show_quota_set
    resp, body = self.get(url)

      File "/root/venvs/tempest/lib/python3.8/site-packages/tempest/lib/common/rest_client.py", line 314, in get
    return self.request('GET', url, extra_headers, headers)

      File "/root/venvs/tempest/lib/python3.8/site-packages/tempest/lib/common/rest_client.py", line 704, in request
    resp, resp_body = self._request(method, url, headers=headers,

      File "/root/venvs/tempest/lib/python3.8/site-packages/tempest/lib/common/rest_client.py", line 580, in _request
    req_url, req_headers, req_body = self.auth_provider.auth_request(

      File "/root/venvs/tempest/lib/python3.8/site-packages/tempest/lib/auth.py", line 185, in auth_request
    auth_url, auth_headers, auth_body = self._decorate_request(

      File "/root/venvs/tempest/lib/python3.8/site-packages/tempest/lib/auth.py", line 277, in _decorate_request
    base_url = self.base_url(filters=filters, auth_data=auth_data)

      File "/root/venvs/tempest/lib/python3.8/site-packages/tempest/lib/auth.py", line 588, in base_url
    filtered_catalog = [service_catalog[0]]

    IndexError: list index out of range

It happens because for system scope account we have to left blank (or don't setup) project and domain. As you can see, there is no domain or project field in tempest-adm1 account mentioned above.

In other hand tempest.api.volume.admin.test_volume_quotas_negative.VolumeQuotasNegativeTestJSON could pick this system-scoped account to use as admin and fails, because admin with system scope don't have project and domain defined.

So It looks like that for system-scoped admin we don't have to add this account to admins without system scope. So right now self.hash_dict of tempest/lib/common/preprov_creds.py contains the following items:

{'roles':

{'member': ['2d5ed5aa957e8197cb931bf510785ca2', 'd024ddf9c4c9ec572953884517ea0c1f', '6f15060a0a48ed4eb4949933d0c02028', 'd0c1b530238784a7c89be65858fbdc8a', 'a6496206775add98aefa8f4c05e87fea', 'b78881f6b48814900b0931e91cd736c8'],

'admin': ['222fd1737bf3bd6e2f2c9dc5048e8cf7', '7ca16ecf29b1758f5b51f274571e4cf8', '33a172f06b3fbb882350e0abb28b0995']}, # !!!!!! admin account

'creds': {'2d5ed5aa957e8197cb931bf510785ca2': {'username': 'tempest-memb1', 'project_name': 'tempest-memb-prj1', 'password': 'pass'}, 'd024ddf9c4c9ec572953884517ea0c1f': {'username': 'tempest-memb2', 'project_name': 'tempest-memb-prj2', 'password': 'pass'}, '6f15060a0a48ed4eb4949933d0c02028': {'username': 'tempest-memb3', 'project_name': 'tempest-memb-prj3', 'password': 'pass'}, 'd0c1b530238784a7c89be65858fbdc8a': {'username': 'tempest-memb4', 'project_name': 'tempest-memb-prj4', 'password': 'pass'}, 'a6496206775add98aefa8f4c05e87fea': {'username': 'tempest-memb5', 'project_name': 'tempest-memb-prj5', 'password': 'pass'}, 'b78881f6b48814900b0931e91cd736c8': {'username': 'tempest-memb6', 'project_name': 'tempest-memb-prj6', 'password': 'pass'}, '222fd1737bf3bd6e2f2c9dc5048e8cf7': {'username': 'tempest-adm2', 'project_name': 'tempest-adm-prj2', 'password': 'pass'}, '7ca16ecf29b1758f5b51f274571e4cf8': {'username': 'tempest-adm3', 'project_name': 'tempest-adm-prj3', 'password': 'pass'}, '33a172f06b3fbb882350e0abb28b0995': {'username': 'tempest-adm1', 'password': 'pass', 'system': 'system'}}, 'networks': {'2d5ed5aa957e8197cb931bf510785ca2': 'tempest-memb-net1', 'd024ddf9c4c9ec572953884517ea0c1f': 'tempest-memb-net2', '6f15060a0a48ed4eb4949933d0c02028': 'tempest-memb-net3', 'd0c1b530238784a7c89be65858fbdc8a': 'tempest-memb-net4', 'a6496206775add98aefa8f4c05e87fea': 'tempest-memb-net5', 'b78881f6b48814900b0931e91cd736c8': 'tempest-memb-net6', '222fd1737bf3bd6e2f2c9dc5048e8cf7': 'tempest-adm-net2', '7ca16ecf29b1758f5b51f274571e4cf8': 'tempest-adm-net3', '33a172f06b3fbb882350e0abb28b0995': 'tempest-adm-net1'},

'scoped_roles': {'system_admin': ['33a172f06b3fbb882350e0abb28b0995']}} # !!!!!! system_scoped account

As we can see role 'admin' has three accounts including this one:

33a172f06b3fbb882350e0abb28b0995

And it is also the system-scoped account without domain and project.

All tests which are using admin account could pick up system-scoped account and could fail. In other hand - some tests needs system_scoped account. So we don't have to add system-scoped account to admin role accounts list in hash

[Ilya Popov (ilya-p)]

Yes, when we use system scope - not all endpoints are returned by keystone. It is reproduced with openstack CLI with the following openrc:

for key in $( set | awk '{FS="="} /^OS_/ {print $1}' ); do unset $key ; done
export OS_USERNAME=tempest-adm1
export OS_PASSWORD=pass
export OS_AUTH_URL=http://172.16.0.18:35357/v3
export OS_INTERFACE=internal
export OS_ENDPOINT_TYPE=internalURL
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne
export OS_AUTH_PLUGIN=password
export OS_SYSTEM_SCOPE=all

and after sourcing it when we issue openstack --debug token issue we will get catalog with some empty endpoints like this:

catalog": [{"endpoints": [], "id": "0a5a95d8c5f04ff394066f190bc4f9d0", "type": "volumev3", "name": "cinderv3"}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/872343

[Ghanshyam Mann (ghanshyammann)]

thanks for reporting.

Everything is scoped now and even with the old account.yaml, the scope will be set to 'project' by L132.

This code of storing the account with role is wrong as we should not store the same account in two places, one with role L145 and the other with scope L147. It should be stored with scope only. We can replace this for loop[1] with the below so that we store the account with only <scope>_<role> hash only

             for role in roles:
                if scope:
                    hash_dict = cls._append_scoped_role(
                        scope, role, temp_hash_key, hash_dict)
                else:
                    hash_dict = cls._append_role(
                        role, temp_hash_key, hash_dict)

[1] https://github.com/openstack/tempest/blob/1632cf5085de995206ec97788b4b546c13466860/tempest/lib/common/preprov_creds.py#L143-L148

[Aaron S (as0)]

Hi there - I found this bug report whilst debugging some Designate tests. We also use a static preprov creds file.

We supply admin creds which are scoped to a project, and also specify 'roles: admin' - despite this, a lot of the Designate tests fail with the same ` Details: No credentials matching role: admin, scope: system specified in the accounts file` error.

Running:
- Tempest tag: xena-last
- Designate Tempest plugin: xena-last (also tried 0.19 0.14 and master)
- OpenStack Victoria release

I have tried set `enforce_scope.designate = false` and `dns.enforce_new_defaults = false` but neither work, so it seems it may be this Tempest bug.

@ghanshyammann - Is the code above still a WIP, or would that be a valid fix to propose? It wasn't clear from your last comment if the changes in https://bugs.launchpad.net/tempest/+bug/2004261/comments/3 were an acceptable

Thank you