[Pre-provisioned credentials] tempest.scenario.test_unified_limits.* tests are failing because of missing system_admin creds

[Lukas Piwowarski]

The tempest.scenario.test_unified_limits.* tests fail when pre-provisioned credentials generated using the account-generator tool are used because of missing system_admin credentials. The failure occurs in the tempest-full-test-account-py3 job (see the attached file).

[Martin Kopec (mkopec)]

Discussed yesterday on the QA office hour IRC meeting - https://meetings.opendev.org/meetings/qa/2023/qa.2023-08-15-15.00.log.html

mainly:
"""
15:39:13 <lpiwowar> The test is failing because of missing system scoped admin creds in the accounts.yaml.
15:39:19 <lpiwowar> https://opendev.org/openstack/tempest/src/branch/master/tempest/scenario/test_unified_limits.py#L32
15:39:32 <lpiwowar> "Details: No credentials matching role: admin, scope: system specified in the accounts file"
15:40:06 <frickler> maybe the test should be skipped in that case, instead of failing?
15:40:15 <lpiwowar> Dynamic creds will create the system scoped admin on the go ...
15:40:47 <lpiwowar> frickler: skipped when pre-prov creds are used?
15:41:12 <frickler> skipped when "No credentials matching role: admin, scope: system specified in the accounts file"
15:41:54 <lpiwowar> frickler: ok, that sounds good to me:)
15:43:46 <frickler> that's it then?
15:44:05 <lpiwowar> frickler, kopecmartin: +1, thanks for the help!
15:44:40 <kopecmartin> lpiwowar: you can try to propose a wip patch to see what happens if you add 'admin' to the test's credentials
15:45:06 <kopecmartin> and meanwhile figure out what's right, not just working, solution, i'd try follow up with gmann
15:45:22 <kopecmartin> let's move on
"""

and the discussion continued after the meeting:

https://meetings.opendev.org/irclogs/%23openstack-qa/%23openstack-qa.2023-08-15.log.html#t2023-08-15T15:58:18

[Martin Kopec (mkopec)]

The job in question:
https://zuul.openstack.org/builds?job_name=tempest-full-test-account-py3&skip=0

Test: tempest.scenario.test_unified_limits
Traceback:

ft1.1: setUpClass (tempest.scenario.test_unified_limits.ImageQuotaTest)testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/test.py", line 206, in setUpClass
    raise value.with_traceback(trace)
  File "/opt/stack/tempest/tempest/test.py", line 191, in setUpClass
    cls.setup_credentials()
  File "/opt/stack/tempest/tempest/scenario/manager.py", line 101, in setup_credentials
    super(ScenarioTest, cls).setup_credentials()
  File "/opt/stack/tempest/tempest/test.py", line 419, in setup_credentials
    manager = cls.get_client_manager(
  File "/opt/stack/tempest/tempest/test.py", line 764, in get_client_manager
    creds = getattr(cred_provider, credentials_method)()
  File "/opt/stack/tempest/tempest/lib/common/preprov_creds.py", line 331, in get_system_admin_creds
    system_admin = self._get_creds(['admin'], scope='system')
  File "/opt/stack/tempest/tempest/lib/common/preprov_creds.py", line 268, in _get_creds
    useable_hashes = self._get_match_hash_list(roles, scope)
  File "/opt/stack/tempest/tempest/lib/common/preprov_creds.py", line 231, in _get_match_hash_list
    raise lib_exc.InvalidCredentials(
tempest.lib.exceptions.InvalidCredentials: Invalid Credentials
Details: No credentials matching role: admin, scope: system specified in the accounts file

[Martin Kopec (mkopec)]

hmm, the job doesn't execute scenario tests regularly .. well, i can see the scenario tests triggered only in 2 cases on August 15th and 11th - the jobs were triggered with 2 independent changes. All builds before, after or in between didn't trigger scenario jobs. That's super weird

[Martin Kopec (mkopec)]

I've just seen the failure in tempest-full-test-account-no-admin-py3 job:

https://16914890dac5609d49b0-6ce456e6f9c28bd8511060ffded7cac7.ssl.cf2.rackcdn.com/890798/10/experimental/tempest-full-test-account-no-admin-py3/339ad65/testr_results.html

[Lukas Piwowarski (lukas-piwowarski)]

As discussed yesterday on PTG during QA session we are going to modify the account-generator tool so that it automatically generates the system admin account which is needed by the ImageQuota tests.