tempest.api.compute fail with policy doesn't allow error

[Martin Kopec]

tempest.api.compute and a few tempest.api.volume tests started failing, example traceback

ft1.1: setUpClass (tempest.api.compute.floating_ips.test_floating_ips_actions.FloatingIPsAssociationTestJSON)testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/test.py", line 206, in setUpClass
    raise value.with_traceback(trace)
  File "/opt/stack/tempest/tempest/test.py", line 199, in setUpClass
    cls.resource_setup()
  File "/opt/stack/tempest/tempest/api/compute/floating_ips/test_floating_ips_actions.py", line 70, in resource_setup
    cls.server = cls.create_test_server(wait_until='ACTIVE')
  File "/opt/stack/tempest/tempest/api/compute/base.py", line 272, in create_test_server
    body, servers = compute.create_test_server(
  File "/opt/stack/tempest/tempest/common/compute.py", line 285, in create_test_server
    body = clients.servers_client.create_server(name=name, imageRef=image_id,
  File "/opt/stack/tempest/tempest/lib/services/compute/servers_client.py", line 115, in create_server
    resp, body = self.post('servers', post_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 300, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/opt/stack/tempest/tempest/lib/services/compute/base_compute_client.py", line 47, in request
    resp, resp_body = super(BaseComputeClient, self).request(
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 847, in _error_checker
    raise exceptions.Forbidden(resp_body, resp=resp)
tempest.lib.exceptions.Forbidden: Forbidden
Details: {'code': 403, 'message': "Policy doesn't allow os_compute_api:servers:create to be performed."}

The affected jobs: tempest-full-test-account-no-admin-py3 and tempest-full-test-account-py3

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_982/880630/13/check/tempest-full-test-account-py3/982332d/testr_results.html
https://0ba57ff75479255f8769-2b6c314b914fb3f996b5c8a41dca9d5c.ssl.cf2.rackcdn.com/880630/13/check/tempest-full-test-account-no-admin-py3/70fe771/testr_results.html

The tests started failing after we enabled NOVA_ENFORCE_SCOPE by default in devstack:
https://review.opendev.org/c/openstack/devstack/+/883556

[Lukas Piwowarski (lukas-piwowarski)]

I did a quick research on this:
- I think that this patch causes the issue: https://review.opendev.org/c/openstack/tempest/+/878074
- Thanks to this patch, a test may get project_reader when it requires primary credentials.
- The reason why the errors appear now is probably the fact that we enabled enforce scope for NOVA and GLANCE in devstack recently.
- Here is DNM patch that might fix the issue: https://review.opendev.org/c/openstack/tempest/+/884509

[Lukas Piwowarski (lukas-piwowarski)]

The above-mentioned patch fixes the issue only partially. Another problem that is associated with pre-provisioned credentials is the fact that it might happen that a test receives credentials from different projects when it is not desirable. For example here [1] a test might get primary user in one project and project_reader from another one.

[1] https://opendev.org/openstack/tempest/src/commit/c3a950b7fe8cc6420c2f320535559340de683bc8/tempest/api/compute/servers/test_servers.py#L31

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/884768

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tempest/+/884768
Committed: https://opendev.org/openstack/tempest/commit/037ea4efe07cbc78aa81388dccc06eed21874d4b
Submitter: "Zuul (22348)"
Branch: master

commit 037ea4efe07cbc78aa81388dccc06eed21874d4b
Author: Ghanshyam Mann <email address hidden>
Date: Tue May 30 13:49:17 2023 -0500

    Enable Nova, Glance RBAC old defaults in pre provisioned account testing

    Pre-provisioned account code and testing it not yet moved to the new
    RBAC design/need so it gor broken when we enabled nova and glance
    new RBAC by default in devstack. We need to fix it but until then
    let's keep testing it with old defaults.

    Related-Bug: #2020859
    Related-Bug: #2020860
    Change-Id: Ib290d4985b93f23dec502a202096d87ff00e9961

[Lukas Piwowarski (lukas-piwowarski)]

We've encountered a similar issue here => https://bugs.launchpad.net/tempest/+bug/2043038

[Martin Kopec (mkopec)]

discussed during office hour:
https://meetings.opendev.org/meetings/qa/2023/qa.2023-11-08-17.00.log.html#l-64