tripleo-ci-centos-8-scenario010-standalone check/gate/periodic jobs failing on tempest tests with error - tempest.lib.exceptions.Forbidden: Forbidden , Details: {'faultcode': 'Client', 'faultstring': 'Policy does not allow this request to be performed.', 'debuginfo': None

[Sandeep Yadav]

tripleo-ci-centos-8-scenario010-standalone jobs failing with below tempest errors since ~02:00 GMT , 16th July. This is blocking Gate.

https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario010-standalone

Logs url:-
https://6aee315f5a36c0c3ea7c-7637ce74125a0f2a5404346b147f7a3a.ssl.cf2.rackcdn.com/714639/5/check/tripleo-ci-centos-8-scenario010-standalone/23cdb0b/logs/undercloud/var/log/tempest/tempest_run.log

https://2c442655f7e6a91a0668-dd7612090ddd37717fc79d2c5d5bec61.ssl.cf2.rackcdn.com/741271/3/gate/tripleo-ci-centos-8-scenario010-standalone/a39a12a/logs/undercloud/var/log/tempest/tempest_run.log

Error snippet:-
~~~
{0} setUpClass (octavia_tempest_plugin.tests.scenario.v2.test_l7policy.L7PolicyScenarioTest) [0.000000s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/tempest/test.py", line 188, in setUpClass
        six.reraise(etype, value, trace)
      File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
        raise value
      File "/usr/lib/python3.6/site-packages/tempest/test.py", line 181, in setUpClass
        cls.resource_setup()
      File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_l7policy.py", line 51, in resource_setup
        lb = cls.mem_lb_client.create_loadbalancer(**lb_kwargs)
      File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/services/load_balancer/v2/loadbalancer_client.py", line 91, in create_loadbalancer
        return self._create_object(**kwargs)
      File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/services/load_balancer/v2/base_client.py", line 101, in _create_object
        response, body = self.post(request_uri, jsonutils.dumps(obj_dict))
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 298, in post
        return self.request('POST', url, extra_headers, headers, body, chunked)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 702, in request
        self._error_checker(resp, resp_body)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 803, in _error_checker
        raise exceptions.Forbidden(resp_body, resp=resp)
    tempest.lib.exceptions.Forbidden: Forbidden
    Details: {'faultcode': 'Client', 'faultstring': 'Policy does not allow this request to be performed.', 'debuginfo': None}

{1} setUpClass (octavia_tempest_plugin.tests.scenario.v2.test_healthmonitor.HealthMonitorScenarioTest) [0.000000s] ... FAILED
~~~

[Sandeep Yadav (sandeepyadav93)]

Hi,

Luigi shared a patch - https://review.opendev.org/#/c/731501/ which might solve this issue:-

~~~
<tosky> ysandeep|rover: the fix for octavia tempest plugin is not there yet; it was added in the next commit
<ysandeep|rover> tosky, Hey o/ thank you, do you have that commit handy?
<tosky> ysandeep|rover: https://opendev.org/osf/python-tempestconf/commit/7ee63b1517b7412c8e25f2842b207339a70f62c6
<ysandeep|rover> tosky, thanks!

~~~
The patch tosky mentioned merged https://review.opendev.org/#/c/731501/ - but is still in component pipeline (consistent) and awaiting further promotion

In the meantime we are making sc10 - Non voting with https://review.opendev.org/#/c/741435/

Need to check if this bug is duplicate of https://bugs.launchpad.net/tripleo/+bug/1885602, But the Gate/check Failure we mentioned started today

[Sandeep Yadav (sandeepyadav93)]

Further Improvements needed:-

* We need to add sc10 in promotion criterian.

[Ronelle Landy (rlandy)]

There is a duplicate bug:
https://bugs.launchpad.net/tripleo/+bug/1885602