octavia tempest tests are failing with Policy does not allow this request to be performed error

[chandan kumar]

https://logserver.rdoproject.org/92/42692/8/check/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-master/408e72b/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

```
{1} setUpClass (octavia_tempest_plugin.tests.scenario.v2.test_load_balancer.LoadBalancerScenarioTest) [0.000000s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/usr/lib/python3.9/site-packages/tempest/test.py", line 168, in setUpClass
    raise value.with_traceback(trace)

      File "/usr/lib/python3.9/site-packages/tempest/test.py", line 161, in setUpClass
    cls.resource_setup()

      File "/usr/lib/python3.9/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_load_balancer.py", line 58, in resource_setup
    cls.lb_admin_flavor_profile_client.create_flavor_profile(

      File "/usr/lib/python3.9/site-packages/octavia_tempest_plugin/common/decorators.py", line 42, in wrapper
    return f(*func_args, **func_kwargs)

      File "/usr/lib/python3.9/site-packages/octavia_tempest_plugin/services/load_balancer/v2/flavor_profile_client.py", line 68, in create_flavor_profile
    return self._create_object(**kwargs)

      File "/usr/lib/python3.9/site-packages/octavia_tempest_plugin/services/load_balancer/v2/base_client.py", line 101, in _create_object
    response, body = self.post(request_uri, jsonutils.dumps(obj_dict))

      File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 299, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)

      File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 720, in request
    self._error_checker(resp, resp_body)

      File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 821, in _error_checker
    raise exceptions.Forbidden(resp_body, resp=resp)

    tempest.lib.exceptions.Forbidden: Forbidden
Details: {'faultcode': 'Client', 'faultstring': 'Policy does not allow this request to be performed.', 'debuginfo': None}
```

It is a tracker bug to do further investigation.

Below is the list of other runs failing with the same error:
[1]. https://logserver.rdoproject.org/56/36356/57/check/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-master/fe0c29b/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

[2]. https://logserver.rdoproject.org/92/42692/8/check/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-master/037c2c5/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

[Dariusz Smigiel (smigiel-dariusz)]

Job `periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-wallaby` started failing on 2022-07-13 14:55:37 tempest test: octavia_tempest_plugin.tests.scenario.v2.test_load_balancer.LoadBalancerScenarioTest

It fails with a message:
> Details: {'faultcode': 'Client', 'faultstring': 'Policy does not allow this request to be performed.', 'debuginfo': None}

In octavia logs [6] we can see a failed attempt to post flavor-profile
>
2022-07-15 17:58:45.027 16 DEBUG octavia.common.policy [req-7929f060-1cb0-4f6b-9bb5-d2b779fd0637 - 13ccbfa09cf2425a9d0e1167b200665e - default default] Policy check for os_load-balancer_api:flavor-profile:post failed with credentials {'is_admin': False, 'user_id': None, 'user_domain_id': 'default', 'system_scope': None, 'domain_id': None, 'project_id': '13ccbfa09cf2425a9d0e1167b200665e', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} authorize /usr/lib/python3.9/site-packages/octavia/common/policy.py:126
2022-07-15 17:58:45.028 16 DEBUG wsme.api [req-7929f060-1cb0-4f6b-9bb5-d2b779fd0637 - 13ccbfa09cf2425a9d0e1167b200665e - default default] Client-side error: Policy does not allow this request to be performed. format_exception /usr/lib/python3.9/site-packages/wsme/api.py:227

[1]: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-wallaby&skip=0
[2]: https://logserver.rdoproject.org/67/41367/15/check/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-wallaby/1315de3/logs/undercloud/var/log/tempest/stestr_results.html.gz
[3]: https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-wallaby/f2c0385/logs/undercloud/var/log/tempest/stestr_results.html.gz
[4]: https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-wallaby/b54b5d3/logs/undercloud/var/log/tempest/stestr_results.html.gz
[5]: https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-wallaby/52f0659/logs/undercloud/var/log/tempest/stestr_results.html.gz
[6]: https://logserver.rdoproject.org/67/41367/15/check/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-wallaby/1315de3/logs/undercloud/var/log/containers/octavia/octavia.log.txt.gz

[chandan kumar (chkumar246)]

It is also failing on periodic-tripleo-ci-rhel-8-scenario010-standalone-octavia-rhos-17

[Gregory Thiemonge (gthiemonge)]

The policy override file was recently changed in stable/wallaby and master

https://review.rdoproject.org/r/q/I85e303b48bd6ebde7a70ec958b89ffe56dcd2fd8

A backport that adds the new roles is under review for wallaby:
* https://review.opendev.org/c/openstack/tripleo-heat-templates/+/850033

then there is this patch for the octavia-tempest-plugin configuration:
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/850118

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/850225

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/850304

[chandan kumar (chkumar246)]

tht patches are merged now.
Currently with this patch https://review.opendev.org/c/openstack/tripleo-ci/+/850225 , It adds
load_balancer.RBAC_test_type: "keystone_default_roles" and removes load_balancer.member_role: '"Member"'

and https://opendev.org/openinfra/python-tempestconf/src/branch/master/config_tempest/services/octavia.py#L48
adds
load_balancer.member_role: member.

But still the tests are failing.
https://logserver.rdoproject.org/38/44038/3/check/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-wallaby-test/2c4b53f/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

and tempest.conf
https://logserver.rdoproject.org/38/44038/3/check/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-wallaby-test/2c4b53f/logs/undercloud/home/zuul/tempest/etc/tempest.conf.txt.gz

and here is the testproject: https://review.rdoproject.org/r/c/testproject/+/44038

[chandan kumar (chkumar246)]

We are seeing these failures in other octavia tests also and is blocking upstream check and gate jobs for wallaby.
```

ft1.1: setUpClass (octavia_tempest_plugin.tests.scenario.v2.test_load_balancer.LoadBalancerScenarioTest)testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/tempest/test.py", line 168, in setUpClass
    raise value.with_traceback(trace)
  File "/usr/lib/python3.9/site-packages/tempest/test.py", line 161, in setUpClass
    cls.resource_setup()
  File "/usr/lib/python3.9/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_load_balancer.py", line 58, in resource_setup
    cls.lb_admin_flavor_profile_client.create_flavor_profile(
  File "/usr/lib/python3.9/site-packages/octavia_tempest_plugin/common/decorators.py", line 42, in wrapper
    return f(*func_args, **func_kwargs)
  File "/usr/lib/python3.9/site-packages/octavia_tempest_plugin/services/load_balancer/v2/flavor_profile_client.py", line 68, in create_flavor_profile
    return self._create_object(**kwargs)
  File "/usr/lib/python3.9/site-packages/octavia_tempest_plugin/services/load_balancer/v2/base_client.py", line 101, in _create_object
    response, body = self.post(request_uri, jsonutils.dumps(obj_dict))
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 299, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 720, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 821, in _error_checker
    raise exceptions.Forbidden(resp_body, resp=resp)
tempest.lib.exceptions.Forbidden: Forbidden
Details: {'faultcode': 'Client', 'faultstring': 'Policy does not allow this request to be performed.', 'debuginfo': None}
```
- https://7114429c7841114b6ab2-a5faeab2c736b5ae3f187165e14e12eb.ssl.cf5.rackcdn.com/847237/1/check/tripleo-ci-centos-9-scenario010-ovn-provider-standalone/9feba20/logs/undercloud/var/log/tempest/stestr_results.html

- https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4a1/847190/2/check/tripleo-ci-centos-9-scenario010-standalone/4a11500/logs/undercloud/var/log/tempest/stestr_results.html

Moving this test to skip list to clear the gate,

[Arx Cruz (arxcruz)]

Patches were merged, and skipped test are now running fine, closing.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "chandan kumar <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/850304

[OpenStack Infra (hudson-openstack)]

Change abandoned by "chandan kumar <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/850225
Reason: no longer needed