You are not authorized to perform the requested action: identity:list_role_assignments.

Bug #1963685 reported by Felipe Reyes
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Keystone Charm
In Progress
High
Felipe Reyes
OpenStack Octavia Charm
In Progress
High
Felipe Reyes

Bug Description

When running Tempest against a juju deployed cloud, the octavia testing fails during the setupClass of octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest when it attempts to get the list of role assignments with dynamically created users.

[Test Case]

git clone https://github.com/openstack-charmers/charmed-openstack-tester
cd charmed-openstack-tester
tox -e func-target -- focal-xena

2022-03-03 19:26:31.221 20767 INFO tempest.lib.common.rest_client [req-9175ea33-e270-48d0-ab28-1caf93f30263 ] Request (TrafficOperationsScenarioTest:setUpClass): 201 POST https://10.5.2.69:5000/v3/auth/tokens 0.628s
2022-03-03 19:26:31.221 20767 DEBUG tempest.lib.common.rest_client [req-9175ea33-e270-48d0-ab28-1caf93f30263 ] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json'}
        Body: <omitted>
    Response - Headers: {'date': 'Thu, 03 Mar 2022 19:26:30 GMT', 'server': 'Apache/2.4.41 (Ubuntu)', 'content-length': '10009', 'x-subject-token': '<omitted>', 'vary': 'X-Auth-Token', 'x-openstack-request-id': 'req-9175ea33-e270-48d0-ab28-1caf93f30263', 'content-type': 'application/json', 'connection': 'close', 'status': '201', 'content-location': 'https://10.5.2.69:5000/v3/auth/tokens'}
        Body: b'{"token": {"methods": ["password"], "user": {"domain": {"id": "c3cc5dc7f4364ea7a1fa1740aad8b85f", "name": "admin_domain"}, "id": "2acc17da5a39459abd8a879e028c52f1", "name": "tempest-TrafficOperationsScenarioTest-1307197676-project-load-balancer_member", "password_expires_at": null}, "audit_ids": ["juQURj6FSyGm3CqbvtQNzA"], "expires_at": "2022-03-03T20:26:31.000000Z", "issued_at": "2022-03-03T19:26:31.000000Z", "project": {"domain": {"id": "c3cc5dc7f4364ea7a1fa1740aad8b85f", "name": "admin_domain"}, "id": "214e73c9934c490c95a7d82d25aa0f8e", "name": "tempest-TrafficOperationsScenarioTest-1307197676"}, "is_domain": false, "roles": [{"id": "e1fa6ee1a2ef41aead97ccbd9fae0960", "name": "load-balancer_member"}], "is_admin_project": false, "catalog": [{"endpoints": [{"id": "5b61f1ea9c9749dab2ac14105e330fe0", "interface": "internal", "region_id": "RegionOne", "url": "https://10.5.3.130:8080", "region": "RegionOne"}, {"id": "5e2749bd83a5422f94b67994e3c1c198", "interface": "admin", "region_id": "RegionOne", "url": "https://10.5.3.130:8080", "region": "RegionOne"}, {"id": "ba7c9e2871fb467aa52789050c78a9ac", "interface": "public", "region_id": "RegionOne", "url": "https://10.5.3.130:8080", "region": "RegionOne"}], "id": "2873e17f7b2e46d0bd71500b8c81639f", "type": "s3", "name": "s3"}, {"endpoints": [{"id": "42024441bab1444c846b0a0cd255d45a", "interface": "internal", "region_id": "RegionOne", "url": "https://10.5.3.168:8000/v1", "region": "RegionOne"}, {"id": "c22afdc16d7349f4ae9508a5d75d46f7", "interface": "admin", "region_id": "RegionOne", "url": "https://10.5.3.168:8000/v1", "region": "RegionOne"}, {"id": "f698bb51670e4efaa9993c81f74469d4", "interface": "public", "region_id": "RegionOne", "url": "https://10.5.3.168:8000/v1", "region": "RegionOne"}], "id": "32878215ed93477ca709e3ca2c7cf6e5", "type": "cloudformation", "name": "heat-cfn"}, {"endpoints": [{"id": "2d33a3f5252c4976a02d7a0ad239bcb9", "interface": "admin", "region_id": "RegionOne", "url": "https://10.5.2.247:9876", "region": "RegionOne"}, {"id": "3276792010014737a7342cfa0380c364", "interface": "internal", "region_id": "RegionOne", "url": "https://10.5.2.247:9876", "region": "RegionOne"}, {"id": "6b2c8291fcca4b6c93a4e3db204cf529", "interface": "public", "region_id": "RegionOne", "url": "https://10.5.2.247:9876", "region": "RegionOne"}], "id": "4bc41ecbfb3a4d0fbc0069de09de9782", "type": "load-balancer", "name": "octavia"}, {"endpoints": [{"id": "1d3d101417144cd4b928a90ae6bc18bd", "interface": "admin", "region_id": "RegionOne", "url": "https://10.5.3.63:9312", "region": "RegionOne"}, {"id": "90bf6076d4494a1a946fa8d81b888a9b", "interface": "public", "region_id": "RegionOne", "url": "https://10.5.3.63:9311", "region": "RegionOne"}, {"id": "c8b9a0910a284c1ea2f03a68e6fc6eaa", "interface": "internal", "region_id": "RegionOne", "url": "https://10.5.3.63:9311", "region": "RegionOne"}], "id": "4fd1b5d008c44a57b6fa66ac39914801", "type": "key-manager", "name": "barbican"}, {"endpoints": [{"id": "12bc6d2985f7417080bd61376fc8028d", "interface": "admin", "region_id": "RegionOne", "url": "https://10.5.3.130:8080/simplestreams/data/", "region": "RegionOne"}, {"id": "4c61773612ea4472b115ce3ae638a021", "interface": "internal", "region_id": "RegionOne", "url": "https://10.5.3.130:8080/v1/AUTH_4ea5904a477e4b98a4a4280119fa1bd7/simplestreams/data/", "region": "RegionOne"}, {"id": "dbf54c0067ee453e9d4e65ad94019968", "interface": "public", "region_id": "RegionOne", "url": "https://10.5.3.130:8080/v1/AUTH_4ea5904a477e4b98a4a4280119fa1bd7/simplestreams/data/", "region": "RegionOne"}], "id": "5a61023ebac0415cad5b09f75d47aa43", "type": "product-streams", "name": "image-stream"}, {"endpoints": [{"id": "333971172573444d95e2c30d794d4a4e", "interface": "internal", "region_id": "RegionOne", "url": "https://10.5.3.168:8004/v1/214e73c9934c490c95a7d82d25aa0f8e", "region": "RegionOne"}, {"id": "4bd8b1367a3040b6b3cdc8fd36aabf96", "interface": "admin", "region_id": "RegionOne", "url": "https://10.5.3.168:8004/v1/214e73c9934c490c95a7d82d25aa0f8e", "region": "RegionOne"}, {"id": "75b1e6c9438746cf96cb22c7d4794aa1", _log_request_full /home/ubuntu/git/charmed-openstack-tester/.tox/func-target/lib/python3.6/site-packages/tempest/lib/common/rest_client.py:456
2022-03-03 19:26:31.372 20767 INFO tempest.lib.common.rest_client [req-f7e30cc3-f01c-4131-b30b-70a649fd3fa8 ] Request (TrafficOperationsScenarioTest:setUpClass): 403 GET https://10.5.2.69:35357/v3/role_assignments?user.id=e2705ac2e28044089be934acdb88cef1&project.id=bacfe18b08fe400d84e2b9b042f44fbe 0.149s
2022-03-03 19:26:31.372 20767 DEBUG tempest.lib.common.rest_client [req-f7e30cc3-f01c-4131-b30b-70a649fd3fa8 ] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Thu, 03 Mar 2022 19:26:31 GMT', 'server': 'Apache/2.4.41 (Ubuntu)', 'content-length': '143', 'vary': 'X-Auth-Token', 'x-openstack-request-id': 'req-f7e30cc3-f01c-4131-b30b-70a649fd3fa8', 'content-type': 'application/json', 'connection': 'close', 'status': '403', 'content-location': 'https://10.5.2.69:35357/v3/role_assignments?user.id=e2705ac2e28044089be934acdb88cef1&project.id=bacfe18b08fe400d84e2b9b042f44fbe'}
        Body: b'{"error":{"code":403,"message":"You are not authorized to perform the requested action: identity:list_role_assignments.","title":"Forbidden"}}\n' _log_request_full /home/ubuntu/git/charmed-openstack-tester/.tox/func-target/lib/python3.6/site-packages/tempest/lib/common/rest_client.py:456

Revision history for this message
Felipe Reyes (freyes) wrote :

I'm adding a task for charm-keystone because it owns the policy.json that could be breaking the assumptions of tempest when creating the users.

Revision history for this message
Felipe Reyes (freyes) wrote :

setUpClass (octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest)
----------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/ubuntu/git/charmed-openstack-tester/.tox/func-target/lib/python3.6/site-packages/tempest/test.py", line 168, in setUpClass
    raise value.with_traceback(trace)

      File "/home/ubuntu/git/charmed-openstack-tester/.tox/func-target/lib/python3.6/site-packages/tempest/test.py", line 153, in setUpClass
    cls.setup_credentials()

      File "/home/ubuntu/git/charmed-openstack-tester/.tox/func-target/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py", line 153, in setup_credentials
    **params)['role_assignments']

      File "/home/ubuntu/git/charmed-openstack-tester/.tox/func-target/lib/python3.6/site-packages/tempest/lib/services/identity/v3/role_assignments_client.py", line 46, in list_role_assignments
    resp, body = self.get(url)

      File "/home/ubuntu/git/charmed-openstack-tester/.tox/func-target/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 314, in get
    return self.request('GET', url, extra_headers, headers)

      File "/home/ubuntu/git/charmed-openstack-tester/.tox/func-target/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 703, in request
    self._error_checker(resp, resp_body)

      File "/home/ubuntu/git/charmed-openstack-tester/.tox/func-target/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 804, in _error_checker
    raise exceptions.Forbidden(resp_body, resp=resp)

    tempest.lib.exceptions.Forbidden: Forbidden
Details: {'code': 403, 'message': 'You are not authorized to perform the requested action: identity:list_role_assignments.', 'title': 'Forbidden'}

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

This sounds like the policy change issue that came in in wallaby: https://docs.openstack.org/releasenotes/octavia/wallaby.html, particularly the bit:

"Legacy Octavia Advanced RBAC policies will continue to function as before as long as the [oslo_policy] enforce_scope = False and enforce_new_defaults = False settings are present (this is the current oslo.policy default). However, we highly recommend you update your user roles to follow the new keystone default roles and start using scoped tokens as appropriate. See the Octavia Policies administration guide for more information."

Revision history for this message
Felipe Reyes (freyes) wrote : Re: [Bug 1963685] Re: You are not authorized to perform the requested action: identity:list_role_assignments.

On Mon, 2022-03-07 at 10:26 +0000, Alex Kavanagh wrote:
> This sounds like the policy change issue that came in in wallaby:
> https://docs.openstack.org/releasenotes/octavia/wallaby.html,
> particularly the bit:
>
> "Legacy Octavia Advanced RBAC policies will continue to function as
> before as long as the [oslo_policy] enforce_scope = False and
> enforce_new_defaults = False settings are present (this is the
> current
> oslo.policy default). However, we highly recommend you update your
> user
> roles to follow the new keystone default roles and start using scoped
> tokens as appropriate. See the Octavia Policies administration guide
> for
> more information."

I came through this, but I moved on from due to the bit "this is the
current oslo.policy default", I'm deploying a focal-victoria cloud to
check if the test passes, if it does, I will definitively circle back
to this and explicitly set these keys to false in octavia.conf.

Revision history for this message
Felipe Reyes (freyes) wrote :

Deploying devstack using this configuration file ( https://gist.github.com/freyes/966393e381316e66b00acc0282ed43db ) and adding the policy.json file generated by the charm (replacing the services project id and the domain id) gives no errors when running octavia tests.

Revision history for this message
Felipe Reyes (freyes) wrote :

Exploring the differences in the tempest configuration now.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-keystone (master)
Changed in charm-keystone:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-octavia (master)
Changed in charm-octavia:
status: New → In Progress
Felipe Reyes (freyes)
Changed in charm-keystone:
importance: Undecided → High
Changed in charm-octavia:
importance: Undecided → High
Changed in charm-keystone:
assignee: nobody → Felipe Reyes (freyes)
Changed in charm-octavia:
assignee: nobody → Felipe Reyes (freyes)
Revision history for this message
Alexander Litvinov (alitvinov) wrote (last edit ):

Facing this issue on fresh deployment of jammy/yoga, all octavia tests fail with the following traceback.

Subsribing ~field-high

octavia_tempest_plugin.tests.api.v2.test_pool.PoolAPITest.test_HTTPS_RR_pool_delete
-----------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/snap/fcbtest/42/.rally/verification/verifier-d60afffa-8574-412b-8e38-a955ff090098/repo/tempest/test.py", line 168, in setUpClass
    raise value.with_traceback(trace)
  File "/home/ubuntu/snap/fcbtest/42/.rally/verification/verifier-d60afffa-8574-412b-8e38-a955ff090098/repo/tempest/test.py", line 153, in setUpClass
    cls.setup_credentials()
  File "/snap/fcbtest/42/lib/python3.10/site-packages/octavia_tempest_plugin/tests/test_base.py", line 154, in setup_credentials
    roles = cls.os_admin.role_assignments_client.list_role_assignments(
  File "/home/ubuntu/snap/fcbtest/42/.rally/verification/verifier-d60afffa-8574-412b-8e38-a955ff090098/repo/tempest/lib/services/identity/v3/role_assignments_client.py", line 46, in list_role_assignments
    resp, body = self.get(url)
  File "/home/ubuntu/snap/fcbtest/42/.rally/verification/verifier-d60afffa-8574-412b-8e38-a955ff090098/repo/tempest/lib/common/rest_client.py", line 314, in get
    return self.request('GET', url, extra_headers, headers)
  File "/home/ubuntu/snap/fcbtest/42/.rally/verification/verifier-d60afffa-8574-412b-8e38-a955ff090098/repo/tempest/lib/common/rest_client.py", line 720, in request
    self._error_checker(resp, resp_body)
  File "/home/ubuntu/snap/fcbtest/42/.rally/verification/verifier-d60afffa-8574-412b-8e38-a955ff090098/repo/tempest/lib/common/rest_client.py", line 821, in _error_checker
    raise exceptions.Forbidden(resp_body, resp=resp)
tempest.lib.exceptions.Forbidden: Forbidden
Details: {'code': 403, 'message': 'You are not authorized to perform the requested action: identity:list_role_assignments.', 'title': 'Forbidden'}

policy.json from keystone unit:
https://pastebin.canonical.com/p/MH5YwvX5dj/

Revision history for this message
Bartosz Woronicz (mastier1) wrote :

I encounter the same issue on Yoga/Jammy
What shall we do to make it work ?

Revision history for this message
Bartosz Woronicz (mastier1) wrote :

this command is run only for debug purposes, to make the test passing

we may add to tempest.conf
[octavia]
log_user_roles = False

or modify the test to add the proper role

Revision history for this message
Corey Bryant (corey.bryant) wrote :

What Bartosz says in comment #11 is correct. tempest.conf needs updating with 'log_user_roles = False' to bypass the 'list_role_assignments' call.

Bas fixed this in https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/867810 (Thanks Bas!)

The tempest tests generate a user and I've found no other octavia tempest config [1] that will give the tempest-generated user the admin privileges required for the keystone policy [2] to allow running list_role_assignments. You can set the admin_role=Admin, and it does take effect, however you can't set the domain=admin_domain or project=admin.

[1] https://github.com/openstack/octavia-tempest-plugin/blob/master/octavia_tempest_plugin/config.py

[2] /etc/keystone/policy.json (this is bionic-ussuri)
    "admin_required": "role:Admin",
    "cloud_admin": "rule:admin_required and (is_admin_project:True or domain_id:c8fd27a31e23422ab7d3b0a6962f6049 or project_id:4f7d8068c88541c688e2fb849d7cb729)",
(where c8fd27a31e23422ab7d3b0a6962f6049==admin_doman, and 4f7d8068c88541c688e2fb849d7cb729==services)
"admin_on_project_filter": "rule:admin_required and project_id:%(scope.project.id)s",
"identity:list_role_assignments": "rule:cloud_admin or rule:admin_on_domain_filter or rule:admin_on_project_filter",

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.