tempest cleanup unauthorized error

[Uemit Seren]

We are running OSP12 (Pike) with domain specific configuration.
In order to validate the OS deployment we are running tempest against it with following active plugins:

+----------------+--------------------------------------------------------+
| Name | EntryPoint |
+----------------+--------------------------------------------------------+
| keystone_tests | keystone_tempest_plugin.plugin:KeystoneTempestPlugin |
| heat_tests | heat_integrationtests.plugin:HeatTempestPlugin |
| ironic_tests | ironic_tempest_plugin.plugin:IronicTempestPlugin |
| neutron_tests | neutron.tests.tempest.plugin:NeutronTempestPlugin |
| horizon | tempest_horizon.plugin:HorizonTempestPlugin |
| cinder_tests | cinder.tests.tempest.plugin:CinderTempestPlugin |
| designate | designate_tempest_plugin.plugin:DesignateTempestPlugin |
+----------------+--------------------------------------------------------+

It seems that the heatintegrationtests (HeatTempestPlugin) are nor properly cleaning up after themselves (maybe a separate bug) because after the tempest run there are around 12 projects that belong to the heat_stack domain. When we run tempest cleanup (we ran tempest cleanup --save-state before the tempest run) we get following error:

Begin cleanup
Process 11 projects
Cleaning project: 19d30893644445fc95eddf10bede6acf-a7487811-5874-4869-a5b3-91b3f08
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 97, in take_action
    self._cleanup()
  File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 152, in _cleanup
    self._clean_project(project)
  File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 193, in _clean_project
    **kwargs))
  File "/usr/lib/python2.7/site-packages/tempest/common/credentials_factory.py", line 298, in get_credentials
    **params)
  File "/usr/lib/python2.7/site-packages/tempest/lib/auth.py", line 648, in get_credentials
    creds = auth_provider.fill_credentials()
  File "/usr/lib/python2.7/site-packages/tempest/lib/auth.py", line 124, in fill_credentials
    auth_data = self.get_auth()
  File "/usr/lib/python2.7/site-packages/tempest/lib/auth.py", line 150, in get_auth
    self.set_auth()
  File "/usr/lib/python2.7/site-packages/tempest/lib/auth.py", line 159, in set_auth
    self.cache = self._get_auth()
  File "/usr/lib/python2.7/site-packages/tempest/lib/auth.py", line 314, in _get_auth
    token, auth_data = auth_func(**auth_params)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/identity/v3/token_client.py", line 183, in get_token
    body = self.auth(**kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/identity/v3/token_client.py", line 132, in auth
    resp, body = self.post(self.auth_url, body=body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 279, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/identity/v3/token_client.py", line 163, in request
    raise exceptions.Unauthorized(resp_body['error']['message'])
Unauthorized: Unauthorized
Details: The request you have made requires authentication.

In the keystone log we get following lines:

2018-04-24 14:42:44.928 23 INFO keystone.common.wsgi [req-c6c7c1db-cce9-4e70-b3f9-eb13f32bc68f d111fd4cf634431cba4b33d8b7d293e2 9894873bc8c3452b81dd196e99064802 - default default] GET http://192.168.24.207:35357/v3/users?name=admin
2018-04-24 14:42:44.970 24 INFO keystone.common.wsgi [req-e4ca2aff-c398-4ee8-9a21-c6b3669d00bb d111fd4cf634431cba4b33d8b7d293e2 9894873bc8c3452b81dd196e99064802 - default default] GET http://192.168.24.207:35357/v3/role_assignments?scope.project.id=9894873bc8c3452b81dd196e99064802
2018-04-24 14:42:44.992 18 INFO keystone.common.wsgi [req-0963cc71-5dc6-42b9-91a6-a27ce5d2e725 d111fd4cf634431cba4b33d8b7d293e2 9894873bc8c3452b81dd196e99064802 - default default] GET http://192.168.24.207:35357/v3/roles
2018-04-24 14:42:45.013 22 INFO keystone.common.wsgi [req-81ba78ee-72fa-4042-bac2-d6f9fbbb14ea d111fd4cf634431cba4b33d8b7d293e2 9894873bc8c3452b81dd196e99064802 - default default] GET http://192.168.24.207:35357/v3/projects
2018-04-24 14:42:45.036 25 INFO keystone.common.wsgi [req-e91c9e7c-72a6-4d70-9235-d5a9cd049a19 d111fd4cf634431cba4b33d8b7d293e2 9894873bc8c3452b81dd196e99064802 - default default] GET http://192.168.24.207:35357/v3/projects/174d1e76b2bd4415aff46b300a4a8852/users/d111fd4cf634431cba4b33d8b7d293e2/roles
2018-04-24 14:42:45.061 29 INFO keystone.common.wsgi [req-931af35a-ce7c-4dfb-b2ee-6deaff3c6009 - - - - -] POST http://172.16.52.19:5000/v3/auth/tokens
2018-04-24 14:42:45.068 29 WARNING keystone.auth.core [req-931af35a-ce7c-4dfb-b2ee-6deaff3c6009 - - - - -] Could not find project: 19d30893644445fc95eddf10bede6acf-a7487811-5874-4869-a5b3-91b3f08.: ProjectNotFound: Could not find project: 19d30893644445fc95eddf10bede6acf-a7487811-5874-4869-a5b3-91b3f08.
2018-04-24 14:42:45.069 29 WARNING keystone.common.wsgi [req-931af35a-ce7c-4dfb-b2ee-6deaff3c6009 - - - - -] Authorization failed. The request you have made requires authentication. from 192.168.25.23: Unauthorized: The request you have made requires authentication.

Deleting the project manually by hand with openstack project delete NAME works fine with the admin user (which is also used for the tempest run)

[Martin Kopec (mkopec)]

Steps to reproduce:

(overcloud) [stack@undercloud-0 cloud]$ openstack project list
+----------------------------------+----------+
| ID | Name |
+----------------------------------+----------+
| 574113dc62b641bea106184056116212 | demo |
| 5d733d38c049440bb795d61a772e4791 | admin |
| b01b7d2e11914c1e86c62b5015c37f15 | alt_demo |
| ff1110058f644c98a231fc509f299ad5 | service |
+----------------------------------+----------+

(overcloud) [stack@undercloud-0 cloud]$ cat etc/tempest.conf | grep "auth]" -A 7
[auth]
tempest_roles = admin
admin_username = admin
admin_project_name = admin
admin_domain_name = Default
use_dynamic_credentials = true
admin_password = vh6FCgSmVQLevRtqhZcVgtPIs
admin_project_id = 5d733d38c049440bb795d61a772e4791

### Notice that my admin user has 'Default' as admin_domain_name.

(overcloud) [stack@undercloud-0 cloud]$ openstack domain list
+----------------------------------+------------+---------+--------------------+
| ID | Name | Enabled | Description |
+----------------------------------+------------+---------+--------------------+
| 6461240771034eeaa5b2ff0615943ece | heat_stack | True | |
| default | Default | True | The default domain |
+----------------------------------+------------+---------+--------------------+

### Let's create a project with a domain name assigned which is different from admin's domain

(overcloud) [stack@undercloud-0 cloud]$ openstack project create --domain heat_stack test
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | |
| domain_id | 6461240771034eeaa5b2ff0615943ece |
| enabled | True |
| id | d06b100928d6468ebe42b5ceea0db8d5 |
| is_domain | False |
| name | test |
| parent_id | 6461240771034eeaa5b2ff0615943ece |
| tags | [] |
+-------------+----------------------------------+
(overcloud) [stack@undercloud-0 cloud]$ openstack project list
+----------------------------------+----------+
| ID | Name |
+----------------------------------+----------+
| 574113dc62b641bea106184056116212 | demo |
| 5d733d38c049440bb795d61a772e4791 | admin |
| b01b7d2e11914c1e86c62b5015c37f15 | alt_demo |
| d06b100928d6468ebe42b5ceea0db8d5 | test |
| ff1110058f644c98a231fc509f299ad5 | service |
+----------------------------------+----------+
(overcloud) [stack@undercloud-0 cloud]$ tempest cleanup
Begin cleanup
Process 1 projects
Cleaning project: test
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/cmd/cleanup.py", line 102, in take_action
    self._cleanup()
  File "/usr/lib/python3.6/site-packages/tempest/cmd/cleanup.py", line 159, in _cleanup
    self._clean_project(project)
  File "/usr/lib/python3.6/site-packages/tempest/cmd/cleanup.py", line 201, in _clean_project
    **kwargs))
  File "/usr/lib/p...

[Martin Kopec (mkopec)]

The unauthorized error is thrown not during deleting the project itself, but before when the tool is assigning [1][2] an admin role to the project for a user running the cleanup (in my case in the previous comment #1, the tool is assigning admin role to the test project for admin user). Apparently that's unauthorized.

I guess that this step (assigning admin role) is there for cases when the user running cleanup doesn't have permissions (roles) by default to delete some projects - therefore the cleanup grants those permissions for the time of deleting the project and then taking the permissions back [3].

[1] https://github.com/openstack/tempest/blob/de20df69549d9aa8fc13554dc196deac60f2e34b/tempest/cmd/cleanup.py#L173
[2] https://github.com/openstack/tempest/blob/de20df69549d9aa8fc13554dc196deac60f2e34b/tempest/cmd/cleanup.py#L275
[3] https://github.com/openstack/tempest/blob/de20df69549d9aa8fc13554dc196deac60f2e34b/tempest/cmd/cleanup.py#L191

[Martin Kopec (mkopec)]

To be more precise than in the previous comment, the error is thrown when the credentials are obtained [1]. It's because the tool is trying to authenticate using the project name which is gonna be deleted. In this case the project is in a different domain than the user - the user's domain has no knowledge of the project - so it fails. It would make more sense if the tool used CONF.auth.admin_project_name instead of the project name which is about to be deleted. CLI client does that the same way, it authenticates using user's credentials (including admin_project_name).

I also looked at the _add_admin method [3] and I can't find a reason/use-case why it's there. The user who's gonna run tempest cleanup has to be an admin otherwise not even project listing will be successful.

[1] https://github.com/openstack/tempest/blob/420155c2348b9e581dd4ffb73c9d88e1488b6f9b/tempest/cmd/cleanup.py#L215
[2] https://github.com/openstack/tempest/blob/420155c2348b9e581dd4ffb73c9d88e1488b6f9b/tempest/cmd/cleanup.py#L212-L216
[3] https://github.com/openstack/tempest/blob/420155c2348b9e581dd4ffb73c9d88e1488b6f9b/tempest/cmd/cleanup.py#L275

[Ghanshyam Mann (ghanshyammann)]

_clean_project() is using project['name'] for creating the creadential manager[1] is fine here as this method cleanup the project's resources do not delete the project itself. So flow is like below
- cleanup() function assigned the admin role to project to be cleaned up if not admin
- _clean_project() will go to each regoster service and delete their resources under that project
- cleanup() will call run on each self.global_services where project is added as one of the service. This will delete the project. And in this we use the admin_mgr[2].

Seems there is no issue on autherization.

In you log, i saw ProjectNotFound exception from keystone, are you sure that you are not trying to delete the already deleted project?

[1] https://github.com/openstack/tempest/blob/420155c2348b9e581dd4ffb73c9d88e1488b6f9b/tempest/cmd/cleanup.py#L215

[2] https://github.com/openstack/tempest/blob/420155c2348b9e581dd4ffb73c9d88e1488b6f9b/tempest/cmd/cleanup.py#L183

[Martin Kopec (mkopec)]

I get the current flow, what I don't understand is, why we need to create another temporary manager [1] when we already have an admin one [2]. Why don't use just the admin one?
In that case we:
  1. don't need _add_admin method, _remove_admin_user_roles method and _remove_admin_role method
  2. we don't need to request new credentials with every project which is about to be deleted
  3. the whole cleanup process will be faster (less methods, less calls, no requesting credentials repeatedly)

[1] https://github.com/openstack/tempest/blob/master/tempest/cmd/cleanup.py#L215
[2] https://github.com/openstack/tempest/blob/master/tempest/cmd/cleanup.py#L128

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/686738

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/686738
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=7ca8602380d5959c273799fb0fae7d2b32cde9ea
Submitter: Zuul
Branch: master

commit 7ca8602380d5959c273799fb0fae7d2b32cde9ea
Author: Martin Kopec <email address hidden>
Date: Fri Oct 4 14:13:59 2019 +0000

    tempest cleanup - use admin_mgr only

    Avoid using temporary managers and use only the admin one. The tool
    has been using the admin_mgr for listing projects, roles, for listing
    all resources during initializing a saved state so why don't use it also
    for deleting all of the resources?
    This wil solve the Unauthorized issue happening when the tool
    was requesting credentials for the project from a different
    domain.

    Closes-bug: #1766582

    Change-Id: Ibb5599ce48712b94dbf591e4a30cf3906cb5cdde

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 23.0.0 release.