When using 'use_dynamic_credentials = false' credentials are being modified somewhere in tempest lib

[Rick Bartra]

When running the Patrole Tempest plugin and testing networking tests that require the Neutron Tempest plugin, the test fail to run when Tempest is configured with [auth] use_dynamic_credentials = false'

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "tempest/test.py", line 213, in setUpClass
        six.reraise(etype, value, trace)
      File "tempest/test.py", line 201, in setUpClass
        cls.setup_clients()
      File "/opt/stack/patrole/patrole_tempest_plugin/rbac_utils.py", line 170, in setup_clients
        cls._override_role(False)
      File "/opt/stack/patrole/patrole_tempest_plugin/rbac_utils.py", line 321, in _override_role
        LOG.exception(exp)
      File "/opt/stack/tempestV/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
        self.force_reraise()
      File "/opt/stack/tempestV/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
        six.reraise(self.type_, self.value, self.tb)
      File "/opt/stack/patrole/patrole_tempest_plugin/rbac_utils.py", line 314, in _override_role
        target_roles)
      File "/opt/stack/patrole/patrole_tempest_plugin/rbac_utils.py", line 385, in _list_and_clear_user_roles_on_project
        cls._project_id, cls._user_id)['roles']
      File "tempest/lib/services/identity/v3/roles_client.py", line 95, in list_user_roles_on_project
        (project_id, user_id))
      File "tempest/lib/common/rest_client.py", line 298, in get
        return self.request('GET', url, extra_headers, headers)
      File "tempest/lib/common/rest_client.py", line 680, in request
        self._error_checker(resp, resp_body)
      File "tempest/lib/common/rest_client.py", line 786, in _error_checker
        raise exceptions.NotFound(resp_body, resp=resp)
    tempest.lib.exceptions.NotFound: Object not found
    Details: {u'message': u'Could not find user: None.', u'code': 404, u'title': u'Not Found'}

Somewhere in Tempest, the 'user_id' and 'tenant_id' are unset
and results in the tests not being executed: "Could not find user: None."

Both user_id and tenant_id are needed in Patrole rbac_utils: https://github.com/openstack/patrole/blob/master/patrole_tempest_plugin/rbac_utils.py#L163

It appears that the manager.credentials are modified when clients.Manager is called: https://github.com/openstack/patrole/blob/master/patrole_tempest_plugin/tests/api/network/rbac_base.py#L70 ... the call ultimately traces back to Tempest.

Please see https://review.opendev.org/#/c/676979/ for reference as a workaround in Patrole, but the correct fix is to find the issue in Tempest that is setting the credentials 'user_id' and 'tenant_id' to None.

[Rick Bartra (rb560u)]

Looks like the "credentials.user_id" and "credentials.tenant_id" is being dropped here: https://github.com/openstack/tempest/blob/master/tempest/lib/auth.py#L251

[Rick Bartra (rb560u)]

'user_id' and 'tenant_id' are not part of Credentials self._initial (https://github.com/openstack/tempest/blob/master/tempest/lib/auth.py#L667) and therefore are not restored when self.clear_auth is called:

The reason why they are not part of self._initial is because with pre-provisioned creds, these fields are not in the accounts.yaml that we use. They can be added to the accounts.yaml manually, but that is overkill.

When self.clear_auth is called:
https://github.com/openstack/tempest/blob/master/tempest/lib/auth.py#L251
the credentials are reset, and only those fields that belong to the Credentials self._initial are restored in the '_apply_credentials' method: https://github.com/openstack/tempest/blob/master/tempest/lib/auth.py#L678

The Credentials reset method could be re-written in https://github.com/openstack/tempest/blob/master/tempest/lib/auth.py to force self._initial to include all keys that were defined in the Credentials object:

    def reset(self):
        # First delete all known attributes
        for key in self.ATTRIBUTES:
            if getattr(self, key) is not None:
                # add all known attributes to self._initial so that no cred attributes are
                # missing when _apply_credentials is called
                self._initial.update({key: getattr(self, key)})
                delattr(self, key)
        # Then re-apply initial setup
        self._apply_credentials(self._initial)

Any other suggestion on how to fix in Tempest would be helpful. I do not think what I am proposing above will be accepted in Tempest and especially into Tempest lib. Until a fix is made in Tempest, can this please be merged as a workaround then later reverted when someone can fix the 'bug' in Tempest? I am not completely sure this is even a bug in Tempest especially given this: https://github.com/openstack/tempest/blob/master/tempest/lib/auth.py#L664

[Martin Kopec (mkopec)]

https://review.opendev.org/#/c/758551/

[Martin Kopec (mkopec)]

https://review.opendev.org/c/openstack/patrole/+/758551 got merged.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/patrole 0.12.0 release.