Identity error on tempest tests

[Konstantinos Kaskavelis]

We have a test run in which two tempest tests failed (test_get_instance_action, test_list_instance_actions) with the following error:

Traceback (most recent call last):
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/test.py", line 181, in setUpClass
    raise value.with_traceback(trace)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/test.py", line 166, in setUpClass
    cls.setup_credentials()
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/api/compute/base.py", line 71, in setup_credentials
    super(BaseV2ComputeTest, cls).setup_credentials()
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/test.py", line 388, in setup_credentials
    credential_type=credentials_type)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/test.py", line 688, in get_client_manager
    cred_provider = cls._get_credentials_provider()
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/test.py", line 665, in _get_credentials_provider
    force_tenant_isolation=force_tenant_isolation)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/common/credentials_factory.py", line 141, in get_credentials_provider
    **get_dynamic_provider_params(identity_version))
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/lib/common/dynamic_creds.py", line 119, in __init__
    self.creds_domain_name)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/lib/common/cred_client.py", line 274, in get_creds_client
    roles_client, domains_client, project_domain_name)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/lib/common/cred_client.py", line 150, in __init__
    name=domain_name)['domains'][0]
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/lib/services/identity/v3/domains_client.py", line 58, in list_domains
    resp, body = self.get(url)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/lib/common/rest_client.py", line 314, in get
    return self.request('GET', url, extra_headers, headers)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/lib/common/rest_client.py", line 688, in request
    body=body, chunked=chunked)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/lib/common/rest_client.py", line 564, in _request
    method, url, headers, body, self.filters)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/lib/auth.py", line 186, in auth_request
    filters, method, url, headers, body)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/lib/auth.py", line 275, in _decorate_request
    auth_data = self.get_auth()
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/lib/auth.py", line 148, in get_auth
    self.set_auth()
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/lib/auth.py", line 157, in set_auth
    self.cache = self._get_auth()
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/lib/auth.py", line 312, in _get_auth
    token, auth_data = auth_func(**auth_params)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/lib/services/identity/v3/token_client.py", line 190, in get_token
    body = self.auth(**kwargs)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/lib/services/identity/v3/token_client.py", line 143, in auth
    resp, body = self.post(self.auth_url, body=body)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/lib/common/rest_client.py", line 299, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-80131ef5-6192-42bd-8987-7f9e1aabb569/repo/tempest/lib/services/identity/v3/token_client.py", line 173, in request
    'Unexpected status code {0}'.format(resp.status))
tempest.lib.exceptions.IdentityError: Got identity error
Details: Unexpected status code 502

Test run:

https://solutions.qa.canonical.com/testruns/testRun/04e8a8ac-dae2-488a-bf6f-cb10047339eb

Test report:

https://oil-jenkins.canonical.com/artifacts/04e8a8ac-dae2-488a-bf6f-cb10047339eb/generated/generated/openstack/tempest_result.html

[Martin Kopec (mkopec)]

Hm, I see tempest is executed via rally, we don't maintain that. Were you able to reproduce the issue when running tempest directly? I cannot open the logs.

[Konstantinos Kaskavelis (kaskavel)]

Hello, you should have access to logs in this link: https://oil-jenkins.canonical.com/artifacts/04e8a8ac-dae2-488a-bf6f-cb10047339eb/index.html

Otherwise I would be happy to attach any logs you may require for investigating.

[Lukas Piwowarski (lukas-piwowarski)]

Hi Konstantin, the link is still not accessible ("Either you have not been granted access to this resource or your entitlement has timed out. Please try again.").

I'm sorry maybe you made it available only for Martin (?).

[Konstantinos Kaskavelis (kaskavel)]

Hello, sorry for the trouble,

I 've uploaded the crashdump from the openstack layer here:

https://people.canonical.com/~kaskavel/1982007/

Any other information that could be useful for this, I 'd be happy to attach it there.

[Lukas Piwowarski (lukas-piwowarski)]

Hi, I'm sorry but I was unable to resolve this issue. For me it came done to the fact that tempest received error code 502 (Bad gateway). I have tested both tests on my local environment (using devstack) and they worked fine. Because of that it seems to me that this is not really an issue with tempest but rather an issue with the environment.

I think it would help to see:
- tempest.conf
- Log file that contains tempest output (I couldn't find it here: https://people.canonical.com/~kaskavel/1982007/)

What I am curious about is whether tempest contacted the correct Keystone API endpoint. If it did not make call to correct Keystone API endpoint then it is tempest issue. In contrast, if it did contact the correct Kyestone API endpoint then it is probably not tempest issue.