Tempest fails in container jobs trying to reach a keystone v2 URI

[Martin André]

Since the tripleo promotion earlier today the CI started picking up newly built containers and we saw failures with tempest trying to talk to keystone v2 API:

2017-11-03 11:30:51 | 2017-11-03 11:30:51.175 14897 DEBUG config_tempest.config_tempest [-] Setting [identity] uri = http://192.168.24.12:5000/v2.0 set /usr/lib/python2.7/site-packages/config_tempest/config_tempest.py:598
2017-11-03 11:30:51 | 2017-11-03 11:30:51.175 14897 DEBUG config_tempest.config_tempest [-] Setting [identity] uri_v3 = http://192.168.24.12:5000/v3 set /usr/lib/python2.7/site-packages/config_tempest/config_tempest.py:598
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 INFO tempest.lib.common.rest_client [req-336d49fb-e3bd-47d3-ba0b-5f5906107eec ] Request (main): 404 POST http://192.168.24.12:5000/v2.0/tokens
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 CRITICAL tempest [-] Unhandled error: IdentityError: Got identity error
2017-11-03 11:30:51 | Details: Unexpected status code 404
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest Traceback (most recent call last):
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest File "/usr/bin/discover-tempest-config", line 10, in <module>
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest sys.exit(main())
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest File "/usr/lib/python2.7/site-packages/config_tempest/config_tempest.py", line 177, in main
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest clients = ClientManager(conf, not args.non_admin)
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest File "/usr/lib/python2.7/site-packages/config_tempest/config_tempest.py", line 548, in __init__
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest tenant_name)['id']
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/common/identity.py", line 20, in get_tenant_by_name
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest tenants = client.list_tenants()['tenants']
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/services/identity/v2/tenants_client.py", line 70, in list_tenants
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest resp, body = self.get(url)
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 285, in get
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest return self.request('GET', url, extra_headers, headers)
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 644, in request
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest body=body, chunked=chunked)
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 533, in _request
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest method, url, headers, body, self.filters)
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/auth.py", line 188, in auth_request
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest filters, method, url, headers, body)
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/auth.py", line 276, in _decorate_request
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest auth_data = self.get_auth()
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/auth.py", line 150, in get_auth
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest self.set_auth()
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/auth.py", line 159, in set_auth
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest self.cache = self._get_auth()
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/auth.py", line 313, in _get_auth
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest token, auth_data = auth_func(**auth_params)
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/services/identity/v2/token_client.py", line 123, in get_token
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest body = self.auth(user, password, tenant)
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/services/identity/v2/token_client.py", line 69, in auth
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest resp, body = self.post(self.auth_url, body=body)
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 270, in post
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest return self.request('POST', url, extra_headers, headers, body, chunked)
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/services/identity/v2/token_client.py", line 117, in request
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest 'Unexpected status code {0}'.format(resp.status))
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest IdentityError: Got identity error
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest Details: Unexpected status code 404
2017-11-03 11:30:51 | 2017-11-03 11:30:51.193 14897 ERROR tempest

http://logs.openstack.org/09/514809/2/gate/legacy-tripleo-ci-centos-7-scenario002-multinode-oooq-container/ccf39c8/logs/undercloud/home/zuul/tempest_output.log.txt.gz#_2017-11-03_11_30_51

[Martin André (mandre)]

It appears that the failing jobs had an older tempest package on the undercloud. This is likely due to the job running at the same time the promotion happened. The undercloud was installed prior to the promotion, and the containers that were pulled later were from the promoted batch.