tempest fails on overcloud keystone admin tenant error

[wes hayutin]

scenario jobs 1-4 are failing on

2017-09-23 07:59:50 | 2017-09-23 07:59:50.066 15459 INFO tempest.lib.common.rest_client [req-a1d17e2b-5985-423e-82e0-d18cd7d7baa6 ] Request (main): 404 GET http://192.168.24.12:35357/v2.0/tenants 0.230s
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 CRITICAL tempest [-] Unhandled error: NotFound: Object not found
2017-09-23 07:59:50 | Details: {u'message': u'(http://192.168.24.12:35357/v2.0/tenants): The resource could not be found.', u'code': 404, u'title': u'Not Found'}
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest Traceback (most recent call last):
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest File "/usr/bin/discover-tempest-config", line 10, in <module>
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest sys.exit(main())
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest File "/usr/lib/python2.7/site-packages/config_tempest/config_tempest.py", line 177, in main
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest clients = ClientManager(conf, not args.non_admin)
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest File "/usr/lib/python2.7/site-packages/config_tempest/config_tempest.py", line 548, in __init__
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest tenant_name)['id']
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/common/identity.py", line 24, in get_tenant_by_name
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest tenants = client.list_tenants()['tenants']
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/services/identity/v2/tenants_client.py", line 70, in list_tenants
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest resp, body = self.get(url)
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 285, in get
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest return self.request('GET', url, extra_headers, headers)
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 659, in request
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest self._error_checker(resp, resp_body)
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 765, in _error_checker
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest raise exceptions.NotFound(resp_body, resp=resp)
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest NotFound: Object not found
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest Details: {u'message': u'(http://192.168.24.12:35357/v2.0/tenants): The resource could not be found.', u'code': 404, u'title': u'Not Found'}
2017-09-23 07:59:50 | 2017-09-23 07:59:50.067 15459 ERROR tempest

LOG: http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-scenario004-multinode-oooq/3600fc5/logs/undercloud/home/jenkins/tempest_output.log.txt.gz#_2017-09-23_07_59_50

Keystone errors in:

http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-scenario004-multinode-oooq/3600fc5/logs/subnode-2/var/log/keystone/keystone.log.txt.gz?level=ERROR

ERROR keystoneclient.common.cms [req-668623c9-7517-4158-bd05-510441765b96 c37b0305bafc468c97ac6426787af855 b5409f6da7c7496899930fbd68097b3d - default default] Signing error: Unable to load certificate - ensure you have configured PKI with "keystone-manage pki_setup"
2017-09-23 07:46:41.173 53334 ERROR keystone.common.wsgi [req-668623c9-7517-4158-bd05-510441765b96 c37b0305bafc468c97ac6426787af855 b5409f6da7c7496899930fbd68097b3d - default default] Command 'openssl' returned non-zero exit status 3: CalledProcessError: Command 'openssl' returned non-zero exit status 3
2017-09-23 07:46:41.173 53334 ERROR keystone.common.wsgi Traceback (most recent call last):
2017-09-23 07:46:41.173 53334 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 228, in __call__
2017-09-23 07:46:41.173 53334 ERROR keystone.common.wsgi result = method(req, **params)
2017-09-23 07:46:41.173 53334 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/controller.py", line 94, in inner
2017-09-23 07:46:41.173 53334 ERROR keystone.common.wsgi return f(self, request, *args, **kwargs)
2017-09-23 07:46:41.173 53334 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/auth/controllers.py", line 350, in revocation_list
2017-09-23 07:46:41.173 53334 ERROR keystone.common.wsgi CONF.signing.keyfile)
2017-09-23 07:46:41.173 53334 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
2017-09-23 07:46:41.173 53334 ERROR keystone.common.wsgi signing_key_file_name, message_digest=message_digest)
2017-09-23 07:46:41.173 53334 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
2017-09-23 07:46:41.173 53334 ERROR keystone.common.wsgi raise subprocess.CalledProcessError(retcode, 'openssl')
2017-09-23 07:46:41.173 53334 ERROR keystone.common.wsgi CalledProcessError: Command 'openssl' returned non-zero exit status 3

[Arx Cruz (arxcruz)]

From my debug it seems that periodic job are no longer enabling keystone v2.0.

For example, the (http://192.168.24.12:35357/v2.0/tenants now is (http://192.168.24.12:35357/v3/projects, so it fails.

I did some changes to enable v3 support on python-tempestconf maintaining the v2.0 support, however, there are several places that still relies on v2, which makes the code harder to fix, I'll talk with Luigi and Chandan Kumar tomorrow to discuss the removal of v2.0 in favor of v3.0

[Arx Cruz (arxcruz)]

https://review.gerrithub.io/#/c/375832/ will fix the problem