Config_tempest fails to gather extensions from nova in featureset001/53 OVB job in master

[Gabriele Cerami]

Deploying the featureset in vexxhost enviroment works correctly, but logs at

https://logs.rdoproject.org/39/17339/1/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost/7d119e0/job-output.txt.gz#_2018-11-13_15_04_04_860770

show that config_tempest is unable to access the url in compute node.
The root cause may be an undiscovered error during deployment.

2018-11-13 15:04:04.860770 | primary | 2018-11-13 15:03:11 | 2018-11-13 15:03:11.117 34 ERROR tempest File "/usr/lib/python2.7/site-packages/config_tempest/services/services.py", line 44, in __init__
2018-11-13 15:04:04.860878 | primary | 2018-11-13 15:03:11 | 2018-11-13 15:03:11.117 34 ERROR tempest self.discover()
2018-11-13 15:04:04.861058 | primary | 2018-11-13 15:03:11 | 2018-11-13 15:03:11.117 34 ERROR tempest File "/usr/lib/python2.7/site-packages/config_tempest/services/services.py", line 106, in discover
2018-11-13 15:04:04.861171 | primary | 2018-11-13 15:03:11 | 2018-11-13 15:03:11.117 34 ERROR tempest service.set_extensions()
2018-11-13 15:04:04.861355 | primary | 2018-11-13 15:03:11 | 2018-11-13 15:03:11.117 34 ERROR tempest File "/usr/lib/python2.7/site-packages/config_tempest/services/compute.py", line 26, in set_extensions
2018-11-13 15:04:04.861860 | primary | 2018-11-13 15:03:11 | 2018-11-13 15:03:11.117 34 ERROR tempest body = self.do_get(self.service_url + '/extensions')
2018-11-13 15:04:04.862023 | primary | 2018-11-13 15:03:11 | 2018-11-13 15:03:11.117 34 ERROR tempest File "/usr/lib/python2.7/site-packages/config_tempest/services/base.py", line 61, in do_get
2018-11-13 15:04:04.862106 | primary | 2018-11-13 15:03:11 | 2018-11-13 15:03:11.117 34 ERROR tempest raise e
2018-11-13 15:04:04.862421 | primary | 2018-11-13 15:03:11 | 2018-11-13 15:03:11.117 34 ERROR tempest MaxRetryError: HTTPConnectionPool(host=u'172.17.0.12', port=8774): Max retries exceeded with url: /v2.1/extensions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6606a10e10>: Failed to establish a new connection: [Errno 101] Network is unreachable',))
2018-11-13 15:04:04.862528 | primary | 2018-11-13 15:03:11 | 2018-11-13 15:03:11.117 34 ERROR tempest 

[chandan kumar (chkumar246)]

If we check the traceback here https://logs.rdoproject.org/33/614633/6/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost/3134b57/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-11-15_06_17_23

2018-11-15 06:15:12 | InsecureRequestWarning)
2018-11-15 06:15:12 | 2018-11-15 06:15:12.205 34 DEBUG config_tempest.constants [-] Setting [identity] uri_v3 = https://10.0.0.5:13000/v3 set /usr/lib/python2.7/site-packages/config_tempest/tempest_conf.py:103
2018-11-15 06:15:12 | 2018-11-15 06:15:12.748 34 INFO tempest.lib.common.rest_client [req-3c9c95bc-c207-4480-a6fd-5ad171087441 ] Request (main): 201 POST https://10.0.0.5:13000/v3/auth/tokens
2018-11-15 06:15:13 | 2018-11-15 06:15:13.054 34 INFO tempest.lib.common.rest_client [req-eb91a003-0df3-4247-a4cc-aaee279ee60e ] Request (main): 200 GET https://10.0.0.5:13000/v3/projects 0.305s
2018-11-15 06:15:13 | 2018-11-15 06:15:13.055 34 DEBUG config_tempest.constants [-] Setting [auth] admin_project_id = a07eeec2a6cc44d5bd52ad0fa20dc012 set /usr/lib/python2.7/site-packages/config_tempest/tempest_conf.py:103
2018-11-15 06:15:13 | 2018-11-15 06:15:13.352 34 INFO tempest.lib.common.rest_client [req-997eefd1-7877-4df9-8543-5bc552971d44 ] Request (main): 200 GET https://10.0.0.5:13000/v3/services 0.293s
2018-11-15 06:15:13 | 2018-11-15 06:15:13.663 34 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc8e272a9d0>: Failed to establish a new connection: [Errno 101] Network is unreachable',)': /v2.1/extensions: NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fc8e272a9d0>: Failed to establish a new connection: [Errno 101] Network is unreachable
2018-11-15 06:15:16 | 2018-11-15 06:15:16.669 34 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc8e272ab50>: Failed to establish a new connection: [Errno 101] Network is unreachable',)': /v2.1/extensions: NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fc8e272ab50>: Failed to establish a new connection: [Errno 101] Network is unreachable
2018-11-15 06:17:23 | 2018-11-15 06:17:23.932 34 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc8e272acd0>: Failed to establish a new connection: [Errno 110] Connection timed out',)': /v2.1/extensions: NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fc8e272acd0>: Failed to establish a new connection: [Errno 110] Connection timed out
2018-11-15 06:17:23 | Traceback (most recent call last):
2018-11-15 06:17:23 | File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
2018-11-15 06:17:23 | msg = self.format(record)
2018-11-15 06:17:23 | File "/usr/lib/python2.7/site-packages/oslo_log/handlers.py", line 168, in fo...

[chandan kumar (chkumar246)]

We are hitting it other than vexhost: https://logs.rdoproject.org/66/610966/9/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053/1530b8d/job-output.txt.gz

[chandan kumar (chkumar246)]

Here in this job https://logs.rdoproject.org/32/614632/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-vexxhost/4374e03/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-11-19_02_10_48 the error occured but the tempestconf generated the config.
It might be related to the environment itself.

[Alan Pevec (apevec)]

@Gabriele Is there any new info from debugging on frozen CI nodes?
When did this start happening in *-vexxhost jobs - they were running fine week or so ago?

[chandan kumar (chkumar246)]

It all started after 13th Nov and this issue is only happening when tempest-deployer-input.conf is not passed with python-tempestconf.

Why it is happening like this?
tempest-deploy-input.conf file contains the correct region for each of the each services https://logs.rdoproject.org/39/17339/1/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost/7d119e0/logs/undercloud/home/zuul/tempest-deployer-input.conf.txt.gz

And when this file is specifically passed the values gets overridden in tempest config otherwise python-tempestconf client will use default region for each service provided by tempest.config.
For example in case Image service region name is 'RegionOne' returned by temepstconf client but on overcloud Image service client region name is 'regionOne' and if regionname is not correct, it will trigger the api and try to reconnect the multiple times and with somehow different IP belonging from different Region leading to failure.

Here is the my debugging output: paste.openstack.org/show/735934/

Here is the fix: https://review.openstack.org/#/c/618994/

[chandan kumar (chkumar246)]

Fixed by this review https://review.openstack.org/#/c/618994/
and now master and rocky vexhost jobs are passing:

http://logs.rdoproject.org/32/614632/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-vexxhost/c159470/logs/tempest.html.gz

and

http://logs.rdoproject.org/33/614633/9/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost/fe436a7/logs/tempest.html.gz