[master][scenario003][standalone] sahara tempest tests failed with .MaxRetryError

[chandan kumar]

On master promotion pipeline scenario003 standalone job failed while running sahara tempest tests with following errors:

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario003-standalone-master/2711c05/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-02-07_16_58_38

2019-02-07 16:58:38 | {0} setUpClass (sahara_tempest_plugin.tests.api.test_data_sources.DataSourceTest) [0.000000s] ... FAILED
2019-02-07 16:58:38 |
2019-02-07 16:58:38 | Captured traceback:
2019-02-07 16:58:38 | ~~~~~~~~~~~~~~~~~~~
2019-02-07 16:58:38 | Traceback (most recent call last):
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/tempest/test.py", line 172, in setUpClass
2019-02-07 16:58:38 | six.reraise(etype, value, trace)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/tempest/test.py", line 165, in setUpClass
2019-02-07 16:58:38 | cls.resource_setup()
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/sahara_tempest_plugin/tests/api/test_data_sources.py", line 31, in resource_setup
2019-02-07 16:58:38 | super(DataSourceTest, cls).resource_setup()
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/sahara_tempest_plugin/tests/api/base.py", line 54, in resource_setup
2019-02-07 16:58:38 | plugin = cls.client.get_plugin(cls.default_plugin)['plugin']
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/sahara_tempest_plugin/services/data_processing/v1_1/data_processing_client.py", line 105, in get_plugin
2019-02-07 16:58:38 | return self._request_check_and_parse_resp(self.get, uri, 200)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/sahara_tempest_plugin/services/data_processing/v1_1/data_processing_client.py", line 49, in _request_check_and_parse_resp
2019-02-07 16:58:38 | resp, body = request_func(uri, headers=headers, *args, **kwargs)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 295, in get
2019-02-07 16:58:38 | return self.request('GET', url, extra_headers, headers)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 661, in request
2019-02-07 16:58:38 | body=body, chunked=chunked)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 552, in _request
2019-02-07 16:58:38 | chunked=chunked
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 587, in raw_request
2019-02-07 16:58:38 | body=body, chunked=chunked)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/http.py", line 112, in request
2019-02-07 16:58:38 | *args, **new_kwargs)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request
2019-02-07 16:58:38 | **urlopen_kw)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url
2019-02-07 16:58:38 | return self.urlopen(method, url, **extra_kw)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen
2019-02-07 16:58:38 | response = conn.urlopen(method, u.request_uri, **kw)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen
2019-02-07 16:58:38 | **response_kw)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen
2019-02-07 16:58:38 | **response_kw)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen
2019-02-07 16:58:38 | **response_kw)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen
2019-02-07 16:58:38 | **response_kw)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen
2019-02-07 16:58:38 | **response_kw)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen
2019-02-07 16:58:38 | **response_kw)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen
2019-02-07 16:58:38 | **response_kw)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen
2019-02-07 16:58:38 | **response_kw)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen
2019-02-07 16:58:38 | **response_kw)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen
2019-02-07 16:58:38 | **response_kw)
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen
2019-02-07 16:58:38 | _stacktrace=sys.exc_info()[2])
2019-02-07 16:58:38 | File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment
2019-02-07 16:58:38 | raise MaxRetryError(_pool, url, error or ResponseError(cause))
2019-02-07 16:58:38 | urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.24.1', port=8386): Max retries exceeded with url: /v1.1/171113ae756f426a837395e78fc75639/plugins/vanilla (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f47f21d5ad0>: Failed to establish a new connection: [Errno 111] Connection refused',))

From sahara logs, we are seeing this https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario003-standalone-master/2711c05/logs/undercloud/var/log/containers/sahara/sahara-api.log.txt.gz#_2019-02-07_16_54_09_141

2019-02-07 16:54:09.141 7 ERROR sahara Traceback (most recent call last):
2019-02-07 16:54:09.141 7 ERROR sahara File "/usr/bin/sahara-api", line 10, in <module>
2019-02-07 16:54:09.141 7 ERROR sahara sys.exit(main())
2019-02-07 16:54:09.141 7 ERROR sahara File "/usr/lib/python2.7/site-packages/sahara/cli/sahara_api.py", line 53, in main
2019-02-07 16:54:09.141 7 ERROR sahara app = setup_api()
2019-02-07 16:54:09.141 7 ERROR sahara File "/usr/lib/python2.7/site-packages/sahara/cli/sahara_api.py", line 43, in setup_api
2019-02-07 16:54:09.141 7 ERROR sahara server.setup_common(possible_topdir, 'API')
2019-02-07 16:54:09.141 7 ERROR sahara File "/usr/lib/python2.7/site-packages/sahara/main.py", line 89, in setup_common
2019-02-07 16:54:09.141 7 ERROR sahara plugins_base.setup_plugins()
2019-02-07 16:54:09.141 7 ERROR sahara File "/usr/lib/python2.7/site-packages/sahara/plugins/base.py", line 162, in setup_plugins
2019-02-07 16:54:09.141 7 ERROR sahara PLUGINS = PluginManager()
2019-02-07 16:54:09.141 7 ERROR sahara File "/usr/lib/python2.7/site-packages/sahara/plugins/base.py", line 84, in __init__
2019-02-07 16:54:09.141 7 ERROR sahara self._load_cluster_plugins()
2019-02-07 16:54:09.141 7 ERROR sahara File "/usr/lib/python2.7/site-packages/sahara/plugins/base.py", line 110, in _load_cluster_plugins
2019-02-07 16:54:09.141 7 ERROR sahara ", ".join(requested_plugins - loaded_plugins))
2019-02-07 16:54:09.141 7 ERROR sahara ConfigurationError: Plugins couldn't be loaded: ambari, vanilla, cdh, mapr, storm, spark
2019-02-07 16:54:09.141 7 ERROR sahara Error ID: 0787d0bf-09e3-4f3d-91c7-9c173e72be44

Currently sahara project has splitted these plugins in a seperate project, It might be related to that.

[chandan kumar (chkumar246)]

https://review.rdoproject.org/r/#/c/18772/ will fix the issue, waiting for a new run

[chandan kumar (chkumar246)]

scenario 003 passed http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario003-standalone-master/a643880/logs/tempest.html