nova client can fail with: Max retries exceeded with url

Bug #1329761 reported by Vincent Ladeuil
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu CI Engine
Fix Released
Medium
Vincent Ladeuil

Bug Description

Seen in the following traceback:

$ ./run-tests --list | wc -l
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): region-a.geo-1.identity.hpcloudsvc.com
WARNING:keystoneclient.httpclient:Failed to retrieve management_url from token
TRACE:dput:No clojure support :(
TRACE:dput:No hython support :(
WARNING:ci_utils.stats:No stats_config found at: /home/vila/ci/uci-engine/exp/fullpath-test-ids/stats_config.py
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): region-a.geo-1.identity.hpcloudsvc.com
14:17:18 INFO requests.packages.urllib3.connectionpool: Starting new HTTPS connection (1): region-a.geo-1.identity.hpcloudsvc.com
Traceback (most recent call last):
  File "./run-tests", line 29, in <module>
    sys.exit(run_tests.main(sys.argv[1:], sys.stdout, sys.stderr))
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/testing/run_tests.py", line 398, in main
    options.exclude_regexps)
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/testing/run_tests.py", line 129, in load_regular_component_tests
    component_suite = load_component_tests(loader, c)
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/testing/run_tests.py", line 99, in load_component_tests
    suite.addTests(sub_loader.loadTestsFromTree('.'))
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/ucitests-0.1.3-py2.7.egg/ucitests/loaders.py", line 145, in loadTestsFromTree
    suite.addTests(self.loadTestsFromFiles(dir_path, names))
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/ucitests-0.1.3-py2.7.egg/ucitests/loaders.py", line 207, in loadTestsFromFiles
    suite.addTests(self.loadTestsFromTree(rel_path))
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/ucitests-0.1.3-py2.7.egg/ucitests/loaders.py", line 140, in loadTestsFromTree
    suite = self.loadTestsFromPackage(dir_path)
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/ucitests-0.1.3-py2.7.egg/ucitests/loaders.py", line 181, in loadTestsFromPackage
    suite.addTests(self.loadTestsFromFiles(dir_path, file_names))
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/ucitests-0.1.3-py2.7.egg/ucitests/loaders.py", line 207, in loadTestsFromFiles
    suite.addTests(self.loadTestsFromTree(rel_path))
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/ucitests-0.1.3-py2.7.egg/ucitests/loaders.py", line 140, in loadTestsFromTree
    suite = self.loadTestsFromPackage(dir_path)
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/ucitests-0.1.3-py2.7.egg/ucitests/loaders.py", line 181, in loadTestsFromPackage
    suite.addTests(self.loadTestsFromFiles(dir_path, file_names))
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/ucitests-0.1.3-py2.7.egg/ucitests/loaders.py", line 204, in loadTestsFromFiles
    suite.addTests(self.loadTestsFromFile(rel_path))
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/ucitests-0.1.3-py2.7.egg/ucitests/loaders.py", line 219, in loadTestsFromFile
    module = self.importFromPath(path)
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/ucitests-0.1.3-py2.7.egg/ucitests/loaders.py", line 274, in importFromPath
    return importlib.import_module(mod_name)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "test_runner/tstrun/tests/test_testbed.py", line 39, in <module>
    @features.requires(features.nova_compute)
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/ucitests-0.1.3-py2.7.egg/ucitests/features.py", line 111, in requires
    if not feature.available():
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/ucitests-0.1.3-py2.7.egg/ucitests/features.py", line 35, in available
    self._available = self._probe()
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/ci-utils/ci_utils/testing/features.py", line 67, in _probe
    client.authenticate()
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/novaclient/v1_1/client.py", line 171, in authenticate
    self.client.authenticate()
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/novaclient/client.py", line 329, in authenticate
    auth_url = self._v2_auth(auth_url)
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/novaclient/client.py", line 411, in _v2_auth
    return self._authenticate(url, body)
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/novaclient/client.py", line 423, in _authenticate
    **kwargs)
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/novaclient/client.py", line 195, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/novaclient/client.py", line 166, in request
    **kwargs)
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/home/vila/ci/uci-engine/exp/fullpath-test-ids/.venv/local/lib/python2.7/site-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='region-a.geo-1.identity.hpcloudsvc.com', port=35357): Max retries exceeded with url: /v2.0/tokens (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known)
0

There should be a way to tweak the nova client to retry more (it's a transient failure, a subsequent run succeed) and/or issue a proper error message.

Tags: airline
Vincent Ladeuil (vila)
Changed in uci-engine:
milestone: none → uce-0
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.