Keystone not available during resources creation

Bug #1886665 reported by Emilien Macchi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Invalid
High
Unassigned

Bug Description

Found in CI/master:

https://9ad4826b71b64bbb4cfa-962c398c9a043149e9c78cc12a8a1726.ssl.cf2.rackcdn.com/737644/7/check/tripleo-ci-centos-8-containers-multinode/8297d69/logs/undercloud/home/zuul/overcloud_deploy.log

2020-07-06 21:01:23 | 2020-07-06 21:01:23.374354 | fa163e95-572d-8225-b716-000000000485 | TIMING | tripleo_keystone_resources : Create Keystone Admin resources | 0:04:24.647 | 0.16s
2020-07-06 21:01:29 | 9, in _new_conn\n self, \"Failed to establish a new connection: %s\" % e\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f474441cb38>: Failed to establish a new connection: [Errno 113] No route to host\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 449, in send\n timeout=timeout\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 720, in urlopen\n method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]\n File \"/usr/lib/python3.6/site-packages/urllib3/util/retry.py\", line 436, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.24.13', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f474441cb38>: Failed to establish a new connection: [Errno 113] No route to host',))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1012, in _send_request\n resp = self.session.request(method, url, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 533, in request\n resp = self.send(prep, **send_kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 646, in send\n r = adapter.send(request, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 516, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.24.13', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f474441cb38>: Failed to establish a new connection: [Errno 113] No route to host',))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 138, in _do_create_plugin\n authenticated=False)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 610, in get_discovery\n authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 1452, in get_discovery\n disc = Discover(session, url, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 536, in __init__\n authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 102, in get_version_data\n resp = session.get(url, headers=headers, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1131, in get\n return self.request(url, 'GET', **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 921, in request\n resp = send(**kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1028, in _send_request\n raise exceptions.ConnectFailure(msg)\nkeystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://192.168.24.13:5000: HTTPConnectionPool(host='192.168.24.13', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f474441cb38>: Failed to establish a new connection: [Errno 113] No route to host',))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<stdin>\", line 102, in <module>\n File \"<stdin>\", line 94, in _ansiballz_main\n File \"<stdin>\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_os_keystone_domain_payload_739i_h84/ansible_os_keystone_domain_payload.zip/ansible/modules/cloud/openstack/os_keystone_domain.py\", line 185, in <module>\n File \"/tmp/ansible_os_keystone_domain_payload_739i_h84/ansible_os_keystone_domain_payload.zip/ansible/modules/cloud/openstack/os_keystone_domain.py\", line 145, in main\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 890, in search_domains\n return self.list_domains(**filters)\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 863, in list_domains\n data = self._identity_client.get(\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 32, in _identity_client\n 'identity', min_version=2, max_version='3.latest')\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py\", line 407, in _get_versioned_client\n if adapter.get_endpoint():\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py\", line 291, in get_endpoint\n return self.session.get_endpoint(auth or self.auth, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1233, in get_endpoint\n return auth.get_endpoint(self, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 380, in get_endpoint\n allow_version_hack=allow_version_hack, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 271, in get_endpoint_data\n service_catalog = self.get_access(session).service_catalog\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n File \"/usr/lib/python3.6/site-packages/keyston2020-07-06 21:01:29.359421 | fa163e95-572d-8225-b716-0000000004cb | FATAL | Create default domain | undercloud | error={"changed": false, "module_stderr": "Failed to discover available identity versions when contacting http://192.168.24.13:5000. Attempting to parse version from URL.\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/urllib3/connection.py\", line 157, in _new_conn\n (self._dns_host, self.port), self.timeout, **extra_kw\n File \"/usr/lib/python3.6/site-packages/urllib3/util/connection.py\", line 84, in create_connection\n raise err\n File \"/usr/lib/python3.6/site-packages/urllib3/util/connection.py\", line 74, in create_connection\n sock.connect(sa)\nOSError: [Errno 113] No route to host\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 672, in urlopen\n chunked=chunked,\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 387, in _make_request\n conn.request(method, url, **httplib_request_kw)\n File \"/usr/lib64/python3.6/http/client.py\", line 1254, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/lib64/python3.6/http/client.py\", line 1300, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/lib64/python3.6/http/client.py\", line 1249, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/lib64/python3.6/http/client.py\", line 1036, in _send_output\n self.send(msg)\n File \"/usr/lib64/python3.6/http/client.py\", line 974, in send\n self.connect()\n File \"/usr/lib/python3.6/site-packages/urllib3/connection.py\", line 184, in connect\n conn = self._new_conn()\n File \"/usr/lib/python3.6/site-packages/urllib3/connection.py\", line 169, in _new_conn\n self, \"Failed to establish a new connection: %s\" % e\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f474441cb38>: Failed to establish a new connection: [Errno 113] No route to host\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 449, in send\n timeout=timeout\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 720, in urlopen\n method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]\n File \"/usr/lib/python3.6/site-packages/urllib3/util/retry.py\", line 436, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.24.13', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f474441cb38>: Failed to establish a new connection: [Errno 113] No route to host',))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1012, in _send_request\n resp = self.session.request(method, url, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 533, in request\n resp = self.send(prep, **send_kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 646, in send\n r = adapter.send(request, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 516, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.24.13', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f474441cb38>: Failed to establish a new connection: [Errno 113] No route to host',))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 138, in _do_create_plugin\n authenticated=False)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 610, in get_discovery\n authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 1452, in get_discovery\n disc = Discover(session, url, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 536, in __init__\n authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 102, in get_version_data\n resp = session.get(url, headers=headers, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1131, in get\n return self.request(url, 'GET', **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 921, in request\n resp = send(**kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1028, in _send_request\n raise exceptions.ConnectFailure(msg)\nkeystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://192.168.24.13:5000: HTTPConnectionPool(host='192.168.24.13', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f474441cb38>: Failed to establish a new connection: [Errno 113] No route to host',))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<stdin>\", line 102, in <module>\n File \"<stdin>\", line 94, in _ansiballz_main\n File \"<stdin>\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_os_keystone_domain_payload_739i_h84/ansible_os_keystone_domain_payload.zip/ansible/modules/cloud/openstack/os_keystone_domain.py\", line 185, in <module>\n File \"/tmp/ansible_os_keystone_domain_payload_739i_h84/ansible_os_keystone_domain_payload.zip/ansible/modules/cloud/openstack/os_keystone_domain.py\", line 145, in main\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 890, in search_domains\n return self.list_domains(**filters)\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 863, in list_domains\n data = self._identity_client.get(\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 32, in _identity_client\n 'identity', min_version=2, max_version='3.latest')\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py\", line 407, in _get_versioned_client\n if adapter.get_endpoint():\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py\", line 291, in get_endpoint\n return self.session.get_endpoint(auth or self.auth, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1233, in get_endpoint\n return auth.get_endpoint(self, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 380, in get_endpoint\n allow_version_hack=allow_version_hack, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 271, in get_endpoint_data\n service_catalog = self.get_access(session).service_catalog\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 206, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 161, in _do_create_plugin\n 'auth_url is correct. %s' % e)\nkeystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.24.13:5000: HTTPConnectionPool(host='192.168.24.13', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f474441cb38>: Failed to establish a new connection: [Errno 113] No route to host',))\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
2020-07-06 21:01:29 |
2020-07-06 21:01:29 | NO MORE HOSTS LEFT *************************************************************
2020-07-06 21:01:29 |
2020-07-06 21:01:29 | PLAY RECAP *********************************************************************
2020-07-06 21:01:29 | centos-8-vexxhost-ca-ymq-1-0017783518 : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

Revision history for this message
Emilien Macchi (emilienm) wrote :

Nevermind, it's not a bug but something's wrong with https://review.opendev.org/#/c/737644/

Changed in tripleo:
milestone: none → victoria-1
importance: Undecided → High
status: New → Triaged
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.