tripleo

Standalone centos 9 job is failing on cleanup legacy cinder catalog entry

Bug #1979300 reported by Arx Cruz on 2022-06-21

This bug report is a duplicate of: Bug #1979276: ovn-dbs-bundle fails to start because ovn-ctl crashes with coredump generated. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Triaged	Critical	Unassigned

Bug Description

Log:

https://f17236bf5e382fcb4363-41718c434e8b2245213b6c590d7bf6c9.ssl.cf1.rackcdn.com/844147/2/gate/tripleo-ci-centos-9-standalone/4d8d634/logs/undercloud/home/zuul/standalone_deploy.log

Example:

2022-06-21 09:56:37.812684 | bc764e20-18fc-7634-579c-0000000000de | TIMING | include_tasks | undercloud | 0:17:29.737476 | 0.04s
2022-06-21 09:56:37.851647 | 83dda902-2490-49eb-9911-c17946f8c576 | INCLUDED | /home/zuul/tripleo-deploy/standalone-ansible-ivl7fs1m/external_deploy_steps_tasks_step4.yaml | undercloud
2022-06-21 09:56:37.877650 | bc764e20-18fc-7634-579c-000000002737 | TASK | Clean up legacy Cinder keystone catalog entries
2022-06-21 09:56:42.732179 | bc764e20-18fc-7634-579c-000000002737 | FATAL | Clean up legacy Cinder keystone catalog entries | undercloud | item={'service_name': 'cinderv2', 'service_type': 'volumev2'} | error={"ansible_index_var": "cinder_api_service", "ansible_loop_var": "item", "changed": false, "cinder_api_service": 0, "item": {"service_name": "cinderv2", "service_type": "volumev2"}, "module_stderr": "Failed to discover available identity versions when contacting http://192.168.24.3:5000. Attempting to parse version from URL.\nTraceback (most recent call last):\n File \"/usr/lib/python3.9/site-packages/urllib3/connection.py\", line 169, in _new_conn\n conn = connection.create_connection(\n File \"/usr/lib/python3.9/site-packages/urllib3/util/connection.py\", line 96, in create_connection\n raise err\n File \"/usr/lib/python3.9/site-packages/urllib3/util/connection.py\", line 86, in create_connection\n sock.connect(sa)\nOSError: [Errno 113] No route to host\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 699, in urlopen\n httplib_response = self._make_request(\n File \"/usr/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 394, in _make_request\n conn.request(method, url, **httplib_request_kw)\n File \"/usr/lib/python3.9/site-packages/urllib3/connection.py\", line 234, in request\n super(HTTPConnection, self).request(method, url, body=body, headers=headers)\n File \"/usr/lib64/python3.9/http/client.py\", line 1285, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/lib64/python3.9/http/client.py\", line 1331, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/lib64/python3.9/http/client.py\", line 1280, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/lib64/python3.9/http/client.py\", line 1040, in _send_output\n self.send(msg)\n File \"/usr/lib64/python3.9/http/client.py\", line 980, in send\n self.connect()\n File \"/usr/lib/python3.9/site-packages/urllib3/connection.py\", line 200, in connect\n conn = self._new_conn()\n File \"/usr/lib/python3.9/site-packages/urllib3/connection.py\", line 181, in _new_conn\n raise NewConnectionError(\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7efd44e0d1c0>: Failed to establish a new connection: [Errno 113] No route to host\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.9/site-packages/requests/adapters.py\", line 439, in send\n resp = conn.urlopen(\n File \"/usr/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 755, in urlopen\n retries = retries.increment(\n File \"/usr/lib/python3.9/site-packages/urllib3/util/retry.py\", line 574, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.24.3', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efd44e0d1c0>: Failed to establish a new connection: [Errno 113] No route to host'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/session.py\", line 1022, in _send_request\n resp = self.session.request(method, url, **kwargs)\n File \"/usr/lib/python3.9/site-packages/requests/sessions.py\", line 542, in request\n resp = self.send(prep, **send_kwargs)\n File \"/usr/lib/python3.9/site-packages/requests/sessions.py\", line 655, in send\n r = adapter.send(request, **kwargs)\n File \"/usr/lib/python3.9/site-packages/requests/adapters.py\", line 516, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.24.3', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efd44e0d1c0>: Failed to establish a new connection: [Errno 113] No route to host'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/identity/generic/base.py\", line 136, in _do_create_plugin\n disc = self.get_discovery(session,\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py\", line 608, in get_discovery\n return discover.get_discovery(session=session, url=url,\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/discover.py\", line 1455, in get_discovery\n disc = Discover(session, url, authenticated=authenticated)\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/discover.py\", line 535, in __init__\n self._data = get_version_data(session, url,\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/discover.py\", line 102, in get_version_data\n resp = session.get(url, headers=headers, authenticated=authenticated)\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/session.py\", line 1141, in get\n return self.request(url, 'GET', **kwargs)\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/session.py\", line 931, in request\n resp = send(**kwargs)\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/session.py\", line 1038, in _send_request\n raise exceptions.ConnectFailure(msg)\nkeystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://192.168.24.3:5000: HTTPConnectionPool(host='192.168.24.3', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efd44e0d1c0>: Failed to establish a new connection: [Errno 113] No route to host'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<stdin>\", line 107, in <module>\n File \"<stdin>\", line 99, in _ansiballz_main\n File \"<stdin>\", line 47, in invoke_module\n File \"/usr/lib64/python3.9/runpy.py\", line 210, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.9/runpy.py\", line 97, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/usr/lib64/python3.9/runpy.py\", line 87, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_musgclb5/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 185, in <module>\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_musgclb5/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 181, in main\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_musgclb5/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/module_utils/openstack.py\", line 407, in __call__\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_musgclb5/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 140, in run\n File \"/usr/lib/python3.9/site-packages/openstack/cloud/_identity.py\", line 517, in search_services\n services = self.list_services()\n File \"/usr/lib/python3.9/site-packages/openstack/cloud/_identity.py\", line 492, in list_services\n if self._is_client_version('identity', 2):\n File \"/usr/lib/python3.9/site-packages/openstack/cloud/openstackcloud.py\", line 460, in _is_client_version\n client = getattr(self, client_name)\n File \"/usr/lib/python3.9/site-packages/openstack/cloud/_identity.py\", line 31, in _identity_client\n self._raw_clients['identity'] = self._get_versioned_client(\n File \"/usr/lib/python3.9/site-packages/openstack/cloud/openstackcloud.py\", line 407, in _get_versioned_client\n if adapter.get_endpoint():\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py\", line 291, in get_endpoint\n return self.session.get_endpoint(auth or self.auth, **kwargs)\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/session.py\", line 1243, in get_endpoint\n return auth.get_endpoint(self, **kwargs)\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py\", line 375, in get_endpoint\n endpoint_data = self.get_endpoint_data(\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py\", line 271, in get_endpoint_data\n service_catalog = self.get_access(session).service_catalog\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py\", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/identity/generic/base.py\", line 206, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/identity/generic/base.py\", line 158, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\nkeystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.24.3:5000: HTTPConnectionPool(host='192.168.24.3', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efd44e0d1c0>: Failed to establish a new connection: [Errno 113] No route to host'))\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
2022-06-21 09:56:42.734069 | bc764e20-18fc-7634-579c-000000002737 | TIMING | Clean up legacy Cinder keystone catalog entries | undercloud | 0:17:34.658878 | 4.85s
2022-06-21 09:56:45.786487 | bc764e20-18fc-7634-579c-000000002737 | FATAL | Clean up legacy Cinder keystone catalog entries | undercloud | item={'service_name': 'cinderv3', 'service_type': 'volume'} | error={"ansible_index_var": "cinder_api_service", "ansible_loop_var": "item", "changed": false, "cinder_api_service": 1, "item": {"service_name": "cinderv3", "service_type": "volume"}, "module_stderr": "Failed to discover available identity versions when contacting http://192.168.24.3:5000. Attempting to parse version from URL.\nTraceback (most recent call last):\n File \"/usr/lib/python3.9/site-packages/urllib3/connection.py\", line 169, in _new_conn\n conn = connection.create_connection(\n File \"/usr/lib/python3.9/site-packages/urllib3/util/connection.py\", line 96, in create_connection\n raise err\n File \"/usr/lib/python3.9/site-packages/urllib3/util/connection.py\", line 86, in create_connection\n sock.connect(sa)\nOSError: [Errno 113] No route to host\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 699, in urlopen\n httplib_response = self._make_request(\n File \"/usr/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 394, in _make_request\n conn.request(method, url, **httplib_request_kw)\n File \"/usr/lib/python3.9/site-packages/urllib3/connection.py\", line 234, in request\n super(HTTPConnection, self).request(method, url, body=body, headers=headers)\n File \"/usr/lib64/python3.9/http/client.py\", line 1285, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/lib64/python3.9/http/client.py\", line 1331, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/lib64/python3.9/http/client.py\", line 1280, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/lib64/python3.9/http/client.py\", line 1040, in _send_output\n self.send(msg)\n File \"/usr/lib64/python3.9/http/client.py\", line 980, in send\n self.connect()\n File \"/usr/lib/python3.9/site-packages/urllib3/connection.py\", line 200, in connect\n conn = self._new_conn()\n File \"/usr/lib/python3.9/site-packages/urllib3/connection.py\", line 181, in _new_conn\n raise NewConnectionError(\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f81a56511c0>: Failed to establish a new connection: [Errno 113] No route to host\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.9/site-packages/requests/adapters.py\", line 439, in send\n resp = conn.urlopen(\n File \"/usr/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 755, in urlopen\n retries = retries.increment(\n File \"/usr/lib/python3.9/site-packages/urllib3/util/retry.py\", line 574, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.24.3', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f81a56511c0>: Failed to establish a new connection: [Errno 113] No route to host'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/session.py\", line 1022, in _send_request\n resp = self.session.request(method, url, **kwargs)\n File \"/usr/lib/python3.9/site-packages/requests/sessions.py\", line 542, in request\n resp = self.send(prep, **send_kwargs)\n File \"/usr/lib/python3.9/site-packages/requests/sessions.py\", line 655, in send\n r = adapter.send(request, **kwargs)\n File \"/usr/lib/python3.9/site-packages/requests/adapters.py\", line 516, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.24.3', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f81a56511c0>: Failed to establish a new connection: [Errno 113] No route to host'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/identity/generic/base.py\", line 136, in _do_create_plugin\n disc = self.get_discovery(session,\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py\", line 608, in get_discovery\n return discover.get_discovery(session=session, url=url,\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/discover.py\", line 1455, in get_discovery\n disc = Discover(session, url, authenticated=authenticated)\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/discover.py\", line 535, in __init__\n self._data = get_version_data(session, url,\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/discover.py\", line 102, in get_version_data\n resp = session.get(url, headers=headers, authenticated=authenticated)\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/session.py\", line 1141, in get\n return self.request(url, 'GET', **kwargs)\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/session.py\", line 931, in request\n resp = send(**kwargs)\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/session.py\", line 1038, in _send_request\n raise exceptions.ConnectFailure(msg)\nkeystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://192.168.24.3:5000: HTTPConnectionPool(host='192.168.24.3', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f81a56511c0>: Failed to establish a new connection: [Errno 113] No route to host'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<stdin>\", line 107, in <module>\n File \"<stdin>\", line 99, in _ansiballz_main\n File \"<stdin>\", line 47, in invoke_module\n File \"/usr/lib64/python3.9/runpy.py\", line 210, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.9/runpy.py\", line 97, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/usr/lib64/python3.9/runpy.py\", line 87, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_r02ctndf/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 185, in <module>\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_r02ctndf/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 181, in main\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_r02ctndf/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/module_utils/openstack.py\", line 407, in __call__\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_r02ctndf/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 140, in run\n File \"/usr/lib/python3.9/site-packages/openstack/cloud/_identity.py\", line 517, in search_services\n services = self.list_services()\n File \"/usr/lib/python3.9/site-packages/openstack/cloud/_identity.py\", line 492, in list_services\n if self._is_client_version('identity', 2):\n File \"/usr/lib/python3.9/site-packages/openstack/cloud/openstackcloud.py\", line 460, in _is_client_version\n client = getattr(self, client_name)\n File \"/usr/lib/python3.9/site-packages/openstack/cloud/_identity.py\", line 31, in _identity_client\n self._raw_clients['identity'] = self._get_versioned_client(\n File \"/usr/lib/python3.9/site-packages/openstack/cloud/openstackcloud.py\", line 407, in _get_versioned_client\n if adapter.get_endpoint():\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py\", line 291, in get_endpoint\n return self.session.get_endpoint(auth or self.auth, **kwargs)\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/session.py\", line 1243, in get_endpoint\n return auth.get_endpoint(self, **kwargs)\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py\", line 375, in get_endpoint\n endpoint_data = self.get_endpoint_data(\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py\", line 271, in get_endpoint_data\n service_catalog = self.get_access(session).service_catalog\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py\", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/identity/generic/base.py\", line 206, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n File \"/usr/lib/python3.9/site-packages/keystoneauth1/identity/generic/base.py\", line 158, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\nkeystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.24.3:5000: HTTPConnectionPool(host='192.168.24.3', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f81a56511c0>: Failed to establish a new connection: [Errno 113] No route to host'))\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
2022-06-21 09:56:45.790254 | bc764e20-18fc-7634-579c-000000002737 | TIMING | Clean up legacy Cinder keystone catalog entries | undercloud | 0:17:37.715091 | 7.91s
2022-06-21 09:56:45.801364 | bc764e20-18fc-7634-579c-000000002737 | TIMING | Clean up legacy Cinder keystone catalog entries | undercloud | 0:17:37.726194 | 7.92s

I checked latest sucessfull was this morning, but then all the other jobs are failing:

https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-standalone&skip=0

I did a diff on the packages between latest passing and latest failing and i see no differences:

https://www.diffchecker.com/DedtKwlE

Tags:

Revision history for this message

Rabi Mishra (rabi) wrote on 2022-06-21:

Is this failing consistently? I see the vip stopped. Maybe some pacemaker latest version issue.

Full List of Resources:
* ip-192.168.24.3 (ocf:heartbeat:IPaddr2): Stopped

Jun 21 09:48:45.422 standalone.localdomain pacemaker-schedulerd[41912] (determine_online_status) info: Node standalone is online
Jun 21 09:48:45.422 standalone.localdomain pacemaker-schedulerd[41912] (pcmk__native_allocate) info: Resource ip-192.168.24.3 cannot run anywhere
Jun 21 09:48:45.422 standalone.localdomain pacemaker-schedulerd[41912] (rsc_action_default) info: Leave ip-192.168.24.3 (Stopped)

And 192.168.24.3 not on br-ctlplane

https://f17236bf5e382fcb4363-41718c434e8b2245213b6c590d7bf6c9.ssl.cf1.rackcdn.com/844147/2/gate/tripleo-ci-centos-9-standalone/4d8d634/logs/undercloud/var/log/extra/network.txt