OpenStack Compute (nova)

oslo.messaging._drivers.impl_rabbit Connection failed: timed out

Bug #1968054 reported by Satish Patel
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Incomplete
Undecided
Unassigned
oslo.messaging
New
Undecided
Unassigned

Bug Description

I am running Wallaby Release on Ubuntu 20.04 (Openstack-Ansible deployment tool)

oslo.messaging=12.7.1
nova=23.1.1

since i upgrade to Wallaby i have started noticed following error message very frequently in nova-compute and solution is to restart nova-compute agent.

Here is the full logs: https://paste.opendev.org/show/bft9znewTxyXHkvIcQO0/

01 19:43:36 compute1.example.net nova-compute[1546242]: AssertionError:
Apr 01 19:45:35 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:35.059 34090 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 110] Connection timed out
Apr 01 19:45:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:40.063 34090 ERROR oslo.messaging._drivers.impl_rabbit [req-707abbfe-8ee0-4af7-900a-e43dc5dec597 - - - - -] [7d350e59-001f-4203-bd41-369650cd5c5c] AMQP server on 172.28.17.24:5671 is unreachable: . Trying again in 1 seconds.: socket.timeout
Apr 01 19:45:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:40.079 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: timed out (retrying in 0 seconds): socket.timeout: timed out
Apr 01 19:45:41 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:41.983 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 0 seconds): OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:45:42 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:42.367 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 2.0 seconds): OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:45:42 compute1.example.net nova-compute[34090]: Traceback (most recent call last):
Apr 01 19:45:42 compute1.example.net nova-compute[34090]: File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
Apr 01 19:45:42 compute1.example.net nova-compute[34090]: timer()
Apr 01 19:45:42 compute1.example.net nova-compute[34090]: File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
Apr 01 19:45:42 compute1.example.net nova-compute[34090]: cb(*args, **kw)
Apr 01 19:45:42 compute1.example.net nova-compute[34090]: File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/semaphore.py", line 152, in _do_acquire
Apr 01 19:45:42 compute1.example.net nova-compute[34090]: waiter.switch()
Apr 01 19:45:42 compute1.example.net nova-compute[34090]: greenlet.error: cannot switch to a different thread
Apr 01 19:45:49 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:49.388 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: timed out (retrying in 0 seconds): socket.timeout: timed out
Apr 01 19:45:50 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:50.303 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] No route to host. Trying again in 1 seconds.: OSError: [Errno 113] No route to host
Apr 01 19:45:51 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:51.199 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 0 seconds): OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:45:51 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:51.583 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:45:51 compute1.example.net nova-compute[34090]: Traceback (most recent call last):
Apr 01 19:45:51 compute1.example.net nova-compute[34090]: File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
Apr 01 19:45:51 compute1.example.net nova-compute[34090]: timer()
Apr 01 19:45:51 compute1.example.net nova-compute[34090]: File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
Apr 01 19:45:51 compute1.example.net nova-compute[34090]: cb(*args, **kw)
Apr 01 19:45:51 compute1.example.net nova-compute[34090]: File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/semaphore.py", line 152, in _do_acquire
Apr 01 19:45:51 compute1.example.net nova-compute[34090]: waiter.switch()
Apr 01 19:45:51 compute1.example.net nova-compute[34090]: greenlet.error: cannot switch to a different thread
Apr 01 19:45:57 compute1.example.net nova-compute[34090]: 2022-04-01 19:45:57.601 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: timed out. Trying again in 1 seconds.: socket.timeout: timed out
Apr 01 19:46:00 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:00.415 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 21 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:18 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:18.464 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 110] Connection timed out. Trying again in 1 seconds.: TimeoutError: [Errno 110] Connection timed out
Apr 01 19:46:22 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:22.559 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:24 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:24.384 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
Apr 01 19:46:24 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:24.384 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:25 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:25.631 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:26 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:26.802 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: . Trying again in 1 seconds.: socket.timeout
Apr 01 19:46:27 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:27.455 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:28 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:28.063 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 24 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:28 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:28.064 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:30 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:30.527 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:31.135 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:31.775 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 21 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:35 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:35.199 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 21 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:52 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:52.575 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:53 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:53.599 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:53 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:53.599 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:56 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:56.671 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 27 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:57 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:57.663 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:57 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:57.664 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:59 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:59.743 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 24 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:46:59 compute1.example.net nova-compute[34090]: 2022-04-01 19:46:59.744 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:02 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:02.816 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 24 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:25 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:25.791 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:26 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:26.783 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:28 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:28.864 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:29 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:29.855 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:29 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:29.856 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:31.935 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 30 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:31.937 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 27 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:31 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:31.939 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:47:35 compute1.example.net nova-compute[34090]: 2022-04-01 19:47:35.007 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 27 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:01 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:01.023 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:05 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:05.055 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:05 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:05.056 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:05 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:05.057 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:07 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:07.167 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 30 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:07 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:07.168 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:08 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:08.159 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:10 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:10.239 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 30 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:11 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:11.231 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 33 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:40.287 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
Apr 01 19:48:40 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:40.288 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:44 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:44.384 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:44 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:44.385 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:44 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:44.386 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [2304697f-2bcd-4377-bbe5-c0e5e9db5225] AMQP server on 172.28.17.24:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:47 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:47.455 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:47 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:47.456 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [bbc499ba-486c-473e-aede-da61bad852ba] AMQP server on 172.28.17.203:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 33 seconds.: OSError: [Errno 113] EHOSTUNREACH
Apr 01 19:48:48 compute1.example.net nova-compute[34090]: 2022-04-01 19:48:48.483 34090 ERROR oslo.messaging._drivers.impl_rabbit [-] [08af61ee-e653-44b0-82bb-155a2a8b7ef3] AMQP server on 172.28.17.98:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 36 seconds.: OSError: [Errno 113] EHOSTUNREACH

My compute nova.conf looks like following

[oslo_messaging_rabbit]
rabbit_retry_interval = 20
rabbit_retry_backoff = 3
rabbit_interval_max = 60
rabbit_transient_queues_ttl = 300
rabbit_qos_prefetch_count = 100
rpc_conn_pool_size = 300

Feel like something crashed in oslo driver and not able to recover itself. I have other deployment running on stein and i haven't seen this kind of issue in last 3 years so definitely this is something related to wallaby oslo bug.

Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

Unfortunately, this doesn't look a Nova issue : this is either an oslo.messaging bug or rather a configuration issue.
Closing this bug for Nova.

Changed in nova:
status: New → Invalid
Revision history for this message
Satish Patel (satish-txt) wrote :

Update: someone point me to this patch so i am trying now to see if that resolve my issue - https://opendev.org/openstack/requirements/commit/887d45e86550bd5ffd25692f61063f78f85d7a2c

Revision history for this message
Isa Fishman (notisa) wrote :

This bug effects me whenever I try to launch an instance. The instance starts the "Scheduling" process and hangs. If I change the connection URL to an invalid one it gives an error that does not allow the instance to enter the "Scheduling" state. The communication seems to be functional as when I try to create an oversized instance it fails with a different message.

Changed in nova:
status: Invalid → New
Revision history for this message
Artom Lifshitz (notartom) wrote :

@Isa that sounds like a different issue - are you sure the instance remains in 'Scheduling' because of RabbitMQ timeouts, or is there something else in the logs?

Changed in nova:
status: New → Incomplete
Revision history for this message
Isa Fishman (notisa) wrote :

@Artom This is the only thing in the logs, it seems as if RABBIT MQ has changed something with the way their system works revolving around environment variables, at least that's what I've found on the internet. Who knows... There where several people saying to downgrade to RABBIT 3.8 I just never got around to it, though I can confirm the only error in any of the logs are that of oslo.messaging and being unable to resolve the broker hostname, though I just realized that this is a connection issue rather than a broker issue.

Changed in nova:
status: Incomplete → Confirmed
status: Confirmed → Incomplete
Revision history for this message
Isa Fishman (notisa) wrote :

The error is as follows: ERROR oslo.messaging._drivers.impl_rabbit Connection failed: failed to resolve broker hostname: OSError: failed to resolve broker hostname

Revision history for this message
farsheed ahmadian (farshan) wrote :

i faced with this problem and fix is adding rabbitmq port in all transport_url in all .cong files:
like this:

by default rabbitmq port is 5672

ex: transport_url = rabbit://openstack:RABBIT_PASS@controller:5672/

then you have to restart all services you changed .conf file, or reboot each node

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.