OpenStack Compute (nova)

Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936

Bug #2044721 reported by Nabil
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Undecided
Unassigned

Bug Description

We were upgrading our server hardware. our compute server was corrupted (unable to enter ubuntu os) , so i reflash the same ubuntu os version to it & with the same IP address/mac address , and redeployed the kolla-ansible with the same configuration . however , after that nova_compute & zun_compute service on compute node seems to be restarting by itself (exit ---> restart ---> exit ---> restart). I have no clue whats the solution to it

expected result
=========================
all services should be working correctly

actual result
=========================
zun_compute is down
nova_compute is down
horizon compute node is down

environment
=======================
kolla-ansible V2023.1
ubuntu version 22.04 for all 3 servers

<class 'oslo_messaging.exceptions.MessagingTimeout'>
2023-11-27 06:26:18.431 22 ERROR nova.api.openstack.wsgi [None req-0c9d29ab-f230-4b26-828f-b1bce144acc9 d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 6fc08dc0c65849d98b55f762441ae230
2023-11-27 06:26:18.431 22 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 6fc08dc0c65849d98b55f762441ae230
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
2023-11-27 06:27:18.420 21 ERROR nova.api.openstack.wsgi [None req-c61791f0-c689-4b37-bc73-183586fd45c1 d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID b3811b5050fb45238d0cce866b8db0d3
2023-11-27 06:27:18.420 21 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID b3811b5050fb45238d0cce866b8db0d3
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
2023-11-27 06:28:18.520 23 ERROR nova.api.openstack.wsgi [None req-25845c63-c886-4878-a976-d427f1fb311f - - - - - -] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID a2f8e06d636f4d96b89c5f5e6b685db0
2023-11-27 06:28:18.520 23 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID a2f8e06d636f4d96b89c5f5e6b685db0
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
2023-11-27 06:29:18.436 19 ERROR nova.api.openstack.wsgi [None req-9eb9d577-c192-4579-a8aa-00d7cabb793b d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 7523a9908d5341a7a667be6c0a1775ea
2023-11-27 06:29:18.436 19 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 7523a9908d5341a7a667be6c0a1775ea
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
2023-11-27 06:30:18.603 23 ERROR nova.api.openstack.wsgi [None req-25f72773-caee-4551-8ebb-92747f12e77f d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID ac1f456f77c94a96b6deea5d2c3186a5
2023-11-27 06:30:18.603 23 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID ac1f456f77c94a96b6deea5d2c3186a5
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
2023-11-27 06:30:51.383 21 ERROR nova.api.openstack.wsgi [None req-0ecdaea3-7494-4bfd-9c47-d571595f2356 d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID cc479a6b25994ae2b1e88b0acdad6fa0
2023-11-27 06:30:51.383 21 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID cc479a6b25994ae2b1e88b0acdad6fa0
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
2023-11-27 06:31:51.444 20 ERROR nova.api.openstack.wsgi [None req-be6e7afb-b3e6-4369-ade3-47ad5323d859 - - - - - -] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID d517cb22d4df45649690a6d5c8b69077
2023-11-27 06:31:51.444 20 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID d517cb22d4df45649690a6d5c8b69077
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
2023-11-27 06:32:51.412 19 ERROR nova.api.openstack.wsgi [None req-eaff062a-22bf-4589-82ce-599937f07984 d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID ff41269181144be98c9d15745063a10d
2023-11-27 06:32:51.412 19 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID ff41269181144be98c9d15745063a10d
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
2023-11-27 06:33:51.421 20 ERROR nova.api.openstack.wsgi [None req-088e4fb0-058d-410f-bd1b-d4d7788ce15b d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID cbbb44e1417540fba9b4405146b13dc2
2023-11-27 06:33:51.421 20 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID cbbb44e1417540fba9b4405146b13dc2
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936
2023-11-27 06:34:51.506 19 ERROR nova.api.openstack.wsgi [None req-9e415a3e-85c4-4bb3-907e-e7cbb3dd2daf d54f447283c843df8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID c44ae10b2f774b428763a1d11368cb16
2023-11-27 06:34:51.506 19 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID c44ae10b2f774b428763a1d11368cb16
<class 'oslo_messaging.exceptions.MessagingTimeout'>
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/nova/api/openstack/wsgi.py:936

Revision history for this message
Nabil (note0-0) wrote :
Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

As you can see here, this is a messaging timeout :
f8c11b038ef198fc0 d865adb13a8d4c15b7a4c07f040efdc5 - - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID c44ae10b2f774b428763a1d11368cb16
2023-11-27 06:34:51.506 19 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID c44ae10b2f774b428763a1d11368cb16

So, either you have an issue with oslo.messaging or you have a wrong configuration, but this is not a nova bug.

Changed in nova:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.