Adding the error trace here since the paste could expire

Apr 18 09:29:40 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:40.553 2499881 INFO nova.compute.manager [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Rebuilding instance
Apr 18 09:29:45 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:45.986 2499881 INFO nova.virt.libvirt.driver [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Instance shutdown successfully after 5 seconds.
Apr 18 09:29:45 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:45.994 2499881 INFO nova.virt.libvirt.driver [-] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Instance destroyed successfully.
Apr 18 09:29:46 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:46.004 2499881 INFO nova.virt.libvirt.driver [-] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Instance destroyed successfully.
Apr 18 09:29:46 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:46.220 2499881 INFO os_vif [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:ea:b9:d7,bridge_name='qbrc6b8410e-fb',has_traffic_filtering=True,id=c6b8410e-fbdc-45b6-b3b3-6dd5515226c5,network=Network(579804e9-8b7c-4402-b23b-02a89c31284d),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapc6b8410e-fb')
Apr 18 09:29:46 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:46.266 2499881 INFO nova.virt.libvirt.driver [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Deleting instance files /var/lib/nova/instances/43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd_del
Apr 18 09:29:46 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:46.268 2499881 INFO nova.virt.libvirt.driver [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Deletion of /var/lib/nova/instances/43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd_del complete
Apr 18 09:29:46 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:46.432 2499881 WARNING nova.virt.libvirt.driver [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] During detach_volume, instance disappeared.: nova.exception.InstanceNotFound: Instance 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd could not be found.
Apr 18 09:29:47 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:47.520 2499881 WARNING nova.compute.manager [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Timeout waiting for ['volume-reimaged-d4b22ad0-dcc7-485f-bab7-b4d82eb61987'] for instance with vm_state active and task_state rebuilding. Event states are: volume-reimaged-d4b22ad0-dcc7-485f-bab7-b4d82eb61987: timed out after 0.00 seconds: eventlet.timeout.Timeout: 0 seconds
Apr 18 09:29:47 dev-compute1 nova-compute[2499881]: /openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_serialization/jsonutils.py:180: UserWarning: Cannot convert <Timeout at 0x7c580c578b80 seconds=0> to primitive, will raise ValueError instead of warning in version 3.0
Apr 18 09:29:47 dev-compute1 nova-compute[2499881]:   warnings.warn("Cannot convert %r to primitive, will raise ValueError "
Apr 18 09:29:48 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:48.523 2499881 INFO nova.compute.manager [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Successfully reverted task state from rebuilding on failure for instance.
Apr 18 09:29:48 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] Exception during message handling: ValueError: Circular reference detected
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/utils.py", line 1439, in decorated_function
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 203, in decorated_function
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 3826, in rebuild_instance
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     self._do_rebuild_instance_with_claim(
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 3912, in _do_rebuild_instance_with_claim
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     self._do_rebuild_instance(
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 4104, in _do_rebuild_instance
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     self._rebuild_default_impl(**kwargs)
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 3693, in _rebuild_default_impl
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     self._rebuild_volume_backed_instance(
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 3613, in _rebuild_volume_backed_instance
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     with self.virtapi.wait_for_instance_event(instance, events,
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     next(self.gen)
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 559, in wait_for_instance_event
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     self._wait_for_instance_events(
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 471, in _wait_for_instance_events
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     actual_event = event.wait()
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 436, in wait
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     instance_event = self.event.wait()
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/eventlet/event.py", line 125, in wait
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     result = hub.switch()
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     return self.greenlet.switch()
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server eventlet.timeout.Timeout: 0 seconds
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server 
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server 
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 244, in inner
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/exception_wrapper.py", line 65, in wrapped
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     self.force_reraise()
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     raise self.value
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/exception_wrapper.py", line 63, in wrapped
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 166, in decorated_function
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     self.force_reraise()
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     raise self.value
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 157, in decorated_function
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/utils.py", line 1437, in decorated_function
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     with EventReporter(context, event_name, host, instance_uuid,
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/utils.py", line 1400, in __exit__
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     objects.InstanceActionEvent.event_finish_with_failure(
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/objects/base.py", line 355, in wrapper
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     return fn.__get__(None, obj)(*args, **kwargs)
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 175, in wrapper
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     result = cls.indirection_api.object_class_action_versions(
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     return cctxt.call(context, 'object_class_action_versions',
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     result = self.transport._send(
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     return self._driver.send(target, ctxt, message,
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     return self._send(target, ctxt, message, wait_for_reply, timeout,
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 644, in _send
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     msg = rpc_common.serialize_msg(msg)
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_messaging/_drivers/common.py", line 292, in serialize_msg
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_serialization/jsonutils.py", line 202, in dumps
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     return json.dumps(obj, default=default, **kwargs)
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     **kw).encode(obj)
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     chunks = self.iterencode(o, _one_shot=True)
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server     return _iterencode(o, 0)
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server ValueError: Circular reference detected
                                                    2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server 
Apr 18 09:29:49 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:49.052 2499881 WARNING nova.compute.manager [req-792db0e8-3379-438e-9fb9-f5e1c8970907 req-c0b887a2-09fc-449a-bf52-7a4400bd9067 51c5ce5de8844debac57ac395dc0b5c1 e1204eb8867846eea31b90b95c76e8e5 - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Received unexpected event network-vif-unplugged-c6b8410e-fbdc-45b6-b3b3-6dd5515226c5 for instance with vm_state active and task_state None.