Description
===========
Following an upgrade from 2023.1 to 2023.2 we are seeing intermittent issues with requests to perform live migrations (and potentially cold migrations). The failed requests result in the impacted instances going into an error state which appears similar to https://bugs.launchpad.net/nova/+bug/2044235, but the fact that a messaging timeout is occurring at all seems to be a new behaviour post-upgrade so I thought it would be worth recording it independently.
After a failure to migrate to a given destination host I believe subsequent requests to migrate to that destination will then succeed, at least until a reasonable time has then expired.
Whether this is a Nova issue directly or an issue in an underlying library I can't be sure.
Steps to reproduce
==================
* Deploy Nova
* Wait for a reasonable period of time. I can't quantify this, but it doesn't seem to happen immediately after Nova services have been restarted.
* Attempt live migration of a number of instances
Expected result
===============
Live migration should proceed as expected
Actual result
=============
Live migration fails for some cases with a 500 error returned to the user via the API (POST /v2.1/servers/<uuid>/action). The instance which migration was attempted for enters the ERROR state.
The post-error message from nova-compute may be a clue, but I'm not familiar enough to know:
2024-04-11 08:22:52.434 3774901 INFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : 1fb21ba089eb47959ac55a29a155a675
Environment
===========
Nova 2023.2 45a926156c863b468318cce462a21027685d07a6 with oslo.messaging 14.4.1
This didn't happen with Nova 2023.1 252e6602f792f24141a9dd59211470601f3b0267 and oslo.messaging 14.2.1
Libvirt + KVM
Ceph
Neutron + LXB
Logs & Configs
==============
Nova API:
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi return self._queues[msg_id].get(block=True, timeout=timeout)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.10/queue.py", line 179, in get
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi raise Empty
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi _queue.Empty
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred:
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi [Previous line repeated 1 more time]
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/api/openstack/compute/migrate_server.py", line 126, in _migrate_live
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi self.compute_api.live_migrate(context, instance, block_migration,
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 389, in inner
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi return function(self, context, instance, *args, **kwargs)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 374, in wrapper
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi return func(self, context, instance, *args, **kwargs)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 310, in inner
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 357, in wrapper
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi return func(self, context, instance, *args, **kwargs)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 279, in inner
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 260, in inner
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 242, in inner
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi return function(self, context, instance, *args, **kwargs)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 168, in inner
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 5578, in live_migrate
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi with excutils.save_and_reraise_exception():
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi self.force_reraise()
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi raise self.value
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 5573, in live_migrate
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi self.compute_task_api.live_migrate_instance(context, instance,
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/api.py", line 110, in live_migrate_instance
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi self.conductor_compute_rpcapi.migrate_server(
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/rpcapi.py", line 352, in migrate_server
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi return cctxt.call(context, 'migrate_server', **kw)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi result = self.transport._send(
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi return self._driver.send(target, ctxt, message,
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi return self._send(target, ctxt, message, wait_for_reply, timeout,
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi result = self._waiter.wait(msg_id, timeout,
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi message = self.waiters.get(msg_id, timeout=timeout)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi raise oslo_messaging.MessagingTimeout(
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 6d20c3260d104854a3c676026ea5abf9
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi
Nova Compute (believed to be destination host for migration):
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server return self._queues[msg_id].get(block=True, timeout=timeout)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/eventlet/queue.py", line 322, in get
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server return waiter.wait()
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/eventlet/queue.py", line 141, in wait
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server return get_hub().switch()
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server return self.greenlet.switch()
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server _queue.Empty
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/exception_wrapper.py", line 65, in wrapped
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server raise self.value
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/exception_wrapper.py", line 63, in wrapped
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/utils.py", line 1439, in decorated_function
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/manager.py", line 213, in decorated_function
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server raise self.value
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/manager.py", line 203, in decorated_function
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/manager.py", line 8520, in check_can_live_migrate_destination
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server dest_check_data = self.driver.check_can_live_migrate_destination(ctxt,
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9664, in check_can_live_migrate_destination
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server if not instance.vcpu_model or not instance.vcpu_model.model:
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 67, in getter
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server self.obj_load_attr(name)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/base.py", line 120, in wrapper
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server return fn(self, attrname)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/instance.py", line 1159, in obj_load_attr
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server self._obj_load_attr(attrname)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/instance.py", line 1187, in _obj_load_attr
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server self._load_vcpu_model()
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/instance.py", line 1000, in _load_vcpu_model
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server self.vcpu_model = objects.VirtCPUModel.get_by_instance_uuid(
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 175, in wrapper
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server result = cls.indirection_api.object_class_action_versions(
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server return cctxt.call(context, 'object_class_action_versions',
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server result = self.transport._send(
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server return self._driver.send(target, ctxt, message,
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server return self._send(target, ctxt, message, wait_for_reply, timeout,
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server result = self._waiter.wait(msg_id, timeout,
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server message = self.waiters.get(msg_id, timeout=timeout)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server raise oslo_messaging.MessagingTimeout(
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 1fb21ba089eb47959ac55a29a155a675
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server
2024-04-11 08:22:52.434 3774901 INFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : 1fb21ba089eb47959ac55a29a155a675
Nova Conductor:
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager Traceback (most recent call last):
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/manager.py", line 505, in _live_migrate
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager task.execute()
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/tasks/base.py", line 25, in wrap
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager with excutils.save_and_reraise_exception():
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager self.force_reraise()
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager raise self.value
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/tasks/base.py", line 23, in wrap
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager return original(self)
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/tasks/base.py", line 40, in execute
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager return self._execute()
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/tasks/live_migrate.py", line 100, in _execute
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager self.destination, dest_node, self.limits = self._find_destination()
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/tasks/live_migrate.py", line 550, in _find_destination
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager self._call_livem_checks_on_host(host, provider_mapping)
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/tasks/live_migrate.py", line 360, in _call_livem_checks_on_host
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager check_can_live_migrate_destination(self.context, self.instance,
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/rpcapi.py", line 604, in check_can_live_migrate_destination
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager return cctxt.call(ctxt, 'check_can_live_migrate_destination', **kwargs)
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager result = self.transport._send(
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager return self._driver.send(target, ctxt, message,
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager return self._send(target, ctxt, message, wait_for_reply, timeout,
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager raise result
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager oslo_messaging.rpc.client.RemoteError: Remote error: MessagingTimeout Timed out waiting for a reply to message ID 1fb21ba089eb47959ac55a29a155a675
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager ['Traceback (most recent call last):\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get\n return self._queues[msg_id].get(block=True, timeout=timeout)\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/eventlet/queue.py", line 322, in get\n return waiter.wait()\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/eventlet/queue.py", line 141, in wait\n return get_hub().switch()\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch\n return self.greenlet.switch()\n', '_queue.Empty\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/exception_wrapper.py", line 65, in wrapped\n with excutils.save_and_reraise_exception():\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/exception_wrapper.py", line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/utils.py", line 1439, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/manager.py", line 213, in decorated_function\n with excutils.save_and_reraise_exception():\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/manager.py", line 203, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/manager.py", line 8520, in check_can_live_migrate_destination\n dest_check_data = self.driver.check_can_live_migrate_destination(ctxt,\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9664, in check_can_live_migrate_destination\n if not instance.vcpu_model or not instance.vcpu_model.model:\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 67, in getter\n self.obj_load_attr(name)\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/base.py", line 120, in wrapper\n return fn(self, attrname)\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/instance.py", line 1159, in obj_load_attr\n self._obj_load_attr(attrname)\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/instance.py", line 1187, in _obj_load_attr\n self._load_vcpu_model()\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/instance.py", line 1000, in _load_vcpu_model\n self.vcpu_model = objects.VirtCPUModel.get_by_instance_uuid(\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 175, in wrapper\n result = cls.indirection_api.object_class_action_versions(\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions\n return cctxt.call(context, \'object_class_action_versions\',\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call\n result = self.transport._send(\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send\n return self._driver.send(target, ctxt, message,\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send\n return self._send(target, ctxt, message, wait_for_reply, timeout,\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send\n result = self._waiter.wait(msg_id, timeout,\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait\n message = self.waiters.get(msg_id, timeout=timeout)\n', ' File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get\n raise oslo_messaging.MessagingTimeout(\n', 'oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 1fb21ba089eb47959ac55a29a155a675\n'].
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager
After some further testing for another issue, this certainly appears to be linked to changes in oslo.messaging between v14.2.1 and v14.4.1. I'm not clear on why I'm not seeing impact to other OpenStack services which are using the same version of the library as things stand.
My best guess at present would be something to do with https:/