Description
===========
When attempting to rescue a volume-based instance using an image without the hw_rescue_device and hw_rescue_bus properties set (i.e. which doesn't support the stable rescue mode) I would expect the API call to fail leaving the state in its existing state. Instead Nova appears to go ahead with the rescue which fails (since non-stable rescues for volume backed instances aren't supported) leading to an unrecoverable, broken instance.
Steps to reproduce
==================
1. Create an instance booting from a volume (in my case the instance is booting from a volume in Ceph)
2. Attempt to trigger a rescue using an image *without* the hw_rescue_device and hw_rescue_bus properties set using:
$ openstack --os-compute-api-version 2.87 server rescue --image <IMAGE UUID> <INSTANCE UUID>
Expected result
===============
I would have expected step 2 to fail with an error along the lines of "Cannot rescue a volume-backed instance", leaving the instance unchanged.
My crude skimming of the Nova source appears to try and perform this check here: https://github.com/openstack/nova/blob/f766db261634c8f95f874ba132159f148de9e8bf/nova/compute/api.py#L4514-L4517.
Actual result
=============
The rescue command returns normally and the instance enters the rescue state very briefly before ending up in the Error state with the message:
Instance 05043c69-3533-4396-a610-98e564fa3ead cannot be rescued: Driver Error: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-43
Looking at the nova-compute logs on the hypervisor gets the following
2022-06-16 11:53:40.765 32936 INFO nova.virt.libvirt.driver [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Attempting rescue
2022-06-16 11:53:40.768 32936 INFO nova.virt.libvirt.driver [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Creating image
2022-06-16 11:53:40.943 32936 WARNING nova.compute.manager [req-a39b605e-e2db-45ce-8f7f-c92a192d9fbe 023555cbd3a2484f89984da17ae9bbfb 0c6d3af6910e429894c203af88fc96c4 - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Received unexpected event network-vif-unplugged-464bd8c6-2b41-4648-abe8-d900c88dd996 for instance with vm_state active a>
2022-06-16 11:53:50.665 32936 INFO nova.virt.libvirt.driver [-] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Instance destroyed successfully.
2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] Error launching a defined domain with XML: <domain type='kvm'>
<name>instance-00015604</name>
<uuid>05043c69-3533-4396-a610-98e564fa3ead</uuid>
...snip...
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
<auth username='cinder'>
<secret type='ceph' uuid='...'/>
</auth>
<source protocol='rbd' name='ephemeral-vms/05043c69-3533-4396-a610-98e564fa3ead_disk.rescue'>
<host name='...snip...' port='6789'/>
<host name='...snip...' port='6789'/>
<host name='...snip...' port='6789'/>
</source>
<target dev='sda' bus='scsi'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
<auth username='cinder'>
<secret type='ceph' uuid='...'/>
</auth>
<source protocol='rbd' name='ephemeral-vms/05043c69-3533-4396-a610-98e564fa3ead_disk'>
<host name='...snip...' port='6789'/>
<host name='...snip...' port='6789'/>
<host name='...snip...' port='6789'/>
</source>
<target dev='sdb' bus='scsi'/>
<address type='drive' controller='0' bus='0' target='0' unit='1'/>
</disk>
...snip...
: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest Traceback (most recent call last):
2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 159, in launch
2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest return self._domain.createWithFlags(flags)
2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 193, in doit
2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest result = proxy_call(self._autowrap, f, *args, **kwargs)
2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 151, in proxy_call
2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest rv = execute(f, *args, **kwargs)
2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 132, in execute
2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest six.reraise(c, e, tb)
2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/six.py", line 719, in reraise
2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest raise value
2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 86, in tworker
2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest rv = meth(*args, **kwargs)
2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/libvirt.py", line 1265, in createWithFlags
2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Error trying to Rescue Instance: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] Traceback (most recent call last):
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 4207, in rescue_instance
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] self.driver.rescue(context, instance, network_info,
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 4145, in rescue
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] self._create_guest(
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 7197, in _create_guest
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] guest.launch(pause=pause)
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 162, in launch
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] LOG.exception('Error launching a defined domain with XML: %s',
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] self.force_reraise()
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] raise self.value
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 159, in launch
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] return self._domain.createWithFlags(flags)
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 193, in doit
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] result = proxy_call(self._autowrap, f, *args, **kwargs)
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 151, in proxy_call
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] rv = execute(f, *args, **kwargs)
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 132, in execute
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] six.reraise(c, e, tb)
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/six.py", line 719, in reraise
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] raise value
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 86, in tworker
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] rv = meth(*args, **kwargs)
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/libvirt.py", line 1265, in createWithFlags
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead]
2022-06-16 11:53:52.310 32936 INFO nova.compute.manager [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Successfully reverted task state from rescuing on failure for instance.
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] Exception during message handling: nova.exception.InstanceNotRescuable: Instance 05043c69-3533-4396-a610-98e564fa3ead cannot be rescued: Driver Error: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 4207, in rescue_instance
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server self.driver.rescue(context, instance, network_info,
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 4145, in rescue
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server self._create_guest(
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 7197, in _create_guest
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server guest.launch(pause=pause)
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 162, in launch
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server LOG.exception('Error launching a defined domain with XML: %s',
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server raise self.value
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 159, in launch
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server return self._domain.createWithFlags(flags)
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 193, in doit
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args, **kwargs)
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 151, in proxy_call
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs)
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 132, in execute
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/six.py", line 719, in reraise
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server raise value
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 86, in tworker
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/libvirt.py", line 1265, in createWithFlags
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/exception_wrapper.py", line 71, in wrapped
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server _emit_versioned_exception_notification(
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server raise self.value
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/exception_wrapper.py", line 63, in wrapped
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 183, in decorated_function
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server LOG.warning("Failed to revert task state for instance. "
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server raise self.value
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 154, in decorated_function
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/utils.py", line 1433, in decorated_function
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 211, in decorated_function
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server raise self.value
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 200, in decorated_function
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 4214, in rescue_instance
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server raise exception.InstanceNotRescuable(
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server nova.exception.InstanceNotRescuable: Instance 05043c69-3533-4396-a610-98e564fa3ead cannot be rescued: Driver Error: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory
2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server
2022-06-16 11:53:55.555 32936 INFO nova.compute.manager [-] [instance: 05043c69-3533-4396-a610-98e564fa3ead] VM Stopped (Lifecycle Event)
2022-06-16 11:53:55.610 32936 INFO nova.compute.manager [req-7790122f-6942-4296-bdfa-37362785baec - - - - -] [instance: 05043c69-3533-4396-a610-98e564fa3ead] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.
Environment
===========
Nova f766db261634c8f95f874ba132159f148de9e8bf (installed from Git by OSA)
Hypervisor: Libvirt + KVM (Libvirt 6.0.0-0ubuntu8.16)
Storage type: Ceph
Networking: Neutron (using Linux networking, no OVS)
Thank you for the descriptive bug report!
I looked at the code you found [1], and while it is indeed a validation check for volume-backed rescue, it's unfortunately only checked that the required API microversion was requested for volume-backed rescue [2].
That said, I think a validation check of the image metadata could easily be added in that method to fail fast if we know the rescue is not going to work.
[1] https:/ /github. com/openstack/ nova/blob/ f766db261634c8f 95f874ba132159f 148de9e8bf/ nova/compute/ api.py# L4514-L4517 /github. com/openstack/ nova/blob/ f766db261634c8f 95f874ba132159f 148de9e8bf/ nova/api/ openstack/ compute/ rescue. py#L60
[2] https:/