Rebuild from BFV Instance Snapshot is failed

[Gökhan]

Issue: Rebuild from Booted From Volume Instance Snapshot is failed

Timeout waiting for ['volume-reimaged-d4b22ad0-dcc7-485f-bab7-b4d82eb61987'] for instance with vm_state active and task_state rebuilding. Event states are: volume-reimaged-d4b22ad0-dcc7-485f-bab7-b4d82eb61987: timed out after 0.00 seconds: eventlet.timeout.Timeout: 0 seconds

Expected State: Min Timeout Per Gb is 20 second.
Analysis:
 I have followed below steps:

 1. Create Booted From Volume Instance
    https://paste.openstack.org/show/bWBSIO7Mr9OIHJna9wk7/

 2. Create snapshot of Booted From Volume Instance
    https://paste.openstack.org/show/bJVPXDYB45FsRbTIhel5/

 3. Rebuild from Booted From Volume Instance Snapshot
    https://paste.openstack.org/show/bHau3AXs789hpYSnLFvW/

Status: It is failed and throws below error logs:
        https://paste.openstack.org/show/bRsJ69NmWzzDVbFiaWW5/

[Rajat Dhasmana (whoami-rajat)]

Adding the error trace here since the paste could expire

Apr 18 09:29:40 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:40.553 2499881 INFO nova.compute.manager [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Rebuilding instance
Apr 18 09:29:45 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:45.986 2499881 INFO nova.virt.libvirt.driver [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Instance shutdown successfully after 5 seconds.
Apr 18 09:29:45 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:45.994 2499881 INFO nova.virt.libvirt.driver [-] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Instance destroyed successfully.
Apr 18 09:29:46 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:46.004 2499881 INFO nova.virt.libvirt.driver [-] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Instance destroyed successfully.
Apr 18 09:29:46 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:46.220 2499881 INFO os_vif [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:ea:b9:d7,bridge_name='qbrc6b8410e-fb',has_traffic_filtering=True,id=c6b8410e-fbdc-45b6-b3b3-6dd5515226c5,network=Network(579804e9-8b7c-4402-b23b-02a89c31284d),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapc6b8410e-fb')
Apr 18 09:29:46 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:46.266 2499881 INFO nova.virt.libvirt.driver [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Deleting instance files /var/lib/nova/instances/43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd_del
Apr 18 09:29:46 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:46.268 2499881 INFO nova.virt.libvirt.driver [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Deletion of /var/lib/nova/instances/43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd_del complete
Apr 18 09:29:46 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:46.432 2499881 WARNING nova.virt.libvirt.driver [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] During detach_volume, instance disappeared.: nova.exception.InstanceNotFound: Instance 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd could not be found.
Apr 18 09:29:47 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:47.520 2499881 WARNING nova.compute.manager [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Timeout waiting ...

[Rajat Dhasmana (whoami-rajat)]

This error is from the nova side, a similar issue can be seen on cinder side while reimaging the volume.
The Cinder side issue is reported here: https://bugs.launchpad.net/cinder/+bug/2062539

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/916409