Rescue BFV instances not working with ceph backend, Tempest new test failing

[Ghanshyam Mann]

Nova implemented the feature for rescuing the BFV instances.
- https://specs.openstack.org/openstack/nova-specs/specs/ussuri/approved/virt-bfv-instance-rescue.html

It is not working with ceph backend as new tests added in Tempest are failing ceph job

- https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_32a/723645/15/check/devstack-plugin-ceph-tempest-py3/32ae221/testr_results.html

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724866
Committed: https://git.openstack.org/cgit/openstack/devstack-plugin-ceph/commit/?id=c1fc88670c0cd1ffa6d13779427c506f5feddb10
Submitter: Zuul
Branch: master

commit c1fc88670c0cd1ffa6d13779427c506f5feddb10
Author: Ghanshyam Mann <email address hidden>
Date: Fri May 1 09:58:57 2020 -0500

    Skip rescue BFV instance tests

    Nova implemented the feature for rescuing the BFV instances.
    - https://specs.openstack.org/openstack/nova-specs/specs/ussuri/approved/virt-bfv-instance-rescue.html

    It is not working with ceph backend as new tests added in Tempest are failing ceph job

    Related-Bug: #1876330

    Change-Id: Ie675610a040e9f4c164b51b6b61b2a8094ecd09f

[Lee Yarwood (lyarwood)]

As I thought, this appears to be a race within the test between our attempt to snapshot the instance to create the rescue disk *and* the subsequent attempt to rescue the instance.

I can reproduce it locally and even see the request to rescue reach the compute just before the request to snapshot:

May 04 04:45:44 localhost.localdomain nova-compute[568186]: INFO nova.virt.libvirt.driver [None
req-df6379e6-86cb-4163-8a2c-a54a2ff65e70 tempest-ServerBootFromVolumeStableRescueTest-1734223307 tempest-ServerBootFromVolumeStableRescueTest-1734223307] [instance: c2c0c393-ead9-4ea3-842c-9148f4039ed6] Attempting a stable>

May 04 04:45:44 localhost.localdomain nova-compute[568186]: INFO nova.virt.libvirt.driver [None req-df6379e6-86cb-4163-8a2c-a54a2ff65e70 tempest-ServerBootFromVolumeStableRescueTest-1734223307 tempest-ServerBootFromVolumeStableRescueTest-1734223307] [instance: c2c0c393-ead9-4ea3-842c-9148f4039ed6] Creating image

[Lee Yarwood (lyarwood)]

Apologies, that wasn't the issue with this test at all.

The issue is that for BFV instances we create a zero sized image that references volume snapshots in the associated block_device_mapping metadata of the image. When the image is then referenced during an instance creation request this then results in new volumes being created from these volume snapshots and used by the new instance. This is all spelled out in the createImage API docs below:

https://docs.openstack.org/api-ref/compute/?expanded=create-image-createimage-action-detail#create-image-createimage-action

Using such an image during instance rescue isn't supported and should be rejected by n-api.

I'll rewrite the test to launch an image based instance, snapshot this to provide a rescue image and optionally launch another BFV instance for tests that require it.

[Dr. Jens Harbott (j-harbott)]

The revert didn't have the bug reference https://review.opendev.org/725819 but I assume that this implies that this bug is fixed.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 25.0.1 release.