"Cannot access backing file, No such file or directory" error when resize instance with ephemeral disks

[alecorps]

Description
===========

When we resize an instance with flavor with ephemeral disk, it finishes in error state and we have the following error : Cannot access backing file \'/var/lib/nova/instances/_base/ephemeral_30_0cf8804\' of storage file \'/var/lib/nova/instances/<instance_uuid>/disk.local.
The issue happened only when the instance has been initially created with flavor without ephemeral disk.

Steps to reproduce
==================
* Have three compute nodes

* create three "dedicated" flavors called small, medium and large. The small flavor has no ephemeral disk.
+----------------------------+--------------------------------------+
| Field | Value |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| access_project_ids | None |
| description | None |
| disk | 50 |
| id | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
| name | medium |
| os-flavor-access:is_public | True |
| properties | |
| ram | 1024 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 1 |
+----------------------------+--------------------------------------+

+----------------------------+--------------------------------------+
| Field | Value |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 30 |
| access_project_ids | None |
| description | None |
| disk | 50 |
| id | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
| name | medium |
| os-flavor-access:is_public | True |
| properties | |
| ram | 2048 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 1 |
+----------------------------+--------------------------------------+

+----------------------------+--------------------------------------+
| Field | Value |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 60 |
| access_project_ids | None |
| description | None |
| disk | 50 |
| id | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
| name | medium |
| os-flavor-access:is_public | True |
| properties | |
| ram | 4096 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 2 |
+----------------------------+--------------------------------------+

* Create the instance with flavor small

* Resize the instance with the flavor medium to the second node

* Check the folder nova_compute/_data/instances/_base on the compute node where is the vm, you should have the ephemeral disk backing file for the flavor medium :

ls nova_compute/_data/instances/_base/ephemeral_30_0cf8804
nova_compute/_data/instances/_base/ephemeral_30_0cf8804

* Ensure this file is not present on the another nodes.

* Resize the instance with the flavor large to the third node

* Then you will see the ERROR in nova_compute logs : libvirt.libvirtError: Cannot access backing file '/var/lib/nova/instances/_base/ephemeral_30_0cf8804' of storage file '/var/lib/nova/instances/<instance_uuid>/disk.local'

Expected result
===============
VM resized successfuly, vm state is active

Actual result
=============

* VM become ERROR

* On dashboard you can see this notice:
Please try again later [Error: libvirtError].

Environment
===========
1. Exact version of OpenStack you are running.

  Train version 20.6.1
  The error happened also in Wallaby

2. Which hypervisor did you use?
   Libvirt + KVM

3. Which storage type did you use?
   local disk

4. Which networking type did you use?
   Neutron with OpenVSwitch

Logs & Configs
==============

2023-07-12 07:51:14.786 7 ERROR nova.compute.manager [req-1e1376df-78f9-44b4-bdec-1841a7cb20c5 6c20cdc6afb344deb2a47ad9b55969a4 3e1174a8a8b74c02966f07c2049d1344 - default default] [instance: 01de72ff-0871
-4ae8-81cd-ef7c3eb8ddf7] Setting instance vm_state to ERROR: libvirt.libvirtError: Cannot access backing file '/var/lib/nova/instances/_base/ephemeral_30_0cf8804' of storage file '/var/lib/nova/instances/
01de72ff-0871-4ae8-81cd-ef7c3eb8ddf7/disk.local' (as uid:42436, gid:42436): No such file or directory