Error nova.exception.VolumeNotFound appears when trying to extend attached volume

[Ilya Popov]

Openstack version: Xena. May have same behaviour in master as code looks like the same

Error appears when we try to extend volume after volume was migrated to ceph beckend. Migration is performed by "change volume type" (in Horizon) or "cinder retype" (in CLI) for attached to running VM cinder volume.

Main condition is os-vol-mig-status-attr:name_id is set for volume. It is performed utomatically during migration to ceph beckend (or to NFS backend - at least using standard NFS driver - cinder.volume.drivers.nfs.NfsDriver)

Steps to reproduce:

1. Create VM
2. Create additional cinder volume on source backend
(I tried on NFS, but I think it could be any backend defferent from destination ceph backend. I suppose it could even be other backend on the same ceph cluster - say other pool with other disk type. E.g. if destination ceph beckend is on SSD pool, source backend could be on the same ceph cluster, but on HDD pool. Main thing is that after migration to destination ceph backend property os-vol-mig-status-attr:name_id is set.)
3. Attach created on p.2 volume to VM
4. Issue "change volume type" (in Horizon) or "cinder retype" (in CLI) and provide volume type for destination ceph backend.
5. After migration is finished, check that os-vol-mig-status-attr:name_id is set to volume ID
6. Try to extend volume using "extend volume" context menu on created in p.2 volume in Horizon. Volume will be extended, but on hypervisor where VM from p.1 is hosted error message will appear in nova-compute.log:

2023-02-20 10:26:27.581 6 INFO nova.compute.manager [req-8eb07f8e-2d68-4521-93cc-94f40007f134 884a8cceeadc4bb1b47455fc9d05f6ac 3e3e26b5910e482a97d188c840114d8f - default default] [instance: 169ab8a2-ad38-49ab-b8f4-14217b709fbb] Cinder extended volume 5a11fb14-6bc5-4c3f-914c-567012c74b01; extending it to detect new size
2023-02-20 10:26:27.633 6 WARNING nova.compute.manager [req-8eb07f8e-2d68-4521-93cc-94f40007f134 884a8cceeadc4bb1b47455fc9d05f6ac 3e3e26b5910e482a97d188c840114d8f - default default] [instance: 169ab8a2-ad38-49ab-b8f4-14217b709fbb] Extend volume failed, volume_id=5a11fb14-6bc5-4c3f-914c-567012c74b01, reason: Volume 584eba48-0baf-4a80-8b8d-ddd6ac6266e6 could not be found.: nova.exception.VolumeNotFound: Volume 584eba48-0baf-4a80-8b8d-ddd6ac6266e6 could not be found.
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server [req-8eb07f8e-2d68-4521-93cc-94f40007f134 884a8cceeadc4bb1b47455fc9d05f6ac 3e3e26b5910e482a97d188c840114d8f - default default] Exception during message handling: nova.exception.VolumeNotFound: Volume 584eba48-0baf-4a80-8b8d-ddd6ac6266e6 could not be found.
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/exception_wrapper.py", line 71, in wrapped
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server _emit_versioned_exception_notification(
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server raise self.value
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/exception_wrapper.py", line 63, in wrapped
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 10581, in external_instance_event
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server self.extend_volume(context, instance, event.tag)
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/utils.py", line 1433, in decorated_function
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 211, in decorated_function
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server raise self.value
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 200, in decorated_function
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 10438, in extend_volume
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server self.driver.extend_volume(context, connection_info, instance,
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 2705, in extend_volume
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server raise exception.VolumeNotFound(volume_id=volume_id)
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server nova.exception.VolumeNotFound: Volume 584eba48-0baf-4a80-8b8d-ddd6ac6266e6 could not be found.
2023-02-20 10:26:27.654 6 ERROR oslo_messaging.rpc.server

Looks like error happens here:
https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2822

Migrated volume has the following connection_info:

{'driver_volume_type': 'rbd', 'data': {'name': 'volumes/volume-584eba48-0baf-4a80-8b8d-ddd6ac6266e6',
 'hosts': ['x.x.x.x', 'x.x.x.x', 'x.x.x.x'], 'ports': ['6789', '6789', '6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder',
 'secret_type': 'ceph', 'secret_uuid': 'xxx', 'volume_id': '584eba48-0baf-4a80-8b8d-ddd6ac6266e6', 'discard': True, 'qos_specs': None, 'access_mode': 'rw',
 'encrypted': False, 'cacheable': False}, 'status': 'reserved', 'instance': '169ab8a2-ad38-49ab-b8f4-14217b709fbb', 'attached_at': '', 'detached_at': '', 'volume_id': '584eba48-0baf-4a80-8b8d-ddd6ac6266e6',
 'serial': '5a11fb14-6bc5-4c3f-914c-567012c74b01'}

Volume serial number is the same as it was before migration (it is the same in xml either), but volume_id in connection_info is new one and it is equal to id in os-vol-mig-status-attr:name_id field of the volume.

So serial number for volume, returned by guest.get_all_disks() doesn't mach volume_id and it cause an exeption raise exception.VolumeNotFound(volume_id=volume_id) as disk variable is set to None

[Ilya Popov (ilya-p)]

Looks like it is related issue:

https://bugzilla.redhat.com/show_bug.cgi?id=1885487

[Ilya Popov (ilya-p)]

Fixed in master in
https://review.opendev.org/c/openstack/nova/+/807025