libvirt KeyError on assisted snapshot calls when using Ceph

[Eric Harney]

Spotted on https://zuul.opendev.org/t/openstack/build/b6b9b728978143c9b3b4d1f914b9a15c/logs

cinder-plugin-ceph-tempest runs tempest's VolumesAssistedSnapshotTest (which is not going to work with Ceph volumes). I think this test exists mostly to test RBAC rules as it passes in fake snapshot paths that wouldn't ever behave normally.

Nonetheless, it throws KeyErrors in the libvirt driver that could be cleaned up by checking if delete_info is None in _volume_snapshot_delete:

Aug 30 14:07:09.906544 np0035104604 nova-compute[107505]: ERROR nova.virt.libvirt.driver Traceback (most recent call last):
Aug 30 14:07:09.906544 np0035104604 nova-compute[107505]: ERROR nova.virt.libvirt.driver File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3840, in volume_snapshot_delete
Aug 30 14:07:09.906544 np0035104604 nova-compute[107505]: ERROR nova.virt.libvirt.driver self._volume_snapshot_delete(context, instance, volume_id,
Aug 30 14:07:09.906544 np0035104604 nova-compute[107505]: ERROR nova.virt.libvirt.driver File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3670, in _volume_snapshot_delete
Aug 30 14:07:09.906544 np0035104604 nova-compute[107505]: ERROR nova.virt.libvirt.driver if delete_info['type'] != 'qcow2':
Aug 30 14:07:09.906544 np0035104604 nova-compute[107505]: ERROR nova.virt.libvirt.driver KeyError: 'type'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/900783

[sean mooney (sean-k-mooney)]

this sound like a cider bug to me.

I'm not convinced its valid to call the assisted snapshot with a ceph backed cinder volume.

we should not clean up the key error by checking for none

we should fix this by retrunnign a 400 bad request here

https://github.com/openstack/nova/blob/96d7c42e2759cd90930a0c5b31ea1b2b412baa2d/nova/api/openstack/compute/assisted_volume_snapshots.py#L88-L89

if we receive a request form cinder with type not set in delete_info

[sean mooney (sean-k-mooney)]

hum prehaps we cant block it in the API.

https://docs.openstack.org/api-ref/compute/#delete-assisted-volume-snapshot

the example we provide only seams to require volume_id

/os-assisted-volume-snapshots/421752a6-acf6-4b2d-bc7a-119f9148cd8c?delete_info='{"volume_id": "521752a6-acf6-4b2d-bc7a-119f9148cd8c"}'

so instead of ignoring the issue we would need to fail the request later just not with a key error

[melanie witt (melwitt)]

I just spent some time looking at this in a recent nova-ceph-multistore job failure and I think:

1. Sean appears to be correct that this API shouldn't be called in the Ceph case. Ceph disks are of source type "network" and the libvirt driver logic intentionally will not consider them.

2. AFAICT the only Cinder driver that will send 'type' is the remotefs driver [1]. None of the others send 'type', so it appears valid to not send 'type'.

[1] https://github.com/openstack/cinder/blob/1ecfffafa6019cf6230a5af675e40c2a985dd6eb/cinder/volume/drivers/remotefs.py#L1833-L1849

-----------------------------

Excerpts from the failed job:

Nova receives a request to attach volume d4a23ed5-c8c0-4fe2-acfa-b2b69db86f21:

May 31 14:19:30.665033 np0037636881 nova-compute[102717]: INFO nova.compute.manager [None req-9aadc198-7fa1-4e7e-812f-242d121238dd tempest-VolumesAssistedSnapshotsTest-1161291479 tempest-VolumesAssistedSnapshotsTest-1161291479-project-admin] [instance: 56d9dfed-ad43-4426-b888-de2e4244e25d] Attaching volume d4a23ed5-c8c0-4fe2-acfa-b2b69db86f21 to /dev/vdb

May 31 14:19:30.784610 np0037636881 nova-compute[102717]: DEBUG os_brick.utils [None req-9aadc198-7fa1-4e7e-812f-242d121238dd tempest-VolumesAssistedSnapshotsTest-1161291479 tempest-VolumesAssistedSnapshotsTest-1161291479-project-admin] ==> get_connector_properties: call "{'root_helper': 'sudo nova-rootwrap /etc/nova/rootwrap.conf', 'my_ip': '10.210.192.62', 'multipath': False, 'enforce_multipath': True, 'host': 'np0037636881', 'execute': None}" {{(pid=102717) trace_logging_wrapper /opt/stack/data/venv/lib/python3.10/site-packages/os_brick/utils.py:176}}

May 31 14:19:32.490131 np0037636881 nova-compute[102717]: DEBUG os_brick.utils [None req-9aadc198-7fa1-4e7e-812f-242d121238dd tempest-VolumesAssistedSnapshotsTest-1161291479 tempest-VolumesAssistedSnapshotsTest-1161291479-project-admin] <== get_connector_properties: return (1705ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.210.192.62', 'host': 'np0037636881', 'multipath': False, 'initiator': 'iqn.2016-04.com.open-iscsi:7ee3f6178548', 'do_local_attach': False, 'uuid': '3fe1b450-6dc2-4e92-9133-7bf0af20715d', 'system uuid': '2afe8293-e7a7-25a4-a5ca-eae3fc88b212', 'nvme_native_multipath': False} {{(pid=102717) trace_logging_wrapper /opt/stack/data/venv/lib/python3.10/site-packages/os_brick/utils.py:203}}
May 31 14:19:32.490566 np0037636881 nova-compute[102717]: DEBUG nova.virt.block_device [None req-9aadc198-7fa1-4e7e-812f-242d121238dd tempest-VolumesAssistedSnapshotsTest-1161291479 tempest-VolumesAssistedSnapshotsTest-1161291479-project-admin] [instance: 56d9dfed-ad43-4426-b888-de2e4244e25d] Updating existing volume attachment record: a603f0d2-3c77-4b62-8a37-1a07beb93e91 {{(pid=102717) _volume_attach /opt/stack/nova/nova/virt/block_device.py:665}}

May 31 14:19:34.185042 np0037636881 nova-compute[102717]: DEBUG nova.virt.libvirt.driver [None req-9aadc198-7fa1-4e7e-812f-242d121238dd tempest-VolumesAssistedSnapshotsTest-1161291479 tempest-VolumesAssistedSnapshotsTest-1161291479-project-admin] [instance: 56d9dfed-ad43-4426-b888-de2e4244e25d] Attempting to attach volume d4a23ed5-c8c0-4fe2-acfa-b2b69db86f21 with discard support enab...