[Manila][NetApp] Share deletion is failing at NetApp storage when DHSS=True is used for ONTAP versions >=9.13.1

[Saravanan Manickam]

Share deletion is failing at NetApp storage when DHSS=True is used for ONTAP versions >=9.13.1.
It happens mainly for the shares created from snapshots. Share from Snapshot is actually creating flexclones in ONTAP.

Starting from ONTAP 13.1, it has been done intentionally that,the deleted clones/volumes are kept back until retention period expires which is 12hours by default. This is to cover the use case of if user has mistakenly deleted flexclone volumes. To avoid waiting for the retention period,and to delete the flexclone shares immediately we can follow one of this.

1) Force delete the flexclone share which will delete the volume immediately in ONTAP
2) Set the retention period to 0 for the new share server created.
3) Do "volume recovery-queue purge-all"

This problem would be more imminent when we use DHSS=True, where the new share server (vserver) and shares created. We'll not be able to delete such shares from OpenStack, and it would be having lot of shares in deleting state. I prefer, we can implement option #2 in NetApp driver code via extra spec option.

This is also related to deferred deletion code done by https://review.opendev.org/c/openstack/manila/+/907051?tab=comments where the fix is taken care of handling it generically.

As part of this bug, we can look at fixing this at NetApp storage.

aff250-astra-01-02::> volume clone show
Parent Parent Parent
Vserver FlexClone Vserver Volume Snapshot State Type
ms-nfs2 vol1_clone ms-nfs2 vol1 vol1-snap online RW

aff250-astra-01-02::>

aff250-astra-01-02::> volume offline -vserver ms-nfs2 -volume vol1_clone
Volume "ms-nfs2:vol1_clone" is now offline.

aff250-astra-01-02::> volume destroy -vserver ms-nfs2 -volume vol1_clone -force
[Job 10400] Job is queued: Delete vol1_clone.

Warning: Unable to list entries for kernel on node "aff250-astra-01": Volume is offline.
Volume "ms-nfs2:vol1_clone" destroyed.

aff250-astra-01-02::> volume clone show
There are no entries matching your query.

aff250-astra-01-02::>

[Saravanan Manickam (msaravan)]

Failure log:

2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server [req-b5fad22e-1367-4445-a152-a159441da7f3 7838dd4373fb4278ad3df4d5370fc47b 48b88fc7191c4130a343db8ad5dcd12c - - -] Exception during message handling: manila.share.drivers.netapp.dataontap.client.api.NaApiError: NetApp API failed. Reason - 15894:Failed to delete volume "share_9b7bca22_d6f4_4603_807a_43c309dec2c3" in Vserver "os_02b6f377-687e-4af8-95ea-090e82cd34b2" because it has one or more clones. Only the cluster administrator can delete the clones associated with this volume.
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/manila/share/manager.py", line 219, in wrapped
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/manila/utils.py", line 574, in wrapper
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/manila/share/manager.py", line 3393, in delete_share_instance
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server self.message_api.create(
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server raise self.value
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/manila/share/manager.py", line 3373, in delete_share_instance
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server self.driver.delete_share(context, share_instance,
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/manila/share/drivers/netapp/dataontap/cluster_mode/drv_multi_svm.py", line 66, in delete_share
2024-02-19 15:35:05.827 12 ERROR oslo_messaging.rpc.server self.library.delete_share(conte...

[Vida Haririan (vhariria)]

See additional triage information at https://meetings.opendev.org/meetings/manila/2024/manila.2024-02-22-15.00.log.txt