[Manila] Share backup deletion stuck at "error_deleting" when the source share and backup volume are on the same vserver

[Saikumar Pulluri]

Steps:

1.Created a share on vserver1

2.Created backup with specific backup_type like mentioned below, backup creation successful.
[backup_type1]
backup_type_name=backup1
netapp_backup_backend_section_name=ontap_cluster_1

3.Tried to delete the backup and the status is stuck at "error_deleting"

Logs:
ERROR manila.share.manager [^[[01;36mNone req-ba85f3d8-31d5-4302-8318-d9d9dfc21fc8 ^[[00;36madmin None] ^[[01;35mFai led to delete share backup e095318d-69ed-4ed7-8e64-91c9222111c5.^[[00m: IndexError: list index out of range
ERROR oslo_messaging.rpc.server [^[[01;36mNone req-ba85f3d8-31d5-4302-8318-d9d9dfc21fc8 ^[[00;36madmin None] ^[[01;3 5mException during message handling^[[00m: IndexError: list index out of range
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00mTraceback (most recent call last):
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messag ing/rpc/server.py", line 165, in _process_incoming
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m res = self.dispatcher.dispatch(message)
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messag ing/rpc/dispatcher.py", line 309, in dispatch
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m return self._do_dispatch(endpoint, method, ctxt, args)
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messag ing/rpc/dispatcher.py", line 229, in _do_dispatch
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m result = func(ctxt, **new_args)
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/manila/manila/share/manager.py", line 5341, in del ete_backup
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m with excutils.save_and_reraise_exception():
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/ excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m self.force_reraise()
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/ excutils.py", line 200, in force_reraise
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m raise self.value
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/manila/manila/share/manager.py", line 5338, in del ete_backup
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m self.driver.delete_backup(context, backup, share_instance,
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/clust er_mode/drv_single_svm.py", line 364, in delete_backup
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m return self.library.delete_backup(context, backup, share, **kwarg s)
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/manila/manila/share/drivers/netapp/utils.py", line 115, in trace_wrapper
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m result = f(self, *args, **kwargs)
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/clust er_mode/lib_base.py", line 4753, in delete_backup
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/clust er_mode/lib_base.py", line 4753, in delete_backup
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m self._resource_cleanup_for_backup(backup,
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/manila/manila/share/drivers/netapp/utils.py", line 115, in trace_wrapper
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m result = f(self, *args, **kwargs)
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/clust er_mode/lib_base.py", line 4976, in _resource_cleanup_for_backup
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m src_vserver_client.delete_vserver_peer(src_vserver, des_vserver)
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/manila/manila/share/drivers/netapp/utils.py", line 115, in trace_wrapper
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m result = f(self, *args, **kwargs)
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/clien t/client_cmode_rest.py", line 4276, in delete_vserver_peer
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m uuid = vserver_peer[0].get('uuid')
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00mIndexError: list index out of range

Expected:
Backup should get deleted.

[Vida Haririan (vhariria)]

See additional triage information at https://meetings.opendev.org/meetings/manila/2024/manila.2024-03-28-15.00.log.html