Description
===========
Restore workflow not working incase of REST client when tried across the ONTAP backends.
Steps to reproduce
==================
1)Created a share on source backend
2)Created 2 files on ONTAP on the above share
OSTK-select6::*> system node run -node OSTK-select6-01 -command mkfile 20MB /vol/share_f6adacb6_5fcf_4bdc_8817_701fcf61c73b/file1
OSTK-select6::*> system node run -node OSTK-select6-01 -command mkfile 20MB /vol/share_f6adacb6_5fcf_4bdc_8817_701fcf61c73b/file2
OSTK-select6::*> system node run -node OSTK-select6-01 -command ls /vol/share_f6adacb6_5fcf_4bdc_8817_701fcf61c73b/
.
..
file1
file2
3)Created backup of the share on the destination backend
stack@astravm-06 ~ $ os backup create --name os_share2_backup1 --backup-options backup_type=backup_type20 os_share2
stack@astravm-06 ~ $ os backup list
+--------------------------------------+-------------------+--------------------------------------+----------------+
| ID | Name | Share ID | Status |
+--------------------------------------+-------------------+--------------------------------------+----------------+
| a58a35b9-4b14-4360-8550-aacf27183bae | os_share2_backup1 | 8dd47c0a-814b-45ba-a044-a51aa25bbb7e | available |
4)Deleted a file from the volume in ONTAP
OSTK-select6::*> system node run -node OSTK-select6-01 -command rm /vol/share_f6adacb6_5fcf_4bdc_8817_701fcf61c73b/file1
OSTK-select6::*> system node run -node OSTK-select6-01 -command ls /vol/share_f6adacb6_5fcf_4bdc_8817_701fcf61c73b/
.
..
file2
5)Restored the backup
stack@astravm-06 ~ $ os backup restore os_share2_backup1
stack@astravm-06 ~ $ os backup list
+--------------------------------------+-------------------+--------------------------------------+----------------+
| ID | Name | Share ID | Status |
+--------------------------------------+-------------------+--------------------------------------+----------------+
| a58a35b9-4b14-4360-8550-aacf27183bae | os_share2_backup1 | 8dd47c0a-814b-45ba-a044-a51aa25bbb7e | error |
Expected result
===============
The file should get restored and status should be back to available.
Actual result
=============
The status of the backup moved to error and restore is not happened.
Environment
===========
1.Manila: Caracal
2.Backend: NetApp
Logs & Configs
==============
Logs:
-------------
DEBUG manila.share.drivers.netapp.dataontap.client.client_cmode_rest [^[[01;36mNone req-b1ac5e19-8b17-4d12-aaa1-402 ccc0692af ^[[00;36madmin None] ^[[01;35mSnapmirror restore has failed. Error: entry doesn't exist. Code: 4^[[00m ^[ [00;33m{{(pid=1898090) snapmirror_restore_vol /opt/stack/manila/manila/share/drivers/netapp/dataontap/client/client _cmode_rest.py:5435}}^[[00m
ERROR manila.share.manager [^[[01;36mNone req-b1ac5e19-8b17-4d12-aaa1-402ccc0692af ^[[00;36madmin None] ^[[01;35mFa iled to restore backup a53483b3-84e3-46e5-b8f2-1d5d536042e6 to share e80d3c28-7d97-405f-9c67-e7d5df7b5c12 by driver .^[[00m: manila.share.drivers.netapp.dataontap.client.api.NaApiError: NetApp API failed. Reason - 4:entry doesn't e xist
ERROR oslo_messaging.rpc.server [^[[01;36mNone req-b1ac5e19-8b17-4d12-aaa1-402ccc0692af ^[[00;36madmin None] ^[[01; 35mException during message handling^[[00m: manila.share.drivers.netapp.dataontap.client.api.NaApiError: NetApp API failed. Reason - 4:entry doesn't exist
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00mTraceback (most recent call last):
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messa ging/rpc/server.py", line 165, in _process_incoming
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m res = self.dispatcher.dispatch(message)
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messa ging/rpc/dispatcher.py", line 309, in dispatch
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m return self._do_dispatch(endpoint, method, ctxt, args)
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messa ging/rpc/dispatcher.py", line 229, in _do_dispatch
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m result = func(ctxt, **new_args)
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/manila/manila/share/manager.py", line 5379, in re store_backup
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m with excutils.save_and_reraise_exception():
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils /excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m self.force_reraise()
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils /excutils.py", line 200, in force_reraise
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m raise self.value
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/manila/manila/share/manager.py", line 5376, in re store_backup
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m self.driver.restore_backup(context, backup, share_instance,
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/clus ter_mode/drv_single_svm.py", line 357, in restore_backup
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m return self.library.restore_backup(context, backup, share, **kwa rgs)
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/manila/manila/share/drivers/netapp/utils.py", lin e 115, in trace_wrapper
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m result = f(self, *args, **kwargs)
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/clus ter_mode/lib_base.py", line 4660, in restore_backup
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m src_vserver_client.snapmirror_restore_vol(source_path=des_path,
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/clie nt/client_cmode_rest.py", line 5432, in snapmirror_restore_vol
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m self.send_request(f"/snapmirror/relationships/{uuid}/restore",
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/clie nt/client_cmode_rest.py", line 195, in send_request
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m code, response = self.connection.invoke_successfully(
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/clie nt/rest_api.py", line 292, in invoke_successfully
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m raise api.NaApiError(code, msg)
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00mmanila.share.drivers.netapp.dataontap.client.api.NaApiError: NetApp
API failed. Reason - 4:entry doesn't exist
ERROR oslo_messaging.rpc.server ^[[01;35m^[[00m
Configs:
-----------
[ontap_select_cluster_1]
vendor_name = NetApp
share_driver = manila.share.drivers.netapp.common.NetAppDriver
driver_handles_share_servers = False
netapp_login = admin
netapp_password = *****
netapp_server_hostname = *****
netapp_server_port = 80
netapp_transport_type = http
netapp_storage_family = ontap_cluster
netapp_storage_protocol = nfs
netapp_vserver = backup11_vs1
netapp_volume_snapshot_reserve_percent = 0
share_backend_name = ontap_select_cluster_1
netapp_enabled_backup_types = backup_type50
netapp_use_legacy_client = False
[ontap_select_cluster_2]
vendor_name = NetApp
share_driver = manila.share.drivers.netapp.common.NetAppDriver
driver_handles_share_servers = False
netapp_login = admin
netapp_password = *****
netapp_server_hostname = *****
netapp_server_port = 80
netapp_transport_type = http
netapp_storage_family = ontap_cluster
netapp_storage_protocol = nfs
netapp_vserver = backup86_vs1
netapp_volume_snapshot_reserve_percent = 0
share_backend_name = ontap_select_cluster_2
netapp_enabled_backup_types = backup_type20
netapp_use_legacy_client = False
[backup_type20]
backup_type_name=backup20
netapp_backup_backend_section_name=ontap_select_cluster_2
netapp_backup_vserver=backup86_vs1
netapp_backup_volume=backup86_vol1
[backup_type50]
backup_type_name=backup50
netapp_backup_backend_section_name=ontap_select_cluster_1
See additional triage information at https:/