NetApp driver randomly fails to continue migration of share

[Rodrigo Barbieri]

Using the NetApp driver, sometimes the migration is successfully started but the volume move job cannot be found when invoked later in migration_driver_continue function.

This problem is random but seems more likely to occur when the ONTAP controller is very busy as it can be observed in CI runs.

2017-04-28 08:57:31.825 4444 ERROR manila.share.manager [req-60fbc85a-1e44-4538-b0cd-3ec05275a720 - - - - -] Driver-assisted migration of share 91c14202-ed58-44c8-af49-456a6fadbf50 failed.
2017-04-28 08:57:31.825 4444 ERROR manila.share.manager Traceback (most recent call last):
2017-04-28 08:57:31.825 4444 ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/manager.py", line 959, in migration_driver_continue
2017-04-28 08:57:31.825 4444 ERROR manila.share.manager src_share_server, dest_share_server)
2017-04-28 08:57:31.825 4444 ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/drv_single_svm.py", line 216, in migration_continue
2017-04-28 08:57:31.825 4444 ERROR manila.share.manager destination_share_server=destination_share_server)
2017-04-28 08:57:31.825 4444 ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py", line 1796, in migration_continue
2017-04-28 08:57:31.825 4444 ERROR manila.share.manager status = self._get_volume_move_status(source_share, share_server)
2017-04-28 08:57:31.825 4444 ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py", line 1789, in _get_volume_move_status
2017-04-28 08:57:31.825 4444 ERROR manila.share.manager status = self._client.get_volume_move_status(share_volume, vserver)
2017-04-28 08:57:31.825 4444 ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/drivers/netapp/utils.py", line 86, in trace_wrapper
2017-04-28 08:57:31.825 4444 ERROR manila.share.manager result = f(self, *args, **kwargs)
2017-04-28 08:57:31.825 4444 ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/client/client_cmode.py", line 3402, in get_volume_move_status
2017-04-28 08:57:31.825 4444 ERROR manila.share.manager raise exception.NetAppException(msg % msg_args)
2017-04-28 08:57:31.825 4444 ERROR manila.share.manager NetAppException: Volume share_7748da39_0c66_47de_98dd_29353cc7a9c4 in Vserver vserver_ed4dca75 is not part of any data motion operations.
2017-04-28 08:57:31.825 4444 ERROR manila.share.manager

[Jason Grosso (jgrosso)]

Is this issue still occurring? Have you been able to test this with a busy ONTAP controller?

[ignazio (cassano)]

I am facing the same issue on centos 7 queens.
Ignazio

[Douglas Viroel (dviroel)]

A query for volume_move_status returning no results isn't something common to see. Today, the code has some retries around that call, but only for checking the current state of the operation, not to treat a 'no results' error. Do we have more evidences of this issue in recent releases? This can be an ONTAP issue that might be fixed already.
@ignazio, can you say which version of ONTAP you faced this issue?

[Vida Haririan (vhariria)]

Hard to reproduce
Additional comments at https://meetings.opendev.org/meetings/manila/2022/manila.2022-12-15-15.00.log.html

[Vida Haririan (vhariria)]

See additional triage information at https://meetings.opendev.org/meetings/manila/2024/manila.2024-03-07-15.00.log.html