NetApp cDOT driver fails Tempest cleanup on clone workflows

[Clinton Knight]

During the create-from-snapshot workflow, the cDOT driver clones the flexvol backing a share, and a job is started to split the clone from its parent. The Tempest tests that exercise this feature attempt to delete all artifacts created for the test (child flexvol, snapshot, parent flexvol) immediately after the test assertions. This rapid deletion wouldn't normally happen in the field, but during the Tempest runs the clone split is still occurring during the cleanup phase, so the cleanup fails. There isn't a great way to detect and prevent the issue, so the workaround is to intercept the error and retry a few times.

[Valeriy Ponomaryov (vponomaryov)]

Quote: "There isn't a great way to detect and prevent the issue"

Why? There is the way! Share driver just should return "ok" response only when some resource is deleted indeed. So, it should not only send request for deletion, but also verify resource absense for some time and only then return answer "ok".

[Clinton Knight (clintonk)]

Hi, Valeriy, I think we're saying the same thing. My point was that there isn't an API we can use to unequivocally know the clone split is done and that subsequent delete APIs won't fail. So the simplest fix is to retry for a few seconds in the failing share-delete workflow step (volume unmount).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/178825

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/178825
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=dd2047a886f48969d543018809b826c117a0ec92
Submitter: Jenkins
Branch: master

commit dd2047a886f48969d543018809b826c117a0ec92
Author: Clinton Knight <email address hidden>
Date: Fri Apr 17 11:17:06 2015 -0400

    NetApp cDOT driver fails Tempest cleanup on clone workflows

    During the create-from-snapshot workflow, the cDOT driver clones
    the flexvol backing a share, and a job is started to split the
    clone from its parent. The Tempest tests that exercise this
    feature attempt to delete all artifacts created for the test
    (child flexvol, snapshot, parent flexvol) immediately after
    the test assertions. This rapid deletion wouldn't normally
    happen in the field, but during the Tempest runs the clone
    split is still occurring during the cleanup phase, so the
    cleanup fails. There isn't a great way to detect and prevent
    the issue, so the workaround is to intercept the error and
    retry a few times.

    Closes-Bug: #1447339
    Change-Id: If272eb3c8060c95510226672a8d1928bea0cc6cf