NetApp driver fails to delete share replica when using a scoped account

[Nahim Alves de Souza]

Description
===========
On ONTAP 9.8, one of the operations used during the process of deleting a replica is broken for users with scoped accounts. The API called by manila returns success when a replica is deleted [1], but if the user tries to delete the share that was being replicated, the ONTAP API will return an error message informing that the share is the source of endpoint of an SnapMirror relationship [2].

Steps to reproduce
==================

For the manila backend I used two clusters configured with ONTAP 9.8, then I did the following steps (using a scoped account):

1. Create a share on manila

2. Create a replica from that share

3. Delete the replica (which returns success [1], but does not delete the SnapMirror relationship)

4. Try to delete the share (which returns the error [2])

Note: The step 3 can be verified on ONTAP, running the command 'snapmirror list-destinations' in the cluster where the source share is hosted. The list of destinations should be cleaned after deleting the share, because the snapmirror relationship should be released in ONTAP.

Expected result
===============
The user should be able to delete the replica and delete the share.

Actual result
=============
The user API return causes the user to think that the replica was deleted, but it prevents the share deletion afterwards.

Environment
===========
1. Devstack deployed on Ubuntu 18.04

2. Manila using the master branch (version 12.1.0)

3. NetApp ONTAP 9.8 + user with scoped account

Logs & Configs
==============
[1] Success message on replica deletion: http://paste.openstack.org/show/807223/
[2] Error message on share deletion: http://paste.openstack.org/show/807224/

[Felipe Rodrigues (felipefutty)]

I could reproduce the bug in my env. Setup 3 backends in the same replication domain. All of them are scoped account.

The manila commands executed: http://paste.openstack.org/show/807271/

The manila-share error: http://paste.openstack.org/show/807272/

[Douglas Viroel (dviroel)]

This issue is one of the root cause of this another issue here: https://bugs.launchpad.net/manila/+bug/1922512

[Vida Haririan (vhariria)]

Additional comments https://meetings.opendev.org/meetings/manila/2021/manila.2021-07-08-15.00.log.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/800552

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/800552
Committed: https://opendev.org/openstack/manila/commit/a745e9c7c48f0bfdd830f13094f588fe085b4845
Submitter: "Zuul (22348)"
Branch: master

commit a745e9c7c48f0bfdd830f13094f588fe085b4845
Author: Nahim Alves de Souza <email address hidden>
Date: Thu Jun 24 14:13:13 2021 -0300

    [NetApp] Fixed scoped account replica delete

    In the ONTAP 9.8 and older, there is a bug in the zapi call
    `snapmirror-release-iter` when using the scoped account
    configuration. The operation is returning success, but the release
    is not occurring.

    To avoid this issue, the Netapp driver was changed to release with
    the `snapmirror-release` zapi call, instead of the one that has a
    bug. This new zapi call requires the field `relationship-id` to be
    specified and it is being retrieved from ONTAP before calling it.

    A small fix, not related to the bug, was made in the zapi call to
    `snapmirror-release-iter` because the `relationship-info-only`
    field was in the wrong place according to the documentation.

    Closes-Bug: #1934889
    Change-Id: I21633d4ffe14983365b0b6239216ed5d0bbfaace

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/manila/+/806525

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/806525
Committed: https://opendev.org/openstack/manila/commit/24545ad9172afcacc1d63a8b972e421c1db24035
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 24545ad9172afcacc1d63a8b972e421c1db24035
Author: Nahim Alves de Souza <email address hidden>
Date: Thu Jun 24 14:13:13 2021 -0300

    [NetApp] Fixed scoped account replica delete

    In the ONTAP 9.8 and older, there is a bug in the zapi call
    `snapmirror-release-iter` when using the scoped account
    configuration. The operation is returning success, but the release
    is not occurring.

    To avoid this issue, the Netapp driver was changed to release with
    the `snapmirror-release` zapi call, instead of the one that has a
    bug. This new zapi call requires the field `relationship-id` to be
    specified and it is being retrieved from ONTAP before calling it.

    A small fix, not related to the bug, was made in the zapi call to
    `snapmirror-release-iter` because the `relationship-info-only`
    field was in the wrong place according to the documentation.

    Closes-Bug: #1934889
    Change-Id: I21633d4ffe14983365b0b6239216ed5d0bbfaace
    (cherry picked from commit a745e9c7c48f0bfdd830f13094f588fe085b4845)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/manila/+/807762

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/807762
Committed: https://opendev.org/openstack/manila/commit/4e7bf1c3d54a57f72493cfb6c821160f34e7d4ca
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 4e7bf1c3d54a57f72493cfb6c821160f34e7d4ca
Author: Nahim Alves de Souza <email address hidden>
Date: Thu Jun 24 14:13:13 2021 -0300

    [NetApp] Fixed scoped account replica delete

    In the ONTAP 9.8 and older, there is a bug in the zapi call
    `snapmirror-release-iter` when using the scoped account
    configuration. The operation is returning success, but the release
    is not occurring.

    To avoid this issue, the Netapp driver was changed to release with
    the `snapmirror-release` zapi call, instead of the one that has a
    bug. This new zapi call requires the field `relationship-id` to be
    specified and it is being retrieved from ONTAP before calling it.

    A small fix, not related to the bug, was made in the zapi call to
    `snapmirror-release-iter` because the `relationship-info-only`
    field was in the wrong place according to the documentation.

    Closes-Bug: #1934889
    Change-Id: I21633d4ffe14983365b0b6239216ed5d0bbfaace
    (cherry picked from commit a745e9c7c48f0bfdd830f13094f588fe085b4845)
    (cherry picked from commit 24545ad9172afcacc1d63a8b972e421c1db24035)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/manila/+/807997

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 13.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 12.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 11.1.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/807997
Committed: https://opendev.org/openstack/manila/commit/ca40f7c80473e062baad4b927d9bc90cbac32d8c
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit ca40f7c80473e062baad4b927d9bc90cbac32d8c
Author: Nahim Alves de Souza <email address hidden>
Date: Thu Jun 24 14:13:13 2021 -0300

    [NetApp] Fixed scoped account replica delete

    In the ONTAP 9.8 and older, there is a bug in the zapi call
    `snapmirror-release-iter` when using the scoped account
    configuration. The operation is returning success, but the release
    is not occurring.

    To avoid this issue, the Netapp driver was changed to release with
    the `snapmirror-release` zapi call, instead of the one that has a
    bug. This new zapi call requires the field `relationship-id` to be
    specified and it is being retrieved from ONTAP before calling it.

    A small fix, not related to the bug, was made in the zapi call to
    `snapmirror-release-iter` because the `relationship-info-only`
    field was in the wrong place according to the documentation.

    Closes-Bug: #1934889
    Change-Id: I21633d4ffe14983365b0b6239216ed5d0bbfaace
    (cherry picked from commit a745e9c7c48f0bfdd830f13094f588fe085b4845)
    (cherry picked from commit 24545ad9172afcacc1d63a8b972e421c1db24035)
    (cherry picked from commit 4e7bf1c3d54a57f72493cfb6c821160f34e7d4ca)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/manila/+/830351

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/830351
Committed: https://opendev.org/openstack/manila/commit/e5189e17bb8869954f2a248673d7f650459bbdeb
Submitter: "Zuul (22348)"
Branch: stable/train

commit e5189e17bb8869954f2a248673d7f650459bbdeb
Author: Nahim Alves de Souza <email address hidden>
Date: Thu Jun 24 14:13:13 2021 -0300

    [NetApp] Fixed scoped account replica delete

    In the ONTAP 9.8 and older, there is a bug in the zapi call
    `snapmirror-release-iter` when using the scoped account
    configuration. The operation is returning success, but the release
    is not occurring.

    To avoid this issue, the Netapp driver was changed to release with
    the `snapmirror-release` zapi call, instead of the one that has a
    bug. This new zapi call requires the field `relationship-id` to be
    specified and it is being retrieved from ONTAP before calling it.

    A small fix, not related to the bug, was made in the zapi call to
    `snapmirror-release-iter` because the `relationship-info-only`
    field was in the wrong place according to the documentation.

    Closes-Bug: #1934889
    Change-Id: I21633d4ffe14983365b0b6239216ed5d0bbfaace
    (cherry picked from commit a745e9c7c48f0bfdd830f13094f588fe085b4845)
    (cherry picked from commit 24545ad9172afcacc1d63a8b972e421c1db24035)
    (cherry picked from commit 4e7bf1c3d54a57f72493cfb6c821160f34e7d4ca)
    (cherry picked from commit ca40f7c80473e062baad4b927d9bc90cbac32d8c)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila ussuri-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila train-eol release.