Unable to delete CephFS manila share if nfs-ganesha export rule doesn't exist

[Goutham Pacha Ravi]

User report:

We have some manila shares which are stuck in 'error_deleting' state and cannot be removed unless their status is reset to 'available'. One thing in common with these shares is that the instances associated with all these shares (share_instance_id) do not exist in the environment. Even so, the 'access_rules_status' is still 'active'.
After the status is reset to active, the 'access_rules_status' gets set to 'error' state after which we are able to delete it.

For example, share ID: 67d12f8b-5bb8-4486-83ce-b3a1b4030a9c

$ manila show 67d12f8b-5bb8-4486-83ce-b3a1b4030a9c
+---------------------------------------+---------------------------------------------------------------------------+
| Property | Value |
+---------------------------------------+---------------------------------------------------------------------------+
| id | 67d12f8b-5bb8-4486-83ce-b3a1b4030a9c |
| size | 5 |
| availability_zone | nova |
| created_at | 2022-12-06T12:31:40.000000 |
| status | error_deleting |
| name | pvc-ce7985c3-c3d0-493b-aa70-37253922b057 |
| description | provisioned-by=manila.csi.openstack.org |
| project_id | 7b4c0133359d44a4ba633f7db8357524 |
| snapshot_id | None |
| share_network_id | None |
| share_proto | NFS |
| metadata | {'manila.csi.openstack.org/cluster': 'cicd-f58nc'} |
| share_type | f6b11e95-1072-44a6-8d89-b660993a505e |
| is_public | False |
| snapshot_support | True |
| task_state | None |
| share_type_name | ceph |
| access_rules_status | active |
| replication_type | None |
| has_replicas | False |
| user_id | 0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 |
| create_share_from_snapshot_support | False |
| revert_to_snapshot_support | False |
| share_group_id | None |
| source_share_group_snapshot_member_id | None |
| mount_snapshot_support | False |
| share_server_id | None |
| host | hostgroup@cephfs#cephfs |
| export_locations | |
| | id = a6b373a8-4c77-42e1-ad03-7dbbab48782a |
| | path = 172.16.33.4:/volumes/_nogroup/a257d35b-d254-4dff-9dac-657bcb1625fc |
| | preferred = False |
| | share_instance_id = a257d35b-d254-4dff-9dac-657bcb1625fc |
| | is_admin_only = False |
+---------------------------------------+---------------------------------------------------------------------------+

$ manila reset-state --state available 67d12f8b-5bb8-4486-83ce-b3a1b4030a9c

$ manila show 67d12f8b-5bb8-4486-83ce-b3a1b4030a9c
+---------------------------------------+---------------------------------------------------------------------------+
| Property | Value |
+---------------------------------------+---------------------------------------------------------------------------+
| id | 67d12f8b-5bb8-4486-83ce-b3a1b4030a9c |
| size | 5 |
| availability_zone | nova |
| created_at | 2022-12-06T12:31:40.000000 |
| status | available |
| name | pvc-ce7985c3-c3d0-493b-aa70-37253922b057 |
| description | provisioned-by=manila.csi.openstack.org |
| project_id | 7b4c0133359d44a4ba633f7db8357524 |
| snapshot_id | None |
| share_network_id | None |
| share_proto | NFS |
| metadata | {'manila.csi.openstack.org/cluster': 'cicd-f58nc'} |
| share_type | f6b11e95-1072-44a6-8d89-b660993a505e |
| is_public | False |
| snapshot_support | True |
| task_state | None |
| share_type_name | ceph |
| access_rules_status | error |
| replication_type | None |
| has_replicas | False |
| user_id | 0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 |
| create_share_from_snapshot_support | False |
| revert_to_snapshot_support | False |
| share_group_id | None |
| source_share_group_snapshot_member_id | None |
| mount_snapshot_support | False |
| share_server_id | None |
| host | hostgroup@cephfs#cephfs |
| export_locations | |
| | id = a6b373a8-4c77-42e1-ad03-7dbbab48782a |
| | path = 172.16.33.4:/volumes/_nogroup/a257d35b-d254-4dff-9dac-657bcb1625fc |
| | preferred = False |
| | share_instance_id = a257d35b-d254-4dff-9dac-657bcb1625fc |
| | is_admin_only = False |
+---------------------------------------+---------------------------------------------------------------------------+

Some of the logs indicate that manila is not able to update the rules of the shares which we are trying to delete where the share_instance_id for that share does not exist in the environment.

For example:

Share ID: 5b5d2a38-2a46-4d81-8690-ba9cc94629a6
Request ID: req-d7e38b71-85db-444c-8689-f0d6298ffbce

~~~
manila-api.log.2.gz:2023-01-30 10:19:00.712 21 INFO manila.api.openstack.wsgi [req-d7e38b71-85db-444c-8689-f0d6298ffbce 0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 7b4c0133359d44a4ba633f7db8357524 - - -] DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13786/v2/7b4c0133359d44a4ba633f7db8357524/shares/5b5d2a38-2a46-4d81-8690-ba9cc94629a6
manila-api.log.2.gz:2023-01-30 10:19:00.713 21 INFO manila.api.v1.shares [req-d7e38b71-85db-444c-8689-f0d6298ffbce 0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 7b4c0133359d44a4ba633f7db8357524 - - -] Delete share with id: 5b5d2a38-2a46-4d81-8690-ba9cc94629a6

manila-share.log:2023-01-30 10:19:01.742 42 DEBUG manila.share.access [req-d7e38b71-85db-444c-8689-f0d6298ffbce 0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 7b4c0133359d44a4ba633f7db8357524 - - -] Updating access rules for share instance cf864d1b-8d5f-4eea-96a0-454ad90ab8b3 belonging to share 5b5d2a38-2a46-4d81-8690-ba9cc94629a6. update_access_rules /usr/lib/python3.6/site-packages/manila/share/access.py:281
manila-share.log:2023-01-30 10:19:02.120 42 DEBUG oslo_concurrency.processutils [req-d7e38b71-85db-444c-8689-f0d6298ffbce 0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 7b4c0133359d44a4ba633f7db8357524 - - -] Running cmd (subprocess): dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:11406 execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:372
manila-share.log:2023-01-30 10:19:02.149 42 DEBUG oslo_concurrency.processutils [req-d7e38b71-85db-444c-8689-f0d6298ffbce 0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 7b4c0133359d44a4ba633f7db8357524 - - -] CMD "dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:11406" returned: 1 in 0.029s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:409
manila-share.log:2023-01-30 10:19:02.149 42 DEBUG oslo_concurrency.processutils [req-d7e38b71-85db-444c-8689-f0d6298ffbce 0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 7b4c0133359d44a4ba633f7db8357524 - - -] 'dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:11406' failed. Not Retrying. execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:457
manila-share.log:2023-01-30 10:19:02.150 42 ERROR manila.share.drivers.ganesha.manager [req-d7e38b71-85db-444c-8689-f0d6298ffbce 0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 7b4c0133359d44a4ba633f7db8357524 - - -] Error while executing management command on Ganesha node <no name>: dbus call exportmgr.RemoveExport.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
~~~

[Goutham Pacha Ravi (gouthamr)]

Some steps to reproduce this bug by causing a "NotFound" error.. (although the error doesn't appear to be the same in the log in the description) :

Set manila's debug logging

$ manila create nfs 1 --name testshare
$ manila access-allow testshare ip 10.10.10.10

check manila logs for the export ID of the rule that was just created; for example if it is 1001; run:

 sudo dbus-send --print-reply \
    --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr \
    org.ganesha.nfsd.exportmgr.RemoveExport uint16:1001

on the node where ganesha is running; then try to delete the share:

$ manila delete testshare

list the shares, the share must be "error_deleting"

+--------------------------------------+-----------+------+-------------+----------------+-----------+-----------------+---------------------------------------+-------------------+
| ID | Name | Size | Share Proto | Status | Is Public | Share Type Name | Host | Availability Zone |
+--------------------------------------+-----------+------+-------------+----------------+-----------+-----------------+---------------------------------------+-------------------+
| b0b24d51-7785-415e-9c10-86218ad727a0 | testshare | 1 | NFS | error_deleting | False | cephfsnfstype | zorilla-cephnfs-rpm@cephfsnfs1#cephfs | manila-zone-0 |
+--------------------------------------+-----------+------+-------------+----------------+-----------+-----------------+---------------------------------------+-------------------+

manila-share log shows the following:
DEBUG oslo_concurrency.lockutils [None req-cc0acbf6-22af-4810-9159-64b7fc6eda98 demo None] Lock "locked_access_rules_operation_by_share_instance_97bf7384-a6ad-43ea-a2da-1b19803babf7" "released" by "manila.share.access.locked_access_rules_operation.<locals>.wrapped.<locals>.locked_operation" :: held 0.020s {{(pid=593653) inner /usr/local/lib/python3.9/site-packages/oslo_concurrency/lockutils.py:423}}
INFO manila.message.api [None req-cc0acbf6-22af-4810-9159-64b7fc6eda98 demo None] Creating message record for request_id = req-cc0acbf6-22af-4810-9159-64b7fc6eda98
ERROR oslo_messaging.rpc.server [None req-cc0acbf6-22af-4810-9159-64b7fc6eda98 demo None] Exception during message handling: manila.exception.GaneshaCommandFailure: Ganesha management command failed.
Command: sudo manila-rootwrap /etc/manila/rootwrap.conf dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:1002
Exit code: 1
Stdout: ''
Stderr: 'Error org.freedesktop.DBus.Error.InvalidArgs: lookup_export failed with Export id not found\n'
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/manager.py", line 233, in _execute
ERROR oslo_messaging.rpc.server return execute(*args, **kwargs)
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/utils.py", line 59, in __call__
ERROR oslo_messaging.rpc.server return self.execute(*args, **exkwargs)
ERROR oslo_messaging.rpc.server File "...

[Vida Haririan (vhariria)]

Adding workaround steps provided by Goutham Pacha Ravi
    "admins can reset the state of the share, and re-attempt share deletion and it'll work.."

[Vida Haririan (vhariria)]

See additional related discussions at https://meetings.opendev.org/meetings/manila/2023/manila.2023-10-05-15.00

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/898992

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/manila/+/900470

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/manila/+/900471

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/manila/+/900472

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/manila/+/900473

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/manila/+/900474

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/manila/+/900475

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/898992
Committed: https://opendev.org/openstack/manila/commit/609b925a51b230c93d8ad408cf8ba351657dd83e
Submitter: "Zuul (22348)"
Branch: master

commit 609b925a51b230c93d8ad408cf8ba351657dd83e
Author: Goutham Pacha Ravi <email address hidden>
Date: Fri Oct 20 15:25:01 2023 -0700

    Don't fail remove-export in NFS-Ganesha

    When dropping an export over DBUS, if there's a problem
    with the removal, we can ignore the failure and move on.
    If not, the share ends up in an "error_deleting" state.

    There's not much that we can do here besides logging
    the failure as an exception.

    Change-Id: I73c2706c9d889f8d08057a543b2a963740bb3f33
    Closes-Bug: #2035572
    Signed-off-by: Goutham Pacha Ravi <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/900470
Committed: https://opendev.org/openstack/manila/commit/d6eed0ea06f662946dcc7cfac3b90de38c168eaf
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit d6eed0ea06f662946dcc7cfac3b90de38c168eaf
Author: Goutham Pacha Ravi <email address hidden>
Date: Fri Oct 20 15:25:01 2023 -0700

    Don't fail remove-export in NFS-Ganesha

    When dropping an export over DBUS, if there's a problem
    with the removal, we can ignore the failure and move on.
    If not, the share ends up in an "error_deleting" state.

    There's not much that we can do here besides logging
    the failure as an exception.

    Change-Id: I73c2706c9d889f8d08057a543b2a963740bb3f33
    Closes-Bug: #2035572
    Signed-off-by: Goutham Pacha Ravi <email address hidden>
    (cherry picked from commit 609b925a51b230c93d8ad408cf8ba351657dd83e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/900472
Committed: https://opendev.org/openstack/manila/commit/b76a035d770093b230c4de4ed3f81c5bc249c1ff
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit b76a035d770093b230c4de4ed3f81c5bc249c1ff
Author: Goutham Pacha Ravi <email address hidden>
Date: Fri Oct 20 15:25:01 2023 -0700

    Don't fail remove-export in NFS-Ganesha

    When dropping an export over DBUS, if there's a problem
    with the removal, we can ignore the failure and move on.
    If not, the share ends up in an "error_deleting" state.

    There's not much that we can do here besides logging
    the failure as an exception.

    Change-Id: I73c2706c9d889f8d08057a543b2a963740bb3f33
    Closes-Bug: #2035572
    Signed-off-by: Goutham Pacha Ravi <email address hidden>
    (cherry picked from commit 609b925a51b230c93d8ad408cf8ba351657dd83e)
    (cherry picked from commit d6eed0ea06f662946dcc7cfac3b90de38c168eaf)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/900473
Committed: https://opendev.org/openstack/manila/commit/13dabf77b9ae931c2d70ea76a0d13fdf0a5d26ae
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 13dabf77b9ae931c2d70ea76a0d13fdf0a5d26ae
Author: Goutham Pacha Ravi <email address hidden>
Date: Fri Oct 20 15:25:01 2023 -0700

    Don't fail remove-export in NFS-Ganesha

    When dropping an export over DBUS, if there's a problem
    with the removal, we can ignore the failure and move on.
    If not, the share ends up in an "error_deleting" state.

    There's not much that we can do here besides logging
    the failure as an exception.

    Change-Id: I73c2706c9d889f8d08057a543b2a963740bb3f33
    Closes-Bug: #2035572
    Signed-off-by: Goutham Pacha Ravi <email address hidden>
    (cherry picked from commit 609b925a51b230c93d8ad408cf8ba351657dd83e)
    (cherry picked from commit d6eed0ea06f662946dcc7cfac3b90de38c168eaf)
    (cherry picked from commit b76a035d770093b230c4de4ed3f81c5bc249c1ff)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/900471
Committed: https://opendev.org/openstack/manila/commit/739301f4d4ac4d7ec3e966eda52996b959bc176f
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 739301f4d4ac4d7ec3e966eda52996b959bc176f
Author: Goutham Pacha Ravi <email address hidden>
Date: Fri Oct 20 15:25:01 2023 -0700

    Don't fail remove-export in NFS-Ganesha

    When dropping an export over DBUS, if there's a problem
    with the removal, we can ignore the failure and move on.
    If not, the share ends up in an "error_deleting" state.

    There's not much that we can do here besides logging
    the failure as an exception.

    Change-Id: I73c2706c9d889f8d08057a543b2a963740bb3f33
    Closes-Bug: #2035572
    Signed-off-by: Goutham Pacha Ravi <email address hidden>
    (cherry picked from commit 609b925a51b230c93d8ad408cf8ba351657dd83e)
    (cherry picked from commit d6eed0ea06f662946dcc7cfac3b90de38c168eaf)
    (cherry picked from commit b76a035d770093b230c4de4ed3f81c5bc249c1ff)
    (cherry picked from commit 13dabf77b9ae931c2d70ea76a0d13fdf0a5d26ae)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/900474
Committed: https://opendev.org/openstack/manila/commit/51baf4e84ac2ecc98cf793cee4ddb2218d89019b
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 51baf4e84ac2ecc98cf793cee4ddb2218d89019b
Author: Goutham Pacha Ravi <email address hidden>
Date: Fri Oct 20 15:25:01 2023 -0700

    Don't fail remove-export in NFS-Ganesha

    When dropping an export over DBUS, if there's a problem
    with the removal, we can ignore the failure and move on.
    If not, the share ends up in an "error_deleting" state.

    There's not much that we can do here besides logging
    the failure as an exception.

    Change-Id: I73c2706c9d889f8d08057a543b2a963740bb3f33
    Closes-Bug: #2035572
    Signed-off-by: Goutham Pacha Ravi <email address hidden>
    (cherry picked from commit 609b925a51b230c93d8ad408cf8ba351657dd83e)
    (cherry picked from commit d6eed0ea06f662946dcc7cfac3b90de38c168eaf)
    (cherry picked from commit b76a035d770093b230c4de4ed3f81c5bc249c1ff)
    (cherry picked from commit 13dabf77b9ae931c2d70ea76a0d13fdf0a5d26ae)
    (cherry picked from commit 739301f4d4ac4d7ec3e966eda52996b959bc176f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/900475
Committed: https://opendev.org/openstack/manila/commit/5e8ab8bb612d2202d4fbba2f5a2623c4afeb528a
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 5e8ab8bb612d2202d4fbba2f5a2623c4afeb528a
Author: Goutham Pacha Ravi <email address hidden>
Date: Fri Oct 20 15:25:01 2023 -0700

    Don't fail remove-export in NFS-Ganesha

    When dropping an export over DBUS, if there's a problem
    with the removal, we can ignore the failure and move on.
    If not, the share ends up in an "error_deleting" state.

    There's not much that we can do here besides logging
    the failure as an exception.

    Change-Id: I73c2706c9d889f8d08057a543b2a963740bb3f33
    Closes-Bug: #2035572
    Signed-off-by: Goutham Pacha Ravi <email address hidden>
    (cherry picked from commit 609b925a51b230c93d8ad408cf8ba351657dd83e)
    (cherry picked from commit d6eed0ea06f662946dcc7cfac3b90de38c168eaf)
    (cherry picked from commit b76a035d770093b230c4de4ed3f81c5bc249c1ff)
    (cherry picked from commit 13dabf77b9ae931c2d70ea76a0d13fdf0a5d26ae)
    (cherry picked from commit 739301f4d4ac4d7ec3e966eda52996b959bc176f)
    (cherry picked from commit 51baf4e84ac2ecc98cf793cee4ddb2218d89019b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 17.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 16.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 15.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 14.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila wallaby-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila xena-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 18.0.0.0rc1 release candidate.