OpenStack Shared File Systems Service (Manila)

Share stuck in "error_deleting" status after running "test_access_rules_metadata" tests

Bug #1971530 reported by Liron Kuchlani on 2022-05-04

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Shared File Systems Service (Manila)	Fix Released	Medium	Goutham Pacha Ravi	OpenStack Shared File Systems Service (Manila) zed-1

Bug Description

Description of problem:
Test "test_set_get_delete_access_metadata" is failed with timeout error
because the waiter method is waiting for the share resource to be deleted,
however, the share stuck in "error_deleting" status.

Version-Release number of selected component (if applicable):
openstack-manila-12.1.2-0.20220404161416.f5b2cdc.el9ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. stestr run manila_tempest_tests.tests.api.test_access_rules_metadata.AccessRulesMetadataTest

Actual results:
Share stuck in "error deleting" status

Expected results:
Share should be deleted successfully

Additional info:

From manila-share.log
======================
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server [req-b2d6f069-58ab-4950-ba97-0655c5e19825 7db6eb0754d543fd96372845c5d4758b 5e8574ca66f44c16b521dac0185f1091 - - -] Exception during message handling: manila.exception.ShareBackendException: json_command failed - prefix=fs subvolume authorize, argdict={'vol_name': 'cephfs', 'sub_name': '859aa3a1-e5af-4eb4-8440-29e263689b55', 'auth_id': 'Joe', 'tenant_id': '5e8574ca66f44c16b521dac0185f1091', 'access_level': 'rw', 'format': 'json'} - exception message: [errno -1] auth ID: Joe is already in use.
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/manila/share/drivers/cephfs/driver.py", line 193, in rados_command
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server raise rados.Error(outs, ret)
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server rados.Error: [errno -1] auth ID: Joe is already in use
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/manila/share/manager.py", line 219, in wrapped
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/manila/utils.py", line 578, in wrapper
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/manila/share/manager.py", line 3920, in update_access
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server self.update_access_for_instances(context, [share_instance_id],
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/manila/share/manager.py", line 3934, in update_access_for_instances
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server self.access_helper.update_access_rules(
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/manila/share/access.py", line 301, in update_access_rules
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server self._update_access_rules(context, share_instance_id,
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/manila/share/access.py", line 338, in _update_access_rules
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server driver_rule_updates = self._update_rules_through_share_driver(
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/manila/share/access.py", line 403, in _update_rules_through_share_driver
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server driver_rule_updates = self.driver.update_access(
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/manila/share/drivers/cephfs/driver.py", line 511, in update_access
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server return self.protocol_helper.update_access(
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/manila/share/drivers/cephfs/driver.py", line 896, in update_access
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server access_key = self._allow_access(context, share, rule)
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/manila/share/drivers/cephfs/driver.py", line 822, in _allow_access
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server auth_result = rados_command(
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/manila/share/drivers/cephfs/driver.py", line 205, in rados_command
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server raise exception.ShareBackendException(msg)
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server manila.exception.ShareBackendException: json_command failed - prefix=fs subvolume authorize, argdict={'vol_name': 'cephfs', 'sub_name': '859aa3a1-e5af-4eb4-8440-29e263689b55', 'auth_id': 'Joe', 'tenant_id': '5e8574ca66f44c16b521dac0185f1091', 'access_level': 'rw', 'format': 'json'} - exception message: [errno -1] auth ID: Joe is already in use.
2022-05-02 11:42:40.973 9 ERROR oslo_messaging.rpc.server

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-05-04: Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/840430

Changed in manila:
status:	New → In Progress

Goutham Pacha Ravi (gouthamr) on 2022-05-04

Changed in manila:
assignee:	nobody → Goutham Pacha Ravi (gouthamr)
milestone:	none → zed-1
importance:	Undecided → Medium

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-05-12: Fix merged to manila (master)

Reviewed: https://review.opendev.org/c/openstack/manila/+/840430
Committed: https://opendev.org/openstack/manila/commit/74e1fe4761b3bc4efb9a60d1b2887766c9ce43a9
Submitter: "Zuul (22348)"
Branch: master

commit 74e1fe4761b3bc4efb9a60d1b2887766c9ce43a9
Author: Goutham Pacha Ravi <email address hidden>
Date: Wed May 4 13:41:54 2022 +0530

[Native CephFS] Don't fail to deny missing rules

    In rare scenarios, an access rule known to manila may
    be missing from CephFS. The driver shouldn't raise an
    exception when this happens.

    Change-Id: Iaeb84f1d9f4c04c23f470ad777d7d6cf2455f543
    Closes-Bug: #1971530
    Signed-off-by: Goutham Pacha Ravi <email address hidden>

Changed in manila:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-05-12: Fix proposed to manila (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/manila/+/841479

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-05-12: Fix proposed to manila (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/manila/+/841480

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-05-12: Fix proposed to manila (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/manila/+/841481

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-05-12: Fix merged to manila (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/manila/+/841479
Committed: https://opendev.org/openstack/manila/commit/a607ed8e999209f8a369f05191a0490ca426f4d6
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit a607ed8e999209f8a369f05191a0490ca426f4d6
Author: Goutham Pacha Ravi <email address hidden>
Date: Wed May 4 13:41:54 2022 +0530

[Native CephFS] Don't fail to deny missing rules

    In rare scenarios, an access rule known to manila may
    be missing from CephFS. The driver shouldn't raise an
    exception when this happens.

tags:

added: in-stable-yoga

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-05-12: Fix merged to manila (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/manila/+/841480
Committed: https://opendev.org/openstack/manila/commit/8805f4f3da2d474b5620f1e9d94eb984e3897db0
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 8805f4f3da2d474b5620f1e9d94eb984e3897db0
Author: Goutham Pacha Ravi <email address hidden>
Date: Wed May 4 13:41:54 2022 +0530

[Native CephFS] Don't fail to deny missing rules

    In rare scenarios, an access rule known to manila may
    be missing from CephFS. The driver shouldn't raise an
    exception when this happens.

    Change-Id: Iaeb84f1d9f4c04c23f470ad777d7d6cf2455f543
    Closes-Bug: #1971530
    Signed-off-by: Goutham Pacha Ravi <email address hidden>
    (cherry picked from commit 74e1fe4761b3bc4efb9a60d1b2887766c9ce43a9)
    (cherry picked from commit a607ed8e999209f8a369f05191a0490ca426f4d6)

tags:

added: in-stable-xena

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-05-16: Fix merged to manila (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/manila/+/841481
Committed: https://opendev.org/openstack/manila/commit/3d844d6bcad5f1a9a6407cae512dc8685bbd12fb
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 3d844d6bcad5f1a9a6407cae512dc8685bbd12fb
Author: Goutham Pacha Ravi <email address hidden>
Date: Wed May 4 13:41:54 2022 +0530

[Native CephFS] Don't fail to deny missing rules

    In rare scenarios, an access rule known to manila may
    be missing from CephFS. The driver shouldn't raise an
    exception when this happens.

tags:

added: in-stable-wallaby

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-09-16: Fix included in openstack/manila 15.0.0.0rc1

This issue was fixed in the openstack/manila 15.0.0.0rc1 release candidate.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-12-05: Fix included in openstack/manila 14.0.1

#10

This issue was fixed in the openstack/manila 14.0.1 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-12-05: Fix included in openstack/manila 13.0.4

#11

This issue was fixed in the openstack/manila 13.0.4 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-06: Fix included in openstack/manila wallaby-eom

#12

This issue was fixed in the openstack/manila wallaby-eom release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.