Share teardown failure with the native cephfs driver

Bug #1970661 reported by Goutham Pacha Ravi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Fix Released
Medium
Goutham Pacha Ravi

Bug Description

Description
===========
When running tempest tests on the CI, we currently set "suppress_errors_in_cleanup=True" [1] within the ceph-native job. I'm not sure why this was done, but I attempted disabling this; i.e., allow cleanup errors to be surfaced as test failures. This resulted in at least one share not being cleaned up:

    ShareCephxRulesForCephFSNegativeTest)
fail
[x]

ft1.1: tearDownClass (manila_tempest_tests.tests.api.test_rules_negative.ShareCephxRulesForCephFSNegativeTest)testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/test.py", line 220, in tearDownClass
    raise value.with_traceback(trace)
  File "/opt/stack/tempest/tempest/test.py", line 192, in tearDownClass
    teardown()
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/manila_tempest_tests/tests/api/base.py", line 1248, in resource_cleanup
    super(BaseSharesMixedTest, cls).resource_cleanup()
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/manila_tempest_tests/tests/api/base.py", line 181, in resource_cleanup
    cls.clear_resources(cls.class_resources)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/manila_tempest_tests/tests/api/base.py", line 887, in clear_resources
    client.wait_for_resource_deletion(share_id=res_id)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/manila_tempest_tests/services/share/json/shares_client.py", line 345, in wait_for_resource_deletion
    raise exceptions.TimeoutException
tempest.lib.exceptions.TimeoutException: Request timed out
Details: None

Steps to reproduce
==================

A chronological list of steps which will help reproduce the issue you hit:
* Disable suppressing of cleanup errors like in [2]

Expected result
===============
No cleanup failures occur

Actual result
=============
Cleanup fails on some shares created during the tests

Environment
===========
1. Exact version of OpenStack Manila you are running. Master

2. Which storage backend did you use? CephFS (Native)

3. Which networking type did you use? Irrelevant

Logs & Configs
==============

Tempest request to delete a share from the failed test above:

Relevant logging in manila-share: https://paste.opendev.org/show/bHw6Xl3oz552BOlIEpYw/

I'm unsure why the driver's using the auth-id as "admin" when we use "manila" in the jobs [3][4]

[1] https://opendev.org/openstack/manila-tempest-plugin/src/commit/c905e77308d48c4582afed88d52bbf3780895383/zuul.d/manila-tempest-jobs.yaml#L324
[2] https://review.opendev.org/c/openstack/manila-tempest-plugin/+/839438
[3] https://github.com/openstack/manila/blob/43ebc12fb53546a286ac6180db52ee8bbb775b1a/manila/share/drivers/cephfs/driver.py#L92-L95
[4] https://opendev.org/openstack/manila-tempest-plugin/src/commit/c905e77308d48c4582afed88d52bbf3780895383/zuul.d/manila-tempest-jobs.yaml#L310

Revision history for this message
Vida Haririan (vhariria) wrote :
Changed in manila:
importance: Undecided → Medium
tags: added: low-hanging-fruit
Changed in manila:
milestone: none → zed-1
Changed in manila:
assignee: nobody → Carlos Eduardo (silvacarlose)
Changed in manila:
milestone: zed-1 → zed-2
Changed in manila:
milestone: zed-2 → zed-3
Changed in manila:
milestone: zed-3 → zed-rc1
Changed in manila:
status: New → In Progress
Changed in manila:
assignee: Carlos Eduardo (silvacarlose) → Goutham Pacha Ravi (gouthamr)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila-tempest-plugin (master)

Reviewed: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/839438
Committed: https://opendev.org/openstack/manila-tempest-plugin/commit/79cad927652abfc99415d32b7d92152b3ecd326d
Submitter: "Zuul (22348)"
Branch: master

commit 79cad927652abfc99415d32b7d92152b3ecd326d
Author: Goutham Pacha Ravi <email address hidden>
Date: Tue Sep 13 11:56:02 2022 -0700

    [CI] Stop suppressing cleanup errors in CephFS jobs

    Change-Id: I44f0bf50c3d7042565dca54949965b2c4df1fb41
    Closes-Bug: #1970661
    Signed-off-by: Goutham Pacha Ravi <email address hidden>

Changed in manila:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila-tempest-plugin 1.10.0

This issue was fixed in the openstack/manila-tempest-plugin 1.10.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.