[victoria][ussuri][train c8]tempest.lib.exceptions.TimeoutException: Request timed out failing in scenario004-standalone

[Bhagyashri Shewale]

ft1.1: setUpClass (manila_tempest_tests.tests.api.test_shares.SharesNFSTest)testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/test.py", line 182, in setUpClass
    six.reraise(etype, value, trace)
  File "/usr/local/lib/python3.6/site-packages/six.py", line 719, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/tempest/test.py", line 175, in setUpClass
    cls.resource_setup()
  File "/usr/lib/python3.6/site-packages/manila_tempest_tests/tests/api/test_shares.py", line 47, in resource_setup
    share_type_id=cls.share_type_id)
  File "/usr/lib/python3.6/site-packages/manila_tempest_tests/tests/api/base.py", line 414, in create_share
    result = cls.create_shares([{"args": args, "kwargs": kwargs}])
  File "/usr/lib/python3.6/site-packages/manila_tempest_tests/tests/api/base.py", line 473, in create_shares
    client, share_id, "available")
  File "/usr/lib/python3.6/site-packages/manila_tempest_tests/common/waiters.py", line 105, in wait_for_resource_status
    raise exceptions.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: share a924aa2d-5767-441e-a0f3-5b90dc9ae19d failed to reach available status (current creating) within the required time (500 s).

Affected jobs:

1. periodic-tripleo-ci-centos-8-scenario004-standalone-victoria
2. periodic-tripleo-ci-centos-8-scenario004-standalone-ussuri
3. periodic-tripleo-ci-centos-8-scenario004-standalone-train

[1]: https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-victoria/8ab71a2/logs/undercloud/var/log/tempest/stestr_results.html.gz
[2]: https://logserver.rdoproject.org/50/29250/10/check/periodic-tripleo-ci-centos-8-scenario004-standalone-ussuri/3665ee6/logs/undercloud/var/log/tempest/stestr_results.html.gz
[3]: https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-train/472eca5/logs/undercloud/var/log/tempest/stestr_results.html.gz

[Bhagyashri Shewale (bhagyashri-shewale)]

Hi All,

Skiplist patch is here: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/799779

[chandan kumar (chkumar246)]

https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-train/472eca5/logs/undercloud/var/log/containers/manila/manila-api.log.txt.gz

```
ERROR manila.db.sqlalchemy.api [req-7e4473ac-9447-48f8-ab60-06e6153633c9 82fe93f6d03841359711abba93726bcd 014272edb16c450e95fd8287a5f3bbf8 - - -] Deletion of share type 1b18c8c3-2de6-4294-a9f4-66a41532f80b failed; it in use by 1 shares and 0 share group types

```

[chandan kumar (chkumar246)]

Similar bug report downstream https://bugzilla.redhat.com/show_bug.cgi?id=1976693

[chandan kumar (chkumar246)]

fix for ceph https://github.com/ceph/ceph/pull/42162 is merged now.
We need to do a build of ceph rpm on CBS then
We need to rebuild the ceph-daemon container with nautalius tag and then rebuild the manila-share container
then only we have working fix in the deployment.

In order to complete the whole step, it is going to take time. The Manila and Ceph team is working on that.

[Bhagyashri Shewale (bhagyashri-shewale)]

 https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/800474

[Marios Andreou (marios-b)]

picking up as ruck this sprint and catching up here...

The fix at [1] is now merged and based on comments in the downstream bug appears to address the issue.

I am testing to see if we can safely revert the skipped tests with [2] - test is at [3]

[1] https://github.com/ceph/ceph/pull/42162
[2] https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/801736
[3] https://review.rdoproject.org/r/c/testproject/+/34642

[Marios Andreou (marios-b)]

hmmm but i think we would have to bump that with the new version containing the fix though right https://opendev.org/openstack/tripleo-common/src/commit/58f20a155b7900665521f4953042cad5391398aa/container-images/tripleo_containers.yaml#L176

[Marios Andreou (marios-b)]

        * Bump nautilus containers to v4.0.19
        * Change-Id: I8cbba356867145a0076c5426f4ac7642195ea357
        * https://review.opendev.org/c/openstack/tripleo-common/+/801711

https://review.opendev.org/q/topic:nautilus_v4.0.19

[Marios Andreou (marios-b)]

So first we need to wait for https://review.opendev.org/q/topic:nautilus_v4.0.19

Then we can revert the skipped tempest tests with https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/801736

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/802618

[Marios Andreou (marios-b)]

issue is fixed as seen in the test results (scenario4) at [1] thanks jfulton!

The test is now passing see [2] manila_tempest_tests.tests.api.test_shares.SharesNFSTest

Moving the bug to fix-released (the revert is starting to work its way through the gate https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/801736)

[1] https://review.opendev.org/c/openstack/tripleo-heat-templates/+/802618/1#message-b24d149c27c258d3307b8034a0e5516302ca4b9f
[2] https://205666bcf957fd7de8bb-509bae49365f8635e29d5b96780095d8.ssl.cf1.rackcdn.com/802618/1/check/tripleo-ci-centos-8-scenario004-standalone/bd3b243/logs/undercloud/var/log/tempest/stestr_results.html

[OpenStack Infra (hudson-openstack)]

Change abandoned by "John Fulton <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/802618
Reason: test-only patch

[Marios Andreou (marios-b)]

So we need something more than what was included in https://review.opendev.org/q/topic:nautilus_v4.0.19.

I will revert the revert for now then we need to continue this skip, e.g. seen yesterday in ussuri https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-ussuri/db63fb8/logs/undercloud/var/log/tempest/stestr_results.html.gz

and again in a test i just ran with testproject https://logserver.rdoproject.org/41/34741/1/check/periodic-tripleo-ci-centos-8-scenario004-standalone-ussuri/7238660/logs/undercloud/var/log/tempest/stestr_results.html.gz

Pasting some comments from Franscesco below.
Thanks @fpantano (copy/paste):

    we need a new, fresh, Ceph nautilus packages rebuild that contains [1], which is needed by the clients to make the test working again.

    In order to get the change [1], someone from the Ceph org should tag a new nautilus release ( > 14.2.22) and make a new build in CentOS, so that the manila containers (and all the clients) can benefit from this fix

    Only after #1 and #2 we can rebuild the new OSP containers (see my previous comment - manila) and revert the skipped test.

[1] ceph/ceph: Pull Request 42162

[Marios Andreou (marios-b)]

there was a patch merged there https://opendev.org/openstack/openstack-tempest-skiplist/commit/cda261ed3271aaabe351145bd808f0d782a66350 which skips for

      - periodic-tripleo-ci-centos-8-scenario004-standalone-tripleo-victoria
      - periodic-tripleo-ci-centos-8-scenario004-standalone-tripleo-ussuri
      - periodic-tripleo-ci-centos-8-scenario004-standalone-tripleo-train

But we still need to re-add the other jobs (as seen today in ussuri see comment #13) posted there https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/802874

[Marios Andreou (marios-b)]

unfortunately ceph is no longer accepting fixes for nautilus

which means that for train ussuri and victoria we will have to continue to skip this with https://opendev.org/openstack/openstack-tempest-skiplist/src/commit/3c94fdcb6c0329e2e758f1c0fded9a18d99ab644/roles/validate-tempest/vars/tempest_skip.yml#L1620-L1626