Wallaby: periodic-tripleo-ci-centos-8-scenario004-standalone-wallaby is failing manila_tempest_tests.tests.api.test_shares.SharesNFSTest setup

[Ronelle Landy]

periodic-tripleo-ci-centos-8-scenario004-standalone-wallaby is failing tempest test:
manila_tempest_tests.tests.api.test_shares.SharesNFSTest

setUpClass (manila_tempest_tests.tests.api.test_shares.SharesNFSTest)
---------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/tempest/test.py", line 182, in setUpClass
        six.reraise(etype, value, trace)
      File "/usr/local/lib/python3.6/site-packages/six.py", line 703, in reraise
        raise value
      File "/usr/lib/python3.6/site-packages/tempest/test.py", line 175, in setUpClass
        cls.resource_setup()
      File "/usr/lib/python3.6/site-packages/manila_tempest_tests/tests/api/test_shares.py", line 47, in resource_setup
        share_type_id=cls.share_type_id)
      File "/usr/lib/python3.6/site-packages/manila_tempest_tests/tests/api/base.py", line 414, in create_share
        result = cls.create_shares([{"args": args, "kwargs": kwargs}])
      File "/usr/lib/python3.6/site-packages/manila_tempest_tests/tests/api/base.py", line 473, in create_shares
        client, share_id, "available")
      File "/usr/lib/python3.6/site-packages/manila_tempest_tests/common/waiters.py", line 99, in wait_for_resource_status
        raise resource_exception(resource_id=resource_id)
    manila_tempest_tests.share_exceptions.ShareBuildErrorException: Share 24f9a9b7-d2d9-49a6-9002-99547d909de6 failed to build and is in ERROR status

The full log is below:
https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-wallaby/25421d3/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

The same test passes in master:

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-master/51d528f/logs/undercloud/var/log/tempest/tempest_results.xml.gz

[Giulio Fidente (gfidente)]

The manila-scheduler seems to be filtering out the -share service on:

2021-04-05 23:01:59.386 8 DEBUG manila.scheduler.filters.capabilities [req-d47c4368-33b9-4942-b7d3-a4af938d24c1 db4fcb254f5b422cb2d02bf295e17e8f 62a4d760ab824b11ba83918ad5659769 - - -] host: 'hostgroup@cephfs#cephfs', free_capacity_gb: 6.95, pools: None fails resource_type extra_specs requirements host_passes /usr/lib/python3.6/site-packages/manila/scheduler/filters/capabilities.py:45

later results in

2021-04-05 23:01:59.387 8 ERROR manila.scheduler.manager [req-d47c4368-33b9-4942-b7d3-a4af938d24c1 db4fcb254f5b422cb2d02bf295e17e8f 62a4d760ab824b11ba83918ad5659769 - - -] Failed to schedule create_share: No valid host was found. Failed to find a weighted host, the last executed filter was CapabilitiesFilter.: manila.exception.NoValidHost: No valid host was found. Failed to find a weighted host, the last executed filter was CapabilitiesFilter.

full log at

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-wallaby/25421d3/logs/undercloud/var/log/containers/manila/manila-scheduler.log.txt.gz

[Giulio Fidente (gfidente)]

In the logs I see:

Share type extra spec requirement "snapshot_support=<is> False" does not match reported capability "True" capabilities_satisfied /usr/lib/python3.6/site-packages/manila/scheduler/utils.py:178

but the config file disables that:

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-wallaby/25421d3/logs/undercloud/var/lib/config-data/puppet-generated/manila/etc/manila/manila.conf.txt.gz

is that the problem?

[Tom Barron (tpb)]

We need to ensure that the capability_create_share_from_snapshot and capability_snapshot_support lines in the share section of tempest.conf are removed [1]. Starting with Wallaby, these are fully supported and always True when CephFS back ends are used for Manila. The lines can simply be removed since their default setting is True.

[share]
capability_create_share_from_snapshot_support = False
capability_snapshot_support = False

[1] https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-master/51d528f/logs/undercloud/home/zuul/tempest/etc/tempest.conf.txt.gz

We can do this in the tripleo-ci jobs definitions.

[Tom Barron (tpb)]

We need to remove these two lines in the tripleo-ci job definition [1] but only for Wallaby and newer.

[1] https://github.com/openstack/tripleo-ci/blob/master/zuul.d/standalone-jobs.yaml#L651-L652

[Goutham Pacha Ravi (gouthamr)]

Partial fix proposed to branch: master (tripleo-ci)
    https://review.opendev.org/c/openstack/tripleo-ci/+/785254

[Goutham Pacha Ravi (gouthamr)]

Related fix proposed to branch: master (python-tempestconf)
    https://review.opendev.org/c/osf/python-tempestconf/+/785450

[Goutham Pacha Ravi (gouthamr)]

TBH, tripleo jobs are running a basic manila integration test that doesn't test snapshots at all; so the tempest plugin could ignore the "snapshot_support" and "create_share_from_snapshot_support" optional capabilities too.

We're working along those lines and limiting the use of these optional directives in the tests: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/785307

This will take some time and testing on our end because these have been around for a long time. In the meanwhile, allowing auto configuration via tempestconf is a decent solution that should work across releases - and that would prevent us from needing to investigate a way to set test configuration per release.

[Ronelle Landy (rlandy)]

periodic-tripleo-ci-centos-8-scenario004-standalone-wallaby success - we can close this out