Manila tests are failing in featureset019 and scenario004

[Arx Cruz]

Started today

http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario004-standalone-master/727e7db/logs/tempest.html

ft1.1: setUpClass (manila_tempest_tests.tests.api.test_shares.SharesCephFSTest)_StringException: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/test.py", line 172, in setUpClass
    six.reraise(etype, value, trace)
  File "/usr/lib/python2.7/site-packages/tempest/test.py", line 165, in setUpClass
    cls.resource_setup()
  File "/usr/lib/python2.7/site-packages/manila_tempest_tests/tests/api/test_shares.py", line 42, in resource_setup
    share_type_id=cls.share_type_id)
  File "/usr/lib/python2.7/site-packages/manila_tempest_tests/tests/api/base.py", line 497, in create_share
    result = cls.create_shares([{"args": args, "kwargs": kwargs}])
  File "/usr/lib/python2.7/site-packages/manila_tempest_tests/tests/api/base.py", line 555, in create_shares
    client.wait_for_share_status(share_id, "available")
  File "/usr/lib/python2.7/site-packages/manila_tempest_tests/services/share/v2/json/shares_client.py", line 370, in wait_for_share_status
    share_id=share_id)
manila_tempest_tests.share_exceptions.ShareBuildErrorException: Share 06c7fd7a-8fcd-4313-8bd0-93b1da0c469d failed to build and is in ERROR status

This affect also featureset019:

http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/8d38ff9/logs/undercloud/home/zuul/tempest.log.txt.gz

[Arx Cruz (arxcruz)]

According Tom:

manila now need RHCS 3.2 but is getting 3.1.

https://review.openstack.org/#/c/630927/ and
https://review.openstack.org/#/c/630925/ should address this.

[Tom Barron (tpb)]

An probably most critical we need the upstream equivalent of 3.2 (12.2.10) baked into the manila-share container image. gfidente is doing this via https://cbs.centos.org/koji/taskinfo?taskID=694345

[Quique Llorente (quiquell)]

Still not addressed http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario004-standalone-master/afaa26c/logs/undercloud/var/log/extra/docker/docker_allinfo.log.txt.gz

[Quique Llorente (quiquell)]

After version upgrade still failing
http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario004-standalone-master/5a484b6/logs/tempest.html

[Quique Llorente (quiquell)]

Looks like we need ceph-12.2.11 it includes the following backport https://github.com/ceph/ceph/pull/25407

[Tom Barron (tpb)]

Apologies for claiming that ceph-12.2.10 would suffice but in the upstream manila CI job [1] for the review that merged the corresponding manila change in master [2] dpkg is reporting 12.2.10 [3].

[1] http://logs.openstack.org/32/614332/13/check/manila-tempest-minimal-dsvm-cephfs-native/6cb8e6d/
[2] https://review.openstack.org/#/c/614332
[3] http://logs.openstack.org/32/614332/13/check/manila-tempest-minimal-dsvm-cephfs-native/6cb8e6d/logs/dpkg-l.txt.gz

[wes hayutin (weshayutin)]

Tom is there a patch that resolves this? /me looking

[wes hayutin (weshayutin)]

<tbarron> weshay: yah, ykarel and quiquell are pulling in ceph 12.2.11 into the CBS and building the OSP containers with it

[Quique Llorente (quiquell)]

ceph 12.2.11 is at cbs now https://cbs.centos.org/koji/, but we need some promotions to happend so we get manila containers builds, we have a pair of promotion blockers to merge https://review.openstack.org/#/c/635108/

[wes hayutin (weshayutin)]

Just to clarify for folks... Upstream jobs seem to be working fine, RDO promotion jobs are failing. A promotion should not be required to see this passing in promotion jobs themselves :)

Upstream:
http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario004-standalone
http://logs.openstack.org/44/630644/35/check/tripleo-ci-centos-7-scenario004-standalone/51ad46c/logs/tempest.html

Promotion:
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-scenario004-standalone-master

If the package is available we should see it in the promotion jobs after any other container build or requisite issues are resolved.

[Quique Llorente (quiquell)]

Manila working fine http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/45976be/logs/tempest.html

[Marios Andreou (marios-b)]

o/ folks adding a note cos I'm seeing this at http://logs.openstack.org/63/636563/5/check/tripleo-ci-centos-7-scenario004-standalone/6658354/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-02-18_14_43_45

https://review.openstack.org/#/c/636563/ here - removing the pacemaker resources from standalones - I got 1-3 green but 4 is failing like this bug afaics

[Tom Barron (tpb)]

@Marios - Yours this is a different failure. This one was about needing a new version of ceph baked into the manila service containers. Yours is about manila not working right when pacemaker support is removed. Signature is (from /var/log/containers/manila/manila-{share,scheduler}.logs:

2019-02-18 14:30:34.431 29 ERROR oslo_service.service [req-2cef942a-4db1-4ffe-a37b-ec75ea9cb1a4 - - - - -] Error starting thread.: ToozConnectionError: [Errno 13] Permission denied: '/var/lib/manila/groups'
2019-02-18 14:30:34.431 29 ERROR oslo_service.service Traceback (most recent call last):
2019-02-18 14:30:34.431 29 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 796, in run_service
2019-02-18 14:30:34.431 29 ERROR oslo_service.service service.start()
2019-02-18 14:30:34.431 29 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/manila/service.py", line 111, in start
2019-02-18 14:30:34.431 29 ERROR oslo_service.service coordination.LOCK_COORDINATOR.start()
2019-02-18 14:30:34.431 29 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/manila/coordination.py", line 68, in start
2019-02-18 14:30:34.431 29 ERROR oslo_service.service self.coordinator.start(start_heart=True)
2019-02-18 14:30:34.431 29 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 690, in start
2019-02-18 14:30:34.431 29 ERROR oslo_service.service super(CoordinationDriverWithExecutor, self).start(start_heart)
2019-02-18 14:30:34.431 29 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 426, in start
2019-02-18 14:30:34.431 29 ERROR oslo_service.service self._start()
2019-02-18 14:30:34.431 29 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/tooz/drivers/file.py", line 273, in _start
2019-02-18 14:30:34.431 29 ERROR oslo_service.service raise coordination.ToozConnectionError(e)
2019-02-18 14:30:34.431 29 ERROR oslo_service.service ToozConnectionError: [Errno 13] Permission denied: '/var/lib/manila/groups'
2019-02-18 14:30:34.431 29 ERROR oslo_service.service

I suspect a missing bind mount but more is likely wrong since there was never any intention to support manila without pacemaker control of the service when it was implemented.