[periodic/check][master] multinode fs016/019 & scenario-10 job failed while deploying overcloud with Permission denied: '/var/lib/mistral/overcloud/ceph-ansible

[chandan kumar]

Multinode fs016/f019 job in promotion pipeline and scenario-10 (fs038) in check jobs are failing while deploying overcloud with following errors:
2019-02-11 00:48:03 | There was an issue creating /var/lib/mistral/overcloud/ceph-ansible as requested: [Errno 13] Permission denied: '/var/lib/mistral/overcloud/ceph-ansible'
2019-02-11 00:48:03 | failed: [undercloud] (item=/var/lib/mistral/overcloud/ceph-ansible/host_vars) => {
2019-02-11 00:48:03 | "changed": false,
2019-02-11 00:48:03 | "item": "/var/lib/mistral/overcloud/ceph-ansible/host_vars",
2019-02-11 00:48:03 | "path": "/var/lib/mistral/overcloud/ceph-ansible/host_vars",
2019-02-11 00:48:03 | "state": "absent"
2019-02-11 00:48:03 | }
2019-02-11 00:48:03 |
2019-02-11 00:48:03 | MSG:
2019-02-11 00:48:03 |
2019-02-11 00:48:03 | There was an issue creating /var/lib/mistral/overcloud/ceph-ansible as requested: [Errno 13] Permission denied: '/var/lib/mistral/overcloud/ceph-ansible'
2019-02-11 00:48:03 | failed: [undercloud] (item=/var/lib/mistral/overcloud/ceph-ansible/fetch_dir) => {
2019-02-11 00:48:03 | "changed": false,
2019-02-11 00:48:03 | "item": "/var/lib/mistral/overcloud/ceph-ansible/fetch_dir",
2019-02-11 00:48:03 | "path": "/var/lib/mistral/overcloud/ceph-ansible/fetch_dir",
2019-02-11 00:48:03 | "state": "absent"
2019-02-11 00:48:03 | }
2019-02-11 00:48:03 |
2019-02-11 00:48:03 | MSG:
2019-02-11 00:48:03 |
2019-02-11 00:48:03 | There was an issue creating /var/lib/mistral/overcloud/ceph-ansible as requested: [Errno 13] Permission denied: '/var/lib/mistral/overcloud/ceph-ansible'

Below is the logs:
fs016: https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/e511b59/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-02-11_00_48_03

fs019: https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/7f72339/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-02-11_00_36_57

scenario10:
http://logs.openstack.org/98/604298/224/check/tripleo-ci-centos-7-scenario010-multinode-oooq-container/c19dd2f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-02-11_01_46_49

It is coming only where multinode ceph is getting tested.

Reason: ceph-ansible is now invoked on the undercloud host with user tripleo-admin, so either this user will need write access to /var/lib/mistral/overcloud/ceph-ansible, or the working directory needs to be moved to ~tripleo-admin/overcloud, or some of those
external_deploy_tasks will need to be become:true

Earlier it was running as the mistral user inside the mistral-executor container, which was really not ideal. mistral user is the owner of that directory, so it was working earlier.
It needs to be fixed.

https://review.openstack.org/#/c/636086/ might fix the issue.
Changes made to address this bug https://bugs.launchpad.net/tripleo/+bug/1813832 leads to this error

[Quique Llorente (quiquell)]

fs016 and fs019 are fixed now

[Quique Llorente (quiquell)]

Wip for scenario010 https://review.openstack.org/#/c/636571/

[wes hayutin (weshayutin)]

all the periodic jobs that were failing are now passing.

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/a4d7ea8/

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/541901c/

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master/e2b8fd6/

Will double check what fixed fs010

[wes hayutin (weshayutin)]

looks like https://review.openstack.org/#/c/636086/ fixed it...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/636844