[master] undercloud reinstall failed with invalid selinux context: [Errno 95] Operation not supported

[chandan kumar]

FS020 master periodic job is consistently failing while doing undercloud reinstall with the following errors:

http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/8ee317f/logs/undercloud/home/zuul/undercloud_reinstall.log.txt.gz

TASK [Create /var/lib/config-data directory] ***********************************
2018-09-24 12:48:21 | fatal: [undercloud]: FAILED! => {"changed": false, "cur_context": ["system_u", "object_r", "proc_t", "s0"], "gid": 0, "group": "root", "input_was": [null, null, "svirt_sandbox_file_t", "s0"], "mode": "0777", "msg": "invalid selinux context: [Errno 95] Operation not supported", "new_context": ["system_u", "object_r", "svirt_sandbox_file_t", "s0"], "owner": "root", "path": "/proc/mounts", "secontext": "system_u:object_r:proc_t:s0", "size": 11, "state": "link", "uid": 0}
2018-09-24 12:48:21 |
2018-09-24 12:48:21 | NO MORE HOSTS LEFT *************************************************************
2018-09-24 12:48:21 |
2018-09-24 12:48:21 | PLAY RECAP *********************************************************************
2018-09-24 12:48:21 | undercloud : ok=178 changed=38 unreachable=0 failed=1

selinux context is preventing to create /var/lib/config-data directory which is needed by various container while installing the undercloud.

There might be some issue with selinux context

Here are the failures from other run:
http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/c273e37/logs/undercloud/home/zuul/undercloud_reinstall.log.txt.gz

Some related issue in same job:
http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/bff71b8/logs/undercloud/home/zuul/undercloud_reinstall.log.txt.gz

2018-09-24 01:27:33 | TASK [Create /var/lib/config-data directory] ***********************************
2018-09-24 01:27:36 | fatal: [undercloud]: FAILED! => {"changed": false, "msg": "path /var/lib/config-data/mysql/etc/../usr/share/zoneinfo/UTC does not exist", "path": "/var/lib/config-data/mysql/etc/../usr/share/zoneinfo/UTC", "state": "absent"}
2018-09-24 01:27:36 |

[Quique Llorente (quiquell)]

Root cause of it https://review.openstack.org/#/c/600532/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/605037

[Cédric Jeanneret (cjeanner)]

So maybe a whole revert isn't the right thing - if I understand the main issue is due to the "recurse" flag https://review.openstack.org/#/c/600532/7/common/deploy-steps-tasks.yaml@115

We might want to drop that one and check again. In // I can ensure this is running as expected with podman+selinux, as the patch being reverted is kind of the "root" of all the podman+selinux work I've done for the last 3-4 weeks ;). I'd rather avoid seeing it reverted like that.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/605039

[Quique Llorente (quiquell)]

Testing 'recurse' here
https://review.rdoproject.org/r/#/c/16354

[OpenStack Infra (hudson-openstack)]

Change abandoned by Quique Llorente (<email address hidden>) on branch: master
Review: https://review.openstack.org/605037
Reason: Not needed

[Cédric Jeanneret (cjeanner)]

My patch still works when deploying podman+selinux. Good point :). CI still running, might be good as well on that side.

[Alex Schultz (alex-schultz)]

We really should land https://review.openstack.org/#/c/602703/ which should address this as we should *not* be running selinux upstream.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605039
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=57154fd084309cacf57fbc1d1e1f2d154ad5b49a
Submitter: Zuul
Branch: master

commit 57154fd084309cacf57fbc1d1e1f2d154ad5b49a
Author: Cédric Jeanneret <email address hidden>
Date: Tue Sep 25 13:20:12 2018 +0200

    Dropped "recurse" for idempotency

    We should not need that recurse anyway.

    Change-Id: I504b52a2bb3c89e75ac3402f259c317889c054e6
    Closes-Bug: #1794251

[Marios Andreou (marios-b)]

adding a note as rover ... so this particular bug seems to be fixed by Cedric review @ https://review.openstack.org/605039 which just merged, in the case that you _do_ have enforcing selinux.

The other bug at https://bugs.launchpad.net/tripleo/+bug/1779005 (with Alex fix pointed at in comment #9 @ https://review.openstack.org/#/c/602703/) isn't a duplicate - that is more generally about enabling/not selinux.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.0.0 release.