RHEL-8 jobs for train and master got broken with Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!

[chandan kumar]

RHEL-8 master and train all standalone and ovb fs01 jobs are broken in periodic and check pipeline

https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-master/1c5b2c3/job-output.txt

2020-02-06 20:14:42.721983 | primary | TASK [tripleo-inventory : create inventory from template] **********************
2020-02-06 20:14:42.733897 | primary | Thursday 06 February 2020 20:14:42 -0500 (0:00:00.101) 0:00:04.209 *****
2020-02-06 20:14:43.815918 | primary | fatal: [undercloud -> localhost]: FAILED! => {
2020-02-06 20:14:43.815962 | primary | "changed": false,
2020-02-06 20:14:43.815971 | primary | "checksum": "fafb8e9944284327c01d32b910d6074549d397cf"
2020-02-06 20:14:43.816000 | primary | }
2020-02-06 20:14:43.816009 | primary |
2020-02-06 20:14:43.816014 | primary | MSG:
2020-02-06 20:14:43.816042 | primary |
2020-02-06 20:14:43.816049 | primary | Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!

In collect logs, this message is seen
https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-master/1c5b2c3/logs/quickstart_install.log

TASK [Gathering Facts] *********************************************************
Thursday 06 February 2020 20:14:31 -0500 (0:00:00.262) 0:00:00.262 *****
[DEPRECATION WARNING]: Distribution redhat 8.1 on host 127.0.0.2 should use
/usr/libexec/platform-python, but is using /usr/bin/python for backward
compatibility with prior Ansible releases. A future Ansible release will
default to using the discovered platform python for this host. See https://docs
.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for
more information. This feature will be removed in version 2.12. Deprecation
warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
ok: [127.0.0.2]

 Looks like /usr/bin/python exist in rhel8.1

Since selinux in permissive mode used on RHEL-8 system and it requires selinux binding.

https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-master/bf79dea/logs/undercloud/var/log/extra/rpm-list.txt.gz

ansible-2.8.3-1.el8ae.noarch is used which does not auto detect the interpreter.
in RHEL-8, we have python2 and python3 binary which might be falling back to python2 ansible python interpreter leading to requires libselinux-python

It also blocks log collections and same error is happening.

Till yesterday the jobs were passing, https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&pipeline=openstack-periodic-latest-released#

Few days back we removed ansible-python-interpreter setting in release file https://review.opendev.org/#/c/705070/

But jobs were passing till yesterday.

RHEL-8.1 nodepool image got built yesterday https://softwarefactory-project.io/nodepool-log/upstream-rhel-8-0000018149.log after that it is happening.

Solution:
1. Either revert the patch or pull ansible-2.9

For info:
https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_2.8.html#python-interpreter-discovery

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/706465

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/706465
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=f81189367e1a2ec1ce15e764d10bef54ac07f9d1
Submitter: Zuul
Branch: master

commit f81189367e1a2ec1ce15e764d10bef54ac07f9d1
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Fri Feb 7 10:47:21 2020 +0000

    Revert "Remove hardcoding of python interpreter for rhel8"

    In RHEL-8, ansible-2.8 is still used.

    Related-Bug: #1862330

    This reverts commit f5c3f2103129fd35119f2671323be12c6aabfb6f.

    Change-Id: I6033b06336854e01b8ff719165d67702174d95fb

[wes hayutin (weshayutin)]

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-rhel-8-standalone-master

the standalone rhel8 jobs are fine..
Digging into ovb in another bug.