Buld-containers-quay-push: libselinux-python aren't installed

[Amol Kahat]

Build containers quay push job failing at containers build.

Traceback
```
2022-11-16 11:29:10 | 2022-11-16 11:29:10.915263 | fa163e61-b74e-4e6a-4c68-000000000073 | FATAL | Create a container file | tripleo-ansible-ee | error={
2022-11-16 11:29:10 | "changed": false,
2022-11-16 11:29:10 | "checksum": "647e2ec42b05c969c07b558ed395681649ecee2a",
2022-11-16 11:29:10 | "diff": [],
2022-11-16 11:29:10 | "invocation": {
2022-11-16 11:29:10 | "module_args": {
2022-11-16 11:29:10 | "_original_basename": "Containerfile.j2",
2022-11-16 11:29:10 | "attributes": null,
2022-11-16 11:29:10 | "backup": false,
2022-11-16 11:29:10 | "checksum": "647e2ec42b05c969c07b558ed395681649ecee2a",
2022-11-16 11:29:10 | "content": null,
2022-11-16 11:29:10 | "delimiter": null,
2022-11-16 11:29:10 | "dest": "/home/zuul/container-builds/85be1fa8-77e4-4318-aaad-fc9a3565d5e4/tripleo-ansible-ee/Dockerfile",
2022-11-16 11:29:10 | "directory_mode": null,
2022-11-16 11:29:10 | "follow": false,
2022-11-16 11:29:10 | "force": true,
2022-11-16 11:29:10 | "group": null,
2022-11-16 11:29:10 | "local_follow": null,
2022-11-16 11:29:10 | "mode": null,
2022-11-16 11:29:10 | "owner": null,
2022-11-16 11:29:10 | "regexp": null,
2022-11-16 11:29:10 | "remote_src": null,
2022-11-16 11:29:10 | "selevel": null,
2022-11-16 11:29:10 | "serole": null,
2022-11-16 11:29:10 | "setype": null,
2022-11-16 11:29:10 | "seuser": null,
2022-11-16 11:29:10 | "src": "/home/zuul/.ansible/tmp/ansible-tmp-1668616149.8660495-44258-195118567696493/source",
2022-11-16 11:29:10 | "unsafe_writes": false,
2022-11-16 11:29:10 | "validate": null
2022-11-16 11:29:10 | }
2022-11-16 11:29:10 | },
2022-11-16 11:29:10 | "msg": "Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!"
2022-11-16 11:29:10 | }
```

Logs:
- https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-9-quay-push-master/dc5c849/logs/build.log
- https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-9-quay-push-master/acb1614/logs/build.log
- https://logserver.rdoproject.org/69/41469/36/check/periodic-tripleo-ci-build-containers-ubi-9-quay-push-zed/48b2722/logs/build.log
- https://logserver.rdoproject.org/69/41469/36/check/periodic-tripleo-ci-build-containers-ubi-9-quay-push-master/4e6fff1/logs/build.log

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/864852

[Marios Andreou (marios-b)]

Looks like that started at some point on the 16th - from build history at [1]

  * periodic-tripleo-ci-build-containers-ubi-9-quay-push-master openstack/tripleo-ci master openstack-periodic-integration-main 24 mins 55 secs 2022-11-17 00:22:11 FAILURE

  * periodic-tripleo-ci-build-containers-ubi-9-quay-push-master openstack/tripleo-ci master openstack-periodic-integration-main 16 mins 22 secs 2022-11-16 16:14:36 FAILURE

  * periodic-tripleo-ci-build-containers-ubi-9-quay-push-master openstack/tripleo-ci master openstack-periodic-integration-main 50 mins 10 secs 2022-11-16 00:16:37 SUCCESS

  * periodic-tripleo-ci-build-containers-ubi-9-quay-push-master openstack/tripleo-ci master openstack-periodic-integration-main 38 mins 41 secs 2022-11-15 17:25:12 SUCCESS

[1] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-build-containers-ubi-9-quay-push-master&pipeline=openstack-periodic-integration-main&skip=0

[Marios Andreou (marios-b)]

haven't found workaround yet - here's what i tried:

* adding dnf install python3-libselinux package [1] (Takashi posted the same but earlier in pre.yaml in comment #1 avove but I wanted to explicitly add it to main) and that didn't work ... see [1] it was already installed on the host

* tried adding into tcib/tripleo-common with [2] but didn't work -> same result [3]

* tried adding pip install (venv for container builds) @ [4] -> same :/ [5]

[1] https://review.opendev.org/c/openstack/tripleo-ci/+/864889/1/roles/build-containers/tasks/main.yaml#230
[2] https://review.opendev.org/c/openstack/tripleo-common/+/864892
[3] https://logserver.rdoproject.org/32/46132/2/check/periodic-tripleo-ci-build-containers-ubi-9-quay-push-master/54de10f/logs/build.log
[4] https://review.opendev.org/c/openstack/tripleo-ci/+/864889/2/roles/build-containers/vars/centos-9.yml
[5] https://logserver.rdoproject.org/32/46132/3/check/periodic-tripleo-ci-build-containers-ubi-9-quay-push-master/29e7d45/job-output.txt

[Ronelle Landy (rlandy)]

https://github.com/openstack/python-tripleoclient/commit/d18b9273859dc2c5909104f1aa7dab0c2c4f982e looks to have added "ANSIBLE_PYTHON_INTERPRETER": "{{ansible_playbook_python}}"

Considering the issue and solution in: https://bugs.launchpad.net/tripleo/+bug/1862330, tried the DNM patch: https://review.opendev.org/c/openstack/python-tripleoclient/+/864952 to remove that line.

Testing in https://review.rdoproject.org/r/c/testproject/+/25325 shows the build to be progressing - with no selinux errors.

[Rabi Mishra (rabi)]

Looks like these jobs are using venv to build containers. Should we change those jobs to not use venv as there is libselinux-python required? Proposed https://review.rdoproject.org/r/c/config/+/46139

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Takashi Kajinami <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/864852
Reason: This does not solve the problem. We are now looking into https://review.rdoproject.org/r/c/config/+/46139

[Sandeep Yadav (sandeepyadav93)]

Thanks to Rabi and Takashi for the fix.

Testproject: https://review.rdoproject.org/r/c/testproject/+/25325/125/.zuul.yaml#7 is green var "buildcontainers_venv: false"

https://logserver.rdoproject.org/25/25325/125/check/periodic-tripleo-ci-build-containers-ubi-9-quay-push-master/e519ee1/logs/build_containers.sh confirms no virtual env used.

Adding a summary of the discussion with Rabi:

Patch[0] brought the issue to notice as we changed that playbook to use the ansible controller interpreter rather than discovering it.

For now, we can merge patches [1] and [2] and look at if there is a case to use venv later. If we plan to use venv for some reason ( we should install all python libraries using pip). But we don't use virtual env for content provider jobs[3].

Rabi plans to cherry-pick [1] at least till Zed.

[0] https://review.opendev.org/c/openstack/python-tripleoclient/+/864647/3/tripleoclient/v2/tripleo_container_image.py
[1] https://review.rdoproject.org/r/c/config/+/46139
[2] https://review.rdoproject.org/r/c/rdo-jobs/+/46141
[3] https://c18ded86635bab17724f-51fb74e788bc28a37082204769947c3c.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/tripleo-heat-templates/master/tripleo-ci-centos-9-content-provider/3352f61/logs/undercloud/home/zuul/tripleo_container_image_build.sh

[Marios Andreou (marios-b)]

We are going with https://review.rdoproject.org/r/c/config/+/46139

One unknown still from me: if this was caused by [1] and that has not yet been backported to stable/zed at [2] (top commit "Pull flake8 from Github" 8b9c3b6bd96fdc452cfe62ad8235abd656796308) then how did we see this issue on zed builds?

From the bug description we have [3]

[1] https://review.opendev.org/c/openstack/python-tripleoclient/+/864647
[2] https://github.com/openstack/python-tripleoclient/commits/stable/zed
[3] https://logserver.rdoproject.org/69/41469/36/check/periodic-tripleo-ci-build-containers-ubi-9-quay-push-zed/48b2722/logs/build.log

[Marios Andreou (marios-b)]

Thanks ysandeep - as just discussed in irc - to answer comment #8 it is because the zed job is checking out master branch for required-projects (i.e. using master tripleoclient).

Patch at https://review.rdoproject.org/r/c/rdo-jobs/+/46142 will fix it.

[Sandeep Yadav (sandeepyadav93)]

Below patches fixed the issue.

https://review.rdoproject.org/r/c/config/+/46139
https://review.rdoproject.org/r/c/rdo-jobs/+/46141