periodic-tripleo-ci-build-containers-ubi-8-push-train is failing to build containers

[Douglas Viroel]

periodic-tripleo-ci-build-containers-ubi-8-push-trai is failing since 08/28:
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-build-containers-ubi-8-push-train&pipeline=openstack-periodic-integration-stable4

Looking at the logs, we identify that a playbook eun is missing:
RUN START: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-buildcontainers/run.yaml@master]
https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push-train/4690f03/job-output.txt

which can be found in a successful run:
https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push-train/b84aefa/job-output.txt

Passing and failing jobs differ their podman versions:
Passing job:
podman.x86_64 3.3.0-4.module_el8.5.0+890+6b136101 @appstream
https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push-train/b84aefa/logs/system/installed_pkgs.txt

Failing job:
podman.x86_64 3.3.1-3.module_el8.5.0+911+f19012f9 @appstream
https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push-train/3795a12/logs/system/installed_pkgs.txt

[Douglas Viroel (dviroel)]

Same issue in ussuri:
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-build-containers-ubi-8-push-ussuri

podman.x86_64 3.3.1-3.module_el8.5.0+911+f19012f9 @appstream
https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push-ussuri/5553f9a/logs/system/installed_pkgs.txt

Both ussuri and train podman version are different from the other passing branches, which are using a older version:
podman.x86_64 3.0.1-6.module_el8.5.0+736+58cc1a5a @appstream
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push/9bb77c9/logs/system/installed_pkgs.txt

[Ronelle Landy (rlandy)]

After digging in the node, podman is installed as a tripleo-ansible dependency before the container-tools module is set to 3.0 - so it installs the rhel8 version.

https://review.opendev.org/c/openstack/tripleo-ci/+/806645/3/playbooks/tripleo-buildcontainers/pre.yaml - this hack fixes the problem in train - but this setting should be done in tripleo-repos before the bindep role is called.

(https://review.rdoproject.org/r/c/testproject/+/35111)

We'd still hav eto investigate why only train/ussuri and report that the newer version of podman fails login.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ci/+/806645
Committed: https://opendev.org/openstack/tripleo-ci/commit/9d5ff817ea8d75d837d26cf34d1dd6b949d7b4e0
Submitter: "Zuul (22348)"
Branch: master

commit 9d5ff817ea8d75d837d26cf34d1dd6b949d7b4e0
Author: Ronelle Landy <email address hidden>
Date: Mon Aug 30 17:18:06 2021 -0400

    Enable supported container-tools before login

    Container-tools module needs to be set before
    the podman login step.
    Going forward, this should be done with tripleo-repos.

    This review also removes unnused container stein
    jobs from check and gate.

    Closes-Bug: #1942078
    Change-Id: I448a8d6d2dad94cb413a8f82adb85867c043f5b7