[CI][CS9][Wallaby] tripleo-ci-centos-9-content-provider-wallaby problem with building containers

[Dariusz Smigiel]

Check has issues with building containers. The problem started around 2022-03-02 20:47:37 [1] [2] [3].

Both providers ovh [4] and rackspace [5] have the same issue.

fatal: [undercloud]: FAILED! => {"ansible_job_id": "233265723236.20869", "attempts": 1, "changed": false, "cmd": "openstack tripleo container image build --base quay.io/centos/centos:stream9 --debug --distro centos --exclude neutron-mlnx-agent --namespace tripleowallabycentos9 --prefix openstack --push --registry 127.0.0.1:5001 --tag 650bb78dc9239e2a6da61965584cc7c2 --volume /etc/yum.repos.d:/etc/distro.repos.d:z --volume /etc/pki/rpm-gpg:/etc/pki/rpm-gpg:z --volume /etc/dnf/vars:/etc/dnf/vars:z --work-dir /home/zuul/container-builds >/home/zuul/container_image_build.log 2>&1 --tcib-extras tcib_release=9 --tcib-extras tcib_python_version=3.9", "delta": "0:05:48.690748", "end": "2022-03-02 21:51:59.135347", "finished": 1, "msg": "non-zero return code", "rc": 1, "start": "2022-03-02 21:46:10.444599", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

[1]:
'time="2022-03-02T21:47:55Z" level=debug msg="Pull Policy for pull [ifnewer]"\nerror building at STEP "RUN if [ \'centos\' == \'centos\' ];then dnf -y install dnf-plugins-core && dnf download centos*release && rpm -ivh --nodeps --replacepkgs --replacefiles *.rpm && if [ -n "$(rpm -qa redhat-release)" ];then rpm -e --nodeps redhat-release; fi && rm -rf *.rpm; fi": error while running runtime: exit status 1\nFailed to write to log, write /home/zuul/container-builds/5effd2c6-794d-413b-acc7-db90da465091/base/base-build.log: file already closed\nFailed to write to log, write /home/zuul/container-builds/5effd2c6-794d-413b-acc7-db90da465091/base/base-build.log: file already closed\n

[2]: https://64655ba8ea8f5130b3ee-f99336677cbcb935a72dddbf18215860.ssl.cf1.rackcdn.com/820530/5/check/tripleo-ci-centos-9-content-provider-wallaby/5f1f0aa/logs/undercloud/home/zuul/container_image_build.log

[3]: https://868d9d181f45f3dc08e4-0ff94bb25fd1b2b22d207854a15b8765.ssl.cf5.rackcdn.com/828911/4/check/tripleo-ci-centos-9-content-provider-wallaby/963ec92/logs/quickstart_install.log

[4]: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3ee/831184/4/gate/tripleo-ci-centos-9-content-provider-wallaby/3ee085e/logs/container-builds/75e6d170-3957-4d7d-acd3-04556d25f64c/base/base-build.log
[5]: https://868d9d181f45f3dc08e4-0ff94bb25fd1b2b22d207854a15b8765.ssl.cf5.rackcdn.com/828911/4/check/tripleo-ci-centos-9-content-provider-wallaby/963ec92/logs/container-builds/d6f26355-1c54-4af1-988b-9a0e75f672ca/base/base-build.log

[Ronelle Landy (rlandy)]

https://kojihub.stream.centos.org/koji/buildinfo?buildID=17492

centos-stream-release-9.0-9.el9.noarch.rpm 3.9 MB/s | 25 kB 00:00
warning: centos-stream-release-9.0-9.el9.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 8483c65d: NOKEY
Verifying... ########################################
Preparing... ########################################
 package centos-stream-release-9.0-12.el9.noarch (which is newer than centos-stream-release-9.0-9.el9.noarch) is already installed

time="2022-03-02T21:06:49Z" level=debug msg="error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LANG=en_US.UTF-8 container=oci] Command:run Args:[if [ 'centos' == 'centos' ];then dnf -y install dnf-plugins-core && dnf download centos*release && rpm -ivh --nodeps --replacepkgs --replacefiles *.rpm && if [ -n \"$(rpm -qa redhat-release)\" ];then rpm -e --nodeps redhat-release; fi && rm -rf *.rpm; fi] Flags:[] Attrs:map[] Message:RUN if [ 'centos' == 'centos' ];then dnf -y install dnf-plugins-core && dnf download centos*release && rpm -ivh --nodeps --replacepkgs --replacefiles *.rpm && if [ -n \"$(rpm -qa redhat-release)\" ];then rpm -e --nodeps redhat-release; fi && rm -rf *.rpm; fi Original:RUN if [ 'centos' == 'centos' ];then dnf -y install dnf-plugins-core && dnf download centos*release && rpm -ivh --nodeps --replacepkgs --replacefiles *.rpm && if [ -n \"$(rpm -qa redhat-release)\" ];then rpm -e --nodeps redhat-release; fi && rm -rf *.rpm; fi}: error while running runtime: exit status 1"

[Ronelle Landy (rlandy)]

there are two basic diffs showing between last passing and first failing jobs:

openstack-tripleo-heat-templates.noarch 14.3.1-0.20220302202939.09c1ce2.el9 @delorean-current

and the above centos stream change plus new stream repos

[chandan kumar (chkumar246)]

Based on this https://bugs.launchpad.net/tripleo/+bug/1962786/comments/1, We have seen this issue in the past.

Here is the fix: https://review.opendev.org/c/openstack/tripleo-common/+/825038 it needs to be backported to stable/wallaby.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-common/+/831580

[chandan kumar (chkumar246)]

Here is the proposed cherry-pick: https://review.opendev.org/c/openstack/tripleo-common/+/831580

[chandan kumar (chkumar246)]

On this patch https://review.opendev.org/c/openstack/tripleo-heat-templates/+/820530/ with above depends on
tripleo-ci-centos-9-content-provider-wallaby success - https://zuul.opendev.org/t/openstack/build/66f9e05a2ce9476384210d00d067c068

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/831580
Committed: https://opendev.org/openstack/tripleo-common/commit/3db8093d0c54db9cd0fae6df851402f270bcf35d
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 3db8093d0c54db9cd0fae6df851402f270bcf35d
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Tue Jan 18 11:20:07 2022 +0530

    Reinstall centos-stream-release from centos repos

    In CentOS Stream 8 and 9 container image, centos-stream-release
    package is already available in the container image. But in UBI,
    it is not available. It can be installed from the base OS repo.

    It removes the logic of downloading the centos-stream-release
    rpm and installing it. As it was added earlier when we were
    adding the initial support for EL9.

    since redhat-release is a part of centos-stream-release
    In UBI 8, redhat-release is a seperate package, so it's rpm
    needs to be removed otherwise it will end up with skip broken
    package.

    On removing redhat-release, /etc/os-release file also gets deleted.
    So we need to first remove redhat-release rpm if present and then
    install centos-stream-release package.

    It also removes dnf-core-plugins package installation as we donot
    need to download the centos-stream-release rpm.

    Closes-Bug: #1958202
    Related-Bug: #1958240
    Closes-Bug: #1962786

    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: Ia06c6770ea567656b5ce380826bdb120ca32a1a8
    (cherry picked from commit b09fb3fef0896e812ca9ce3e6dc74f297494203a)