tripleo

tripleo-ci-centos-8-content-provider failing consistently

Bug #1939674 reported by Rabi Mishra on 2021-08-12

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	Critical	Unassigned	tripleo xena-3

Bug Description

Noticed at:

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_102/804331/3/check/tripleo-ci-centos-8-content-provider/102c8a9/logs/undercloud/home/zuul/container_image_build.log

Looks like it's failing to build mistral container. I don't see anything in the traceback pointing to any issue, other than "Failed to write to log, write /home/zuul/container-builds/d98a7d66-eb57-42b9-b48f-7c9ce123d6c2/base/os/mistral-base/mistral-base-build.log: file already closed\n': RuntimeError:
The following errors were detected during container build(s):" in the logs

Full traceback:

Exception information: Unexpected error while running command.
Command: sudo buildah --log-level=debug bud --volume /etc/yum.repos.d:/etc/distro.repos.d:z --volume /etc/pki/rpm-gpg:/etc/pki/rpm-gpg:z --volume /etc/yum.repos.d:/etc/distro.repos.d:z --volume /etc/pki/rpm-gpg:/etc/pki/rpm-gpg:z --volume /etc/dnf/vars:/etc/dnf/vars:z --volume /opt/gating_repo:/opt/gating_repo:z --loglevel=3 --format docker --tls-verify=False --logfile /home/zuul/container-builds/d98a7d66-eb57-42b9-b48f-7c9ce123d6c2/base/os/mistral-base/mistral-base-build.log -t 127.0.0.1:5001/tripleomaster/openstack-mistral-base:91b798cf1820c86b98dcd828866e3afd /home/zuul/container-builds/d98a7d66-eb57-42b9-b48f-7c9ce123d6c2/base/os/mistral-base
Exit code: 1
Stdout: ''
Stderr: 'level=debug msg="Pull Policy for pull [PullIfNewer]"\nerror building at STEP "RUN dnf -y install openstack-mistral-common openstack-tripleo-common python3-shade python3dist\\(ansible\\) && dnf clean all && rm -rf /var/cache/dnf": error while running runtime: exit status 1\nFailed to write to log, write /home/zuul/container-builds/d98a7d66-eb57-42b9-b48f-7c9ce123d6c2/base/os/mistral-base/mistral-base-build.log: file already closed\nFailed to write to log, write /home/zuul/container-builds/d98a7d66-eb57-42b9-b48f-7c9ce123d6c2/base/os/mistral-base/mistral-base-build.log: file already closed\n'
2021-08-12 07:53:54.067 56166 ERROR openstack Traceback (most recent call last):
2021-08-12 07:53:54.067 56166 ERROR openstack File "/usr/lib/python3.6/site-packages/cliff/app.py", line 402, in run_subcommand
2021-08-12 07:53:54.067 56166 ERROR openstack result = cmd.run(parsed_args)
2021-08-12 07:53:54.067 56166 ERROR openstack File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
2021-08-12 07:53:54.067 56166 ERROR openstack super(Command, self).run(parsed_args)
2021-08-12 07:53:54.067 56166 ERROR openstack File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
2021-08-12 07:53:54.067 56166 ERROR openstack return super(Command, self).run(parsed_args)
2021-08-12 07:53:54.067 56166 ERROR openstack File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
2021-08-12 07:53:54.067 56166 ERROR openstack return_code = self.take_action(parsed_args) or 0
2021-08-12 07:53:54.067 56166 ERROR openstack File "/usr/lib/python3.6/site-packages/tripleoclient/v2/tripleo_container_image.py", line 688, in take_action
2021-08-12 07:53:54.067 56166 ERROR openstack bb.build_all()
2021-08-12 07:53:54.067 56166 ERROR openstack File "/usr/lib/python3.6/site-packages/tripleo_common/image/builder/buildah.py", line 248, in build_all
2021-08-12 07:53:54.067 56166 ERROR openstack self._multi_build(containers=containers)
2021-08-12 07:53:54.067 56166 ERROR openstack File "/usr/lib/python3.6/site-packages/tripleo_common/image/builder/buildah.py", line 362, in _multi_build
2021-08-12 07:53:54.067 56166 ERROR openstack exceptions='\n'.join(exceptions)
2021-08-12 07:53:54.067 56166 ERROR openstack RuntimeError:
2021-08-12 07:53:54.067 56166 ERROR openstack The following errors were detected during container build(s):
2021-08-12 07:53:54.067 56166 ERROR openstack
2021-08-12 07:53:54.067 56166 ERROR openstack Exception information: Unexpected error while running command.
2021-08-12 07:53:54.067 56166 ERROR openstack Command: sudo buildah --log-level=debug bud --volume /etc/yum.repos.d:/etc/distro.repos.d:z --volume /etc/pki/rpm-gpg

Tags:

Revision history for this message

Sagi (Sergey) Shnaidman (sshnaidm) wrote on 2021-08-12:

There is an error in mistral build log:

The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Transaction test error:
file /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release from install of containers-common-1:1.4.0-5.module_el8.5.0+878+851f435b.x86_64 conflicts with file from package redhat-release-8.4-0.6.el8.x86_64

tags:

added: alert promotion-blocker

Revision history for this message

chandan kumar (chkumar246) wrote on 2021-08-12:

We are also seeing it in master promotion integration line: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push/6fae76a/logs/container-builds/b3808f9e-cff5-41cd-b767-17f6cf6ae14c/base/os/mistral-base/mistral-base-build.log

```
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Transaction test error:
file /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release from install of containers-common-1:1.4.0-5.module_el8.5.0+878+851f435b.x86_64 conflicts with file from package redhat-release-8.2-1.0.el8.x86_64

time="2021-08-12T08:44:32Z" level=debug msg="\"/var/tmp/buildah838002395/mnt/buildah-bind-target-12\" is apparently not really mounted, skipping"
time="2021-08-12T08:44:32Z" level=debug msg="\"/var/tmp/buildah838002395/mnt/buildah-bind-target-11\" is apparently not really mounted, skipping"
time="2021-08-12T08:44:32Z" level=debug msg="\"/var/tmp/buildah838002395/mnt/buildah-bind-target-7\" is apparently not really mounted, skipping"
time="2021-08-12T08:44:32Z" level=debug msg="\"/var/tmp/buildah838002395/mnt/buildah-bind-target-3\" is apparently not really mounted, skipping"
time="2021-08-12T08:44:32Z" level=debug msg="\"/var/tmp/buildah838002395/mnt/rootfs\" is apparently not really mounted, skipping"
time="2021-08-12T08:44:32Z" level=debug msg="\"/var/tmp/buildah838002395/mnt\" is apparently not really mounted, skipping"
time="2021-08-12T08:44:32Z" level=debug msg="error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LANG=en_US.UTF-8 container=oci] Command:run Args:[dnf -y install openstack-mistral-common openstack-tripleo-common python3-shade python3dist\\(ansible\\) && dnf clean all && rm -rf /var/cache/dnf] Flags:[] Attrs:map[] Message:RUN dnf -y install openstack-mistral-common openstack-tripleo-common python3-shade python3dist\\(ansible\\) && dnf clean all && rm -rf /var/cache/dnf Original:RUN dnf -y install openstack-mistral-common openstack-tripleo-common python3-shade python3dist\\(ansible\\) && dnf clean all && rm -rf /var/cache/dnf}: error while running runtime: exit status 1"
```

```
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Transaction test error:
  file /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release from install of containers-common-1:1.4.0-5.module_el8.5.0+878+851f435b.x86_64 conflicts with file from package redhat-release-8.2-1.0.el8.x86_64

Revision history for this message

Rabi Mishra (rabi) wrote on 2021-08-12:

I was looking at the wrong place for the error.

You can remove cached packages by executing 'dnf clean packages'.
Error: Transaction test error:
file /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release from install of containers-common-1:1.4.0-5.module_el8.5.0+878+851f435b.x86_64 conflicts with file from package redhat-release-8.4-0.6.el8.x86_64

https://4fa0950c9393dd01b78a-8ef78dfb3db63968baa844dac74dc0d0.ssl.cf2.rackcdn.com/804353/1/check/tripleo-ci-centos-8-content-provider/e9b335a/logs/undercloud/home/zuul/workspace/logs/container-builds/e322e895-1c25-4db3-bb00-56fd2815bae6/base/os/mistral-base/mistral-base-build.log

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-12: Fix proposed to tripleo-common (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-common/+/804379

Changed in tripleo:
status:	Triaged → In Progress

yatin (yatinkarel) on 2021-08-12

Changed in tripleo:
milestone:	none → xena-3

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-12: Fix proposed to tripleo-common (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-common/+/804342

Revision history for this message

Alex Schultz (alex-schultz) wrote on 2021-08-12:

We need to file a bz against containers-common as they likely should not be including that key
Error: Transaction test error:
file /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release from install of containers-common-1:1.4.0-5.module_el8.5.0+878+851f435b.x86_64 conflicts with file from package redhat-release-8.4-0.6.el8.x86_64

Revision history for this message

Alex Schultz (alex-schultz) wrote on 2021-08-12:

Filed bz https://bugzilla.redhat.com/show_bug.cgi?id=1993209

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-12: Fix proposed to tripleo-common (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-common/+/804421

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-12: Fix proposed to tripleo-common (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-common/+/804422

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-12: Fix proposed to tripleo-common (stable/train)

#10

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-common/+/804423

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-12: Fix merged to tripleo-common (master)

#11

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/804379
Committed: https://opendev.org/openstack/tripleo-common/commit/906265c433003886d87ac29d9daf68eccb69a40c
Submitter: "Zuul (22348)"
Branch: master

commit 906265c433003886d87ac29d9daf68eccb69a40c
Author: Sagi Shnaidman <email address hidden>
Date: Thu Aug 12 13:53:28 2021 +0300

Remove rehdat-release package from UBI8 image

Also install centos*release package when tcib_distro
is centos.

*release packages are protected by dnf, so used
rpm commands to remove.

    Disable upgrades pipeline to get this fix in.
    Closes-Bug: #1939674
    Change-Id: I5467a5faa36aa1359ff9b8135a512566dff12a79

Changed in tripleo:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-12: Change abandoned on tripleo-common (stable/wallaby)

#12

Change abandoned by "wes hayutin <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-common/+/804342
Reason: restart

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-13: Fix merged to tripleo-common (stable/ussuri)

#13

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/804422
Committed: https://opendev.org/openstack/tripleo-common/commit/a83b83449d87f0cde1f83e0a0ab6be8b5d591f3c
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit a83b83449d87f0cde1f83e0a0ab6be8b5d591f3c
Author: Sagi Shnaidman <email address hidden>
Date: Thu Aug 12 13:53:28 2021 +0300

Remove redhat-release package from UBI8 image

Also install centos*release package when tcib_distro
is centos.

*release packages are protected by dnf, so used
rpm commands to remove.

Disable upgrades pipeline to get this fix in.

    (cherry picked from commit a7513bc0b970d8317f8b598a967c7c61c4ed1b6e)
    (cherry picked from commit 963c55bd4423e47eb243fdd746ff484dc3a4d6f2)
    (cherry picked from commit 906265c433003886d87ac29d9daf68eccb69a40c)

Closes-Bug: #1939674
Change-Id: I5467a5faa36aa1359ff9b8135a512566dff12a79

tags:

added: in-stable-ussuri

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-13: Fix merged to tripleo-common (stable/victoria)

#14

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/804421
Committed: https://opendev.org/openstack/tripleo-common/commit/a7513bc0b970d8317f8b598a967c7c61c4ed1b6e
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit a7513bc0b970d8317f8b598a967c7c61c4ed1b6e
Author: Sagi Shnaidman <email address hidden>
Date: Thu Aug 12 13:53:28 2021 +0300

Remove redhat-release package from UBI8 image

Also install centos*release package when tcib_distro
is centos.

*release packages are protected by dnf, so used
rpm commands to remove.

Disable upgrades pipeline to get this fix in.

(cherry picked from commit 963c55bd4423e47eb243fdd746ff484dc3a4d6f2)
(cherry picked from commit 906265c433003886d87ac29d9daf68eccb69a40c)

Closes-Bug: #1939674
Change-Id: I5467a5faa36aa1359ff9b8135a512566dff12a79

tags:

added: in-stable-victoria

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-13: Fix merged to tripleo-common (stable/train)

#15

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/804423
Committed: https://opendev.org/openstack/tripleo-common/commit/04a64aa292feae5d5f4dc5ab2c9ebe9321ded103
Submitter: "Zuul (22348)"
Branch: stable/train

commit 04a64aa292feae5d5f4dc5ab2c9ebe9321ded103
Author: Sagi Shnaidman <email address hidden>
Date: Thu Aug 12 13:53:28 2021 +0300

Remove redhat-release package from UBI8 image

Also install centos*release package when tcib_distro
is centos.

*release packages are protected by dnf, so used
rpm commands to remove.

Disable upgrades pipeline to get this fix in.

    (cherry picked from commit f26401d3fb70ba9f01056cf88812c25aba639c69)
    (cherry picked from commit a7513bc0b970d8317f8b598a967c7c61c4ed1b6e)
    (cherry picked from commit 963c55bd4423e47eb243fdd746ff484dc3a4d6f2)
    (cherry picked from commit 906265c433003886d87ac29d9daf68eccb69a40c)

Closes-Bug: #1939674
Change-Id: I5467a5faa36aa1359ff9b8135a512566dff12a79

tags:

added: in-stable-train

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-13: Fix merged to tripleo-common (stable/wallaby)

#16

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/804342
Committed: https://opendev.org/openstack/tripleo-common/commit/963c55bd4423e47eb243fdd746ff484dc3a4d6f2
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 963c55bd4423e47eb243fdd746ff484dc3a4d6f2
Author: Sagi Shnaidman <email address hidden>
Date: Thu Aug 12 13:53:28 2021 +0300

Remove redhat-release package from UBI8 image

Also install centos*release package when tcib_distro
is centos.

*release packages are protected by dnf, so used
rpm commands to remove.

Disable upgrades pipeline to get this fix in.

(cherry picked from commit 906265c433003886d87ac29d9daf68eccb69a40c)

Closes-Bug: #1939674
Change-Id: I5467a5faa36aa1359ff9b8135a512566dff12a79

tags:

added: in-stable-wallaby

Revision history for this message

Rabi Mishra (rabi) wrote on 2021-08-16:

#17

We probably need a promotion as jobs are still failing with the same error.

Revision history for this message

yatin (yatinkarel) wrote on 2021-08-16:

#18

<< We probably need a promotion as jobs are still failing with the same error.
Actually tripleo common is used from current repo, so promotion not involved.
DLRN process for c8 master was stuck since 12th, and after killing stuck process, all pending commits were built https://trunk.rdoproject.org/centos8-master/report.html.