Kolla fails to build containers for tripleo in Ci build-container job

Bug #1771090 reported by Sagi (Sergey) Shnaidman
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Sagi (Sergey) Shnaidman

Bug Description

Kolla fails to build container when trying to find http://mirror.centos.org/%24contentdir/7/virt/x86_64/kvm-common/repodata/repomd.xml

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-centos-7-master-containers-build/c178b0c/console.txt.gz#_2018-05-14_04_38_34_561

INFO:kolla.common.utils.base:Loaded plugins: fastestmirror, ovl, priorities
INFO:kolla.common.utils.base:Cleaning repos: base centos-ceph-luminous centos-opstools-release centos-qemu-ev
INFO:kolla.common.utils.base: : delorean delorean-master-build-deps delorean-master-testing
INFO:kolla.common.utils.base: : extras rdo-qemu-ev updates
INFO:kolla.common.utils.base:Cleaning up everything
INFO:kolla.common.utils.base:Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
INFO:kolla.common.utils.base:Cleaning up list of fastest mirrors
INFO:kolla.common.utils.base: ---> 604da8e1fda0
INFO:kolla.common.utils.base:Removing intermediate container 59e8eb6888e2
INFO:kolla.common.utils.base:Step 17/27 : RUN rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-OpsTools && rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage && rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Virtualization && yum -y update --security --sec-severity=Important --sec-severity=Critical && yum clean all
INFO:kolla.common.utils.base: ---> Running in c2aec2984677
INFO:kolla.common.utils.base:\u001b[91m\u001b[0m
INFO:kolla.common.utils.base:Loaded plugins: fastestmirror, ovl, priorities
INFO:kolla.common.utils.base:\u001b[91mhttp://mirror.centos.org/%24contentdir/7/virt/x86_64/kvm-common/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
INFO:kolla.common.utils.base:\u001b[0m
INFO:kolla.common.utils.base:\u001b[91mTrying other mirror.
INFO:kolla.common.utils.base:\u001b[0m
INFO:kolla.common.utils.base:\u001b[91mTo address this issue please refer to the below wiki article
INFO:kolla.common.utils.base:https://wiki.centos.org/yum-errors
INFO:kolla.common.utils.base:If above article doesn't help to resolve this issue please use https://bugs.centos.org/.
INFO:kolla.common.utils.base:\u001b[0m
INFO:kolla.common.utils.base:\u001b[91m
INFO:kolla.common.utils.base: One of the configured repositories failed (CentOS-7 - QEMU EV),
INFO:kolla.common.utils.base: and yum doesn't have enough cached data to continue. At this point the only
INFO:kolla.common.utils.base: safe thing yum can do is fail. There are a few ways to work "fix" this:
INFO:kolla.common.utils.base: 1. Contact the upstream for the repository and get them to fix the problem.
INFO:kolla.common.utils.base: 2. Reconfigure the baseurl/etc. for the repository, to point to a working
INFO:kolla.common.utils.base: upstream. This is most often useful if you are using a newer
INFO:kolla.common.utils.base: distribution release than is supported by the repository (and the
INFO:kolla.common.utils.base: packages for the previous distribution release still work).
INFO:kolla.common.utils.base: 3. Run the command with the repository temporarily disabled
INFO:kolla.common.utils.base: yum --disablerepo=centos-qemu-ev ...
INFO:kolla.common.utils.base: 4. Disable the repository permanently, so yum won't use it by default. Yum
INFO:kolla.common.utils.base: will then just ignore the repository until you permanently enable it
INFO:kolla.common.utils.base: again or use --enablerepo for temporary usage:
INFO:kolla.common.utils.base: yum-config-manager --disable centos-qemu-ev
INFO:kolla.common.utils.base: or
INFO:kolla.common.utils.base: subscription-manager repos --disable=centos-qemu-ev
INFO:kolla.common.utils.base: 5. Configure the failing repository to be skipped, if it is unavailable.
INFO:kolla.common.utils.base: Note that yum will try to contact the repo. when it runs most commands,
INFO:kolla.common.utils.base: so will have to try and fail each time (and thus. yum will be be much
INFO:kolla.common.utils.base: slower). If it is a very temporary problem though, this is often a nice
INFO:kolla.common.utils.base: compromise:
INFO:kolla.common.utils.base: yum-config-manager --save --setopt=centos-qemu-ev.skip_if_unavailable=true
INFO:kolla.common.utils.base:failure: repodata/repomd.xml from centos-qemu-ev: [Errno 256] No more mirrors to try.
INFO:kolla.common.utils.base:http://mirror.centos.org/$contentdir/7/virt/x86_64/kvm-common/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
INFO:kolla.common.utils.base:\u001b[0m
INFO:kolla.common.utils.base:Removing intermediate container c2aec2984677
ERROR:kolla.common.utils.base:Error'd with the following message
ERROR:kolla.common.utils.base:The command '/bin/sh -c rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-OpsTools && rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage && rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Virtualization && yum -y update --security --sec-severity=Important --sec-severity=Critical && yum clean all' returned a non-zero code: 1
INFO:kolla.common.utils:Attempt number: 2 to run task: BuildTask(base)
INFO:kolla.common.utils.base:Building
INFO:kolla.common.utils.base:Step 1/27 : FROM centos:latest
INFO:kolla.common.utils.base: ---> e934aafc2206
INFO:kolla.common.utils.base:Step 2/27 : LABEL maintainer "TripleO Project (http://tripleo.org)" name "base" build-date "20180514"
INFO:kolla.common.utils.base: ---> Using cache
INFO:kolla.common.utils.base: ---> 716a8c92b28c
INFO:kolla.common.utils.base:Step 3/27 : RUN groupadd --force --gid 42475

Revision history for this message
Sandro Bonazzola (sandro-bonazzola) wrote :

Please cross check kolla code.
The problem is not in the package.
On a centos instance installed with anaconda the $contendir variable is filled by /etc/yum/vars/contentdir which is supposed to be installed by centos-release rpm.

The introduction of this variable has been proposed in https://lists.centos.org/pipermail/centos-devel/2018-March/016542.html
and has been implemented in CentOS 7.5 GA.

Looking at it, deeper I see:

# LC_ALL=C rpm -qf /etc/yum/vars/contentdir
file /etc/yum/vars/contentdir is not owned by any package
Seems like centos-release rpm has a packaging issue, being the file genreated by centos-release in %post instead of being shipped.

A bug has been opened on https://bugs.centos.org/view.php?id=14786 which identified the bug happening in OpenStack CI as a bug in diskimage-builder.

I hope this help solving the issue on kolla side.

Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

Patches that are related to the issue:
DIB: https://review.openstack.org/#/c/568180/ rpm-distro: set the contentdir yum var
Kolla: https://review.openstack.org/#/c/568126/ base: set $contentdir for CentOS

Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

Queens waits for backports:
https://review.openstack.org/#/c/568239/ base: set $contentdir for CentOS
https://review.openstack.org/#/c/568270/ Add yum update to base

Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :
Revision history for this message
Matt Young (halcyondude) wrote :

update, queens $contentdir patch has merged, pike is in progress

tags: removed: alert
Revision history for this message
Matt Young (halcyondude) wrote :

pike is thru check, needs +w

Revision history for this message
Alex Schultz (alex-schultz) wrote :

Looks like the Pike patch has landed. Can we close this or is it still failing?

Changed in tripleo:
milestone: rocky-2 → rocky-3
wes hayutin (weshayutin)
Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.