yum repo failures in upstream mirrors

[wes hayutin]

http://logs.openstack.org/87/603187/1/check/tripleo-ci-centos-7-containers-multinode/91ad246/logs/undercloud/home/zuul/install-undercloud.log.txt.gz

  + yum install -y yum-plugin-priorities
    [0mLoaded plugins: fastestmirror, ovl, priorities
    Determining fastest mirrors
    [91mhttp://mirror.dfw.rax.openstack.org:8080/cbs.centos/virt7-container-common-candidate/x86_64/os/repodata/bf30f430c78a804f68bc1d61804ca33f16b59ba197d2868c45564ae1b39d2dda-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
    [0m[91mTrying other mirror.
    [0m[91mTo address this issue please refer to the below wiki article

    https://wiki.centos.org/yum-errors

    If above article doesn't help to resolve this issue please use https://bugs.centos.org/.

    [0m[91mhttp://mirror.dfw.rax.openstack.org:8080/cbs.centos/virt7-container-common-candidate/x86_64/os/repodata/bf30f430c78a804f68bc1d61804ca33f16b59ba197d2868c45564ae1b39d2dda-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
    [0m[91mTrying other mirror.
    [0m[91mhttp://mirror.dfw.rax.openstack.org:8080/cbs.centos/virt7-container-common-candidate/x86_64/os/repodata/bf30f430c78a804f68bc1d61804ca33f16b59ba197d2868c45564ae1b39d2dda-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
    Trying other mirror.
    [0m[91m

     One of the configured repositories failed (quickstart-centos-virt7-container-common-candidate),
     and yum doesn't have enough cached data to continue. At this point the only
     safe thing yum can do is fail. There are a few ways to work "fix" this:

         1. Contact the upstream for the repository and get them to fix the problem.

         2. Reconfigure the baseurl/etc. for the repository, to point to a working
            upstream. This is most often useful if you are using a newer
            distribution release than is supported by the repository (and the
            packages for the previous distribution release still work).

         3. Run the command with the repository temporarily disabled
                yum --disablerepo=quickstart-centos-virt7-container-common-candidate ...

         4. Disable the repository permanently, so yum won't use it by default. Yum
            will then just ignore the repository until you permanently enable it
            again or use --enablerepo for temporary usage:

                yum-config-manager --disable quickstart-centos-virt7-container-common-candidate
            or
                subscription-manager repos --disable=quickstart-centos-virt7-container-common-candidate

         5. Configure the failing repository to be skipped, if it is unavailable.
            Note that yum will try to contact the repo. when it runs most commands,
            so will have to try and fail each time (and thus. yum will be be much
            slower). If it is a very temporary problem though, this is often a nice
            compromise:

                yum-config-manager --save --setopt=quickstart-centos-virt7-container-common-candidate.skip_if_unavailable=true

    failure: repodata/bf30f430c78a804f68bc1d61804ca33f16b59ba197d2868c45564ae1b39d2dda-primary.sqlite.bz2 from quickstart-centos-virt7-container-common-candidate: [Errno 256] No more mirrors to try.
    http://mirror.dfw.rax.openstack.org:8080/cbs.centos/virt7-container-common-candidate/x86_64/os/repodata/bf30f430c78a804f68bc1d61804ca33f16b59ba197d2868c45564ae1b39d2dda-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found

[wes hayutin (weshayutin)]

This is blocking https://bugs.launchpad.net/tripleo/+bug/1792892

[Alan Pevec (apevec)]

OOOQ config/general_config/featureset033.yml uses Koji generated repo https://cbs.centos.org/repos/virt7-container-common-candidate/x86_64/os/
which is updated after each new build in that Koji tag and keeps only one set of repodata. This can end up in failures like reported, where repomd is out of sync with the rest of repodata files.
Safer is to use buildlogs.c.o repo with the builds tagged in -testing tag (assuming after at least sanity and ideally proper CI) and where few previous sets of repodata are kept [*]

baseurl: https://buildlogs.centos.org/centos/7/virt/x86_64/container/

Drawback is that it requires Virt SIG to do the tagging and updates to buildlogs.c.o take up to 4h after tagging into -testing Koji tag.

[*] compare
https://buildlogs.centos.org/centos/7/virt/x86_64/container/repodata/?C=M;O=D
vs
https://cbs.centos.org/repos/virt7-container-common-candidate/x86_64/os/repodata/

[Alan Pevec (apevec)]

BTW mirrors should have settled by now, can you retry the job?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/603273

[wes hayutin (weshayutin)]

I suspect the repo was added to get the latest versions of podman.

[wes hayutin (weshayutin)]

@apevec this issue was not just w/ featureset033, it's with ALL the upstream master jobs.

http://codesearch.openstack.org/?q=virt7-container-common-candidate&i=nope&files=&repos=

[Alan Pevec (apevec)]

@Wes yes, it is bad idea to use CBS Koji repos, https://review.openstack.org/#/c/603273/3 should fix it.

[Emilien Macchi (emilienm)]

I think this error is transient, it already happened once a few weeks ago, and it was the index in the mirror not updated. Nothing we can do unless we remove the mirror, then no podman.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/603273
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=77f995548973584c89419efd32c0ab5d00395017
Submitter: Zuul
Branch: master

commit 77f995548973584c89419efd32c0ab5d00395017
Author: Wes Hayutin <email address hidden>
Date: Mon Sep 17 20:43:15 2018 -0600

    use a more stable centos repo for k8s and container rpms

    Closes-Bug: #1793006
    Change-Id: I6555109551ee2fd8a88d2d7f2636edbd4390585c

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-quickstart 2.1.1 release.