centos-7 train jobs are failing on dependency resolution: glibc-common = 2.17-307.el7.1 and python3-libs(x86-64) = 3.6.8-13.el7

[Ronelle Landy]

centos-7 train jobs are failing in the integration line with the following error:

2020-11-13 15:13:39.877679 | primary | ---> Package python3-devel.x86_64 0:3.6.8-13.el7 will be installed
2020-11-13 15:13:39.878286 | primary | --> Processing Dependency: python3-libs(x86-64) = 3.6.8-13.el7 for package: python3-devel-3.6.8-13.el7.x86_64
2020-11-13 15:13:40.228034 | primary | --> Finished Dependency Resolution
2020-11-13 15:13:40.264151 | primary | Error: Package: glibc-2.17-307.el7.1.i686 (base)
2020-11-13 15:13:40.264268 | primary | Requires: glibc-common = 2.17-307.el7.1
2020-11-13 15:13:40.264314 | primary | Installed: glibc-common-2.17-317.el7.x86_64 (@base)
2020-11-13 15:13:40.264346 | primary | glibc-common = 2.17-317.el7
2020-11-13 15:13:40.264384 | primary | Available: glibc-common-2.17-307.el7.1.x86_64 (base)
2020-11-13 15:13:40.264415 | primary | glibc-common = 2.17-307.el7.1
2020-11-13 15:13:40.264489 | primary | Error: Package: python3-devel-3.6.8-13.el7.x86_64 (base)
2020-11-13 15:13:40.264529 | primary | Requires: python3-libs(x86-64) = 3.6.8-13.el7
2020-11-13 15:13:40.264567 | primary | Installed: python3-libs-3.6.8-17.el7.x86_64 (@base)
2020-11-13 15:13:40.264600 | primary | python3-libs(x86-64) = 3.6.8-17.el7
2020-11-13 15:13:40.264637 | primary | Available: python3-libs-3.6.8-13.el7.x86_64 (base)
2020-11-13 15:13:40.264669 | primary | python3-libs(x86-64) = 3.6.8-13.el7
2020-11-13 15:13:40.264707 | primary | You could try using --skip-broken to work around the problem
2020-11-13 15:13:40.734339 | primary | You could try running: rpm -Va --nofiles --nodigest

The full buildset is linked below:
https://review.rdoproject.org/zuul/buildset/62fe003eaabd45fea6f61ce0bd9b3119

Example log:
https://logserver.rdoproject.org/openstack-periodic-integration-stable3-centos7/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-train/24b0696/job-output.txt

[Ronelle Landy (rlandy)]

Mirrors are missing centos 7.9 content:

<amoralej> rlandy|rover, so it seems http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/ didn't sync 7.9

<jpena> we might check where does it sync from, maybe that one seed does not have 7.9 yet
<jpena> so, what I see in https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-mirror-update#L51 is that the opendev AFS mirrors sync from mirror.lstn.net/centos
<jpena> looking at http://mirror.lstn.net/centos/7/os/x86_64/Packages/?C=M;O=D, it looks like they don't have the 7.9 content yet

[Ronelle Landy (rlandy)]

https://3214acddd9362a1b98b2-6b3b04e0e3cb61f476109a58566f92f2.ssl.cf1.rackcdn.com/761892/5/check/tripleo-buildimage-overcloud-full-centos-7/f62d019/build.log

2020-11-13 14:06:24.083 | >
2020-11-13 14:06:24.083 | > yum-config-manager --save --setopt=centos7-rt.skip_if_unavailable=true
2020-11-13 14:06:24.083 | >
2020-11-13 14:06:24.083 | > failure: repodata/repomd.xml from centos7-rt: [Errno 256] No more mirrors to try.
2020-11-13 14:06:24.084 | > http://mirror.centos.org/centos/7/rt/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found

Taking centos-7 image-build jobs

[Marios Andreou (marios-b)]

Per comment #1 and #2 about the mirrors, it is still ongoing today

https://efbed3978079cc645f09-4b7d4da09de7dae33770cacbfd550cfb.ssl.cf5.rackcdn.com/761892/5/check/tripleo-buildimage-overcloud-full-centos-7/871dc5d/build.log

https://6350d29ea0552d9b896d-16024773d98de775c76d8d6d7d6dfaac.ssl.cf5.rackcdn.com/761892/5/check/tripleo-buildimage-overcloud-full-centos-7/6374f38/build.log

to be clear the mirrors 404 is not the issue reported in this bug.

Instead this bug is about the dependency resolution. That issue may be fixed by https://review.opendev.org/#/c/761854/ (in which case the bug @ https://bugs.launchpad.net/tripleo/+bug/1902986 is a duplicate)

[Marios Andreou (marios-b)]

Indeed, I don't think we are seeing the issue reported in the bug any more - I think the fix was https://review.opendev.org/#/c/761854/ which merged to tripleo-ci on the 12th

Now looking at https://zuul.openstack.org/builds?job_name=tripleo-buildimage-overcloud-full-centos-7 the errors are consistently about the 404/mirror issue.

That may be resolved if https://review.opendev.org/#/c/762812 merges

Until then however, the mirror issue is blocking train/queens gates so I think we should mark the jobs non voting until the issue is resolved.

Patch to do that at https://review.opendev.org/762829

[Marios Andreou (marios-b)]

k let's close this bug out to prevent confusion - per comments 3 and 4 above this should be resolved with https://review.opendev.org/#/c/761854/

I just filed a new bug @ https://bugs.launchpad.net/tripleo/+bug/1904418 to track the issue with mirrors.

Moving this fix-released move back if you disagree or see it is still happening thanks