CentOS-9 stream mirrors are returning Status code: 404

[Ronelle Landy]

Starting on 05/01, on both https://zuul.opendev.org and https://review.rdoproject.org/zuul, all CentOS-9 stream jobs are failing with:

repo: downloading from remote: baseos
error: Status code: 404 for https://mirror.bhs1.ovh.opendev.org/centos-stream/9-stream/BaseOS/x86_64/os/repodata/3ea088796d71ec43bd0450022bddc9365606b1996065fac43595e4ef6798af11-primary.xml.gz (IP: 158.69.73.218) (https://mirror.bhs1.ovh.opendev.org/centos-stream/9-stream/BaseOS/x86_64/os/repodata/3ea088796d71ec43bd0450022bddc9365606b1996065fac43595e4ef6798af11-primary.xml.gz).

or

rror: Failed to download metadata for repo 'baseos': Yum repo downloading error: Downloading error(s): repodata/3ea088796d71ec43bd0450022bddc9365606b1996065fac43595e4ef6798af11-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/e9e075843272c90115f8a5c44c1f3b23be8714c443aa0543d40bc38a2cf49de5-filelists.xml.gz - Cannot download, all mirrors were already tried without success
2023-05-01 21:15:43.533453 | primary | ok: Item: dnf makecache -v Result: 1

or other similar 404 errors accessing CentOS-9 stream mirrors.

Example logs:

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-9-wallaby-promote-promoted-components-to-tripleo-ci-testing/5151150/job-output.txt

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_746/880600/1/check/tripleo-ci-centos-9-content-provider/746b54f/job-output.txt

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a30/881471/1/check/tripleo-ci-centos-9-content-provider/a30e090/job-output.txt

[Ronelle Landy (rlandy)]

From chatting on #opendev:

<fungi> rlandy: https://review.opendev.org/868392 switched us from mirror.facebook.net to mirror.rackspace.com in December because facebooks mirrors stopped updating, according to the commit message

Looking at the patch and the mirror link referenced: mirror.rackspace.com/centos-stream/9-stream/:

http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/repodata/

vs.

http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/repodata/

[Ronelle Landy (rlandy)]

https://mirror.bhs1.ovh.opendev.org/centos-stream/timestamp.txt - shows 2023-05-02T00:09:03,058927131+00:00 - so that latest mirror update.

[Marios Andreou (marios-b)]

I am not sure that switching back to facebook mirror would help us

For example, taking a job that just failed a little while ago at https://zuul.opendev.org/t/openstack/build/b74caf7c42ae47d68d810457eb916a10

it failed for:
  - Status code: 404 for https://mirror-int.ord.rax.opendev.org/centos-stream/9-stream/BaseOS/x86_64/os/repodata/e9e075843272c90115f8a5c44c1f3b23be8714c443aa0543d40bc38a2cf49de5-filelists.xml.gz (IP: 10.209.128.57)
  - Status code: 404 for https://mirror-int.ord.rax.opendev.org/centos-stream/9-stream/BaseOS/x86_64/os/repodata/3ea088796d71ec43bd0450022bddc9365606b1996065fac43595e4ef6798af11-primary.xml.gz (IP: 10.209.128.57)

So we want e9e075843272c90115f8a5c44c1f3b23be8714c443aa0543d40bc38a2cf49de5 and 3ea088796d71ec43bd0450022bddc9365606b1996065fac43595e4ef6798af11

BUT neither facebook nor rax mirror has that currently i.e.

http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml
&
http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml

do not have those.

[Cédric Jeanneret (cjeanner)]

I just checked the CERN mirror[1] and I apparently can't match either of the mentioned hashes. I'd suspect some issue deeper in the sync thing then...

[1] https://linuxsoft.cern.ch/cern/centos/s9/BaseOS/x86_64/os/repodata/repomd.xml

[Marios Andreou (marios-b)]

in fact i cannot find those in https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/repodata/repomd.xml either

e9e075843272c90115f8a5c44c1f3b23be8714c443aa0543d40bc38a2cf49de5 and 3ea088796d71ec43bd0450022bddc9365606b1996065fac43595e4ef6798af11

neither are there

[Marios Andreou (marios-b)]

possible workaround with https://review.opendev.org/c/openstack/tripleo-common/+/881949

overriding the 'baseos-source': true for the configure_mirrors_components_9_stream:

found via [1][2]

The job is still running there but it seems to be passed the setup stage and actually into container build now https://zuul.openstack.org/stream/e4498f721cef44d78b63b6782554e55f?logfile=console.log

2023-05-02 09:54:07.098525 | primary | TASK [tripleo.operator.tripleo_container_image_build : Run tripleo container image build] ***
2023-05-02 09:54:07.098652 | primary | Tuesday 02 May 2023 09:54:07 +0000 (0:00:02.410) 0:24:07.155 ***********

[1] https://opendev.org/zuul/zuul-jobs/src/commit/839de7f8996838162ae0de6a9f6ba28f968381bc/roles/configure-mirrors/vars/CentOS-9.yaml#L5

[2] https://opendev.org/zuul/zuul-jobs/src/commit/839de7f8996838162ae0de6a9f6ba28f968381bc/roles/configure-mirrors/templates/centos9-stream/etc/yum.repos.d/centos.repo.j2#L31-L34

[Ronelle Landy (rlandy)]

looks like http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/repodata/ got updated with multiple entries today.

There are some held (successful) c9 content-provider jobs. And RDO jobs started clearing about one/two hours ago.

So we may have cleared this block

[Marios Andreou (marios-b)]

yeah so per comment #6 ... the workaround may/not have helped but it seems it was just in time for the mirrors to update, and that is why it was passing there.

[Alan Pevec (apevec)]

this looks like an old copy of repomd.xml referencing previous versions of repodata XML files was cached somewhere

[Alan Pevec (apevec)]

Marking as Incomplete until we get more info on CentOS Stream repodata caching.

[Alan Pevec (apevec)]

iirc this was fixed on the centos infra side