yum repo mirrors are down 5/24/2021

[wes hayutin]

Most jobs in 3rd party TripleO are failing on mirror setup.

2021-05-24 19:24:41.273400 |
2021-05-24 19:24:41.273544 | LOOP [configure-mirrors : Update yum/dnf cache]
2021-05-24 19:24:42.422761 | primary | 21 files removed
2021-05-24 19:24:42.948067 | primary | Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync
2021-05-24 19:24:42.948252 | primary | DNF version: 4.4.2
2021-05-24 19:24:42.948322 | primary | cachedir: /var/cache/dnf
2021-05-24 19:24:42.948382 | primary | Making cache files for all metadata files.
2021-05-24 19:24:42.948441 | primary | appstream: has expired and will be refreshed.
2021-05-24 19:24:42.948497 | primary | baseos: has expired and will be refreshed.
2021-05-24 19:24:42.948608 | primary | extras: has expired and will be refreshed.
2021-05-24 19:24:42.948670 | primary | ha: has expired and will be refreshed.
2021-05-24 19:24:42.948726 | primary | powertools: has expired and will be refreshed.
2021-05-24 19:24:42.948782 | primary | repo: downloading from remote: appstream
2021-05-24 19:24:42.948841 | primary | error: Status code: 404 for http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz (IP: 38.102.83.237) (http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz).
2021-05-24 19:24:42.948905 | primary | error: Status code: 404 for http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/d0435a46fff272cacc6c2d5433e8e2b0b2d70d57141116c8b9fa7624aaf01aaf-filelists.xml.gz (IP: 38.102.83.237) (http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/d0435a46fff272cacc6c2d5433e8e2b0b2d70d57141116c8b9fa7624aaf01aaf-filelists.xml.gz).
2021-05-24 19:24:42.948964 | primary | error: Status code: 404 for http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz (IP: 38.102.83.237) (http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz).
2021-05-24 19:24:42.949058 | primary | error: Status code: 404 for http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/d0435a46fff272cacc6c2d5433e8e2b0b2d70d57141116c8b9fa7624aaf01aaf-filelists.xml.gz (IP: 38.102.83.237) (http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/d0435a46fff272cacc6c2d5433e8e2b0b2d70d57141116c8b9fa7624aaf01aaf-filelists.xml.gz).
2021-05-24 19:24:42.949121 | primary | error: Status code: 404 for http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz (IP: 38.102.83.237) (http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz).
2021-05-24 19:24:42.949178 | primary | CentOS-Stream - AppStream 14 MB/s | 458 kB 00:00
2021-05-24 19:24:42.949234 | primary | Errors during downloading metadata for repository 'appstream':
2021-05-24 19:24:42.949292 | primary | - Status code: 404 for http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz (IP: 38.102.83.237)
2021-05-24 19:24:42.949352 | primary | - Status code: 404 for http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/d0435a46fff272cacc6c2d5433e8e2b0b2d70d57141116c8b9fa7624aaf01aaf-filelists.xml.gz (IP: 38.102.83.237)
2021-05-24 19:24:42.949422 | primary | Error: Failed to download metadata for repo 'appstream': Yum repo downloading error: Downloading error(s): repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/d0435a46fff272cacc6c2d5433e8e2b0b2d70d57141116c8b9fa7624aaf01aaf-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/7e8f79b6889aebe5f13c31af5add1aa27e21d71056a5645567f64e9427925a0b-modules.yaml.xz - Cannot download, all mirrors were already tried without success
2021-05-24 19:24:42.978407 |

[wes hayutin (weshayutin)]

Also see this in psi public..

2021-05-24 19:59:01.725520 | primary | powertools: has expired and will be refreshed.
2021-05-24 19:59:01.725555 | primary | repo: downloading from remote: appstream
2021-05-24 19:59:01.725591 | primary | error: Status code: 404 for http://mirror.regionone.psi-public-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz (IP: 38.145.38.112) (http://mirror.regionone.psi-public-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz).
2021-05-24 19:59:01.725629 | primary | error: Status code: 404 for http://mirror.regionone.psi-public-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz (IP: 38.145.38.112) (http://mirror.regionone.psi-public-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz).
2021-05-24 19:59:01.725669 | primary | error: Status code: 404 for http://mirror.regionone.psi-public-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/d0435a46fff272cacc6c2d5433e8e2b0b2d70d57141116c8b9fa7624aaf01aaf-filelists.xml.gz (IP: 38.145.38.112) (http://mirror.regionone.psi-public-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/d0435a46fff272cacc6c2d5433e8e2b0b2d70d57141116c8b9fa7624aaf01aaf-filelists.xml.gz).
2021-05-24 19:59:01.725706 | primary | error: Status code: 404 for http://mirror.regionone.psi-public-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz (IP: 38.145.38.112) (http://mirror.regionone.psi-public-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz).
2021-05-24 19:59:01.725743 | primary | error: Status code: 404 for http://mirror.regionone.psi-public-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/d0435a46fff272cacc6c2d5433e8e2b0b2d70d57141116c8b9fa7624aaf01aaf-filelists.xml.gz (IP: 38.145.38.112) (http://mirror.regionone.psi-public-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/d0435a46fff272cacc6c2d5433e8e2b0b2d70d57141116c8b9fa7624aaf01aaf-filelists.xml.gz).
2021-05-24 19:59:01.725779 | primary | CentOS-Stream - AppStream 447 kB/s | 458 kB 00:01
2021-05-24 19:59:01.725815 | primary | Errors during downloading metadata for repository 'appstream':
2021-05-24 19:59:01.725891 | primary | - Status code: 404 for http://mirror.regionone.psi-public-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz (IP: 38.145.38.112)
2021-05-24 19:59:01.725940 | primary | - Status code: 404 for http://mirror.regionone.psi-public-nodepool-tripleo.rdoproject.org/c...

[wes hayutin (weshayutin)]

2021-05-24 20:18:58.910906 | primary | error: Status code: 404 for https://mirror-int.ord.rax.opendev.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz (IP: 10.209.128.57) (https://mirror-int.ord.rax.opendev.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz).
2021-05-24 20:18:58.910986 | primary | error: Status code: 404 for https://mirror-int.ord.rax.opendev.org/centos/8-stream/AppStream/x86_64/os/repodata/d0435a46fff272cacc6c2d5433e8e2b0b2d70d57141116c8b9fa7624aaf01aaf-filelists.xml.gz (IP: 10.209.128.57) (https://mirror-int.ord.rax.opendev.org/centos/8-stream/AppStream/x86_64/os/repodata/d0435a46fff272cacc6c2d5433e8e2b0b2d70d57141116c8b9fa7624aaf01aaf-filelists.xml.gz).
2021-05-24 20:18:58.911060 | primary | error: Status code: 404 for https://mirror-int.ord.rax.opendev.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz (IP: 10.209.128.57) (https://mirror-int.ord.rax.opendev.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz).
2021-05-24 20:18:58.911133 | primary | error: Status code: 404 for https://mirror-int.ord.rax.opendev.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz (IP: 10.209.128.57) (https://mirror-int.ord.rax.opendev.org/centos/8-stream/AppStream/x86_64/os/repodata/18234d890dc28f97d6aaa4796cb5dc6f8f37c029109b2bba04e388158804f171-primary.xml.gz).
2021-05-24 20:18:58.911211 | primary | error: Status code: 404 for https://mirror-int.ord.rax.opendev.org/centos/8-stream/AppStream/x86_64/os/repodata/d0435a46fff272cacc6c2d5433e8e2b0b2d70d57141116c8b9fa7624aaf01aaf-filelists.xml.gz (IP: 10.209.128.57) (https://mirror-int.ord.rax.opendev.org/centos/8-stream/AppStream/x86_64/os/repodata/d0435a46fff272cacc6c2d5433e8e2b0b2d70d57141116c8b9fa7624aaf01aaf-filelists.xml.gz).
2021-05-24 20:18:58.911283 | primary | CentOS-Stream - AppStream 151 kB/s | 457 kB 00:03
2021-05-24 20:18:58.911355 | primary | Errors during downloading metadata for re

[wes hayutin (weshayutin)]

seems resolved

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/793200

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/793200
Committed: https://opendev.org/openstack/tripleo-ci-health-queries/commit/39eefe04e2d6594e64c9cb4363767bf5c699bc6d
Submitter: "Zuul (22348)"
Branch: master

commit 39eefe04e2d6594e64c9cb4363767bf5c699bc6d
Author: Wesley Hayutin <email address hidden>
Date: Wed May 26 10:05:36 2021 -0600

    add mirror timeout query

    Related-Bug: #1929461
    Change-Id: I5b3b6467cd2ad10a5cad346deb5812bc47a1825d

[Sandeep Yadav (sandeepyadav93)]

We have again hit the issue today with limestone mirror.

Reached out to infra on #opendev and they have disabled the limestone mirror:

https://review.opendev.org/c/openstack/project-config/+/79414

Cause: appears to be slow operations on cache volume.