various jobs failing repo setup Cannot download repodata/repomd.xml: All mirrors were tried

[Marios Andreou]

At [1][2][3][4] various jobs are failing to setup repos. Though the exact failing repo may differ in each case inevitably there is "Cannot download repodata/repomd.xml: All mirrors were tried" - for example

"2022-03-16 11:18:05 | - Status code: 403 for http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/BaseOS/x86_64/os/repodata/repomd.xml (IP: 38.102.83.237)", "2022-03-16 11:18:05 | Error: Failed to download metadata for repo 'quickstart-centos-base': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried"]}
2022-03-16 11:18:06.233951 | primary |

Looks like there may be a problem with the mirror http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream

This blocks centos 8 stable/wallaby promotion... we have seen this issue since at least the 14th March [4] but it is hitting the latest run of the stable/wallaby line particularly badly (multiple jobs failing on this).

[1] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-wallaby/cdd744b/job-output.txt
[2] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-wallaby/3115a4d/job-output.txt
[3] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-wallaby/fd6c74c/job-output.txt
[4] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-undercloud-containers-wallaby/113fc3f/job-output.txt

[Douglas Viroel (dviroel)]

Mirrors issues are also happening in CentOS-9 jobs. For instance [1] just failed in periodic line with:
"msg": "Failed to download packages: libpmem-1.10.1-2.el9.x86_64: Cannot download, all mirrors were already tried without success"

[1]https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-9-buildimage-overcloud-hardened-uefi-full-wallaby/3ea5304/job-output.txt

[chandan kumar (chkumar246)]

Few recent examples:
https://logserver.rdoproject.org/00/834100/4/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/ed357ce/job-output.txt

```
Failed to download packages: device-mapper-persistent-data-0.9.0-12.el9.x86_64: Cannot download, all mirrors were already tried without success", "results": []}
```
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/c80caa6/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz

```
2022-03-16 14:57:13 | + sudo dnf install -yq ansible-core ansible-tripleo-ipa
2022-03-16 14:57:24 | Error: Error downloading packages:
2022-03-16 14:57:24 | ansible-macros-2021.1.2-2.el9s.noarch: Cannot download, all mirrors were already tried without success
2022-03-16 14:57:24 |
```

[Douglas Viroel (dviroel)]

Still facing mirror issue even after vexxhost internal migrations.

https://logserver.rdoproject.org/56/36356/39/check/periodic-tripleo-ci-centos-9-standalone-compute-wallaby/90c6987/job-output.txt

Errors during downloading metadata for repository 'quickstart-centos-appstreams':", "2022-03-17 15:00:28 | - Curl error (28): Timeout was reached for http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos-stream/9-stream/AppStream/x86_64/os/repodata/repomd.xml [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]", "2022-03-17 15:00:28 | Error: Failed to download metadata for repo 'quickstart-centos-appstreams': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried"]}

[Ronelle Landy (rlandy)]

This is still hitting us - as of yesterday

[Marios Andreou (marios-b)]

i don't think we are seeing this any more - at least not in the last couple days looking at [1][2][3][4][5]

moving to fixed we can move back if we see it again

[1] https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main

[2] https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable1

[3] https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable1-cs9

[4] https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable2
[5] https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable4

[Jakob Meng (jm1337)]

Still hitting this every week, e.g. in job:

* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby

with error [1]:

  + sudo dnf install -yq ansible-core ansible-tripleo-ipa
  Error: Error downloading packages:
    ansible-macros-2021.1.2-2.el9s.noarch: Cannot download, all mirrors were already tried without success

[1] https://logserver.rdoproject.org/07/45107/1/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby/9f1a7a4/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz