All CentOS upstream and rdo third party jobs broken with Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried.

[chandan kumar]

All CentOS based upstream and RDO third party jobs are broken due to following errors:

on Upstream:
2020-06-14 12:50:40.084792 | LOOP [configure-mirrors : Update yum/dnf cache]
2020-06-14 12:50:41.539581 | primary | 28 files removed
2020-06-14 12:50:42.918033 | primary | Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync
2020-06-14 12:50:42.918254 | primary | DNF version: 4.2.7
2020-06-14 12:50:42.918362 | primary | cachedir: /var/cache/dnf
2020-06-14 12:50:42.918464 | primary | Making cache files for all metadata files.
2020-06-14 12:50:42.918564 | primary | AppStream: has expired and will be refreshed.
2020-06-14 12:50:42.918662 | primary | BaseOS: has expired and will be refreshed.
2020-06-14 12:50:42.918759 | primary | extras: has expired and will be refreshed.
2020-06-14 12:50:42.918857 | primary | HighAvailability: has expired and will be refreshed.
2020-06-14 12:50:42.918954 | primary | PowerTools: has expired and will be refreshed.
2020-06-14 12:50:42.919051 | primary | epel-modular: has expired and will be refreshed.
2020-06-14 12:50:42.919148 | primary | repo: downloading from remote: AppStream
2020-06-14 12:50:42.919252 | primary | error: Status code: 403 for https://mirror.kna1.airship-citycloud.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml (https://mirror.kna1.airship-citycloud.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml).
2020-06-14 12:50:42.919358 | primary | CentOS-8 - AppStream 678 B/s | 199 B 00:00
2020-06-14 12:50:42.919456 | primary | Cannot download 'https://mirror.kna1.airship-citycloud.opendev.org/centos/8/AppStream/x86_64/os/': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried.
2020-06-14 12:50:42.919564 | primary | Failed to download metadata for repo 'AppStream'
2020-06-14 12:50:42.919661 | primary | Error: Failed to download metadata for repo 'AppStream'
2020-06-14 12:50:42.945052 |
2020-06-14 12:50:42.945185 | PLAY RECAP
2020-06-14 12:50:42.945326 | localhost | ok: 6 changed: 2 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0

https://a27160e83f5589b215d5-41718c434e8b2245213b6c590d7bf6c9.ssl.cf2.rackcdn.com/727200/69/check/tripleo-ci-centos-8-undercloud-containers/4d8f67f/job-output.txt

On RDO:
2020-06-14 12:45:20.680671 | LOOP [configure-mirrors : Update yum/dnf cache]
2020-06-14 12:45:21.664355 | primary | 28 files removed
2020-06-14 12:45:22.758786 | primary | Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync
2020-06-14 12:45:22.759078 | primary | DNF version: 4.2.7
2020-06-14 12:45:22.759207 | primary | cachedir: /var/cache/dnf
2020-06-14 12:45:22.759305 | primary | Making cache files for all metadata files.
2020-06-14 12:45:22.759457 | primary | AppStream: has expired and will be refreshed.
2020-06-14 12:45:22.759613 | primary | BaseOS: has expired and will be refreshed.
2020-06-14 12:45:22.759716 | primary | extras: has expired and will be refreshed.
2020-06-14 12:45:22.759809 | primary | HighAvailability: has expired and will be refreshed.
2020-06-14 12:45:22.759903 | primary | PowerTools: has expired and will be refreshed.
2020-06-14 12:45:22.759996 | primary | epel-modular: has expired and will be refreshed.
2020-06-14 12:45:22.760135 | primary | repo: downloading from remote: AppStream
2020-06-14 12:45:22.760245 | primary | error: Status code: 403 for http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml (http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml).

https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/configure-mirrors/handlers/main.yaml#L10 it is failing due to mirror issue.

Adding this bug as a tracker.
2020-06-14 12:45:22.760341 | primary | CentOS-8 - AppStream 716 B/s | 250 B 00:00
2020-06-14 12:45:22.760501 | primary | Cannot download 'http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org/centos/8/AppStream/x86_64/os/': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried.
2020-06-14 12:45:22.760651 | primary | Failed to download metadata for repo 'AppStream'
2020-06-14 12:45:22.760749 | primary | Error: Failed to download metadata for repo 'AppStream'
2020-06-14 12:45:22.799312 |

https://logserver.rdoproject.org/00/727200/69/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/8c09bf7/job-output.txt

It is happening across the openstack board. As per openstackstatus bot:
-openstackstatus/#rdo- NOTICE: The opendev specific CentoOS and openSUSE mirror disappeared and thus CentOS and openSUSE jobs are all broken.

[wes hayutin (weshayutin)]

upstream mirror monitoring is here..

http://cacti.openstack.org/cacti/graph_view.php

[wes hayutin (weshayutin)]

[02:35:24] -openstackstatus/#oooq- NOTICE: The opendev specific CentoOS and openSUSE mirror disappeared and thus CentOS and openSUSE jobs are all broken.
[07:40:00] -openstackstatus/#oooq- NOTICE: Package mirrors should be back in working order; any jobs which logged package retrieval failures between 19:35 UTC yesterday and 13:20 UTC today can be safely rechecked

[yatin (yatinkarel)]

Seems it's back:- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_552/727200/73/check/tripleo-ci-centos-8-containers-multinode/5524ccf/job-output.txt

[wes hayutin (weshayutin)]

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22All%20mirrors%20were%20tried%5C%22%20AND%20filename%3A%5C%22job-output.txt%5C%22

[Andreas Jaeger (jaegerandi)]

Regarding comment #3: The vexxhost mirror - and only that one - was broken. It's fixed now, all should be good again.

If you see anything new *after* this, please speak up on #opendev directly.

[Marios Andreou (marios-b)]

This is an automated action. Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo

[Launchpad Janitor (janitor)]

[Expired for tripleo because there has been no activity for 60 days.]