All CentOS upstream and rdo third party jobs broken with Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried.

Bug #1883430 reported by chandan kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Expired
Critical
Unassigned

Bug Description

All CentOS based upstream and RDO third party jobs are broken due to following errors:

on Upstream:
2020-06-14 12:50:40.084792 | LOOP [configure-mirrors : Update yum/dnf cache]
2020-06-14 12:50:41.539581 | primary | 28 files removed
2020-06-14 12:50:42.918033 | primary | Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync
2020-06-14 12:50:42.918254 | primary | DNF version: 4.2.7
2020-06-14 12:50:42.918362 | primary | cachedir: /var/cache/dnf
2020-06-14 12:50:42.918464 | primary | Making cache files for all metadata files.
2020-06-14 12:50:42.918564 | primary | AppStream: has expired and will be refreshed.
2020-06-14 12:50:42.918662 | primary | BaseOS: has expired and will be refreshed.
2020-06-14 12:50:42.918759 | primary | extras: has expired and will be refreshed.
2020-06-14 12:50:42.918857 | primary | HighAvailability: has expired and will be refreshed.
2020-06-14 12:50:42.918954 | primary | PowerTools: has expired and will be refreshed.
2020-06-14 12:50:42.919051 | primary | epel-modular: has expired and will be refreshed.
2020-06-14 12:50:42.919148 | primary | repo: downloading from remote: AppStream
2020-06-14 12:50:42.919252 | primary | error: Status code: 403 for https://mirror.kna1.airship-citycloud.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml (https://mirror.kna1.airship-citycloud.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml).
2020-06-14 12:50:42.919358 | primary | CentOS-8 - AppStream 678 B/s | 199 B 00:00
2020-06-14 12:50:42.919456 | primary | Cannot download 'https://mirror.kna1.airship-citycloud.opendev.org/centos/8/AppStream/x86_64/os/': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried.
2020-06-14 12:50:42.919564 | primary | Failed to download metadata for repo 'AppStream'
2020-06-14 12:50:42.919661 | primary | Error: Failed to download metadata for repo 'AppStream'
2020-06-14 12:50:42.945052 |
2020-06-14 12:50:42.945185 | PLAY RECAP
2020-06-14 12:50:42.945326 | localhost | ok: 6 changed: 2 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0

https://a27160e83f5589b215d5-41718c434e8b2245213b6c590d7bf6c9.ssl.cf2.rackcdn.com/727200/69/check/tripleo-ci-centos-8-undercloud-containers/4d8f67f/job-output.txt

On RDO:
2020-06-14 12:45:20.680671 | LOOP [configure-mirrors : Update yum/dnf cache]
2020-06-14 12:45:21.664355 | primary | 28 files removed
2020-06-14 12:45:22.758786 | primary | Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync
2020-06-14 12:45:22.759078 | primary | DNF version: 4.2.7
2020-06-14 12:45:22.759207 | primary | cachedir: /var/cache/dnf
2020-06-14 12:45:22.759305 | primary | Making cache files for all metadata files.
2020-06-14 12:45:22.759457 | primary | AppStream: has expired and will be refreshed.
2020-06-14 12:45:22.759613 | primary | BaseOS: has expired and will be refreshed.
2020-06-14 12:45:22.759716 | primary | extras: has expired and will be refreshed.
2020-06-14 12:45:22.759809 | primary | HighAvailability: has expired and will be refreshed.
2020-06-14 12:45:22.759903 | primary | PowerTools: has expired and will be refreshed.
2020-06-14 12:45:22.759996 | primary | epel-modular: has expired and will be refreshed.
2020-06-14 12:45:22.760135 | primary | repo: downloading from remote: AppStream
2020-06-14 12:45:22.760245 | primary | error: Status code: 403 for http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml (http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml).

https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/configure-mirrors/handlers/main.yaml#L10 it is failing due to mirror issue.

Adding this bug as a tracker.
2020-06-14 12:45:22.760341 | primary | CentOS-8 - AppStream 716 B/s | 250 B 00:00
2020-06-14 12:45:22.760501 | primary | Cannot download 'http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org/centos/8/AppStream/x86_64/os/': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried.
2020-06-14 12:45:22.760651 | primary | Failed to download metadata for repo 'AppStream'
2020-06-14 12:45:22.760749 | primary | Error: Failed to download metadata for repo 'AppStream'
2020-06-14 12:45:22.799312 |

https://logserver.rdoproject.org/00/727200/69/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/8c09bf7/job-output.txt

It is happening across the openstack board. As per openstackstatus bot:
-openstackstatus/#rdo- NOTICE: The opendev specific CentoOS and openSUSE mirror disappeared and thus CentOS and openSUSE jobs are all broken.

summary: - All CentOS upstream and rdo thrird party jobs broken with Cannot
+ All CentOS upstream and rdo third party jobs broken with Cannot
download repomd.xml: Cannot download repodata/repomd.xml: All mirrors
were tried.
Revision history for this message
wes hayutin (weshayutin) wrote :

upstream mirror monitoring is here..

http://cacti.openstack.org/cacti/graph_view.php

Revision history for this message
wes hayutin (weshayutin) wrote :

[02:35:24] -openstackstatus/#oooq- NOTICE: The opendev specific CentoOS and openSUSE mirror disappeared and thus CentOS and openSUSE jobs are all broken.
[07:40:00] -openstackstatus/#oooq- NOTICE: Package mirrors should be back in working order; any jobs which logged package retrieval failures between 19:35 UTC yesterday and 13:20 UTC today can be safely rechecked

wes hayutin (weshayutin)
Changed in tripleo:
status: Triaged → Fix Released
Revision history for this message
yatin (yatinkarel) wrote :
Changed in tripleo:
status: Fix Released → Triaged
Revision history for this message
wes hayutin (weshayutin) wrote :
Revision history for this message
wes hayutin (weshayutin) wrote :
Revision history for this message
Andreas Jaeger (jaegerandi) wrote :

Regarding comment #3: The vexxhost mirror - and only that one - was broken. It's fixed now, all should be good again.

If you see anything new *after* this, please speak up on #opendev directly.

Changed in tripleo:
milestone: victoria-1 → victoria-3
Changed in tripleo:
milestone: victoria-3 → wallaby-1
Changed in tripleo:
milestone: wallaby-1 → wallaby-2
Changed in tripleo:
milestone: wallaby-2 → wallaby-3
Changed in tripleo:
milestone: wallaby-3 → wallaby-rc1
Changed in tripleo:
milestone: wallaby-rc1 → xena-1
Changed in tripleo:
milestone: xena-1 → xena-2
Revision history for this message
Marios Andreou (marios-b) wrote :

This is an automated action. Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo

Changed in tripleo:
milestone: xena-2 → none
status: Triaged → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for tripleo because there has been no activity for 60 days.]

Changed in tripleo:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers