Check/gate jobs failing on task "configure-mirrors : Update yum/dnf cache" with Error: Failed to download metadata for repo 'AppStream': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

[Sandeep Yadav]

We noticed that intermittently some check/gate jobs failing on task "configure-mirrors : Update yum/dnf cache" with Error: Failed to download metadata for repo 'AppStream': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7d5/756492/1/gate/tripleo-ci-centos-8-content-provider/7d526c8/job-output.txt
~~~
2020-10-23 07:36:25.265465 | LOOP [configure-mirrors : Update yum/dnf cache]
2020-10-23 07:36:26.180518 | primary | 20 files removed
2020-10-23 07:36:26.812485 | primary | Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync
2020-10-23 07:36:26.812689 | primary | DNF version: 4.2.17
2020-10-23 07:36:26.812787 | primary | cachedir: /var/cache/dnf
2020-10-23 07:36:26.812879 | primary | Making cache files for all metadata files.
2020-10-23 07:36:26.812970 | primary | AppStream: has expired and will be refreshed.
2020-10-23 07:36:26.813060 | primary | BaseOS: has expired and will be refreshed.
2020-10-23 07:36:26.813149 | primary | extras: has expired and will be refreshed.
2020-10-23 07:36:26.813237 | primary | HighAvailability: has expired and will be refreshed.
2020-10-23 07:36:26.813327 | primary | PowerTools: has expired and will be refreshed.
2020-10-23 07:36:26.813415 | primary | repo: downloading from remote: AppStream
2020-10-23 07:36:26.813505 | primary | error: Status code: 403 for https://mirror.regionone.limestone.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml (IP: 2607:ff68:100:54:f816:3eff:feb5:4635) (https://mirror.regionone.limestone.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml).
2020-10-23 07:36:26.813605 | primary | error: Status code: 403 for https://mirror.regionone.limestone.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml (IP: 2607:ff68:100:54:f816:3eff:feb5:4635) (https://mirror.regionone.limestone.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml).
2020-10-23 07:36:26.814120 | primary | error: Status code: 403 for https://mirror.regionone.limestone.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml (IP: 2607:ff68:100:54:f816:3eff:feb5:4635) (https://mirror.regionone.limestone.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml).
2020-10-23 07:36:26.814213 | primary | error: Status code: 403 for https://mirror.regionone.limestone.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml (IP: 2607:ff68:100:54:f816:3eff:feb5:4635) (https://mirror.regionone.limestone.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml).
2020-10-23 07:36:26.814303 | primary | CentOS-8 - AppStream 5.1 kB/s | 199 B 00:00
2020-10-23 07:36:26.814749 | primary | Errors during downloading metadata for repository 'AppStream':
2020-10-23 07:36:26.814847 | primary | - Status code: 403 for https://mirror.regionone.limestone.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml (IP: 2607:ff68:100:54:f816:3eff:feb5:4635)
2020-10-23 07:36:26.814939 | primary | Error: Failed to download metadata for repo 'AppStream': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
~~~

https://bc4b4befb95a58dee4cd-414923073fac46561b63670d4adbb9d2.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/tripleo-quickstart-extras/master/tripleo-ci-centos-8-containers-multinode-train/8608b45/job-output.txt
~~~
2020-10-23 06:34:05.229898 | LOOP [configure-mirrors : Update yum/dnf cache]
2020-10-23 06:34:06.151017 | secondary | 20 files removed
2020-10-23 06:34:06.166717 | primary | 20 files removed
2020-10-23 06:34:06.769535 | primary | Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync
2020-10-23 06:34:06.769737 | primary | DNF version: 4.2.17
2020-10-23 06:34:06.769837 | primary | cachedir: /var/cache/dnf
2020-10-23 06:34:06.769953 | primary | Making cache files for all metadata files.
2020-10-23 06:34:06.770051 | primary | AppStream: has expired and will be refreshed.
2020-10-23 06:34:06.770143 | primary | BaseOS: has expired and will be refreshed.
2020-10-23 06:34:06.770233 | primary | extras: has expired and will be refreshed.
2020-10-23 06:34:06.770323 | primary | HighAvailability: has expired and will be refreshed.
2020-10-23 06:34:06.770414 | primary | PowerTools: has expired and will be refreshed.
2020-10-23 06:34:06.770504 | primary | repo: downloading from remote: AppStream
2020-10-23 06:34:06.770595 | primary | error: Status code: 403 for https://mirror.regionone.limestone.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml (IP: 2607:ff68:100:54:f816:3eff:feb5:4635) (https://mirror.regionone.limestone.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml).
2020-10-23 06:34:06.770695 | primary | error: Status code: 403 for https://mirror.regionone.limestone.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml (IP: 2607:ff68:100:54:f816:3eff:feb5:4635) (https://mirror.regionone.limestone.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml).
2020-10-23 06:34:06.770787 | primary | error: Status code: 403 for https://mirror.regionone.limestone.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml (IP: 2607:ff68:100:54:f816:3eff:feb5:4635) (https://mirror.regionone.limestone.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml).
2020-10-23 06:34:06.770877 | primary | error: Status code: 403 for https://mirror.regionone.limestone.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml (IP: 2607:ff68:100:54:f816:3eff:feb5:4635) (https://mirror.regionone.limestone.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml).
2020-10-23 06:34:06.770983 | primary | CentOS-8 - AppStream 5.0 kB/s | 199 B 00:00
2020-10-23 06:34:06.771075 | primary | Errors during downloading metadata for repository 'AppStream':
2020-10-23 06:34:06.771165 | primary | - Status code: 403 for https://mirror.regionone.limestone.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml (IP: 2607:ff68:100:54:f816:3eff:feb5:4635)
2020-10-23 06:34:06.771255 | primary | Error: Failed to download metadata for repo 'AppStream': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
2020-10-23 06:34:06.823861 | secondary | Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync
~~~

[Sandeep Yadav (sandeepyadav93)]

There were issues with one of a mirror mirror.regionone.limestone.opendev.org

It went unaccessible. Infra team rebooted the affected server.

[Sandeep Yadav (sandeepyadav93)]

We have not observed the issue again after the Infra team rebooted the affected server.

Closing this bug, for now, please feel free to reopen if the issue reappears.