Issue with Upstream Mirror - error: Status code: 403

[Sandeep Yadav]

Description:-

Issue with Upstream Mirror - error: Status code: 403

Logs:-

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9b6/775310/3/check/tripleo-validations-centos-8-molecule-ceph/9b6b259/job-output.txt
~~~
error: Status code: 403 for https://mirror.kna1.airship-citycloud.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml (IP: 188.212.109.26) (https://mirror.kna1.airship-citycloud.opendev.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml).
~~~

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_99a/775067/1/check/openstack-tox-py37/99aa3cd/job-output.txt
~~~
Failed to fetch https://mirror.regionone.limestone.opendev.org/ubuntu/dists/bionic/universe/binary-amd64/Packages 403 Forbidden [IP: 2607:ff68:100:54:f816:3eff:feb5:4635 443]
~~

We are working with infra.

[Sandeep Yadav (sandeepyadav93)]

Workaround:-

Infra team added afs server - afsdb01 and afsdb02 to emergency disable list and added back missing public UDP ports in firewall rules.

Permanent solution:-

https://review.opendev.org/c/opendev/system-config/+/775311

Keeping the situation under observation.

[Sandeep Yadav (sandeepyadav93)]

We didn't notice the issue again after the workaround was applied, further permanent fix[1] is now merged.

[1] https://review.opendev.org/c/opendev/system-config/+/775311