resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated and expected checksum different

[chandan kumar]

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider job started failing
from 2022-06-16 00:20:27 onwards.

By taking a deep look, we saw following error:
```
[MIRROR] resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796(sha256) Expected: 3a4a37810d503f5eefb6b114d3b9f65d82ede7b9983dd1b3d62f975f8081e94b(sha256)
[MIRROR] resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796(sha256) Expected: 3a4a37810d503f5eefb6b114d3b9f65d82ede7b9983dd1b3d62f975f8081e94b(sha256)
[MIRROR] resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796(sha256) Expected: 3a4a37810d503f5eefb6b114d3b9f65d82ede7b9983dd1b3d62f975f8081e94b(sha256)
[MIRROR] resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796(sha256) Expected: 3a4a37810d503f5eefb6b114d3b9f65d82ede7b9983dd1b3d62f975f8081e94b(sha256)
[FAILED] resource-agents-4.10.0-17.el9.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success

The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Error downloading packages:
  resource-agents-4.10.0-17.el9.x86_64: Cannot download, all mirrors were already tried without success
```
And Nodepool mirror used [2]is
```
NODEPOOL_CENTOS_MIRROR=http://mirror.iad3.inmotion.opendev.org/centos-stream
```
We sync content from facebook mirror to OpenDev afs mirror.
Ianw has investigated the issue further and found that resource-agents-4.10.0-17.el9.x86_64 present on facebook mirror is corrupted.
Below is the investigation traceback:
```
# upstream

$ curl -s http://mirror.stream.centos.org/9-stream/HighAvailability/x86_64/os/Packages/resource-agents-4.10.0-17.el9.x86_64.rpm | sha256sum
3a4a37810d503f5eefb6b114d3b9f65d82ede7b9983dd1b3d62f975f8081e94b -

# our upstream mirror

$ curl -s http://mirror.facebook.net/centos-stream/9-stream/HighAvailability/x86_64/os/Packages/resource-agents-4.10.0-17.el9.x86_64.rpm | sha256sum
b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796 -

# our mirror

$ curl -s http://mirror.iad.rax.opendev.org/centos-stream/9-stream/HighAvailability/x86_64/os/Packages/resource-agents-4.10.0-17.el9.x86_64.rpm | sha256sum
b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796 -

# ergo we have synced the bad file from the facebook mirror
```
As we can see the checksum of the file resource-agents-4.10.0-17.el9.x86_64 present on mirror.stream.centos.org is different from facebook and AFS mirror leading to
following error
```
[MIRROR] resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796(sha256) Expected: 3a4a37810d503f5eefb6b114d3b9f65d82ede7b9983dd1b3d62f975f8081e94b(sha256)
```
There is no idea what caused the corrupt file presence on facebook mirror.
As a temporary workaround, Ian replaced the corrupted file on OpenDev mirror with the actual file and removed the lock on AFS mirror to avoid sync for today.
It will unblock the CI temporarily.
Here is the corrupted file: http://mirror.iad.rax.opendev.org/centos-stream/9-stream/HighAvailability/x86_64/os/Packages/resource-agents-4.10.0-17.el9.x86_64.rpm.corrupt

It is a tracker bug for the same.

Links:
[1]. https://1ca2ee7583d21788b1d8-42b9b3ca9891e58d539431fcfb5b799d.ssl.cf2.rackcdn.com/841114/2/check/tripleo-ci-centos-9-content-provider/547bca5/logs/undercloud/home/zuul/workspace/logs/container-builds/505f9cce-8572-4172-b2c7-470f0678749f/base/rabbitmq/rabbitmq-build.log

[2]. https://1ca2ee7583d21788b1d8-42b9b3ca9891e58d539431fcfb5b799d.ssl.cf2.rackcdn.com/841114/2/check/tripleo-ci-centos-9-content-provider/547bca5/job-output.txt

[yatin (yatinkarel)]

Adding discussion from centos-devel wrt this issue:-

<ykarel> we are seeing errors like with 9-stream repos Downloading successful, but checksum doesn't match. Calculated:
<ykarel> can you check if it's related to todays mirror push
<ianw_> i have to run out, but https://paste.opendev.org/show/b68rv7RD3lkuV1RwpuB8/ shows where we started with this
<arrfab> mirrormanager seems to be confused and while some mirrors are up2date, mirrormanager still has the previous copy/checksum so invalidating all mirrors I guess
<arrfab> problem is that we don't control mirrormanager and it will still consider everything out of sync, until it fetches up2date info from somewhere else
<arrfab> adrian: ^
<ykarel> arrfab, and how that will be fixed?
<arrfab> ykarel: well, we should contact fedora infra
<arrfab> but it's mainly managed by adrian (doing that in his spare time) so let's see if we can reach him
<ykarel> arrfab, yes please

Also facebook mirror from where we sync seems to have issue likely due to above as below fails but with mirror.stream.centos.org it passes
sudo dnf install --repofrompath=tmp,http://mirror.facebook.net/centos-stream/9-stream/HighAvailability/x86_64/os --disablerepo=* --enablerepo=tmp resource-agents
vs
sudo dnf install --repofrompath=tmp,http://mirror.stream.centos.org/9-stream/HighAvailability/x86_64/os --disablerepo=* --enablerepo=tmp resource-agents

[yatin (yatinkarel)]

Update:-
It's fixed now https://pagure.io/centos-infra/issue/812
<adrian> arrfab: MM DB is now in sync with the primary mirror, fixes should go live 20 minutes after the next hour

Following works now.
sudo dnf install --repofrompath=tmp,http://mirror.facebook.net/centos-stream/9-stream/HighAvailability/x86_64/os --disablerepo=* --enablerepo=tmp resource-agents

[Arx Cruz (arxcruz)]

I am no longer seeing failures on https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider closing it.