| 2022-06-16 08:09:15 |
chandan kumar |
description |
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider job started failing
from 2022-06-16 00:20:27 onwards.
By taking a deep look, we saw following error:
```
[MIRROR] resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796(sha256) Expected: 3a4a37810d503f5eefb6b114d3b9f65d82ede7b9983dd1b3d62f975f8081e94b(sha256)
[MIRROR] resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796(sha256) Expected: 3a4a37810d503f5eefb6b114d3b9f65d82ede7b9983dd1b3d62f975f8081e94b(sha256)
[MIRROR] resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796(sha256) Expected: 3a4a37810d503f5eefb6b114d3b9f65d82ede7b9983dd1b3d62f975f8081e94b(sha256)
[MIRROR] resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796(sha256) Expected: 3a4a37810d503f5eefb6b114d3b9f65d82ede7b9983dd1b3d62f975f8081e94b(sha256)
[FAILED] resource-agents-4.10.0-17.el9.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Error downloading packages:
resource-agents-4.10.0-17.el9.x86_64: Cannot download, all mirrors were already tried without success
```
And Nodepool mirror used [2]is
```
NODEPOOL_CENTOS_MIRROR=http://mirror.iad3.inmotion.opendev.org/centos-stream
```
We sync content from facebook mirror to OpenDev afs mirror.
Ianw has investigated the issue further and found that resource-agents-4.10.0-17.el9.x86_64 present on facebook mirror is corrupted.
Below is the investigation traceback:
```
# upstream
$ curl -s http://mirror.stream.centos.org/9-stream/HighAvailability/x86_64/os/Packages/resource-agents-4.10.0-17.el9.x86_64.rpm | sha256sum
3a4a37810d503f5eefb6b114d3b9f65d82ede7b9983dd1b3d62f975f8081e94b -
# our upstream mirror
$ curl -s http://mirror.facebook.net/centos-stream/9-stream/HighAvailability/x86_64/os/Packages/resource-agents-4.10.0-17.el9.x86_64.rpm | sha256sum
b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796 -
# our mirror
$ curl -s http://mirror.iad.rax.opendev.org/centos-stream/9-stream/HighAvailability/x86_64/os/Packages/resource-agents-4.10.0-17.el9.x86_64.rpm | sha256sum
b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796 -
# ergo we have synced the bad file from the facebook mirror
```
As we can see the checksum of the file resource-agents-4.10.0-17.el9.x86_64 present on mirror.stream.centos.org is different from facebook and AFS mirror leading to
following error
```
[MIRROR] resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796(sha256) Expected: 3a4a37810d503f5eefb6b114d3b9f65d82ede7b9983dd1b3d62f975f8081e94b(sha256)
```
There is no idea what caused the corrupt file presence on facebook mirror.
As a temporary workaround, Ian replaced the corrupted file on OpenDev mirror with the actual file and removed the lock on AFS mirror to avoid sync for today.
It will unblock the CI temporarily.
It is a tracker bug for the same.
Links:
[1]. https://1ca2ee7583d21788b1d8-42b9b3ca9891e58d539431fcfb5b799d.ssl.cf2.rackcdn.com/841114/2/check/tripleo-ci-centos-9-content-provider/547bca5/logs/undercloud/home/zuul/workspace/logs/container-builds/505f9cce-8572-4172-b2c7-470f0678749f/base/rabbitmq/rabbitmq-build.log
[2]. https://1ca2ee7583d21788b1d8-42b9b3ca9891e58d539431fcfb5b799d.ssl.cf2.rackcdn.com/841114/2/check/tripleo-ci-centos-9-content-provider/547bca5/job-output.txt |
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider job started failing
from 2022-06-16 00:20:27 onwards.
By taking a deep look, we saw following error:
```
[MIRROR] resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796(sha256) Expected: 3a4a37810d503f5eefb6b114d3b9f65d82ede7b9983dd1b3d62f975f8081e94b(sha256)
[MIRROR] resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796(sha256) Expected: 3a4a37810d503f5eefb6b114d3b9f65d82ede7b9983dd1b3d62f975f8081e94b(sha256)
[MIRROR] resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796(sha256) Expected: 3a4a37810d503f5eefb6b114d3b9f65d82ede7b9983dd1b3d62f975f8081e94b(sha256)
[MIRROR] resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796(sha256) Expected: 3a4a37810d503f5eefb6b114d3b9f65d82ede7b9983dd1b3d62f975f8081e94b(sha256)
[FAILED] resource-agents-4.10.0-17.el9.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Error downloading packages:
resource-agents-4.10.0-17.el9.x86_64: Cannot download, all mirrors were already tried without success
```
And Nodepool mirror used [2]is
```
NODEPOOL_CENTOS_MIRROR=http://mirror.iad3.inmotion.opendev.org/centos-stream
```
We sync content from facebook mirror to OpenDev afs mirror.
Ianw has investigated the issue further and found that resource-agents-4.10.0-17.el9.x86_64 present on facebook mirror is corrupted.
Below is the investigation traceback:
```
# upstream
$ curl -s http://mirror.stream.centos.org/9-stream/HighAvailability/x86_64/os/Packages/resource-agents-4.10.0-17.el9.x86_64.rpm | sha256sum
3a4a37810d503f5eefb6b114d3b9f65d82ede7b9983dd1b3d62f975f8081e94b -
# our upstream mirror
$ curl -s http://mirror.facebook.net/centos-stream/9-stream/HighAvailability/x86_64/os/Packages/resource-agents-4.10.0-17.el9.x86_64.rpm | sha256sum
b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796 -
# our mirror
$ curl -s http://mirror.iad.rax.opendev.org/centos-stream/9-stream/HighAvailability/x86_64/os/Packages/resource-agents-4.10.0-17.el9.x86_64.rpm | sha256sum
b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796 -
# ergo we have synced the bad file from the facebook mirror
```
As we can see the checksum of the file resource-agents-4.10.0-17.el9.x86_64 present on mirror.stream.centos.org is different from facebook and AFS mirror leading to
following error
```
[MIRROR] resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: b24e6a8a70066918658ffc390d3dc4e3f91eb19eead295db9f14b7af988d8796(sha256) Expected: 3a4a37810d503f5eefb6b114d3b9f65d82ede7b9983dd1b3d62f975f8081e94b(sha256)
```
There is no idea what caused the corrupt file presence on facebook mirror.
As a temporary workaround, Ian replaced the corrupted file on OpenDev mirror with the actual file and removed the lock on AFS mirror to avoid sync for today.
It will unblock the CI temporarily.
Here is the corrupted file: http://mirror.iad.rax.opendev.org/centos-stream/9-stream/HighAvailability/x86_64/os/Packages/resource-agents-4.10.0-17.el9.x86_64.rpm.corrupt
It is a tracker bug for the same.
Links:
[1]. https://1ca2ee7583d21788b1d8-42b9b3ca9891e58d539431fcfb5b799d.ssl.cf2.rackcdn.com/841114/2/check/tripleo-ci-centos-9-content-provider/547bca5/logs/undercloud/home/zuul/workspace/logs/container-builds/505f9cce-8572-4172-b2c7-470f0678749f/base/rabbitmq/rabbitmq-build.log
[2]. https://1ca2ee7583d21788b1d8-42b9b3ca9891e58d539431fcfb5b799d.ssl.cf2.rackcdn.com/841114/2/check/tripleo-ci-centos-9-content-provider/547bca5/job-output.txt |
|
