periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master is failing image build - dependency fence-agents-common = 4.10.0-28.el9

[Ronelle Landy]

periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master is failing to build overclous images with:

2022-08-10 15:10:32.174 | > --> Starting dependency resolution
2022-08-10 15:10:32.205 | > --> Finished dependency resolution
2022-08-10 15:10:32.205 | > Error:
2022-08-10 15:10:32.205 | > Problem: package fence-agents-sbd-4.10.0-27.el9.noarch requires fence-agents-common = 4.10.0-27.el9, but none of the providers can be installed
2022-08-10 15:10:32.205 | > - cannot install both fence-agents-common-4.10.0-28.el9.noarch and fence-agents-common-4.10.0-27.el9.noarch
2022-08-10 15:10:32.205 | > - cannot install both fence-agents-common-4.10.0-27.el9.noarch and fence-agents-common-4.10.0-28.el9.noarch
2022-08-10 15:10:32.206 | > - package fence-agents-compute-4.10.0-28.el9.x86_64 requires fence-agents-common = 4.10.0-28.el9, but none of the providers can be installed
2022-08-10 15:10:32.206 | > - cannot install the best candidate for the job
2022-08-10 15:10:32.206 | > (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
2022-08-10 15:10:32.206 | returncode: 1

Example logs:

https://logserver.rdoproject.org/54/36254/146/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/22b7ab2/logs/undercloud/home/zuul/build.log.txt.gz

https://logserver.rdoproject.org/54/36254/145/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/8c0e86a/logs/undercloud/home/zuul/build.log.txt.gz

This jobs has been failing since 08/09:

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master&skip=0

[Marios Andreou (marios-b)]

will check here today

might be related to/same root cause as https://bugs.launchpad.net/tripleo/+bug/1984175 i.e. mirrors/sync issue but I need to check to be more sure of that.

[Alan Pevec (apevec)]

same as the other issue, FB CS9 mirror used as a source for opendev's AFS is out of date
need to find FB mirror admin and ask them to force rsync from CS9 origin mirror

[Marios Andreou (marios-b)]

@apevec ack as commented earlier I wanted to cycle back and check the packages... indeed looks like fence-agents is coming from the same repo that is problematic in the +bug/1984175

http://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/Packages/

[ ] fence-agents-common-4.10.0-23.el9.noarch.rpm 2022-05-05 15:10 426K
[ ] fence-agents-common-4.10.0-24.el9.noarch.rpm 2022-05-17 13:45 426K
[ ] fence-agents-common-4.10.0-26.el9.noarch.rpm 2022-06-02 09:46 426K
[ ] fence-agents-common-4.10.0-27.el9.noarch.rpm 2022-06-23 10:48 426K
[ ] fence-agents-common-4.10.0-28.el9.noarch.rpm 2022-08-02 08:35 428K

waiting on https://review.opendev.org/c/opendev/system-config/+/852793

[Ronelle Landy (rlandy)]

This is also hitting wallaby c9 fs002:

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-wallaby/c9a4e13/logs/undercloud/home/zuul/build.log.txt.gz

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-wallaby

[Sandeep Yadav (sandeepyadav93)]

FB Mirror is back in sync, undercloud upgrade issue for the bug/1984175 is solved now.

Rekicking fs002 via testproject[1] to confirm all okay.

[1] https://review.rdoproject.org/r/c/testproject/+/28537

[Sandeep Yadav (sandeepyadav93)]

fs002 passed

periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master https://review.rdoproject.org/zuul/build/6b659ded062c46109d36c229fbaf3da4 : SUCCESS in 1h 58m 06s

periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-wallaby https://review.rdoproject.org/zuul/build/ffe38d9ed80e4a9ba58d32d6920ab138 : SUCCESS in 2h 06m 46s

Issue got cleared after FB mirror sync clears and also the change at https://review.opendev.org/c/opendev/system-config/+/852793 merged - where we moved to rax mirror.