periodic-tripleo-centos-8-buildimage-overcloud-full-train is failing with : nothing provides pacemaker-libs(x86-64) = 2.1.2-3.el8 needed by pacemaker-2.1.2-3.el8.x86_64

[Ronelle Landy]

periodic-tripleo-centos-8-buildimage-overcloud-full-train is failing with

2022-01-30 18:31:54.651 | Error:
2022-01-30 18:31:54.652 | Problem 1: cannot install the best candidate for the job
2022-01-30 18:31:54.652 | - nothing provides pacemaker-libs(x86-64) = 2.1.2-3.el8 needed by pacemaker-2.1.2-3.el8.x86_64
2022-01-30 18:31:54.652 | - nothing provides pacemaker-cluster-libs(x86-64) = 2.1.2-3.el8 needed by pacemaker-2.1.2-3.el8.x86_64
2022-01-30 18:31:54.652 | Problem 2: cannot install the best candidate for the job
2022-01-30 18:31:54.652 | - nothing provides pacemaker-libs(x86-64) = 2.1.2-3.el8 needed by pacemaker-remote-2.1.2-3.el8.x86_64
2022-01-30 18:31:54.652 | (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Full log is below:

https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-train/8128424/build.log

The error started on 01/28:

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-8-buildimage-overcloud-full-train

[Ronelle Landy (rlandy)]

Looks like this is hitting wallaby as well:

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-wallaby/a9d5a26/build.log

[Marios Andreou (marios-b)]

and master :/ looks like it is all 8 lines?

        * https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-master/9031fc4/build.log
        * 2022-01-31 05:30:59.548 | Problem: cannot install the best candidate for the job
2022-01-31 05:30:59.548 | - nothing provides pacemaker-libs(x86-64) = 2.1.2-3.el8

strange thing is, I can see the package it is complaining about there [1]

        * pacemaker-cluster-libs-2.1.2-3.el8.x86_64.rpm 2022-01-27 16:13 139K

[1] http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/Packages/

[Ronelle Landy (rlandy)]

Checking the vexx mirror:

http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/Packages/

pacemaker-cluster-li..> 2022-01-27 16:13 139K

 http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/AppStream/x86_64/os/Packages/pacemaker-cluster-libs-2.1.2-3.el8.x86_64.rpm

[Alfredo Moralejo (amoralej)]

So, according to https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-master/9031fc4/yum.repos.d/:

- appstream and baseos repos are used from AFS mirror under http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/
- tripleo-centos-highavailability repo is taken from mirror.centos.org [1]

note that content in mirror.centos.org takes some time to land to mirrors.

in this case pacemaker-remote is being pulled from tripleo-centos-highavailability but the pacemaker-libs it requires is pulled from appstream repo in AFS mirror and when it was not synced yet. According to logs, AFS mirror has happened on 2022-01-31T06:33:26,086665596+00:00

So it should be fine now.

To avoid this situacion in future, note we have two diferent highavailability repos, one pointing to AFS mirror and one to mirror.centos.org:

https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-master/9031fc4/yum.repos.d/CentOS-Stream-HighAvailability.repo
https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-master/9031fc4/yum.repos.d/tripleo-centos-highavailability.repo

Wes should leave only the one pointing to afs mirror

[1] https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-master/9031fc4/yum.repos.d/tripleo-centos-highavailability.repo

[Marios Andreou (marios-b)]

looks like this is trending closed at least it is no longer blocking

we have a green run for the buildimage jobs today at [1][2]

[1] https://review.rdoproject.org/zuul/build/6c7d7db1c0d048798ffc3aede0e550d9
[2] https://review.rdoproject.org/zuul/build/15c27028099f4570b2f6638d647da76e

[Marios Andreou (marios-b)]

mostly green but seen today on train

        * https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/051457b7cd20347240a871126d2943d20722b78c/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-train/19b7d46/logs/undercloud/home/zuul/build.log.txt.gz
        * 2022-02-02 03:16:45.060 | > Error:
 2022-02-02 03:16:45.060 | > Problem: cannot install the best candidate for the job
 2022-02-02 03:16:45.060 | > - nothing provides libgcc >= 8.5.0-10.el8 needed by gcc-8.5.0-10.el8.x86_64
 2022-02-02 03:16:45.060 | > - nothing provides libgomp = 8.5.0-10.el8 needed by gcc-8.5.0-10.el8.x86_64