Centos-8 wallaby buildimage jobs fail dependency issue for centos-gpg-keys-1:8-5.el8.noarch

[Marios Andreou]

At [1][2][3] the centos8 wallaby buildimage jobs are failing to build with dependency issue that looks like:

 2022-03-28 21:48:50.405 | Error:
 2022-03-28 21:48:50.405 | Problem: package centos-stream-repos-8-4.el8.noarch requires centos-gpg-keys = 1:8-4.el8, but none of the providers can be installed
 2022-03-28 21:48:50.405 | - cannot install both centos-gpg-keys-1:8-5.el8.noarch and centos-gpg-keys-1:8-4.el8.noarch
 2022-03-28 21:48:50.405 | - cannot install both centos-gpg-keys-1:8-4.el8.noarch and centos-gpg-keys-1:8-5.el8.noarch
 2022-03-28 21:48:50.405 | - cannot install the best update candidate for package centos-stream-repos-8-2.el8.noarch
 2022-03-28 21:48:50.405 | - cannot install the best update candidate for package centos-gpg-keys-1:8-2.el8.noarch

Wallaby promotion blocker :/

[1] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-wallaby/29f2d7a/build.log
[2] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-hardened-uefi-full-wallaby/9c28d67/build.log
[3] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-ironic-python-agent-wallaby/29190f5/job-output.txt

[Marios Andreou (marios-b)]

also discussed there https://bugzilla.redhat.com/show_bug.cgi?id=2069547

[Alan Pevec (apevec)]

and being resolved in https://pagure.io/centos-infra/issue/724

offending centos-gpg-keys-8-5 was removed from mirror.centos.org but CS8 mirror used as opendev AFS source still has it http://mirror.facebook.net/centos/8-stream/extras/x86_64/os/Packages/

If mirror sync takes much longer, CI could temp.revert to baseurl=mirror.centos.org for CS8 Extras repo to avoid centos-gpg-keys-8-5

[Marios Andreou (marios-b)]

thanks Alan

we have started seeing it in the gates too now

example there https://zuul.opendev.org/t/openstack/build/2fc70ccd10004a8fa2a0a21382b57872

  - cannot install the best update candidate for package centos-stream-repos-8-2.el8.noarch

if it doesnt clear in the next few hours we will prepare pin for baseurl mirror

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/835645

[Marios Andreou (marios-b)]

also seeing on train gate there https://b0d0b874198de991f976-e66afa9c5b07ebbf36793ab7ddc56f1f.ssl.cf2.rackcdn.com/834285/3/check/tripleo-ci-centos-8-content-provider/fc12a2f/logs/undercloud/var/log/dnf.log

we can try to avoid the extras mirror as suggested by alan - review there https://review.opendev.org/c/openstack/tripleo-quickstart/+/835645 (test there https://review.rdoproject.org/r/c/testproject/+/41119 )

as another data point - fail @ https://zuul.opendev.org/t/openstack/build/c685638851ba42d38c643d01b55679f5 has a different package conflict

2022-03-29 15:08:35.149236 | primary | Depsolve Error occured:
2022-03-29 15:08:35.149258 | primary | Problem: cannot install the best update candidate for package platform-python-devel-3.6.8-37.el8.x86_64
2022-03-29 15:08:35.149285 | primary | - nothing provides python3-libs(x86-64) = 3.6.8-46.el8 needed by platform-python-devel-3.6.8-46.el8.x86_64

assuming it is the same issue for now

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/835648

[Marios Andreou (marios-b)]

@Rlandy - we see this on stable/train gates (see comment 3 for link) and I expect we should also see it on stable/wallaby gates.

For tripleo-ci - it specifically blocks us on the tripleo-tox-molecule job. The proposed temp workaround @ https://review.opendev.org/c/openstack/tripleo-quickstart/+/835645 won't help us I don't think that job uses the release files.

We may just consider making that non voting to unblock tripleo-ci gate with https://review.opendev.org/c/openstack/tripleo-ci/+/835648 (as an aside to tripleo-quickstart/+/835645 )

[Takashi Kajinami (kajinamit)]

Alternatively you can exclude that package causing conflict.

reference: https://review.opendev.org/c/openstack/puppet-openstack-integration/+/835535

$ sudo dnf config-manager --save --setopt exclude=centos-gpg-keys-*8-5*

We are holding that patch in puppet repos hoping the fixed version becomes available soon, but we confirmed that exclude resolves the issue.

[Ronelle Landy (rlandy)]

centos-gpg-keys noarch 1:8-6.el8 quickstart-centos-base 14 k

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-hardened-uefi-full-wallaby/f8594d0/repo_setup.log

Updated centos-gpg-keys are being installed now

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ronelle Landy <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/835648

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Marios Andreou <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/835645
Reason: issue was resolved so we no longer need this https://bugs.launchpad.net/tripleo/+bug/1966914/comments/9

[Marios Andreou (marios-b)]

k done here too mirrors caught up we didn't merge the temp/workaround.

moving bug fix-released