tripleo-ci-centos-8-content-provider is failing DLRN build with "Error: Failed to download metadata for repo 'extras-common': Cannot prepare internal mirrorlist: No URLs in mirrorlist"

[Ronelle Landy]

tripleo-ci-centos-8-content-provider is failing DLRN build with "Error: Failed to download metadata for repo 'extras-common': Cannot prepare internal mirrorlist: No URLs in mirrorlist"

CentOS Stream 8 - Extras common packages 189 B/s | 38 B 00:00
Error: Failed to download metadata for repo 'extras-common': Cannot prepare internal mirrorlist: No URLs in mirrorlist

+ finalize
+ rm -rf /tmp/tmp.4twrhLaILf

Example logs:

https://42a6936e390c46a01522-80f03e44f8bcbed0ce6b7d82324742a0.ssl.cf5.rackcdn.com/835762/1/check/tripleo-ci-centos-8-content-provider/54baa4d/logs/undercloud/home/zuul/DLRN/data/repos/component/tripleo/37/2e/372ee52c63857df49e64df301544a85c44c55c6f_dev/rpmbuild.log

https://c10e3b99137a410a82c8-4b6e23057e269cbb9e47c65b7470b96b.ssl.cf5.rackcdn.com/834940/2/check/tripleo-ci-centos-8-content-provider/838125d/logs/undercloud/home/zuul/DLRN/data/repos/component/tripleo/8e/6a/8e6a8ea4086705b53f33a121b8d6bdcb376003b9_dev/rpmbuild.log

The failure started on 03/29:
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-content-provider&skip=0

The extras-common repo is in:

https://c10e3b99137a410a82c8-4b6e23057e269cbb9e47c65b7470b96b.ssl.cf5.rackcdn.com/834940/2/check/tripleo-ci-centos-8-content-provider/838125d/logs/undercloud/etc/yum.repos.d/CentOS-Stream-Extras-common.repo

[Ronelle Landy (rlandy)]

https://c10e3b99137a410a82c8-4b6e23057e269cbb9e47c65b7470b96b.ssl.cf5.rackcdn.com/834940/2/check/tripleo-ci-centos-8-content-provider/838125d/logs/undercloud/etc/yum.repos.d/CentOS-Stream-Extras.repo.rpmnew is still enabled

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/835764

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ronelle Landy <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/835764
Reason: This does not fix it - only build test packages

[Ronelle Landy (rlandy)]

The above change will not fix the problem. The exrtras-common repo only comes in build-test-packages:

https://db9b0bc9c1f2447902ad-66179e131883a8ab832a0afb9e9b5999.ssl.cf1.rackcdn.com/835762/2/check/tripleo-ci-centos-8-content-provider/d4f2b35/logs/undercloud/home/zuul/DLRN/data/repos/component/tripleo/72/bc/72bcac9799d78d4351deda6b4d710991ba812b49_dev/rpmbuild.log

It is not on the node by default.

[Alfredo Moralejo (amoralej)]

The extras-common repo has been recently added to the centos-stream-repos and it's enabled by default. IIRC the build-test-packages role addes the official centos repos to the dlrn config. My doubt is why it's failing to get it in mirrorlist. It may be just a matter of time. In my test environment it works as expected. May we recheck some of the failed tests?

[Alfredo Moralejo (amoralej)]

Actually, here is the code which adds the centos official repos to the dlrn config:

https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/tasks/main.yml#L149-L150

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/835815

[Alfredo Moralejo (amoralej)]

This is the explanation of what is happening:

CentOS has added a new version of centos-stream-repos which adds a new repo extras-common enabled by default. Last image doesn't have that package.

Relese configs disable all the repos as in:

https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-8/master.yml#L74

But at that point, extras-common is not yet in the system.

Then, they do "dnf update -y" at the end and the new "extras-common" repo is added:

https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-8/master.yml#L228

final question is whay it fails to pull the repos from mirrorlist. That may be only a temporary issue until the repos are propagated but i'm not sure. In my environment works fine.

Note that this would be automatically fixed just by waiting until the next image when the centos-stream-repos is included in the image or when mirrorlist is fixed. Anyway, it'd be probably would to update centos-stream-repos package before doint the "dnf config-manager --disabe "*".

[Marios Andreou (marios-b)]

with the patch at https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/835815/ and in the test @ https://review.opendev.org/c/openstack/tripleo-heat-templates/+/835762/3

got a green run on the content provider there https://zuul.opendev.org/t/openstack/build/b132add59df64031bafc8d49e40a8d77

cant see any more 'extras-common' in the rpmbuild log: https://c9fbb7c04753bc93e94d-0bcd974829c5636eec53c54a828a88e5.ssl.cf2.rackcdn.com/835762/3/check/tripleo-ci-centos-8-content-provider/b132add/logs/undercloud/home/zuul/DLRN/data/repos/component/tripleo/52/f2/52f25290d7058d15885ff7a6c9cc1bdc0a4bb6d2_dev/rpmbuild.log

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/835815
Committed: https://opendev.org/openstack/tripleo-quickstart-extras/commit/b8f09e015927ada8acdb3640f4652621013ba86c
Submitter: "Zuul (22348)"
Branch: master

commit b8f09e015927ada8acdb3640f4652621013ba86c
Author: Marios Andreou <email address hidden>
Date: Wed Mar 30 13:44:28 2022 +0300

    Disable extras common for build-test-packages

    This explicitly removes the extras-common repo from being
    included by build-test-packages role for the dlrn build.
    See related-bug for more information

    Related-Bug: 1967012

    Change-Id: Iec7c3baad8ddffe692bd3317aa1753d6241b8203

[Alan Pevec (apevec)]

Can you confirm CS8 jobs work fine now?