ovb featureset 64 (ipa) wallaby fails downloading packages 'all mirrors were tried' ansible-tripleo-ipa

[Marios Andreou]

At [1][2] the periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby fails during supplemental node setup while downloading ansible-tripleo-ipa

 2022-03-10 00:53:58 | + sudo dnf install -yq ansible-core ansible-tripleo-ipa
 2022-03-10 00:54:02 | Error: Error downloading packages:
 2022-03-10 00:54:02 | ansible-tripleo-ipa-0.2.3-0.20220301014811.6b0ed82.el9.noarch: Cannot download, all mirrors were already tried without success
 2022-03-10 00:54:02 |

This is a wallaby promotion blocker.

[1] https://logserver.rdoproject.org/69/40169/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/5fa35c3/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz
[2] https://logserver.rdoproject.org/48/48dcad8c810111579a41db528c5e490dd2dcc559/openstack-periodic-integration-stable1-cs9/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/a12777d/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz

[Marios Andreou (marios-b)]

I think this is something to do with the supplemental node *not* using the mirror but instead trying trunk.rdoproject.org directly..

-----------
on *undercloud*:

        * https://logserver.rdoproject.org/69/40169/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/5fa35c3/logs/undercloud/etc/yum.repos.d/delorean.repo.txt.gz
        * baseurl=http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org:8080/rdo/centos9-wallaby/component/security/6b/0e/6b0ed8286154c72971f9f1232d571bb1425500d2_cb5e4589
        * ansible-tripleo-ipa-0.2.3-0.20220301014811.6b0ed82.el9.noarch.rpm

-----------
on *supplemental*:
        * https://logserver.rdoproject.org/69/40169/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/5fa35c3/logs/supplemental/etc/yum.repos.d/delorean.repo.txt.gz
        * baseurl=https://trunk.rdoproject.org/centos9-wallaby/component/security/6b/0e/6b0ed8286154c72971f9f1232d571bb1425500d2_cb5e4589
        * ansible-tripleo-ipa-0.2.3-0.20220301014811.6b0ed82.el9.noarch.rpm

Same package version available in both cases but the supplemental is failing to download it from trunk.rdoproject.org

[Ronelle Landy (rlandy)]

This is also an issue on master:

https://logserver.rdoproject.org/60/39960/7/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/fd4a310/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz

[Marios Andreou (marios-b)]

trying to create /etc/mirror_info.sh for the supplemental node with https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/833027 & test https://review.rdoproject.org/r/c/testproject/+/40286

[chandan kumar (chkumar246)]

last run https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby and https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master on 14th Mar was successful. Will keep an eye on these jobs.

[Marios Andreou (marios-b)]

@Chandan - per comment #4 - the issue is not seen consistently. The problem is as described in comment #1 - the supplemental node was not using the RDO mirror.

I was testing with https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/833027/13/playbooks/baremetal-full-freeipa.yml#57 and it should address the issue - supplemental node is picking up the mirror config from the undercloud before repo-setup.

[Marios Andreou (marios-b)]

related-fix

https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/833027 Add mirror_info.sh creation for for supplemental node

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/833027
Committed: https://opendev.org/openstack/tripleo-quickstart-extras/commit/56f0af5b7ec30eb0b6fe68cb56b39d5de0f799aa
Submitter: "Zuul (22348)"
Branch: master

commit 56f0af5b7ec30eb0b6fe68cb56b39d5de0f799aa
Author: Marios Andreou <email address hidden>
Date: Thu Mar 10 12:46:13 2022 +0200

    Add mirror_info.sh creation for supplemental node repo setup

    As discussed in related-bug the supplemental node used in the IPA
    jobs is not using the mirror_info.sh so not using mirrors in the
    repos created by repo-setup. This adds a slurp to grab the mirrors
    config from the undercloud before repo-setup on supplemental.
    Test and results at [1].

    [1] https://review.rdoproject.org/r/c/testproject/+/40286/1#message-f825e6369163ad1f5aa98eb7c6c67b2798d5788b
    Related-Bug: 1964457
    Change-Id: I7b3c6f0a3b9fa7114670656df7b39cef8047fd85