centos9 fs1/fs2 ovb failing install_packages "package buildah filtered out by modular filtering"

[Marios Andreou]

at [1][2] the centos9 fs1 and fs2 ovb jobs are failing during install_packages.sh and when trying to install python-tripleoclient.
Trace looks like:

2021-11-03 08:13:51 | Error:
2021-11-03 08:13:51 | Problem: package python3-tripleoclient-17.1.1-0.20211101131424.ce6a555.el9.noarch requires buildah, but none of the providers can be installed
2021-11-03 08:13:51 | - cannot install the best candidate for the job
2021-11-03 08:13:51 | - package buildah-1.24.0-0.15.el9.x86_64 is filtered out by modular filtering
2021-11-03 08:13:51 | (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

The buildah package in the appstream repo [3] is being filtered out.

For now trying to disable the filtering with module_hotfixes: true at [4] (testing at [5])

[1] https://logserver.rdoproject.org/67/36267/22/check/tripleo-stream9-development-centos-9-ovb-3ctlr_1comp-featureset001-master/dd05362/logs/undercloud/home/zuul-worker/install_packages.sh.log.txt.gz
[2] https://logserver.rdoproject.org/67/36267/22/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/e2db8c9/logs/undercloud/home/zuul-worker/install_packages.sh.log.txt.gz
[3] https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/AppStream/x86_64/os/Packages/
[4] https://review.opendev.org/c/openstack/tripleo-quickstart/+/816313/3/config/release/tripleo-ci/CentOS-9/promotion-testing-hash-master.yml#137
[5] https://review.rdoproject.org/r/c/testproject/+/36267/22#message-6d93fa175e229a811b46ff54e3ed214761140e9e

[Marios Andreou (marios-b)]

confirmed that setting module_hotfixes: true with https://review.opendev.org/c/openstack/tripleo-quickstart/+/816313/3/config/release/tripleo-ci/CentOS-9/promotion-testing-hash-master.yml#137

worked and allowed the installation to proceed

https://logserver.rdoproject.org/67/36267/15/check/tripleo-stream9-development-centos-9-ovb-3ctlr_1comp-featureset001-master/e73d2f2/logs/undercloud/home/zuul-worker/install_packages.sh.log.txt.gz

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/816648

[Sandeep Yadav (sandeepyadav93)]

This issue is also affecting standalone jobs:-

https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-baremetal-master/2d1c5e3/logs/undercloud/home/zuul-worker/install_packages.sh.log.txt.gz

~~~
2021-11-07 19:23:53 | Last metadata expiration check: 0:01:53 ago on Sun 07 Nov 2021 07:22:00 PM EST.
2021-11-07 19:23:54 | Error:
2021-11-07 19:23:54 | Problem: package python3-tripleoclient-17.1.1-0.20211101131424.ce6a555.el9.noarch requires buildah, but none of the providers can be installed
2021-11-07 19:23:54 | - cannot install the best candidate for the job
2021-11-07 19:23:54 | - package buildah-1.24.0-0.15.el9.x86_64 is filtered out by modular filtering
2021-11-07 19:23:54 | (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
~~~

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/817161

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/816648
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/134ab2410a159548471a4d97838ca4b6d72371ec
Submitter: "Zuul (22348)"
Branch: master

commit 134ab2410a159548471a4d97838ca4b6d72371ec
Author: Marios Andreou <email address hidden>
Date: Thu Nov 4 13:33:53 2021 +0200

    Set module_hotfixes to true for centos9 appstream repos

    To workaround issue described in the related-bug we have
    to disable module filtering for now

    Note: We will switch to CentOS Stream mirror contents
    once https://review.opendev.org/c/opendev/system-config/+/817136
    gets merged and synced on OpenDev mirrors.

    Related-Bug: 1949601
    Change-Id: Ie24243f7c20c9b274bae0972721496d33a8ca46c

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ronelle Landy <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/817161

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/817161
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/162299297201776484b2700f499dd34e7706701f
Submitter: "Zuul (22348)"
Branch: master

commit 162299297201776484b2700f499dd34e7706701f
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Tue Nov 9 13:47:44 2021 +0530

    Use centos mirror content instead of compose

    Currently CentOS stream contents are pre-released on
    http://mirror.stream.centos.org/9-stream.

    In TripleO Deployment, we are using compose to lay down the repos.
    But on the base system, It has contents from CentOS stream mirror
    where module metadata is coming and but those contents are not
    available in Compose.

    Due to mixing of compose and mirror content, packages
    which are not available in compose are getting filtered out.

    In order to avoid this issue, we are switching to
    CentOS Stream mirror content. Patch at [1] adds c9 stream mirror
    for opendev system-config repo.

    Also removes the use of lookup plugin as it caused problems
    for zuul native execution see [2]. Instead ansible_env can
    give us the same value.
    Also carries workaround for opendev centos9 stream repos
    until [3] or similar fix merges.

    [1] https://review.opendev.org/c/opendev/system-config/+/817136
    [2] https://logserver.rdoproject.org/17/36817/4/check/tripleo-stream9-development-buildimage-overcloud-hardened-uefi-full-master/31ebe5e/job-output.txt
    [3] https://review.opendev.org/c/opendev/base-jobs/+/820018

    Closes-Bug: 1949601

    Co-Authored-By: Marios Andreou (marios) <email address hidden>
    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: I480dbb83d4721a30c0d96913df3e2a9d57ea4f75

[Marios Andreou (marios-b)]

moving this back to in progress so we can see if we can remove the workaround from https://review.opendev.org/c/openstack/tripleo-quickstart/+/816648.

we are now using the 'proper' repos not compose with https://review.opendev.org/c/openstack/tripleo-quickstart/+/817161 (and https://review.opendev.org/c/openstack/tripleo-ci/+/818222 for images/containers) so we should be able to remove the hotfixes workaround but lets confirm

I posted a revert for the module_hotfixes @ https://review.opendev.org/c/openstack/tripleo-quickstart/+/820494

testing with https://review.rdoproject.org/r/c/testproject/+/36267/36#message-09a1b99b0380df50c6ac905d66c1eeed194d73e4

[Marios Andreou (marios-b)]

OK moving the bug to fix-released... the latest run @ https://review.rdoproject.org/r/c/testproject/+/36267/38#message-9ad78ec2c3f0c37f922184666dc774eb98db203a fails for FS1 but it does get far enough for us to verify this and close it.

There is now no more module_hotfixes: True @ appstreams repo (with [1] which we should merge )

the buildah package is now installed OK see [2] which was the failing step per the description

        * 2021-12-07 12:38:22 | buildah x86_64 1.24.0-0.15.el9 quickstart-centos-appstreams 7.7 M

No more module_hotfixes=True in [3][4] (eg. compared to [5] which still has the workaround as a comparison)

[1] https://review.opendev.org/c/openstack/tripleo-quickstart/+/820494

[2] https://logserver.rdoproject.org/67/36267/37/check/tripleo-stream9-development-centos-9-ovb-3ctlr_1comp-featureset001-master/299f3d1/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz

[3] https://logserver.rdoproject.org/67/36267/37/check/periodic-tripleo-ci-build-containers-centos-9-push-master/cdcd5d6/etc/yum.repos.d/quickstart-centos-appstreams.repo

[4] https://logserver.rdoproject.org/67/36267/37/check/tripleo-stream9-development-centos-9-ovb-3ctlr_1comp-featureset001-master/299f3d1/logs/undercloud/etc/yum.repos.d/quickstart-centos-appstreams.repo.txt.gz

[5] https://logserver.rdoproject.org/17/36817/13/check/periodic-tripleo-ci-build-containers-centos-9-push-master/7298bb0/etc/yum.repos.d/quickstart-centos-appstreams.repo

[Marios Andreou (marios-b)]

moving back to in progress until we merge https://review.opendev.org/c/openstack/tripleo-quickstart/+/820494 too

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/820494
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/46b5014ee2922b2632f6e897c3dcfd735f4c0890
Submitter: "Zuul (22348)"
Branch: master

commit 46b5014ee2922b2632f6e897c3dcfd735f4c0890
Author: Marios Andreou <email address hidden>
Date: Mon Dec 6 09:09:21 2021 +0000

    Revert "Set module_hotfixes to true for centos9 appstream repos"

    This reverts commit 134ab2410a159548471a4d97838ca4b6d72371ec.

    Reason for revert: We are now using centos9 stream instead of stream composes so this should no longer be a thing.

    Related-Bug: 1949601
    Change-Id: I6191cbf2b102f9cc9d15fc52ccce726fc06e7dea