package podman-1.6.4-23.module_el8.3.0+566+4759265c.x86_64 is filtered out by modular filtering

[chandan kumar]

Below is the list of jobs which are failing with following reasons with retry_limit status
https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-hardened-full-ussuri/2d56775/job-output.txt
================
2020-12-08 04:49:45.811865 | TASK [bindep : Install distro packages from bindep]
2020-12-08 04:49:46.880029 | primary | ERROR
2020-12-08 04:49:46.880356 | primary | {
2020-12-08 04:49:46.880428 | primary | "failures": [],
2020-12-08 04:49:46.880462 | primary | "msg": "Depsolve Error occured: \n Problem: package tripleo-ansible-1.5.2-0.20201124210415.633348b.el8.noarch requires python3-paunch, but none of the providers can be installed\n - package python3-paunch-7.0.3-0.20201118200429.5dcc4df.el8.noarch requires podman = 1.6.4, but none of the providers can be installed\n - conflicting requests\n - package podman-1.6.4-23.module_el8.3.0+566+4759265c.x86_64 is filtered out by modular filtering",
2020-12-08 04:49:46.880498 | primary | "rc": 1,
2020-12-08 04:49:46.880527 | primary | "results": []
2020-12-08 04:49:46.880554 | primary | }
2020-12-08 04:49:46.893681 |
2020-12-08 04:49:46.893788 | TASK [bindep : Remove bindep temp dir]
2020-12-08 04:49:46.924177 |

================

We are also seeing the same issue in ussuri content provider.
it is blocking gate jobs.
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_520/762124/2/gate/tripleo-ci-centos-8-content-provider-ussuri/520e2d8/logs/undercloud/home/zuul/container-builds/3e05429e-566a-4fea-9f55-7b804ebdab85/base/os/mistral-base/mistral-executor/mistral-executor-build.log

===

Error:
 Problem: package openstack-tripleo-heat-templates-12.4.2-0.20201207051843.300283c.el8.noarch requires python3-paunch, but none of the providers can be installed
  - package python3-paunch-7.0.3-0.20201118200429.5dcc4df.el8.noarch requires podman = 1.6.4, but none of the providers can be installed
  - cannot install the best candidate for the job
  - package podman-1.6.4-23.module_el8.3.0+566+4759265c.x86_64 is filtered out by modular filtering
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
===

List of impacted jobs:
https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-hardened-full-ussuri/2d56775/job-output.txt

https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-ironic-python-agent-ussuri/88a21ff/job-output.txt

https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-ussuri/ff3508d/job-output.txt

https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push-ussuri/7106c1c/job-output.txt

https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push-train/38f7746/job-output.txt

https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-ironic-python-agent-train/e46186d/job-output.txt

It is failing at this point https://opendev.org/openstack/tripleo-ci/src/branch/master/playbooks/tripleo-buildimages/pre.yaml#L28 which pulls tripleo-ansible from here https://opendev.org/openstack/python-tripleoclient/src/branch/master/bindep.txt#L12

It seems to be a tripleo-ansible packaging issue.

[chandan kumar (chkumar246)]

https://review.rdoproject.org/r/#/c/31307/ in pauch distgit causes the issue.

[chandan kumar (chkumar246)]

EDIT: https://review.rdoproject.org/r/#/c/31307/ added in train so it does not appear there, we need similar fix for ussuri release

[chandan kumar (chkumar246)]

https://review.rdoproject.org/r/#/c/31310/ added the fix for ussuri

[chandan kumar (chkumar246)]

https://review.rdoproject.org/r/#/c/31309/ fixes the issue, abandoning my patch

[wes hayutin (weshayutin)]

thanks all! well done

[Marios Andreou (marios-b)]

so this is now hitting the ussuri periodic integration pipeline eg at [1]

it should clear once we get the fix through the component pipeline (tripleo ussuri component [2])

        * 2020-12-09 04:55:03.729400 | primary | "msg": "Depsolve Error occured: \n Problem: package tripleo-ansible-1.5.2-0.20201124210415.633348b.el8.noarch requires python3-paunch, but none of the providers can be installed\n - package python3-paunch-7.0.3-0.20201118200429.5dcc4df.el8.noarch requires podman = 1.6.4, but none of the providers can be installed\n - conflicting requests\n - package podman-1.6.4-23.module_el8.3.0+566+4759265c.x86_64 is filtered out by modular filtering",

[1] https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-hardened-full-ussuri/a4aacd7/job-output.txt
[2] http://dashboard-ci.tripleo.org/d/MhDLSiOGz/component-pipeline-ussuri?orgId=1&fullscreen&panelId=443

[Marios Andreou (marios-b)]

also seen on train today https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push-train/a31a094/job-output.txt

so we'll need a similar fix to https://review.rdoproject.org/r/#/c/31309/ - however, i just checked and it already has >= 1.6.4 ? https://github.com/rdo-packages/paunch-distgit/blob/8521a057b919db641ef67b153915acfbaad852c7/python-paunch.spec#L70-L75

[Alex Schultz (alex-schultz)]

That error points to it not having an rpm that has the >= 1.6.4 update. Can we check package versions/repos?

[Alex Schultz (alex-schultz)]

python3-paunch-7.0.3-0.20201118200429.5dcc4df.el8.noarch is too old since we merged the patch for the requirements this week. That's from nov 11th

[Alex Schultz (alex-schultz)]

I don't know what repo the train one is using, but it's not tripleo current which has python3-paunch from Dec 8th.

python3-paunch-7.0.3-0.20201208164406.5dcc4df.el8.noarch.rpm

[Marios Andreou (marios-b)]

still hitting this on train
https://review.opendev.org/c/openstack/tripleo-common/+/762892/
  https://zuul.opendev.org/t/openstack/build/8b0372c223754334a631d14f6be8a91e

[Alex Schultz (alex-schultz)]

need a tripleo promotion to get newer version python3-paunch

[Bhagyashri Shewale (bhagyashri-shewale)]

Hi All,

This issue is causing failure on featureset050-upgrades-master, multinode-oooq-container-updates-master and featureset037-updates-master.

[Bhagyashri Shewale (bhagyashri-shewale)]

Error logs link:

- featureset050-upgrades-master

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-singlenode-featureset050-upgrades-master/7a8585d/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz

- multinode-oooq-container-updates-master

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-master/0a6b4b6/logs/undercloud/home/zuul/overcloud_update_run-Controller.log.txt.gz

- featureset037-updates-master

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset037-updates-master/288cabe/logs/undercloud/home/zuul/overcloud_update_run-Controller.log.txt.gz

[chandan kumar (chkumar246)]

Based on the logs, it is a different issue ?

FATAL | Ensure DNF modules have the right stream | node-0000744824 | item={'module': 'container-tools', 'stream': '2.0'} | error={"ansible_loop_var": "item", "changed": false, "failures": [], "item": {"module": "container-tools", "stream": "2.0"}, "msg": "Depsolve Error occured: \n Problem: cannot install both podman-1.6.4-23.module_el8.3.0+566+4759265c.x86_64 and podman-2.0.5-5.module_el8.3.0+512+b3b58dca.x86_64\n - package podman-catatonit-2.0.5-5.module_el8.3.0+512+b3b58dca.x86_64 requires podman = 2.0.5-5.module_el8.3.0+512+b3b58dca, but none of the providers can be installed\n - conflicting requests\n - problem with installed package podman-catatonit-2.0.5-5.module_el8.3.0+512+b3b58dca.x86_64\n - package podman-2.0.5-5.module_el8.3.0+512+b3b58dca.x86_64 is filtered out by modular filtering", "rc": 1, "results": []}

```
Depsolve Error occured: \n Problem: cannot install both podman-1.6.4-23.module_el8.3.0+566+4759265c.x86_64 and podman-2.0.5-5.module_el8.3.0+512+b3b58dca.x86_64\n

```

[chandan kumar (chkumar246)]

Under container-tools', 'stream': '2.0', podman-1.6.4-23.module_el8.3.0+566+4759265c.x86_64 is available, but we need higher version of podman which is not there.

[Alex Schultz (alex-schultz)]

the last error points to rhel8 podman being installed prior to the thing that tries to require 1.6.24. We need container-tools:2.0 to be configured prior to any installation of podman

[Sandeep Yadav (sandeepyadav93)]

Hello,

As per discussion with Chandan and Bhagyashris over irc chat, New issue reported in #comment 14 [1] looks different.

We have filed[2] for tracking this new issue

[1] https://bugs.launchpad.net/tripleo/+bug/1907193/comments/14
[2] https://bugs.launchpad.net/tripleo/+bug/1909445