tripleo-tox-molecule (tripleo-ci) failing with package conflicts "Transaction test error"

[Marios Andreou]

At [1][2][3] the tripleo-tox-molecule job is failing with trace like:

        2021-06-21 17:53:55.582345 | primary | TASK [tripleo-repos : Update all installed packages after new repos are setup] ***
 2021-06-21 17:53:55.582359 | primary | fatal: [stream8]: FAILED! => changed=false 
 2021-06-21 17:53:55.582374 | primary |  failures: []
 2021-06-21 17:53:55.582392 | primary |  msg: |-
 2021-06-21 17:53:55.582406 | primary |  Unknown Error occured: Transaction test error:
 2021-06-21 17:53:55.582421 | primary |  file /usr/lib/rpm/ansible-generator conflicts between attempted installs of ansible-2.9.21-1.el8.noarch and ansible-macros-2021.1.2-1.el8.noarch

This was first seen yesterday @ 2021-06-21 17:36:27 via [4]

This job is only wired up for the tripleo-ci and tripleo-upgrades repos and so far has only been seen to affect tripleo-ci.

Gate blocker for tripleo-ci

[1] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_32f/796812/2/check/tripleo-tox-molecule/32f76c0/tox/reports.html
[2] https://ec3dc25d6c69ad1d5c12-91b98dcf3667661ff385730b6a5b33e6.ssl.cf1.rackcdn.com/797360/1/check/tripleo-tox-molecule/799b219/tox/reports.html
[3] https://2838ba1bfcdbf51ffe25-b44e5239a98f7c2f735e8a48ea32ba93.ssl.cf5.rackcdn.com/796812/2/check/tripleo-tox-molecule/b2442b0/tox/reports.html
[4] https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-tox-molecule

[wes hayutin (weshayutin)]

https://zuul.openstack.org/builds?job_name=tripleo-tox-molecule&project=openstack%2Ftripleo-ci

[wes hayutin (weshayutin)]

hrm..

ansible.noarch 2.9.20-1.el8 @quickstart-centos-ceph-pacific

https://0caa56a8fe761affce01-dc55dfdaf5877a4d04198310cb0fd46a.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-8-standalone/36762b4/logs/undercloud/var/log/extra/package-list-installed.txt

regular jobs are pulling 2.9.20-1

[wes hayutin (weshayutin)]

tripleo-repos should pull it from https://trunk.rdoproject.org/centos8-master/deps/latest/noarch/ which should be ansible-2.9.19-1.el8.noarch.rpm

[wes hayutin (weshayutin)]

test invocation of tripleo-repos..

tripleo-repos -b master ceph opstools current-tripleo --stream -d centos8 -o /tmp/wes

based on:

    if [ "{{ is_stream_node|string }}" == "True" ]; then
        tripleo-repos \
          -b "{{ ci_branch | default('master') }}" \
          $MIRROR \
          --rdo-mirror $RDO_MIRROR_HOST $REPO ceph opstools \
          --stream
    else
        tripleo-repos \
          -b "{{ ci_branch | default('master') }}" \
          $MIRROR \
          --rdo-mirror $RDO_MIRROR_HOST $REPO ceph opstools \
          --no-stream

These come up and do not serve 2.9.21
[whayutin@localhost wes]$ ls
delorean-master-testing.repo tripleo-centos-highavailability.repo
delorean.repo tripleo-centos-opstools.repo
tripleo-centos-ceph-pacific.repo tripleo-centos-powertools.repo

[wes hayutin (weshayutin)]

molecule has epel enabled..

YOUR KEYBOARD IS NOW ON FIRE
https://dl.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/a/

[wes hayutin (weshayutin)]

See https://quay.io/repository/pycontribs/centos/manifest/sha256:d1c995c244438b4082a26f048a996cc7c4c0754409bb10d4becce735020a919a

https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/tripleo-repos/molecule/default

[wes hayutin (weshayutin)]

https://review.opendev.org/c/openstack/tripleo-ci/+/797528

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/797557

[wes hayutin (weshayutin)]

also playing around w/ https://review.opendev.org/c/openstack/tripleo-ci/+/797557/1/roles/tripleo-repos/molecule/default/playbook.yml

[wes hayutin (weshayutin)]

This patch works : ) https://review.opendev.org/c/openstack/tripleo-ci/+/797528

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ci/+/797528
Committed: https://opendev.org/openstack/tripleo-ci/commit/29dbd915ec68c3e93fd841eff806f118c854903f
Submitter: "Zuul (22348)"
Branch: master

commit 29dbd915ec68c3e93fd841eff806f118c854903f
Author: Sorin Sbarnea <email address hidden>
Date: Tue Jun 22 19:15:49 2021 +0100

    Remove EPEL from test images

    Closes-Bug: #1933193
    Change-Id: I4f032471a1b1ab7fbf1051a4e3d8f85870f31dcc

[OpenStack Infra (hudson-openstack)]

Change abandoned by "James Slagle <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/797557
Reason: Abandoning this patch per the TripleO Patch Abandonment guidelines
(https://specs.openstack.org/openstack/tripleo-specs/specs/policy/patch-abandonment.html).
If you wish to have this restored and cannot do so yourself, please reach out
via #tripleo on OFTC or the openstack-discuss mailing list.