tripleo-ci-centos-9-undercloud-upgrade is failing check/gate with "cannot install both NetworkManager-1:1.39.2-1.el9.x86_64 and NetworkManager-1:1.39.5-1.el9.x86_64"

[Ronelle Landy]

tripleo-ci-centos-9-undercloud-upgrade is failing check/gate on master jobs with the following error:

https://6e4bbc8ab1d82885b9df-970b30994ec26ff19befc99de6943775.ssl.cf5.rackcdn.com/843161/1/check/tripleo-ci-centos-9-undercloud-upgrade/25332e6/logs/undercloud/home/zuul/undercloud_upgrade.log

2022-05-24 17:27:48 | 2022-05-24 17:27:48.838272 | bc764e20-1929-3098-17aa-00000000053c | FATAL | Update all packages | undercloud | error={"changed": false, "failures": [], "msg": "Depsolve Error occurred: \n Problem: package os-net-config-15.2.1-0.20220520152213.49bdd97.el9.noarch requires NetworkManager-ovs, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.36.0-2.el9.x86_64 requires NetworkManager(x86-64) = 1:1.36.0-2.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.37.2-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.37.2-1.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.0-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.0-1.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.2-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.2-1.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.3-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.3-1.el9, but none of the providers can be installed\n - cannot install both NetworkManager-1:1.36.0-2.el9.x86_64 and NetworkManager-1:1.39.5-1.el9.x86_64\n - cannot install both NetworkManager-1:1.37.2-1.el9.x86_64 and NetworkManager-1:1.39.5-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.0-1.el9.x86_64 and NetworkManager-1:1.39.5-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.2-1.el9.x86_64 and NetworkManager-1:1.39.5-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.3-1.el9.x86_64 and NetworkManager-1:1.39.5-1.el9.x86_64\n - cannot install the best update candidate for package os-net-config-14.2.1-0.20220419122345.19307e0.el9.noarch\n - problem with installed package NetworkManager-1:1.39.5-1.el9.x86_64", "rc": 1, "results": []}

The failure started on 05/24:

https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade&skip=0

[Ronelle Landy (rlandy)]

Passing job log shows:

https://db636a808fe8ed51f6c0-c97a7f6b7423ddbb4161c783b6bd4b21.ssl.cf1.rackcdn.com/820668/3/check/tripleo-ci-centos-9-undercloud-upgrade/28dda14/logs/undercloud/var/log/extra/package-list-installed.txt

NetworkManager.x86_64 1:1.39.3-1.el9 @baseos
NetworkManager-libnm.x86_64 1:1.39.3-1.el9 @baseos
NetworkManager-ovs.x86_64 1:1.39.3-1.el9 @quickstart-centos-appstreams

os-collect-config.noarch 13.1.0-0.20211215160701.3a7d05a.el9 @delorean-current
os-net-config.noarch 15.2.1-0.20220520152213.49bdd97.el9 @delorean-current

[Ronelle Landy (rlandy)]

Failing job logs shows:

https://6e4bbc8ab1d82885b9df-970b30994ec26ff19befc99de6943775.ssl.cf5.rackcdn.com/843161/1/check/tripleo-ci-centos-9-undercloud-upgrade/25332e6/logs/undercloud/var/log/extra/package-list-installed.txt

NetworkManager.x86_64 1:1.39.5-1.el9 @baseos
NetworkManager-libnm.x86_64 1:1.39.5-1.el9 @baseos

no NetworkManager-ovs

os-collect-config.noarch 13.1.0-0.20211215160701.3a7d05a.el9 @delorean-current
os-net-config.noarch 14.2.1-0.20220419122345.19307e0.el9 @delorean-current

[Ronelle Landy (rlandy)]

Only recent change in os-net-config:

https://opendev.org/openstack/os-net-config/commit/49bdd97a73cb580ac8371cca955d20137e397657

[Ronelle Landy (rlandy)]

Possible mirror sync issue:

http://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/Packages/

[ ] NetworkManager-ovs-1.37.2-1.el9.x86_64.rpm 2022-03-10 13:16 51K
[ ] NetworkManager-ovs-1.39.0-1.el9.x86_64.rpm 2022-04-08 15:16 52K
[ ] NetworkManager-ovs-1.39.2-1.el9.x86_64.rpm 2022-04-22 11:46 52K
[ ] NetworkManager-ovs-1.39.3-1.el9.x86_64.rpm 2022-05-05 13:21 53K
[ ] NetworkManager-ovs-1.39.5-1.el9.x86_64.rpm 2022-05-22 19:16 53K

[Sandeep Yadav (sandeepyadav93)]

Passing now:-

https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade&skip=0

Keeping under observation.

[Sandeep Yadav (sandeepyadav93)]

We are not seeing this issue anymore, seems it solved automatically after mirror sync.

Closing this bug.