tripleo-ci-centos-9-undercloud-upgrade - cannot install both NetworkManager-1:1.39.5-1.el9.x86_64 and NetworkManager-1:1.39.12

[Marios Andreou]

At [1][2][3] the tripleo-ci-centos-9-undercloud-upgrade fails during the undercloud upgrade and specifically package conflict when upgrading packages. Trace looks like:

        * 2022-08-10 10:02:29 | 2022-08-10 10:02:29.642254 | bc764e04-48fc-91a2-43a1-000000000550 | FATAL | Update all packages | undercloud | error={"changed": false, "failures": [], "msg": "Depsolve Error occurred: \n Problem: package os-net-config-15.2.1-0.20220629114404.6505f24.el9.noarch requires NetworkManager-ovs, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.10-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.10-1.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.3-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.3-1.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.5-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.5-1.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.6-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.6-1.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.7-2.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.7-2.el9, but none of the providers can be installed\n - cannot install both NetworkManager-1:1.39.10-1.el9.x86_64 and NetworkManager-1:1.39.12-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.3-1.el9.x86_64 and NetworkManager-1:1.39.12-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.5-1.el9.x86_64 and NetworkManager-1:1.39.12-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.6-1.el9.x86_64 and NetworkManager-1:1.39.12-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.7-2.el9.x86_64 and NetworkManager-1:1.39.12-1.el9.x86_64\n - cannot install the best update candidate for package os-net-config-14.2.1-0.20220624004921.755f151.el9.noarch\n - problem with installed package NetworkManager-1:1.39.12-1.el9.x86_64", "rc": 1, "results": []}

This blocks the master gate across most tripleo repos - I can see tripleo-heat-templates, tripleo-common, tripleo-ansible, tripleo-quickstart, python-tripleoclient in builds at [4]. Attaching a screenshot that shows the failed builds started 9th august.

[1] https://6e96b467b536ead9d981-83b5a515fd4f8ade122b2cc60973de6d.ssl.cf2.rackcdn.com/848640/2/check/tripleo-ci-centos-9-undercloud-upgrade/2a07467/logs/undercloud/home/zuul/undercloud_upgrade.log
[2] https://1eed5b8dc2d4d2c1a513-67e15cc69bd7500d8d48cf199d18d0c8.ssl.cf2.rackcdn.com/851958/4/gate/tripleo-ci-centos-9-undercloud-upgrade/8093ae0/logs/undercloud/home/zuul/undercloud_upgrade.log
[3] https://8cc16a860ef78508d8bf-63e0619ac9734be8bebe39a119169a2b.ssl.cf5.rackcdn.com/836979/2/gate/tripleo-ci-centos-9-undercloud-upgrade/fbfa910/logs/undercloud/home/zuul/undercloud_upgrade.log
[4] https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade&skip=0

[Marios Andreou (marios-b)]

noting that a similar bug was filed a few months ago there https://bugs.launchpad.net/tripleo/+bug/1975623 root cause was mirror/sync issue

[Marios Andreou (marios-b)]

at http://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/Packages/ I see

[ ] NetworkManager-ovs-1.39.5-1.el9.x86_64.rpm 2022-05-22 19:16 53K
[ ] NetworkManager-ovs-1.39.6-1.el9.x86_64.rpm 2022-06-07 07:46 53K
[ ] NetworkManager-ovs-1.39.7-2.el9.x86_64.rpm 2022-06-29 10:16 53K
[ ] NetworkManager-ovs-1.39.10-1.el9.x86_64.rpm 2022-07-19 09:16 54K
[ ] NetworkManager-ovs-1.39.12-1.el9.x86_64.rpm 2022-08-03 10:57 54K

So the 1.39.12-1 was updated over a week ago why is it complaining about it now?

[Marios Andreou (marios-b)]

        * http://mirror.stream.centos.org/9-stream/COMPOSE_ID

        * CentOS-Stream-9-20220808.1

That was 08th. We started seeing this on the 9th, might be related if this is mirror sync issue again

[Marios Andreou (marios-b)]

On the failing jobs the nodes have this networkmanager installed:

NetworkManager.x86_64 1:1.39.12-1.el9 @baseos
NetworkManager-libnm.x86_64 1:1.39.12-1.el9 @baseos

https://6e96b467b536ead9d981-83b5a515fd4f8ade122b2cc60973de6d.ssl.cf2.rackcdn.com/848640/2/check/tripleo-ci-centos-9-undercloud-upgrade/2a07467/logs/undercloud/var/log/extra/package-list-installed.txt

[Marios Andreou (marios-b)]

sent a note to list about rechecks @ https://lists.openstack.org/pipermail/openstack-discuss/2022-August/029942.html [tripleo] gate blocker tripleo-ci-centos-9-undercloud-upgrade

[Alfredo Moralejo (amoralej)]

Looks like AFS is out of sync.

Standard CentOS repo:

[root@77b891bf9fd8 ~]# dnf repoquery -q NetworkManager-ovs
NetworkManager-ovs-1:1.39.10-1.el9.x86_64
NetworkManager-ovs-1:1.39.12-1.el9.x86_64
NetworkManager-ovs-1:1.39.5-1.el9.x86_64
NetworkManager-ovs-1:1.39.6-1.el9.x86_64
NetworkManager-ovs-1:1.39.7-2.el9.x8

Checking AFS:

# dnf repoquery --repofrompath=afs,http://mirror.regionone.vexxhost-nodepool-sf.rdoproject.org/centos-stream/9-stream/AppStream/x86_64/os/ --disablerepo="*" --enablerepo=afs -q NetworkManager-ovs

NetworkManager-ovs-1:1.39.10-1.el9.x86_64
NetworkManager-ovs-1:1.39.3-1.el9.x86_64
NetworkManager-ovs-1:1.39.5-1.el9.x86_64
NetworkManager-ovs-1:1.39.6-1.el9.x86_64
NetworkManager-ovs-1:1.39.7-2.el9.x86_64

[Jeremy Stanley (fungi)]

  wget -qO- http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/ | grep NetworkManager-ovs | sed 's/.*href="\([^"]*\)".*/\1/'

  NetworkManager-ovs-1.39.3-1.el9.x86_64.rpm
  NetworkManager-ovs-1.39.5-1.el9.x86_64.rpm
  NetworkManager-ovs-1.39.6-1.el9.x86_64.rpm
  NetworkManager-ovs-1.39.7-2.el9.x86_64.rpm
  NetworkManager-ovs-1.39.10-1.el9.x86_64.rpm

[Marios Andreou (marios-b)]

adding for context as it isn't mentioned here only on the mailing list

amoralej posted https://review.opendev.org/c/opendev/system-config/+/852793 Use rackspace mirror to sync centos stream repos

the issue is ongoing at time of writing and the mirror has not yet synced.

https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade&skip=0

if we merge 852793 it should unblock us

[Marios Andreou (marios-b)]

another similar bug there different packages but same repo and root cause https://bugs.launchpad.net/bugs/1984453

[chandan kumar (chkumar246)]

CentOS Stream 9 build infra was down on Aug 11 (https://lists.centos.org/pipermail/centos-devel/2022-August/120524.html). It might have impacted the facebook mirror, leading to out of sync for AFS mirror.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/852880

[Marios Andreou (marios-b)]

posted https://review.opendev.org/c/openstack/tripleo-ci/+/852880 to unblock gates if needed
even though i don't want to do that as we risk breaking that job or letting something through that should not be...

editing to add that i proposed the non-voting as this issue is blocking gates for 24 hours now

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ronelle Landy <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/852880

[Sandeep Yadav (sandeepyadav93)]

As per [1], Issue on Centos stream 9 build infra is solved

[1] https://lists.centos.org/pipermail/centos-devel/2022-August/120525.html

I see NetworkManager-ovs-1.39.12-1.el9.x86_64.rpm on FB mirror, rechecked the job to confirm if the issue is resolved.

~~~
$ wget -qO- http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/ | grep NetworkManager-ovs | sed 's/.*href="\([^"]*\)".*/\1/'

NetworkManager-ovs-1.39.5-1.el9.x86_64.rpm
NetworkManager-ovs-1.39.6-1.el9.x86_64.rpm
NetworkManager-ovs-1.39.7-2.el9.x86_64.rpm
NetworkManager-ovs-1.39.10-1.el9.x86_64.rpm
NetworkManager-ovs-1.39.12-1.el9.x86_64.rpm
~~~

[chandan kumar (chkumar246)]

Thank you Sandeep for checking it, the job https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade&skip=0

https://zuul.openstack.org/build/a8d0b2716fbb45aeba63a8dd0a797584
tripleo-ci-centos-9-undercloud-upgrade SUCCESS is back to green now. I think we can mark this bug to resolved.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Marios Andreou <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/852880
Reason: luckily no longer needed - the bug is resolved/mirrors are OK again

[Marios Andreou (marios-b)]

the change at https://review.opendev.org/c/opendev/system-config/+/852793 merged so I think that is actually what fixed us in this case.

The times seem to match up looking at the green runs in build history (attaching screenshot here).

[EDIT]: perhaps both things... per https://lists.centos.org/pipermail/centos-devel/2022-August/120525.html root cause of facebook mirror issue was fixed. and we also switched to rax mirror which seemed to be fine throughout.