tripleo-ci-centos-9-undercloud-upgrade failing consistently while updating all packages

[Pooja Jadhav]

tripleo-ci-centos-9-undercloud-upgrade failing consistently while updating all packages with below traceback :

2022-09-01 07:06:25 | 2022-09-01 07:06:25.280385 | fa163ee2-35df-fd17-55b8-000000000550 | TASK | Update all packages
2022-09-01 07:06:26 | 2022-09-01 07:06:26.573497 | fa163ee2-35df-fd17-55b8-000000000550 | FATAL | Update all packages | undercloud | error={"changed": false, "failures": [], "msg": "Depsolve Error occurred: \n Problem: package os-net-config-15.2.1-0.20220629114404.6505f24.el9.noarch requires NetworkManager-ovs, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.10-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.10-1.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.12-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.12-1.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.6-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.6-1.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.7-2.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.7-2.el9, but none of the providers can be installed\n - package NetworkManager-ovs-1:1.39.90-1.el9.x86_64 requires NetworkManager(x86-64) = 1:1.39.90-1.el9, but none of the providers can be installed\n - cannot install both NetworkManager-1:1.39.10-1.el9.x86_64 and NetworkManager-1:1.40.0-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.12-1.el9.x86_64 and NetworkManager-1:1.40.0-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.6-1.el9.x86_64 and NetworkManager-1:1.40.0-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.7-2.el9.x86_64 and NetworkManager-1:1.40.0-1.el9.x86_64\n - cannot install both NetworkManager-1:1.39.90-1.el9.x86_64 and NetworkManager-1:1.40.0-1.el9.x86_64\n - cannot install the best update candidate for package os-net-config-14.2.1-0.20220624004921.755f151.el9.noarch\n - problem with installed package NetworkManager-1:1.40.0-1.el9.x86_64", "rc": 1, "results": []}
2022-09-01 07:06:26 | 2022-09-01 07:06:26.575110 | fa163ee2-35df-fd17-55b8-000000000550 | TIMING | Update all packages | undercloud | 0:01:08.655448 | 1.29s

Detailed Logs :

https://32f730b73845cbdfd0fc-4cf4bb096d4774f141f32d30757edf40.ssl.cf1.rackcdn.com/853411/2/check/tripleo-ci-centos-9-undercloud-upgrade/b6a6d75/logs/undercloud/home/zuul/undercloud_upgrade.log

https://ef8e3d7fa760b4ab7420-4082203e55009ffc8095c87eb70265ba.ssl.cf5.rackcdn.com/854602/1/gate/tripleo-ci-centos-9-undercloud-upgrade/dc4a2fd/logs/undercloud/home/zuul/undercloud_upgrade.log

[Jakob Meng (jm1337)]

      <ysandeep> we still have mirror issues:
      <ysandeep> job have .40 version installed
      <ysandeep> https://32f730b73845cbdfd0fc-4cf4bb096d4774f141f32d30757edf40.ssl.cf1.rackcdn.com/853411/2/check/tripleo-ci-centos-9-undercloud-upgrade/b6a6d75/logs/undercloud/var/log/extra/package-list-installed.txt
      <ysandeep> ~~~
      <ysandeep> NetworkManager.x86_64 1:1.40.0-1.el9 @baseos
      <ysandeep> NetworkManager-libnm.x86_64 1:1.40.0-1.el9 @baseos
      <ysandeep> ~~~
      <ysandeep> I don't see .40 rpms for NetworkManager on rackspace mirror: http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/
      <ykarel> facebook mirror looks good
      <ykarel> rackspace one not updated since 10 hours
      <ykarel> http://dfw.mirror.rackspace.com/centos-stream/timestamp.txt
      <jm1> ysandeep, ykarel: and neither on http://mirror.mtl01.iweb.opendev.org/centos-stream/9-stream/AppStream/x86_64/os/Packages/
      <ykarel> yes ^ syncs from mirror which is rax currently so should be same
      <ykarel> http://mirror.mtl01.iweb.opendev.org/centos-stream/timestamp.txt
      <ykarel> 2022-09-01T06:08:45,769043757+00:00

[yatin (yatinkarel)]

Pushed https://review.opendev.org/c/opendev/system-config/+/855457 to switch back to facebook mirror as that's up to date currently.

Since CI jobs use infra mirrors, it makes sense nodepool images also use infra mirrors to avoid these issues issues in future.

While discussing it on #opendev[1] it's found that CentOS images not using infra mirrors like some non centos image using(via DIB_DISTRIBUTION_MIRROR)[1], so need to make CentOS images such that infra mirror repos get's used, currently centos repos are getting used[3]

[1] https://meetings.opendev.org/irclogs/%23opendev/%23opendev.2022-09-01.log.html#t2022-09-01T12:09:25
[2] https://opendev.org/openstack/project-config/src/branch/master/nodepool/nodepool.yaml#L225
[3] https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/centos-minimal/yum.repos.d/9-stream