glibc-2.28-181.el8.x86_64 and glibc-2.28-180.el8.x86_64 - failing tripleo-ci tripleo-tox-molecule

[Ronelle Landy]

tripleo-tox-molecule on tripleo-ci jobs are failing with:

 2022-01-12 18:26:04.394447 | primary | TASK [tripleo-repos : Update all installed packages after new repos are setup] ***
2022-01-12 18:26:04.394460 | primary | fatal: [stream8]: FAILED! => {
2022-01-12 18:26:04.394473 | primary |  "changed": false,
2022-01-12 18:26:04.394486 | primary |  "failures": [],
2022-01-12 18:26:04.394499 | primary |  "rc": 1,
2022-01-12 18:26:04.394512 | primary |  "results": []
2022-01-12 18:26:04.394525 | primary | }
2022-01-12 18:26:04.394537 | primary |
2022-01-12 18:26:04.394550 | primary | MSG:

2022-01-12 18:26:04.394575 | primary | Depsolve Error occured: 
2022-01-12 18:26:04.394588 | primary |  Problem 1: package glibc-gconv-extra-2.28-180.el8.x86_64 requires glibc(x86-64) = 2.28-180.el8, but none of the providers can be installed
2022-01-12 18:26:04.394602 | primary |  - cannot install both glibc-2.28-181.el8.x86_64 and glibc-2.28-180.el8.x86_64
2022-01-12 18:26:04.394615 | primary |  - cannot install the best update candidate for package glibc-gconv-extra-2.28-180.el8.x86_64
2022-01-12 18:26:04.394628 | primary |  - cannot install the best update candidate for package glibc-2.28-180.el8.x86_64
2022-01-12 18:26:04.394641 | primary |  Problem 2: problem with installed package glibc-gconv-extra-2.28-180.el8.x86_64
2022-01-12 18:26:04.394656 | primary |  - package glibc-gconv-extra-2.28-180.el8.x86_64 requires glibc-common = 2.28-180.el8, but none of the providers can be installed
2022-01-12 18:26:04.394670 | primary |  - cannot install both glibc-common-2.28-181.el8.x86_64 and glibc-common-2.28-180.el8.x86_64
2022-01-12 18:26:04.394683 | primary |  - cannot install the best update candidate for package glibc-common-2.28-180.el8.x86_64

Also noted a number of FTBS errors.

The error started on 01/12/2022:

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-tox-molecule+

[Jiri Podivin (jpodivin)]

openstack-tripleo-validations sees something similar in recent stable branch builds.[1]

---
DEBUG: delorean-gnocchi-c757cb4a95f19d6d445bfad87376d9 28 kB/s | 3.0 kB 00:00
DEBUG: delorean-ansible-role-collect-logs-1809de92a657 31 kB/s | 3.0 kB 00:00
DEBUG: delorean-openstack-nova-8906552cfc2525a44251d4c 30 kB/s | 3.0 kB 00:00
DEBUG: delorean-python-glance-store-79e043af2a86f4547f 29 kB/s | 3.0 kB 00:00
DEBUG: delorean-openstack-manila-2cebaa9d0bb3e892597a5 29 kB/s | 3.0 kB 00:00
DEBUG: delorean-python-networking-ovn-57d27d6e2187066e 27 kB/s | 3.0 kB 00:00
DEBUG: delorean-openstack-octavia-51c6d8bd9006041c782b 28 kB/s | 3.0 kB 00:00
DEBUG: delorean-openstack-barbican-0e526998303ec7fa8e6 29 kB/s | 3.0 kB 00:00
DEBUG: delorean-openstack-swift-2bffa1e205c499768abd76 28 kB/s | 3.0 kB 00:00
DEBUG: delorean-openstack-tempest-271f8201d4d927830d28 29 kB/s | 3.0 kB 00:00
DEBUG: delorean-ansible-tripleo-ipsec-0c8693cecc2a358d 30 kB/s | 3.0 kB 00:00
DEBUG: delorean-openstack-manila-ui-1442151bd3b2b8f63e 29 kB/s | 3.0 kB 00:00
DEBUG: delorean-python-validations-libs-17e689536f1c0e 29 kB/s | 3.0 kB 00:00
DEBUG: CentOS-Stream - Base 14 kB/s | 3.9 kB 00:00
DEBUG: CentOS-Stream - AppStream 17 kB/s | 4.4 kB 00:00
DEBUG: CentOS-Stream - Extras 75 kB/s | 16 kB 00:00
DEBUG: CentOS-Stream - PowerTools 18 kB/s | 4.4 kB 00:00
DEBUG: dlrn-train-testing 28 kB/s | 3.0 kB 00:00
DEBUG: dlrn-train-build-deps 28 kB/s | 3.0 kB 00:00
DEBUG: Advanced Virtualization mirror 100 kB/s | 3.0 kB 00:00
DEBUG: Messaging RabbitMQ 65 kB/s | 3.0 kB 00:00
DEBUG: opstools 101 kB/s | 3.0 kB 00:00
DEBUG: centos-8-fix 29 kB/s | 2.9 kB 00:00
DEBUG: Error:
DEBUG: Problem: cannot install the best update candidate for package glibc-2.28-181.el8.x86_64
DEBUG: - problem with installed package glibc-2.28-181.el8.x86_64
DEBUG: - glibc-gconv-extra-2.28-167.el8.i686 has inferior architecture
DEBUG: - package redhat-rpm-config-127-1.el8.noarch requires glibc-gconv-extra, but none of the providers can be installed
DEBUG: - glibc-gconv-extra-2.28-170.el8.i686 has inferior architecture
DEBUG: - glibc-gconv-extra-2.28-174.el8.i686 has inferior architecture
DEBUG: - glibc-gconv-extra-2.28-180.el8.i686 has inferior architecture
DEBUG: - package glibc-gconv-extra-2.28-167.el8.x86_64 requires glibc(x86-64) = 2.28-167.el8, but none of the providers can be installed
DEBUG: - package glibc-gconv-extra-2.28-170.el8.x86_64 requires glibc(x86-64) = 2.28-170.el8, but none of the providers can be installed
DEBUG: - package glibc-gconv-extra-2.28-174.el8.x86_64 requires glibc(x86-64) = 2.28-174.el8, but none of the providers can be installed
DEBUG: - package glibc-gconv-extra-2.28-180.el8.x86_64 requires glibc(x86-64) = 2.28-180.e...

[Alfredo Moralejo (amoralej)]

Caused by issue in CentOS mirror, it's working fine now. I've asked CentOS team, i'll update if i get any further info.

[Alfredo Moralejo (amoralej)]

From conversation on #centos-devel:

<arrfab> it seems (based on time report and what I could find in zabbix) that it was pushed "too fast" (there are two steps) without waiting for the whole mirror.centos.org to be stable/up2date itself
<arrfab> so external mirrors were fetching content and refreshing but not from same state

[Ronelle Landy (rlandy)]

This is fixed now - closing out