os_tempest: python_venv_build failing with "Depsolve Error occured: \n Problem: cannot install both gcc-8.5.0-7.el8.x86_64 and gcc-8.5.0-8.el8.x86_64\n

[Ronelle Landy]

Tests running os_tempest are failing at the following step:

2022-01-30 06:09:51.120455 | primary | TASK [python_venv_build : Install distro packages for venv build] **************
2022-01-30 06:09:51.120561 | primary | Sunday 30 January 2022 06:09:51 +0000 (0:00:00.061) 0:40:46.553 ********
2022-01-30 06:09:54.199270 | primary | FAILED - RETRYING: Install distro packages for venv build (5 retries left).
2022-01-30 06:09:59.408360 | primary | FAILED - RETRYING: Install distro packages for venv build (4 retries left).
2022-01-30 06:10:04.410529 | primary | FAILED - RETRYING: Install distro packages for venv build (3 retries left).
2022-01-30 06:10:09.404302 | primary | FAILED - RETRYING: Install distro packages for venv build (2 retries left).
2022-01-30 06:10:14.448819 | primary | FAILED - RETRYING: Install distro packages for venv build (1 retries left).
2022-01-30 06:10:19.504967 | primary | fatal: [undercloud]: FAILED! => {"attempts": 5, "changed": false, "failures": [], "msg": "Depsolve Error occured: \n Problem: cannot install both gcc-8.5.0-7.el8.x86_64 and gcc-8.5.0-8.el8.x86_64\n - package gcc-c++-8.5.0-7.el8.x86_64 requires gcc = 8.5.0-7.el8, but none of the providers can be installed\n - cannot install the best update candidate for package gcc-8.5.0-8.el8.x86_64\n - cannot install the best candidate for the job", "rc": 1, "results": []}

https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-victoria/e9a13df/job-output.txt

[Ronelle Landy (rlandy)]

This is hitting all tests:

https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-tripleo-master/baa75b9/job-output.txt

2022-01-30 20:27:44.192978 | primary | fatal: [undercloud]: FAILED! => {"attempts": 5, "changed": false, "failures": [], "msg": "Depsolve Error occured: \n Problem: cannot install both gcc-8.5.0-7.el8.x86_64 and gcc-8.5.0-8.el8.x86_64\n - package gcc-c++-8.5.0-7.el8.x86_64 requires gcc = 8.5.0-7.el8, but none of the providers can be installed\n - cannot install the best update candidate for package gcc-8.5.0-8.el8.x86_64\n - cannot install the best candidate for the job", "rc": 1, "results": []}

Error started on 01/28:
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario004-standalone-tripleo-master

[Ronelle Landy (rlandy)]

gcc.x86_64 8.5.0-8.el8 @appstream

possible update from appstream?

[Marios Andreou (marios-b)]

The error comes from [1] - a recent commit there is [2] but i don't know yet if that is responsible
as it merged 2 weeks ago... may still be a repo issue

[1] https://opendev.org/openstack/ansible-role-python_venv_build/src/commit/4a1490377ccde785d4f066f94608d18978894e0c/tasks/python_venv_install.yml#L32
[2] https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/824180

[Marios Andreou (marios-b)]

this is hitting all 8 branches e.g. master from today there [1]

2022-01-31 07:43:22.200095 | primary | fatal: [undercloud]: FAILED! => {"attempts": 5, "changed": false, "failures": [], "msg": "Depsolve Error occured: \n Problem: cannot install both gcc-8.5.0-7.el8.x86_64 and gcc-8.5.0-8.el8.x86_64\n - package gcc-c++-8.5.0-7.el8.x86_64 requires gcc = 8.5.0-7.el8, but none of the providers can be installed\n - cannot install the best update candidate for package gcc-8.5.0-8.el8.x86_64\n - cannot install the best candidate for the job", "rc": 1, "results": []}

[1] https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-master/204a336/job-output.txt

[Marios Andreou (marios-b)]

http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/Packages/

        * gcc-c++-8.5.0-8.el8.x86_64.rpm 2022-01-27 16:09 12M
        * gcc-c++-8.5.0-7.el8.x86_64.rpm 2021-12-18 01:59 12M
        * gcc-8.5.0-7.el8.x86_64.rpm 2021-12-18 01:59 23M
        * gcc-8.5.0-8.el8.x86_64.rpm 2022-01-27 16:09 23M

[Alfredo Moralejo (amoralej)]

Synchronization to AFS happened properly on 28th January. Both gcc and gcc-c++ were updated together:

centos.log.1:2022-01-28T16:43:02,784921420+00:00
centos.log.1:AppStream/aarch64/os/Packages/gcc-8.5.0-8.el8.aarch64.rpm
centos.log.1:AppStream/aarch64/os/Packages/gcc-c++-8.5.0-8.el8.aarch64.rpm
centos.log.1:AppStream/aarch64/os/Packages/gcc-gdb-plugin-8.5.0-8.el8.aarch64.rpm
centos.log.1:AppStream/aarch64/os/Packages/gcc-gfortran-8.5.0-8.el8.aarch64.rpm
centos.log.1:AppStream/aarch64/os/Packages/libstdc++-devel-8.5.0-8.el8.aarch64.rpm
centos.log.1:AppStream/aarch64/os/Packages/libstdc++-docs-8.5.0-8.el8.aarch64.rpm
centos.log.1:AppStream/x86_64/os/Packages/gcc-8.5.0-8.el8.x86_64.rpm
centos.log.1:AppStream/x86_64/os/Packages/gcc-c++-8.5.0-8.el8.x86_64.rpm
centos.log.1:AppStream/x86_64/os/Packages/gcc-gdb-plugin-8.5.0-8.el8.i686.rpm
centos.log.1:AppStream/x86_64/os/Packages/gcc-gdb-plugin-8.5.0-8.el8.x86_64.rpm
centos.log.1:AppStream/x86_64/os/Packages/gcc-gfortran-8.5.0-8.el8.x86_64.rpm
centos.log.1:AppStream/x86_64/os/Packages/gcc-offload-nvptx-8.5.0-8.el8.x86_64.rpm
centos.log.1:AppStream/x86_64/os/Packages/libstdc++-devel-8.5.0-8.el8.i686.rpm
centos.log.1:AppStream/x86_64/os/Packages/libstdc++-devel-8.5.0-8.el8.x86_64.rpm
centos.log.1:AppStream/x86_64/os/Packages/libstdc++-docs-8.5.0-8.el8.x86_64.rpm
centos.log.1:BaseOS/aarch64/os/Packages/libgcc-8.5.0-8.el8.aarch64.rpm
centos.log.1:BaseOS/aarch64/os/Packages/libgfortran-8.5.0-8.el8.aarch64.rpm
centos.log.1:BaseOS/aarch64/os/Packages/libstdc++-8.5.0-8.el8.aarch64.rpm
centos.log.1:BaseOS/x86_64/os/Packages/libgcc-8.5.0-8.el8.i686.rpm
centos.log.1:BaseOS/x86_64/os/Packages/libgcc-8.5.0-8.el8.x86_64.rpm
centos.log.1:BaseOS/x86_64/os/Packages/libgfortran-8.5.0-8.el8.i686.rpm
centos.log.1:BaseOS/x86_64/os/Packages/libgfortran-8.5.0-8.el8.x86_64.rpm
centos.log.1:BaseOS/x86_64/os/Packages/libstdc++-8.5.0-8.el8.i686.rpm
centos.log.1:BaseOS/x86_64/os/Packages/libstdc++-8.5.0-8.el8.x86_64.rpm

Something i don't understan well is why in those jobs we have installed:

libgfortran-8.5.0-7.el8.x86_64 (note 7)
gcc-8.5.0-8.el8.x86_64

https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-master/204a336/logs/undercloud/var/log/extra/rpm-list.txt.gz

still investigating

[Marios Andreou (marios-b)]

the openstack-periodic-integration-main-centos8 is currently running but it looks like the issue is resolved in today's run

we have green scen1/2/3/4/7/12

e.g.

https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/0bebe01/

https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-master/031ed9f/

https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario003-standalone-master/91d3bab/

... so confirmed 'mirror issue' ? :/

[Alfredo Moralejo (amoralej)]

For the record, it seems we were only hitting this in jobs running in SF zuul:

<marios|ruck> amoralej: i don't think we saw that one in gates it was only in the rdo SF/periodic jobs

I couldn't find anything wrong in the repos sync.

I suspect that the issue may have been caused for something in the image, but i'm not sure what.

[Marios Andreou (marios-b)]

we are mostly clear of this but for some reason the train line failed today there

        * https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/051457b7cd20347240a871126d2943d20722b78c/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-train/19b7d46/logs/undercloud/home/zuul/build.log.txt.gz

        * 2022-02-02 03:16:45.060 | > Error:
          2022-02-02 03:16:45.060 | > Problem: cannot install the best candidate for the job
          2022-02-02 03:16:45.060 | > - nothing provides libgcc >= 8.5.0-10.el8 needed by gcc-8.5.0-10.el8.x86_64
          2022-02-02 03:16:45.060 | > - nothing provides libgomp = 8.5.0-10.el8 needed by gcc-8.5.0-10.el8.x86_64

[Alfredo Moralejo (amoralej)]

I've investigated this particular error which is different that the one reported originally. So the issue in this case (as we had previously) it's appstream and baseos repos being out of sync because they are picking different mirrors.

This shouldn't happen if we use AFS mirrors for the mirrors as they sync from the same source repo server, but in this case we are including both the repos config with AFS mirrors and the default ones using mirrorlist:

2022-02-02 02:51:13.289 | CentOS Stream 8 - AppStream 17 MB/s | 19 MB 00:01
2022-02-02 02:51:20.665 | CentOS Stream 8 - BaseOS 5.7 MB/s | 18 MB 00:03
2022-02-02 02:51:25.086 | CentOS Stream 8 - Extras 19 kB/s | 17 kB 00:00
2022-02-02 02:51:25.572 | dlrn-train-testing 5.8 MB/s | 1.5 MB 00:00
2022-02-02 02:51:26.369 | dlrn-train-build-deps 1.5 MB/s | 395 kB 00:00
2022-02-02 02:51:26.732 | Advanced Virtualization mirror 3.1 MB/s | 207 kB 00:00
2022-02-02 02:51:27.048 | Messaging RabbitMQ 1.3 MB/s | 80 kB 00:00
2022-02-02 02:51:27.348 | opstools 943 kB/s | 149 kB 00:00
2022-02-02 02:51:27.732 | centos-8-fix 119 kB/s | 19 kB 00:00
2022-02-02 02:51:28.286 | quickstart-centos-appstreams 55 MB/s | 19 MB 00:00
2022-02-02 02:51:33.005 | quickstart-centos-base 62 MB/s | 18 MB 00:00
2022-02-02 02:51:36.977 | quickstart-centos-ceph-nautilus 3.0 MB/s | 549 kB 00:00
2022-02-02 02:51:37.346 | quickstart-centos-extras 275 kB/s | 16 kB 00:00
2022-02-02 02:51:37.579 | quickstart-centos-highavailability 21 MB/s | 2.2 MB 00:00
2022-02-02 02:51:38.557 | quickstart-centos-opstools 418 kB/s | 149 kB 00:00
2022-02-02 02:51:38.837 | quickstart-centos-powertools 32 MB/s | 4.0 MB 00:00

2022-02-02 03:16:44.429 | > repo: using cache for: appstream
2022-02-02 03:16:44.429 | > appstream: using metadata from Tue Feb 1 18:18:39 2022.
2022-02-02 03:16:44.429 | > repo: using cache for: baseos
2022-02-02 03:16:44.429 | > baseos: using metadata from Fri Jan 28 17:04:05 2022.

2022-02-02 03:16:44.429 | > repo: using cache for: quickstart-centos-appstreams
2022-02-02 03:16:44.429 | > quickstart-centos-appstreams: using metadata from Fri Jan 28 17:05:04 2022.
2022-02-02 03:16:44.429 | > repo: using cache for: quickstart-centos-base
2022-02-02 03:16:44.430 | > quickstart-centos-base: using metadata from Fri Jan 28 17:04:05 2022.

CentOS should provide some way to avoid this kind of issues but there is not a solution from their side yet.

I'd say we should ensure we are only using the AFS ones for all centos repos, i think that's our best bet so far.