Error: cannot install the best update candidate for package gcc-11.2.1-7.5.el9.x86_64 cs9 failing on master content provider

[Bhagyashri Shewale]

Error:

022-01-27 06:55:55.569824 | primary | Failed to store expired repos cache: [Errno 13] Permission denied: '/var/cache/dnf/expired_repos.json'
2022-01-27 06:55:55.572365 | primary | Error: This command has to be run with superuser privileges (under the root user on most systems).
2022-01-27 06:55:57.214953 | primary | Last metadata expiration check: 0:11:06 ago on Thu Jan 27 06:44:51 2022.
2022-01-27 06:55:57.313839 | primary | Error:
2022-01-27 06:55:57.313928 | primary | Problem: cannot install the best update candidate for package gcc-11.2.1-7.5.el9.x86_64
2022-01-27 06:55:57.313945 | primary | - nothing provides libgomp = 11.2.1-7.6.el9 needed by gcc-11.2.1-7.6.el9.x86_64
2022-01-27 06:55:57.313958 | primary | - nothing provides libgcc >= 11.2.1-7.6.el9 needed by gcc-11.2.1-7.6.el9.x86_64
2022-01-27 06:55:57.314045 | primary | (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
2022-01-27 06:55:57.350619 | primary |
2022-01-27 06:55:57.350729 | primary |  SUCCESS: installed required packages via bindep. 
2022-01-27 06:55:57.350835 | primary | +(./toci_quickstart.sh:107): ./quickstart.sh --bootstrap --no-clone --working-dir /home/zuul/workspace/.quickstart --playbook noop.yml --retain-inventory 127.0.0.2
2022-01-27 06:55:57.362444 | primary | Starting tripleo quickstart bootstrap...
2022-01-27 06:55:57.362571 | primary | Run install_deps from bootstrap...
2022-01-27 06:55:57.362608 | primary | Print out the commit hash of the git repo
2022-01-27 06:55:57.365408 | primary | commit ed58fe978aaf2ba5c5bd831eb1d8266746ed1653
2022-01-27 06:55:57.365464 | primary | Author: Sandeep Yadav <email address hidden>
2022-01-27 06:55:57.365529 | primary | Date: Tue Jan 25 12:23:03 2022 +0530
2022-01-27 06:55:57.365562 | primary |
2022-01-27 06:55:57.365586 | primary | Wallaby CS9 release file
2022-01-27 06:55:57.365614 | primary |
2022-01-27 06:55:57.365637 | primary | With this patch, We are adding CS9 release file for wallaby
2022-01-27 06:55:57.365659 | primary | branch.
2022-01-27 06:55:57.365688 | primary |
2022-01-27 06:55:57.365716 | primary | CS9 wallaby intergration release file was earlier added with [1].
2022-01-27 06:55:57.365758 | primary |
2022-01-27 06:55:57.365781 | primary | [1] https://review.opendev.org/c/openstack/tripleo-quickstart/+/823206
2022-01-27 06:55:57.365803 | primary |
2022-01-27 06:55:57.365823 | primary | Change-Id: I71a5f6b0d94b2fb7dcea71465f72fd538eb1608b
2022-01-27 06:55:57.365843 | primary |
2022-01-27 06:55:57.365864 | primary | create mode 100644 config/release/tripleo-ci/CentOS-9/wallaby.yml
2022-01-27 06:55:57.365892 | primary | Python Command is:
2022-01-27 06:55:57.366107 | primary | python3
2022-01-27 06:55:57.366135 | primary | Running install_deps
2022-01-27 06:55:57.377156 | primary | sudo-1.9.5p2-7.el9.x86_64
2022-01-27 06:55:57.439904 | primary | setting up for python3
2022-01-27 06:55:57.501791 | primary | failed to link /usr/bin/pip -> /etc/alternatives/pip: /usr/bin/pip exists and it is either not a symlink or --keep-foreign was set and link points outside /etc/alternatives
2022-01-27 06:55:57.505742 | primary | Installing RPM packages python3-libselinux python3-setuptools python3-virtualenv python3-pip
2022-01-27 06:55:58.005502 | primary | Last metadata expiration check: 0:11:07 ago on Thu Jan 27 06:44:51 2022.
2022-01-27 06:55:58.059214 | primary | Package python3-libselinux-3.3-2.el9.x86_64 is already installed.
2022-01-27 06:55:58.059953 | primary | Package python3-setuptools-53.0.0-9.el9.noarch is already installed.
2022-01-27 06:55:58.060718 | primary | Package python3-pip-21.2.3-5.el9.noarch is already installed.
2022-01-27 06:55:58.088478 | primary | Dependencies resolved.
2022-01-27 06:55:58.090740 | primary | Nothing to do.
2022-01-27 06:55:58.090882 | primary | Complete!
2022-01-27 06:55:58.127578 | primary | checking python modules
2022-01-27 06:55:58.403571 | primary | pip:21.3.1 setuptools:53.0.0
2022-01-27 06:55:58.403647 | primary |  SUCCESS: install-deps succeeded. 
2022-01-27 06:55:58.403693 | primary | Run install_virtual_env from bootstrap...
2022-01-27 06:55:58.403710 | primary | Creating virtualenv at /home/zuul/workspace/.quickstart
2022-01-27 06:56:02.706007 | primary | Looking in indexes: https://mirror-int.ord.rax.opendev.org/pypi/simple, https://mirror-int.ord.rax.opendev.org/wheel/centos-9-x86_64
2022-01-27 06:56:02.949850 | primary | Collecting wheel
2022-01-27 06:56:02.955750 | primary | Downloading https://mirror-int.ord.rax.opendev.org/pypifiles/packages/27/d6/003e593296a85fd6ed616ed962795b2f87709c3eee2bca4f6d0fe55c6d00/wheel-0.37.1-py2.py3-none-any.whl (35 kB)
2022-01-27 06:56:03.017811 | primary | Installing collected packages: wheel
2022-01-27 06:56:03.064742 | primary | Successfully installed wheel-0.37.1
2022-01-27 06:56:03.560629 | primary | Looking in indexes: https://mirror-int.ord.rax.opendev.org/pypi/simple, https://mirror-int.ord.rax.opendev.org/wheel/centos-9-x86_64
2022-01-27 06:56:03.561899 | primary | Requirement already satisfied: pip in /home/zuul/workspace/.quickstart/lib/python3.9/site-packages (21.2.3)
2022-01-27 06:56:03.842352 | primary | Collecting pip
2022-01-27 06:56:03.848054 | primary | Downloading https://mirror-int.ord.rax.opendev.org/pypifiles/packages/a4/6d/6463d49a933f547439d6b5b98b46af8742cc03ae83543e4d7688c2420f8b/pip-21.3.1-py3-none-any.whl (1.7 MB)
2022-01-27 06:56:03.969627 | primary | Installing collected packages: pip
2022-01-27 06:56:03.969694 | primary | Attempting uninstall: pip
2022-01-27 06:56:03.969990 | primary | Found existing installation: pip 21.2.3
2022-01-27 06:56:04.265152 | primary | Uninstalling pip-21.2.3:
2022-01-27 06:56:04.276659 | primary | Successfully uninstalled pip-21.2.3
2022-01-27 06:56:05.622012 | primary | Successfully installed pip-21.3.1
2022-01-27 06:56:05.689996 | primary | Installing bindep
2022-01-27 06:56:06.126946 | primary | Looking in indexes: https://mirror-int.ord.rax.opendev.org/pypi/simple, https://mirror-int.ord.rax.opendev.org/wheel/centos-9-x86_64
2022-01-27 06:56:06.297735 | primary | Collecting bindep
2022-01-27 06:56:06.301954 | primary | Using cached https://mirror-int.ord.rax.opendev.org/pypifiles/packages/28/05/cd85dca3800d7aa002ef733aad55c367ea2bbd79a10d98fe50a556c85c3b/bindep-2.10.2-py2.py3-none-any.whl (33 kB)
2022-01-27 06:56:06.344173 | primary | Collecting Parsley
2022-01-27 06:56:06.348747 | primary | Using cached https://mirror-int.ord.rax.opendev.org/pypifiles/packages/2b/d6/4fed8d65e28a970e1c5cb33ce9c7e22e3de745e1b2ae37af051ef16aea3b/Parsley-1.3-py2.py3-none-any.whl (88 kB)
2022-01-27 06:56:06.428581 | primary | Collecting packaging
2022-01-27 06:56:06.432642 | primary | Using cached https://mirror-int.ord.rax.opendev.org/pypifiles/packages/05/8e/8de486cbd03baba4deef4142bd643a3e7bbe954a784dc1bb17142572d127/packaging-21.3-py3-none-any.whl (40 kB)
2022-01-27 06:56:06.544025 | primary | Collecting pbr>=2.0.0
2022-01-27 06:56:06.548645 | primary | Using cached https://mirror-int.ord.rax.opendev.org/pypifiles/packages/73/c3/d45171501210b0305f4c93fafe50950f0c2228e87034ceb51744bd03ff08/pbr-5.8.0-py2.py3-none-any.whl (112 kB)
2022-01-27 06:56:06.589950 | primary | Collecting distro
2022-01-27 06:56:06.594421 | primary | Using cached https://mirror-int.ord.rax.opendev.org/pypifiles/packages/b3/8d/a0a5c389d76f90c766e956515d34c3408a1e18f60fbaa08221d1f6b87490/distro-1.6.0-py2.py3-none-any.whl (19 kB)
2022-01-27 06:56:06.736752 | primary | Collecting pyparsing!=3.0.5,>=2.0.2
2022-01-27 06:56:06.741080 | primary | Using cached https://mirror-int.ord.rax.opendev.org/pypifiles/packages/80/c1/23fd82ad3121656b585351aba6c19761926bb0db2ebed9e4ff09a43a3fcc/pyparsing-3.0.7-py3-none-any.whl (98 kB)
2022-01-27 06:56:06.806942 | primary | Installing collected packages: pyparsing, pbr, Parsley, packaging, distro, bindep
2022-01-27 06:56:07.242733 | primary | Successfully installed Parsley-1.3 bindep-2.10.2 distro-1.6.0 packaging-21.3 pbr-5.8.0 pyparsing-3.0.7
2022-01-27 06:56:07.775005 | primary | Looking in indexes: https://mirror-int.ord.rax.opendev.org/pypi/simple, https://mirror-int.ord.rax.opendev.org/wheel/centos-9-x86_64
2022-01-27 06:56:07.940733 | primary | Collecting selinux
2022-01-27 06:56:07.948527 | primary | Downloading https://mirror-int.ord.rax.opendev.org/pypifiles/packages/65/11/67955ff6fbd5aedee3b4d47847c24510e7de970aa352cacac8444ef33059/selinux-0.2.1-py2.py3-none-any.whl (4.3 kB)
2022-01-27 06:56:07.957449 | primary | Requirement already satisfied: distro>=1.3.0 in /home/zuul/workspace/.quickstart/lib/python3.9/site-packages (from selinux) (1.6.0)
2022-01-27 06:56:07.958225 | primary | Requirement already satisfied: setuptools>=39.0 in /home/zuul/workspace/.quickstart/lib/python3.9/site-packages (from selinux) (53.0.0)
2022-01-27 06:56:08.026710 | primary | Installing collected packages: selinux
2022-01-27 06:56:08.044684 | primary | Successfully installed selinux-0.2.1
2022-01-27 06:56:08.104313 | primary | Run install_package_deps_via_bindep from bootstrap...
2022-01-27 06:56:08.104360 | primary | install_package_deps_via_bindep
2022-01-27 06:56:09.682879 | primary | Failed to store expired repos cache: [Errno 13] Permission denied: '/var/cache/dnf/expired_repos.json'
2022-01-27 06:56:09.683896 | primary | Error: This command has to be run with superuser privileges (under the root user on most systems).
2022-01-27 06:56:11.309705 | primary | Last metadata expiration check: 0:11:20 ago on Thu 27 Jan 2022 06:44:51 AM UTC.
2022-01-27 06:56:11.406960 | primary | Error:
2022-01-27 06:56:11.406996 | primary | Problem: cannot install the best update candidate for package gcc-11.2.1-7.5.el9.x86_64
2022-01-27 06:56:11.407012 | primary | - nothing provides libgomp = 11.2.1-7.6.el9 needed by gcc-11.2.1-7.6.el9.x86_64
2022-01-27 06:56:11.407024 | primary | - nothing provides libgcc >= 11.2.1-7.6.el9 needed by gcc-11.2.1-7.6.el9.x86_64
2022-01-27 06:56:11.407189 | primary | (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
2022-01-27 06:56:11.537907 | primary | ERROR
2022-01-27 06:56:11.538203 | primary | {
2022-01-27 06:56:11.538281 | primary | "delta": "0:00:35.881404",
2022-01-27 06:56:11.538356 | primary | "end": "2022-01-27 06:56:11.445415",
2022-01-27 06:56:11.538466 | primary | "msg": "non-zero return code",
2022-01-27 06:56:11.538564 | primary | "rc": 1,
2022-01-27 06:56:11.538639 | primary | "start": "2022-01-27 06:55:35.564011"
2022-01-27 06:56:11.538740 | primary | }
2022-01-27 06:56:11.606451 |
2022-01-27 06:56:11.606908 | PLAY RECAP
2022-01-27 06:56:11.607033 | primary | ok: 8 changed: 5 unreachable: 0 failed: 1 skipped: 11 rescued: 0 ignored: 0

[1]: https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider&skip=0
[2]: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54f/826417/2/check/tripleo-ci-centos-9-content-provider/54f6f3a/job-output.txt

[Marios Andreou (marios-b)]

https://bugs.launchpad.net/tripleo/+bug/1959187/comments/1

Alfredo Moralejo (amoralej) wrote 3 minutes ago: #1

The problem is caused by broken centos9 repos after last compose promotion in centos infra https://bugzilla.redhat.com/show_bug.cgi?id=2046690

[Marios Andreou (marios-b)]

Patch at https://review.opendev.org/c/zuul/zuul-jobs/+/826603/ will help in future (and would have prevented this issue as the AFS mirrors were OK in this case)

The issue is no longer blocking us as mirrors were eventually synced yesterday and gate unblocked.

[Alan Pevec (apevec)]

zuul-jobs 826603 merged can we close this now?