Master Periodic jobs: Sc001/002 failing with Error: \", \" Problem: package rdma-core-26.0-8.el8.x86_64 requires dracut, but none of the providers can be installed

Bug #1889192 reported by Sandeep Yadav
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
yatin

Bug Description

Master periodic jobs Sc001/002 are failing with below error:-

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-master/dc4bd03/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/7be4f48/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

~~~
Error: \", \" Problem: package rdma-core-26.0-8.el8.x86_64 requires dracut, but none of the providers can be installed\", \" - package dracut-049-70.git20200228.el8.x86_64 requires systemd-udev >= 219, but none of the providers can be installed\", \" - package libibverbs-26.0-8.el8.x86_64 requires rdma-core(x86-64) = 26.0-8.el8, but none of the providers can be installed\", \" - package systemd-udev-239-30.el8_2.x86_64 requires systemd(x86-64) = 239-30.el8_2, but none of the providers can be installed\", \" - package librados2-2:14.2.9-1.el8.x86_64 requires libibverbs.so.1()(64bit), but none of the providers can be installed\", \" - package librados2-2:14.2.9-1.el8.x86_64 requires libibverbs.so.1(IBVERBS_1.0)(64bit), but none of the providers can be installed\", \" - package librados2-2:14.2.9-1.el8.x86_64 requires libibverbs.so.1(IBVERBS_1.1)(64bit), but none of the providers can be installed\", \" - cannot install both systemd-239-30.el8_2.x86_64 and systemd-239-31.el8_2.2.x86_64\", \" - cannot install the best update candidate for package librados2-1:12.2.7-9.el8.x86_64\", \" - problem with installed package systemd-239-31.el8_2.2.x86_64\", \"level=debug msg=\\\\\"error running [bash -x /tmp/yum_update.sh delorean-current,quickstart-centos-ceph-nautilus] in container \\\\\\\\\\\\\"centos-binary-gnocchi-statsd-working-container\\\\\\\\\\\\\": error while running runtime: exit status 1\\\\\"\"], \"stdout\": \"Updating Subscription Management repositories.\\\\nUnable to read consumer identity\\\\nSubscription Manager is operating in container mode.\\\\nThis system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.\\\\nLast metadata expiration check: 0:00:27 ago on Mon 27 Jul 2020 09:25:49 AM UTC.\\\\n(try to add \\'--allowerasing\\' to command line to replace conflicting packages or \\'--skip-broken\\' to skip uninstallable packages or \\'--nobest\\' to use not only best candidate packages)\", \"stdout_lines\": [\"Updating Subscription Management repositories.\", \"Unable to read consumer identity\", \"Subscription Manager is operating in container mode.\", \"This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.\", \"Last metadata expiration check: 0:00:27 ago on Mon 27 Jul 2020 09:25:49 AM UTC.\", \"(try to add \\'--allowerasing\\' to command line to replace conflicting packages or \\'--skip-broken\\' to skip uninstallable packages or \\'--nobest\\' to use not only best candidate packages)\"]}\\n\\x1b[Ke30=\\x1b[4D\\x1b[K\\x1b[KeyJ1dWlkIjogImQzYTViNWQ1LTA4OGItNGVmNS05MzBiLWJjNzQ5NGVkZGFhYiJ9\\x1b[64D\\x1b[K\\nPLAY RECAP
~~~

This issue is also discuss in patch: https://review.rdoproject.org/r/#/c/28663/

Snippet:-
~~~
For fixing librados2 issue, librados2 can be added explicitly to the package installation list https://github.com/openstack/tripleo-common/blob/master/container-images/tcib/base/os/gnocchi-base/gnocchi-base.yaml#L4-L5, which will pull librados and trigger upgrade/downgrade.

The issue happened after new ubi8 image released recently which have new systemd packages installed then centos8 base repos, So when there is attempt to install python3-cradox, due to dnf behavior mentioned above it avoids downgrade to systemd from centos8-base repos and picks old librados2-12 from appstream repo.

Mixing centos8 and ubi8 can cause similar issue in future too, so should keep enabled ubi8 repos instead of removing them(happening currently as we bind mount /etc/yum.repos.d:/etc/yum.repos.d) during container build.
~~~

A work in progress patch[1] is already proposed and test run[2] is green using it.

[1] https://review.opendev.org/#/c/743263/
[2] https://review.rdoproject.org/r/#/c/28723/

Changed in tripleo:
assignee: nobody → yatin (yatinkarel)
status: Triaged → In Progress
Changed in tripleo:
milestone: victoria-1 → victoria-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/743739

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/743741

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (master)

Reviewed: https://review.opendev.org/743263
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=c6e680e180f457b8b9085d503ea8e8b6ccdad90d
Submitter: Zuul
Branch: master

commit c6e680e180f457b8b9085d503ea8e8b6ccdad90d
Author: Emilien Macchi <email address hidden>
Date: Mon Jul 27 11:48:42 2020 -0400

    container-images: add librados2 to gnocchi-base

    This package is needed so the right dependencies are installed in the
    gnocchi base container images and db_sync can run fine when ceph is
    used as a backend.

    Partial-Bug: #1889192
    Change-Id: Ic0bfdec756a23460f9257e78c508d167d21f0792

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/ussuri)

Reviewed: https://review.opendev.org/743739
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=8a09eb94d926051a4588fd71440468a48852f940
Submitter: Zuul
Branch: stable/ussuri

commit 8a09eb94d926051a4588fd71440468a48852f940
Author: Emilien Macchi <email address hidden>
Date: Mon Jul 27 11:48:42 2020 -0400

    container-images: add librados2 to gnocchi-base

    This package is needed so the right dependencies are installed in the
    gnocchi base container images and db_sync can run fine when ceph is
    used as a backend.

    Partial-Bug: #1889192
    Change-Id: Ic0bfdec756a23460f9257e78c508d167d21f0792
    (cherry picked from commit c6e680e180f457b8b9085d503ea8e8b6ccdad90d)

tags: added: in-stable-ussuri
Revision history for this message
Marios Andreou (marios-b) wrote :

so we are still waiting for https://review.opendev.org/#/c/743741/2 stable/train and that needs the patch above it https://review.opendev.org/#/c/741228/8 to merge first.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/train)

Reviewed: https://review.opendev.org/743741
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=4dac873004a8eecb20274cc078cffd1ff3cd4ade
Submitter: Zuul
Branch: stable/train

commit 4dac873004a8eecb20274cc078cffd1ff3cd4ade
Author: Emilien Macchi <email address hidden>
Date: Mon Jul 27 11:48:42 2020 -0400

    container-images: add librados2 to gnocchi-base

    This package is needed so the right dependencies are installed in the
    gnocchi base container images and db_sync can run fine when ceph is
    used as a backend.

    Partial-Bug: #1889192
    Change-Id: Ic0bfdec756a23460f9257e78c508d167d21f0792
    (cherry picked from commit c6e680e180f457b8b9085d503ea8e8b6ccdad90d)

tags: added: in-stable-train
Changed in tripleo:
milestone: victoria-3 → wallaby-1
Changed in tripleo:
milestone: wallaby-1 → wallaby-2
Changed in tripleo:
milestone: wallaby-2 → wallaby-3
Changed in tripleo:
milestone: wallaby-3 → wallaby-rc1
Changed in tripleo:
milestone: wallaby-rc1 → xena-1
Changed in tripleo:
milestone: xena-1 → xena-2
Changed in tripleo:
milestone: xena-2 → xena-3
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers