build fails due to ceph dependency issues

[Davlet Panech]

Brief Description
-----------------
The following RPMs fail to build:

ceph-14.2.22-0.el7.tis.15486.src.rpm
libvirt-4.7.0-1.tis.28.src.rpm
libvirt-python-4.7.0-1.tis.5.src.rpm
qemu-kvm-ev-3.0.0-0.tis.19.src.rpm

Severity
--------
Major

Steps to Reproduce
------------------
rpm-buildpkgs

Expected Behavior
------------------
Build succeds

Actual Behavior
----------------
Build fails

Reproducibility
---------------
Seen once (so far)

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
master/2021-10-27

Last Pass
---------
master/2021-10-26

Timestamp/Logs
--------------

*** Ceph ***

DEBUG util.py:587: Error: Package: sudo-1.8.23-10.el7_9.1.tis.10.x86_64 (1270018088_localdisk_loadbuild_jenkins_masterdistro_20211028T021723Z_std_results_jenkinsmasterdistro40std_)
DEBUG util.py:587: Requires: /usr/bin/vi
DEBUG util.py:587: Error: Package: cmake3-3.17.5-1.el7.x86_64 (StxCentos7Distro)
DEBUG util.py:587: Requires: libzstd.so.1()(64bit)
DEBUG util.py:587: You could try using --skip-broken to work around the problem
DEBUG util.py:587: You could try running: rpm -Va --nofiles --nodigest
DEBUG util.py:734: Child return code was: 1

*** libvirt ***
DEBUG util.py:587: --> yajl-devel-2.0.4-4.el7.x86_64
DEBUG util.py:587: Error: No Package found for /usr/bin/qemu-img
DEBUG util.py:587: Error: No Package found for librados2-devel
DEBUG util.py:587: Error: No Package found for librbd1-devel

*** libvirt-python ***
DEBUG util.py:587: If above article doesn't help to resolve this issue please use https://bugs.centos.org/.
DEBUG util.py:587: Getting requirements for libvirt-python-4.7.0-1.tis.5.src
DEBUG util.py:587: --> Already installed : gcc-4.8.5-36.el7.x86_64
DEBUG util.py:587: --> Already installed : python-devel-2.7.5-89.el7.tis.8.x86_64
DEBUG util.py:587: --> Already installed : python-lxml-3.2.1-4.el7.x86_64
DEBUG util.py:587: --> python-nose-1.3.7-7.el7.noarch
DEBUG util.py:587: --> Already installed : python2-pip-9.0.3-6.fc29.noarch
DEBUG util.py:587: --> Already installed : 1:python2-wheel-0.31.1-3.fc30.noarch
DEBUG util.py:587: Error: No Package found for libvirt-devel = 4.7.0
DEBUG util.py:734: Child return code was: 1

*** qemu-kvm-ev ***
DEBUG util.py:587: Error: No Package found for librados2-devel
DEBUG util.py:587: Error: No Package found for librbd1-devel
DEBUG util.py:734: Child return code was: 1

Test Activity
-------------
[Sanity, Feature Testing, Regression Testing, Developer Testing, Evaluation, Other - Please specify]

Workaround
----------
Describe workaround if available

[Ghada Khalil (gkhalil)]

Is this fixed by the following commits to stx-libvirt?
https://github.com/starlingx-staging/stx-libvirt/commit/8c46e49ba0880f6eddbd4d4c3ffe1ed5653a8dd6
https://github.com/starlingx-staging/stx-libvirt/commit/e8c2aed278a51062778b5f79252d2bb8a3b759ce
https://github.com/starlingx-staging/stx-libvirt/commit/dee98c789c3fc8d1d1f51d93acea9e3003db049c

Commits to the starlingx-staging repos are not integrated with launchpad. The LP needs to be updated manually.

[Ghada Khalil (gkhalil)]

There are reports that the stx build is still failing even with the above commits.
The following dependencies are still missing: libzstd.so.1 and /usr/bin/vi

[M. Vefa Bicakci (vbicakci)]

Hello all,

I have a fix for this issue. I will publish a patch either later tonight or tomorrow morning. Hence, I am assigning this issue to myself.

The problem arises from the fact that libzstd and vim-minimal packages were missing from the distro layer's chroot package list. vim-minimal is a dependency of sudo, whereas libzstd is a dependency of cmake3.

[M. Vefa Bicakci (vbicakci)]

Here is a preliminary patch I prepared for the build chroot package lists.

However, after applying this patch, I encountered the following failure:

build-iso
...
02:21:23 Warning: Infinite loop detected in dependency resolution. See /loadbuild/loadbuild/.../stx-master-container-flock/export/deps.txt for details -- exiting
02:21:23 These RPMS had problems (likely version conflicts)
02:21:23 librabbitmq.so.4()(64bit)
02:21:23 librdkafka.so.1()(64bit)
02:21:23 libstoragemgmt
02:21:23 Could not install dependencies

[.../stx-master-container-flock/export/dist/isolinux/Packages]$ for pkg in *.rpm ; do rpm -q --requires -p ${pkg} 2>/dev/null | grep --label ${pkg} -e libstorage -e librdkafka -e librabbitmq - /dev/null ; done
ceph-common-14.2.22-0.el7.tis.15486.x86_64.rpm:librabbitmq.so.4()(64bit)
ceph-common-14.2.22-0.el7.tis.15486.x86_64.rpm:librdkafka.so.1()(64bit)
ceph-osd-14.2.22-0.el7.tis.15486.x86_64.rpm:libstoragemgmt
ceph-radosgw-14.2.22-0.el7.tis.15486.x86_64.rpm:librabbitmq.so.4()(64bit)
librgw2-14.2.22-0.el7.tis.15486.x86_64.rpm:librabbitmq.so.4()(64bit)
^C

[Ghada Khalil (gkhalil)]

I'm assigning this back to the ceph team. As per my discussion with Linda, even with the fix to address the libzstd and vi dependencies, the build is still failing:

build-iso
...
02:21:23 Warning: Infinite loop detected in dependency resolution. See /localdisk/loadbuild/vbicakci/stx-master-container-flock/export/deps.txt for details -- exiting
02:21:23 These RPMS had problems (likely version conflicts)
02:21:23 librabbitmq.so.4()(64bit)
02:21:23 librdkafka.so.1()(64bit)
02:21:23 libstoragemgmt
02:21:23 Could not install dependencies

[.../stx-master-container-flock/export/dist/isolinux/Packages]$ for pkg in *.rpm ; do rpm -q --requires -p ${pkg} 2>/dev/null | grep --label ${pkg} -e libstorage -e librdkafka -e librabbitmq - /dev/null ; done
ceph-common-14.2.22-0.el7.tis.15486.x86_64.rpm:librabbitmq.so.4()(64bit)
ceph-common-14.2.22-0.el7.tis.15486.x86_64.rpm:librdkafka.so.1()(64bit)
ceph-osd-14.2.22-0.el7.tis.15486.x86_64.rpm:libstoragemgmt
ceph-radosgw-14.2.22-0.el7.tis.15486.x86_64.rpm:librabbitmq.so.4()(64bit)
librgw2-14.2.22-0.el7.tis.15486.x86_64.rpm:librabbitmq.so.4()(64bit)
^C

So this needs further investigation by the ceph team.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/tools/+/816086

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/tools/+/816086
Committed: https://opendev.org/starlingx/tools/commit/4ef251dddcd8caf0e448324787456064c1110772
Submitter: "Zuul (22348)"
Branch: master

commit 4ef251dddcd8caf0e448324787456064c1110772
Author: Scott Little <email address hidden>
Date: Sat Oct 30 02:17:58 2021 -0400

    fix ceph build failure: missing vi and libzstd

    Since upgrade, ceph build failes with.

    DEBUG util.py:587: Error: Package: sudo-1.8.23-10.el7_9.1.tis.10.x86_64
    (local_build_repo)
    DEBUG util.py:587: Requires: /usr/bin/vi
    DEBUG util.py:587: Error: Package: cmake3-3.17.5-1.el7.x86_64
    (StxCentos7Distro)
    DEBUG util.py:587: Requires: libzstd.so.1()(64bit)

    Add the missing packages to the distro build.

    Closes-Bug: 1949112
    Signed-off-by: Scott Little <email address hidden>
    Change-Id: Iab324070fd8b0f01589bc03007f627b6dce884a6

[M. Vefa Bicakci (vbicakci)]

Hello all,

This issue is not yet resolved. In summary, build-iso still fails due to the following messages:

02:21:23 Warning: Infinite loop detected in dependency resolution. See /loadbuild/loadbuild/.../stx-master-container-flock/export/deps.txt for details -- exiting
02:21:23 These RPMS had problems (likely version conflicts)
02:21:23 librabbitmq.so.4()(64bit)
02:21:23 librdkafka.so.1()(64bit)
02:21:23 libstoragemgmt
02:21:23 Could not install dependencies

I am attaching a patch that fixes that issue as well, and that makes build-iso succeed.

Please note that the ceph team still needs to look at the run-time dependencies though, because the library librdmacm has two *different* versions specified in the rpms_centos.lst files for distro and flock layers:

stx-tools/centos-mirror-tools]$ git grep librdmacm
config/centos/distro/rpms_centos.lst:librdmacm-22.4-5.el7.x86_64.rpm
config/centos/distro/rpms_centos.lst:librdmacm-devel-1.1.0-2.el7.x86_64.rpm
config/centos/flock/rpms_centos.lst:librdmacm-17.2-3.el7.x86_64.rpm

Scott, please feel free to adapt my patch to fix the build-iso issue. I can publish it as well, if you would like.

Thank you.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/tools/+/816090

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/tools/+/816091

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/tools/+/816090
Committed: https://opendev.org/starlingx/tools/commit/867ae970adbec89c980b7d014c37e4f3b4c86e30
Submitter: "Zuul (22348)"
Branch: master

commit 867ae970adbec89c980b7d014c37e4f3b4c86e30
Author: Scott Little <email address hidden>
Date: Sat Oct 30 09:51:39 2021 -0400

    fix ceph build failure: wrong vi

    ceph now needs sudo, sudo needs vim-minimal not vim-enhanced.

    Partial-Bug: 1949112
    Signed-off-by: Scott Little <email address hidden>
    Change-Id: I56ef778a8403d22d673426cd231fc3092b5268aa

[OpenStack Infra (hudson-openstack)]

Change abandoned by "M. Vefa Bicakci <email address hidden>" on branch: master
Review: https://review.opendev.org/c/starlingx/tools/+/816091

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/tools/+/816092
Committed: https://opendev.org/starlingx/tools/commit/f0581faeaa79f72417a8a87e48e80270e9f55a8f
Submitter: "Zuul (22348)"
Branch: master

commit f0581faeaa79f72417a8a87e48e80270e9f55a8f
Author: M. Vefa Bicakci <email address hidden>
Date: Sat Oct 30 09:11:28 2021 -0400

    flock: Add run-time dependencies of ceph-14

    According to the layered build referense at

      https://docs.starlingx.io/developer_resources/Layered_Build.html

    all run-time dependencies of packages in all layers need to be included
    in the flock layer's rpms_centos.lst. Quoting from the linked guide:

      If the package will be installed to iso, the package’s ‘Requires’ as
      well as the transitive Requires of those Requires, should be added to
      a lst file under stx-tools/centos-mirror-tools/config/<os>/flock. Yes
      I said ‘flock, and not <layer>, because the ISO is build from the
      flock layer.

    This commit adds the missing runtime dependencies of ceph-14 to ensure
    that "build-iso" does not fail with the following errors:

      Warning: Infinite loop detected in dependency resolution. \
        See .../loadbuild/.../flock/export/deps.txt for details -- exiting
      These RPMS had problems (likely version conflicts)
              librabbitmq.so.4()(64bit)
              librdkafka.so.1()(64bit)
              libstoragemgmt
      Could not install dependencies

    Verification: Layered build and build-iso are successful.

    Partial-Bug: #1949112

    Signed-off-by: M. Vefa Bicakci <email address hidden>
    Change-Id: I7c061b1f219061ad8d34e3ea9b5aafe237476440