anaconda: build fails due to expat package change

[Davlet Panech]

Brief Description
-----------------
Anaconda RPM fails to build due to the missing expat-devel package. Caused by https://review.opendev.org/c/starlingx/tools/+/847409 .

Severity
--------
Major.
From Scott Little:
This is only affecting the monolithic installer build.
We are still getting successful layered builds in CENGN.
Developers can still do monolithic builds if they don't build the installer.

Steps to Reproduce
------------------
Try to build anaconda

Expected Behavior
------------------
Build succeeds

Actual Behavior
----------------
Build fails

Reproducibility
---------------
Seen once

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
master/2022-07-06

Last Pass
---------
master/2022-07-05

Timestamp/Logs
--------------

From: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/monolithic/20220706T043001Z/logs/std/failed-packages/anaconda-21.48.22.147-1.el7.centos.tis.5/

DEBUG util.py:679: Executing command: ['/localdisk/designer/jenkins/master/cgcs-root/build-tools/yum-builddep-wrapper', '--installroot', '/localdisk/loadbuild/jenkins/master/20220706T043001Z/installer/mock/b0/root/', '--releasever', '7', '/localdisk/loadbuild/jenkins/master/20220706T043001Z/installer/mock/b0/root//builddir/build/SRPMS/anaconda-21.48.22.147-1.el7.centos.tis.5.src.rpm'] with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/builddir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\\033]0;<mock-chroot>\\007"', 'PS1': '<mock-chroot> \\s-\\v\\$ ', 'LANG': 'en_US.UTF-8', 'BUILD_BY': 'jenkins', 'BUILD_DATE': '2022-07-06 08:52:14 +0000', 'REPO': '/localdisk/designer/jenkins/master/cgcs-root', 'WRS_GIT_BRANCH': 'HEAD', 'CGCS_GIT_BRANCH': 'HEAD', 'LC_MESSAGES': 'C.UTF-8'} and shell False
DEBUG util.py:585: BUILDSTDERR: Failed to set locale, defaulting to C
DEBUG util.py:587: Getting requirements for anaconda-21.48.22.147-1.el7.centos.tis.5.src
DEBUG util.py:587: --> 1:NetworkManager-glib-devel-1.12.0-8.el7_6.x86_64
[...]
DEBUG util.py:587: --> Processing Dependency: pkgconfig(expat) for package: fontconfig-devel-2.13.0-4.3.el7.x86_64
[...]
DEBUG util.py:587: Error: Package: fontconfig-devel-2.13.0-4.3.el7.x86_64 (StxCentos7Distro)
DEBUG util.py:587: Requires: pkgconfig(expat)

Test Activity
-------------
N/A

Workaround
----------
N/A

[Ghada Khalil (gkhalil)]

screening: stx.7.0 / high priority - recent submission causing the build to fail

[Joe Slater (jslater0wind)]

How does one build anaconda? Even with expat-devel available, if I run

$ build-pkgs anaconda

I get this, extracted from root.log in the build.

...
DEBUG util.py:417: Getting requirements for anaconda-21.48.22.147-1.el7.centos.tis.5.src
DEBUG util.py:417: --> audit-libs-devel-2.8.4-4.el7.x86_64
DEBUG util.py:417: --> gettext-0.19.8.1-3.el7.x86_64
DEBUG util.py:417: --> gtk3-devel-3.22.30-3.el7.x86_64
DEBUG util.py:417: --> gtk-doc-1.28-2.el7.x86_64
DEBUG util.py:417: --> gtk3-devel-docs-3.22.30-3.el7.x86_64
DEBUG util.py:417: --> glib2-doc-2.56.1-7.el7.noarch
DEBUG util.py:417: --> gobject-introspection-devel-1.56.1-1.el7.x86_64
DEBUG util.py:417: --> glade-devel-3.20.0-1.el7.x86_64
DEBUG util.py:417: --> python-gobject-3.22.0-1.el7_4.1.x86_64
DEBUG util.py:417: --> intltool-0.50.2-7.el7.noarch
DEBUG util.py:417: --> libgnomekbd-devel-3.26.0-1.el7.x86_64
DEBUG util.py:417: --> libxklavier-devel-5.4-7.el7.x86_64
DEBUG util.py:417: --> pango-devel-1.42.4-1.el7.x86_64
DEBUG util.py:417: --> pykickstart-1.99.66.19-2.el7.noarch
DEBUG util.py:417: --> python-devel-2.7.5-89.el7.x86_64
DEBUG util.py:417: --> Already installed : python-urlgrabber-3.10-9.el7.noarch
DEBUG util.py:417: --> python-nose-1.3.7-7.el7.noarch
DEBUG util.py:417: --> Already installed : systemd-219-78.el7_9.3.x86_64
DEBUG util.py:417: --> Already installed : yum-3.4.3-161.el7.centos.noarch
DEBUG util.py:417: --> 1:NetworkManager-glib-devel-1.12.0-8.el7_6.x86_64
DEBUG util.py:417: --> 1:NetworkManager-glib-devel-1.12.0-8.el7_6.x86_64
DEBUG util.py:417: --> 1:dbus-devel-1.10.24-15.el7.x86_64
DEBUG util.py:417: --> dbus-python-1.1.1-9.el7.x86_64
DEBUG util.py:417: --> libarchive-devel-3.1.2-10.el7_2.x86_64
DEBUG util.py:417: --> desktop-file-utils-0.23-1.el7.x86_64
DEBUG util.py:417: --> libtimezonemap-devel-0.4.4-1.el7.x86_64
DEBUG util.py:417: --> gsettings-desktop-schemas-3.28.0-2.el7.x86_64
DEBUG util.py:417: Error: No Package found for rpm-devel >= 4.14.0
...

[Davlet Panech (dpanech)]

Jenkins does this:

% build-pkgs
% build-iso
% build-pkgs --installer

Its the last command that tries to build anaconda. I'm not sure whether build-iso is necessary, but that's what Jenkins does.

[Joe Slater (jslater0wind)]

I think the "build-pkgs --installer" is the key.

[Joe Slater (jslater0wind)]

To have the installer build expat from source add expat to cgcs-root/stx/compile/centos_pkg_dirs_installer.

To be sure no-one uses the centos7 expat-devel, remove it from all rpms_centos.lst files. (This is what caused the failure addressed by this case, but I think the commit has been reverted.)

[Ghada Khalil (gkhalil)]

@Joe, Can you please add a link to the revert? Looking at the stx main branch this morning, I don't see it:
https://review.opendev.org/q/projects:starlingx%20is:merged%20branch:master

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/compile/+/849558

[Joe Slater (jslater0wind)]

Anthony Nowell contacted me about reverting to fix build failures, but has not done it. If we get the fix referenced above in soon, he won't have to.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/compile/+/849558
Committed: https://opendev.org/starlingx/compile/commit/48a7d2713b2b810ee69bc26ded6429a82e93b798
Submitter: "Zuul (22348)"
Branch: master

commit 48a7d2713b2b810ee69bc26ded6429a82e93b798
Author: Joe Slater <email address hidden>
Date: Tue Jul 12 11:44:20 2022 -0400

    installer: build expat from source

    Since expat is built from source, we do not want to use
    the centos version by accident, or require it to be
    available.

    === testing

    $ build-pkgs --installer # should succeed

    Go to the build installer/results directory and
    note that anaconda, rpm and expat were built.

    ===

    Closes-Bug: 1980838
    Change-Id: Ia1317bd4e8e1b703c65bf786e6ddbae22ef52cfe
    Signed-off-by: Joe Slater <email address hidden>