yum deadlocks during the build

[Davlet Panech]

Brief Description
-----------------
Occasionally mock hangs during the build due to a stale yum.lock file in the chroot directory. This is caused by a bug in pre-DNF yum-builddep: it creates the pid file, but never deletes it. If the stale PID happens to match an existing process during a subsequent yum run, it thinks the file is locked and hangs.

Severity
--------
Minor

Steps to Reproduce
------------------

Call build-pkgs on a CentOS 7 host or docker container.

Expected Behavior
------------------
Mock, yum & related commands called by build-pkgs work correctly.

Actual Behavior
---------------
Yum deadlocks.

Reproducibility
---------------
Intermittent. Happens very infrequently.

System Configuration
--------------------
CentOS 7 - based build host

Branch/Pull Time/Commit
-----------------------
Master branch, 2021-03-22

Last pass
---------
N/A

Timestamp/Logs
--------------

DEBUG package_manager.py:96: ['/usr/bin/yum-builddep', '--installroot', '/localdisk/loadbuild/jenkins/WRCP_Dev_Build/2021-03-20_00-00-07/rt/mock/b0/root/', '--releasever', '7', '/localdisk/loadbuild/jenkins/WRCP_Dev_Build/2021-03-20_00-00-07/rt/mock/b0/root//builddir/build/SRPMS/wrs_avp-kmod-rt-1.2.14-0.tis.2.src.rpm']
DEBUG util.py:563: child environment: None
DEBUG util.py:495: Executing command: ['/usr/bin/yum-builddep', '--installroot', '/localdisk/loadbuild/jenkins/WRCP_Dev_Build/2021-03-20_00-00-07/rt/mock/b0/root/', '--releasever', '7', '/localdisk/loadbuild/jenkins/WRCP_Dev_Build/2021-03-20_00-00-07/rt/mock/b0/root//builddir/build/SRPMS/wrs_avp-kmod-rt-1.2.14-0.tis.2.src.rpm'] with env {'LANG': 'en_US.UTF-8', 'BUILD_DATE': '2021-03-20 02:19:06 -0400', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'LC_MESSAGES': 'C', 'WRS_GIT_BRANCH': 'HEAD', 'HOSTNAME': 'mock', 'REPO': '/localdisk/designer/jenkins/WRCP_Dev_Build/cgcs-root', 'CGCS_GIT_BRANCH': 'HEAD', 'PROMPT_COMMAND': 'printf "\x1b]0;<mock-chroot>\x07<mock-chroot>"', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'HOME': '/builddir', 'BUILD_BY': 'jenkins'} and shell False
DEBUG util.py:417: http://127.0.0.1:8088/localdisk/loadbuild/jenkins/WRCP_Dev_Build/2021-03-20_00-00-07/installer/rpmbuild/RPMS/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
DEBUG util.py:417: Trying other mirror.
DEBUG util.py:417: To address this issue please refer to the below wiki article
DEBUG util.py:417: https://wiki.centos.org/yum-errors
DEBUG util.py:417: If above article doesn't help to resolve this issue please use https://bugs.centos.org/.
DEBUG util.py:417: Getting requirements for wrs_avp-kmod-rt-1.2.14-0.tis.2.src
DEBUG util.py:417: --> Already installed : gcc-4.8.5-36.el7.x86_64
DEBUG util.py:417: --> Already installed : glibc-headers-2.17-317.el7.x86_64
DEBUG util.py:417: --> Already installed : kernel-rt-devel-3.10.0-1127.rt56.1093.el7.tis.8.x86_64
DEBUG util.py:417: --> Already installed : 4:perl-5.16.3-299.el7_9.x86_64
DEBUG util.py:417: --> Already installed : 1:openssl-1.0.2k-16.el7.x86_64
DEBUG util.py:417: --> Already installed : elfutils-libelf-devel-0.176-2.el7.x86_64
DEBUG util.py:417: Error: No Package found for cgcs-dpdk-rt
DEBUG util.py:417: Error: No Package found for cgcs-dpdk-rt-dev
DEBUG util.py:542: Child return code was: 1
DEBUG util.py:176: kill orphans
...
DEBUG util.py:495: Executing command: ['/usr/bin/yum', '--installroot', '/localdisk/loadbuild/jenkins/WRCP_Dev_Build/2021-03-20_00-00-07/rt/mock/b3/root/', '-y', '--releasever', '7', 'update'] with env {'LANG': 'en_US.UTF-8', 'BUILD_DATE': '2021-03-20 02:19:06 -0400', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'LC_MESSAGES': 'C', 'WRS_GIT_BRANCH': 'HEAD', 'HOSTNAME': 'mock', 'REPO': '/localdisk/designer/jenkins/WRCP_Dev_Build/cgcs-root', 'CGCS_GIT_BRANCH': 'HEAD', 'PROMPT_COMMAND': 'printf "\x1b]0;<mock-chroot>\x07<mock-chroot>"', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'HOME': '/builddir', 'BUILD_BY': 'jenkins'} and shell False
DEBUG util.py:417: Existing lock /localdisk/loadbuild/jenkins/WRCP_Dev_Build/2021-03-20_00-00-07/rt/mock/b3/root/var/run/yum.pid: another copy is running as pid 68378.
DEBUG util.py:417: Another app is currently holding the yum lock; waiting for it to exit...
DEBUG util.py:417: The other application is: mock
DEBUG util.py:417: Memory : 16 M RSS (268 MB VSZ)
DEBUG util.py:417: Started: Sat Mar 20 02:35:19 2021 - 00:01 ago
DEBUG util.py:417: State : Running, pid: 68378
DEBUG util.py:417: Another app is currently holding the yum lock; waiting for it to exit...
DEBUG util.py:417: The other application is: mock
DEBUG util.py:417: Memory : 16 M RSS (268 MB VSZ)
DEBUG util.py:417: Started: Sat Mar 20 02:35:19 2021 - 00:03 ago
DEBUG util.py:417: State : Sleeping, pid: 68378
...

[Ghada Khalil (gkhalil)]

screening: marked as low priority given the issue happens very infrequently

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/root/+/792232

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/root/+/792232
Committed: https://opendev.org/starlingx/root/commit/7cb55d28111dbf7458ed3f01a5eda5c4dfdb124b
Submitter: "Zuul (22348)"
Branch: f/centos8

commit f457bd15b9ce0be512ec96abcb9b858b0bab2ea8
Author: Scott Little <email address hidden>
Date: Fri Apr 16 01:39:17 2021 -0400

    Improved branching tools

    create_branches_and_tags.sh:
    - Update the .gitreview files in branched git repos.
    - When updating a manifest, add the ability to update and
      use the default revision field.
    - Create two levels of manifest lockdown, soft and hard.
      Soft lockdown only sets sha revisions on unbranched projects
      that lack a revision, or set the revision to master.
      Hard lockdown applies to all unbranched projects.

    push_branches_tags.sh:
    - opendev no longer accepts 'git push' for the delivery of
      new branches with updates. Instead we must now
      use separate commands to deliver the tag, the branch,
      and any updates.

    Closes-Bug: 1924762
    Signed-off-by: Scott Little <email address hidden>
    Change-Id: I6d669ddc80cc9b3cb9e72d65a64589dbccf43ae3

commit 0babd33b6d851dd11492c26e92c8a6ac2c2557de
Author: Rafael Jardim <email address hidden>
Date: Wed May 12 11:25:54 2021 -0300

    Update stx-platformclients tag to stx.5.0-v1.4.3

    This commit updates the image with the updated clients.

    Test:
    Some normal commands
    Commands related with https dcmanager that wasn't working
    System application-upload that wasn't working when executed
    from remote cli

    Closes-Bug: 1928233
    Closes-Bug: 1928231
    Signed-off-by: Rafael Jardim <email address hidden>
    Change-Id: I8a0d12f699336a4412be5ff3c73cfb8d59038780

commit a163d7723e659e89c37ec933d1b0f9aa638a6a73
Author: Cole Walker <email address hidden>
Date: Wed May 5 09:39:17 2021 -0400

    Update image tag for notificationservice-base

    Update image tag to stx.5.0-v1.0.4 for notificationservice-base

    Closes-Bug: 1924201
    Closes-Bug: 1924197

    Signed-off-by: Cole Walker <email address hidden>
    Change-Id: Id863c4f154cc0b39e30ee8986fc07f9856a22826

commit 84c45f5e3a241237887af9db89d8c0aa1f8923e0
Author: Charles Short <email address hidden>
Date: Sun May 2 12:04:04 2021 -0400

    Fix wheels tarball generation

    In d7c5a54ab94bce6635b83d91a807d28f97836a81, django was dropped in favor
    of rfc3986. However the wheel was not added to the build-wheels
    generation which breaks the docker images. Also add the migrate wheel
    since it was mising as well. Add the required wheels in order to
    build the docker image properly.

    Test:
    - Build new centos-stable-wheels tarball
    - Build stx-keystone-api-proxy container

    Closes-Bug: 1926795

    Signed-off-by: Charles Short <email address hidden>
    Change-Id: Ib6f0abfdcc82ca14f92ebc5b45fe8df961e804ee

commit 062353b6de7d4b0e017203c3e6086891bd6b9213
Author: Cole Walker <email address hidden>
Date: Mon May 3 09:48:54 2021 -0400

    Update image tag for notificationclient-base...