yum cache corruption causes build-iso to fail

Bug #1867817 reported by Scott Little
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
High
Davlet Panech

Bug Description

Brief Description
-----------------
build-iso fails with

18:08:17 Not using downloaded local-std/repomd.xml because it is older than what we have:
18:08:17 Current : Sat Mar 14 18:07:32 2020
18:08:17 Downloaded: Sat Mar 14 18:07:08 2020
18:08:17 rpm2cpio: ////localdisk/loadbuild/jenkins/WRCP_20.01_PRESTAGING_Build/2020-03-14_16-00-00/std/rpmbuild/RPMS/platform-kickstarts-1.0.0-5.tis.noarch.rpm: No such file or directory

Severity
--------
Major: unable to build an iso

Steps to Reproduce
------------------
A little unclear ... here is what I know
- At minimum two different projects must build in the same environment i.e. docker instance.
- The two projects must not agree on the version of platform-kickstarts.
e.g. platform-kickstarts-1.0.0-3.tis.noarch.rpm vs platform-kickstarts-1.0.0-5.tis.noarch.rpm
- Alternate running build-pkgs and build-iso in the two different projects
- At some point the yum cache of one project contaminated the other, and build-iso starts looking for package versions only available in the other.

Expected Behavior
------------------
build-iso passes for both projects.

Actual Behavior
----------------
build-iso fails.

Reproducibility
---------------
Intermittent: In terms of inducing the initial yum cache contamination. After contamination, build-iso fails consistently.

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
2020-03-14_16-00-00

Last Pass
---------
2020-03-07_16-00-00

Timestamp/Logs
--------------
18:07:14 Installing PKG=platform-kickstarts PKG_FILE=platform-kickstarts-1.0.0-3.tis.noarch.rpm PKG_REL_PATH=platform-kickstarts-1.0.0-3.tis.noarch.rpm PKG_PATH=/localdisk/loadbuild/jenkins/WRCP_20.01_PRESTAGING_Build/2020-03-14_16-00-00/std/rpmbuild/RPMS/platform-kickstarts-1.0.0-3.tis.noarch.rpm from repo local-std
...
18:08:17 Not using downloaded local-std/repomd.xml because it is older than what we have:
18:08:17 Current : Sat Mar 14 18:07:32 2020
18:08:17 Downloaded: Sat Mar 14 18:07:08 2020
18:08:17 rpm2cpio: ////localdisk/loadbuild/jenkins/WRCP_20.01_PRESTAGING_Build/2020-03-14_16-00-00/std/rpmbuild/RPMS/platform-kickstarts-1.0.0-5.tis.noarch.rpm: No such file or directory

Test Activity
-------------
Build

 Workaround
 ----------
Clean yum cache with ...
yum clean all -c $MY_WORKSPACE/yum.conf

Revision history for this message
Ghada Khalil (gkhalil) wrote :

stx.4.0 / high priority - results in build iso failures

tags: added: stx.build
tags: added: stx.4.0
Changed in starlingx:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Scott Little (slittle1)
Revision history for this message
Davlet Panech (dpanech) wrote :

Caused by yum using a shared meta data cache for all concurrent Jenkins builds.

Working on a solution now.

Davlet Panech (dpanech)
Changed in starlingx:
assignee: Scott Little (slittle1) → Davlet Panech (dpanech)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to root (master)

Fix proposed to branch: master
Review: https://review.opendev.org/713736

Changed in starlingx:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to root (master)

Reviewed: https://review.opendev.org/713736
Committed: https://git.openstack.org/cgit/starlingx/root/commit/?id=f5985aa83ad6c5275f97db53db9a095a1f4a1bb1
Submitter: Zuul
Branch: master

commit f5985aa83ad6c5275f97db53db9a095a1f4a1bb1
Author: Davlet Panech <email address hidden>
Date: Wed Mar 18 16:29:40 2020 -0400

    Use private yum cache dir in build scripts

    YUM & friends use a cache directory named /tmp/yum-$USER-xxx or
    similar, even when used concurrently e.g. when bulding unrelated
    source trees. This causes interference between CI pipelines.

    This patch sets TMPDIR=$MY_WORKSPACE/tmp when calling repoquery,
    etc.

    Change-Id: Ieeb4d6fd9447f1c2988380c3427975893be365a5
    Closes-Bug: 1867817
    Signed-off-by: Davlet Panech <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to root (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/716131

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to root (f/centos8)
Download full text (11.6 KiB)

Reviewed: https://review.opendev.org/716131
Committed: https://git.openstack.org/cgit/starlingx/root/commit/?id=e5faf8bd5cebfa275823de8cc10be1abbdc2f8f8
Submitter: Zuul
Branch: f/centos8

commit b31cd4b01ee0e6c02cc35094b17aef0edd4f6071
Author: Don Penney <email address hidden>
Date: Wed Mar 25 15:19:07 2020 -0400

    Update tag for stx-oidc-client to stx.4.0-v1.0.2

    Change-Id: I6deab3e81b2210968c93f82d667003e5805a1136
    Partial-Bug: 1868895
    Signed-off-by: Don Penney <email address hidden>

commit bd0d9015fa095948443e29f40a701bc3f81c328f
Author: Don Penney <email address hidden>
Date: Wed Mar 18 10:56:08 2020 -0400

    Add managed tag for stx-platformclients image

    Change-Id: Id37f2eaf5bacaad941a7abaf835d3c7204d7ca97
    Story: 2006711
    Task: 39096
    Signed-off-by: Don Penney <email address hidden>

commit f5985aa83ad6c5275f97db53db9a095a1f4a1bb1
Author: Davlet Panech <email address hidden>
Date: Wed Mar 18 16:29:40 2020 -0400

    Use private yum cache dir in build scripts

    YUM & friends use a cache directory named /tmp/yum-$USER-xxx or
    similar, even when used concurrently e.g. when bulding unrelated
    source trees. This causes interference between CI pipelines.

    This patch sets TMPDIR=$MY_WORKSPACE/tmp when calling repoquery,
    etc.

    Change-Id: Ieeb4d6fd9447f1c2988380c3427975893be365a5
    Closes-Bug: 1867817
    Signed-off-by: Davlet Panech <email address hidden>

commit 7dc6b23b517d4d14cad38f29d47af64e66e105b9
Author: Stefan Dinescu <email address hidden>
Date: Wed Mar 18 16:23:59 2020 +0200

    Support custom docker registries for remote cli

    The build script for remote cli was receiving as parameters
    only the desired tag for the clients docker images; the docker
    images were always pulled from docker.io/starlinx.

    Nowthe build script take the full link of the docker image
    as parameter, so we can use any docker registry as source
    for the client images.

    Change-Id: Ibc4584a9401c1ff30f8e657ca27d31b35c4e94d5
    Depends-On: https://review.opendev.org/#/c/713665/
    Story: 2006711
    Task: 39095
    Signed-off-by: Stefan Dinescu <email address hidden>

commit 3b0e2dd4ea92b4fe5b53b1fd4b38a5e673e9c92c
Author: Davlet Panech <email address hidden>
Date: Mon Mar 9 14:54:10 2020 -0400

    Fixed broken setuptools import

    pyScss tries to import a module called "Feature" from setuptools,
    which was removed from setuptools 46.0.0 (mar 8 2020). This causes
    compilation erros. The fix updates "pip install" in Dockerfile
    to install setuptools < 46.0.0

    Change-Id: Ib5c00aafb934efaf1413e72ede30638f2dc35230
    Signed-off-by: Davlet Panech <email address hidden>
    Closes-Bug: 1866699

commit 3a95ab163f4cb4371fa4c6a94f5f64e27b1980f2
Author: Don Penney <email address hidden>
Date: Wed Mar 4 10:59:37 2020 -0500

    Add tag for rvmc image

    Update tag management file to publish a static tag for the rvmc image.

    Change-Id: I459e4f676ffa0a28c2c1fca5d61df16f499ded93
    Story: 2006980
    Task: 38935
    Signed-off-by: Don Penney <don.penney...

tags: added: in-f-centos8
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.