StarlingX

CENGN build failure, stx-nova and stx-neutron images

Bug #1863957 reported by Scott Little on 2020-02-19

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	StarlingX	Fix Released	Critical	Don Penney

Bug Description

Brief Description
-----------------
CENGN build of docker images failed on stx-neutron and stx-nova

Severity
--------
Major: unable to build new docker images

Steps to Reproduce
------------------
docker build /localdisk/loadbuild/jenkins/master/20200219T150016Z/std/build-images/loci --no-cache --build-arg PROJECT=nova --build-arg PROJECT_REPO=https://opendev.org/openstack/nova.git --build-arg FROM=docker.io/starlingx/stx-centos:master-stable-20200219T150016Z --build-arg WHEELS=http://build.starlingx.cengn.ca:80//mirror/starlingx/master/centos/20200219T150016Z/outputs/wheels/stx-centos-stable-wheels.tar --build-arg PROJECT_REF=stable/train --build-arg PIP_PACKAGES=pycrypto httplib2 pylint python-ironicclient --build-arg DIST_PACKAGES=openssh-clients openssh-server libvirt e2fsprogs --build-arg PROFILES=fluent nova ceph linuxbridge openvswitch configdrive qemu apache --tag jenkins/stx-nova:master-centos-stable-build

Expected Behavior
------------------
successful image build

Actual Behavior
----------------
fail to build the docker image

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
20200219T150016Z

Last Pass
---------
20200204T162546Z

Timestamp/Logs
--------------

03:00:42.174 Downloading contextlib2-0.5.5-py2.py3-none-any.whl (8.1 kB)
03:00:42.244 [91mERROR: Could not find a version that satisfies the requirement ovsdbapp===0.17.2 (from -c /tmp/wheels/upper-constraints.txt (line 619)) (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.4.4, 0.5.0, 0.6.0, 0.7.0, 0.8.0, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.10.2, 0.10.3, 0.10.4, 0.10.5, 0.11.0, 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.12.4, 0.12.5, 0.14.0, 0.15.0, 0.16.0, 0.17.0, 0.17.1, 0.18.0)

Test Activity
-------------
Build

Workaround
----------
From Don Penney ...

The image build failures are due to openstack constraint updates pointing to a newer python3-only version of the module. Since our images are currently python2, this fails to install. We can push a temporary update to use a previous constraint file to enable the builds, and then work on migrating the images to python3.

Tags:

Don Penney (dpenney) on 2020-02-19

Changed in starlingx:
assignee:	nobody → Don Penney (dpenney)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-02-19: Fix proposed to root (master)

Fix proposed to branch: master
Review: https://review.opendev.org/708766

Changed in starlingx:
status:	New → In Progress

Ghada Khalil (gkhalil) on 2020-02-20

tags:

added: stx.build stx.containers

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-02-21: Fix merged to root (master)

Reviewed: https://review.opendev.org/708766
Committed: https://git.openstack.org/cgit/starlingx/root/commit/?id=b1532a41dacaa4b08fbf0d378adf51e17705c32c
Submitter: Zuul
Branch: master

commit b1532a41dacaa4b08fbf0d378adf51e17705c32c
Author: Don Penney <email address hidden>
Date: Wed Feb 19 18:44:04 2020 -0500

Lock down upper-constraints for wheels

    Currently, the images built via loci are python2-based. Given that
    support for python2 has ended, most new module versions in pypi are
    python3-only. As a result, these new versions will fail to install in
    the image during the build.

    A recent change to the upstream upper-constraints upversioned the
    ovsdbapp module to a python3-only version, which caused the stx-nova
    and stx-neutron image builds to fail. As a temporary workaround until
    we can move these images to python3, this update locks down the
    upper-constraints and global-requirements files to a version prior to
    this change.

    Change-Id: Iaf0e9c76c96b21c398345dfc6b0b8715e978e879
    Closes-Bug: 1863957
    Signed-off-by: Don Penney <email address hidden>

Changed in starlingx:
status:	In Progress → Fix Released

Ghada Khalil (gkhalil) on 2020-02-21

Changed in starlingx:
importance:	Undecided → Critical
tags:	added: stx.4.0

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2020-02-22:

stx.4.0 / critical as this results in a build failure of the images.

As per my discussion with Don, this will likely be an issue if we attempt to build the stx.3.0 docker images for an upcoming maintenance release.

tags:

added: stx.3.0

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-03-31: Fix proposed to root (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/716131

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-03-31: Fix merged to root (f/centos8)

Download full text (11.6 KiB)

Reviewed: https://review.opendev.org/716131
Committed: https://git.openstack.org/cgit/starlingx/root/commit/?id=e5faf8bd5cebfa275823de8cc10be1abbdc2f8f8
Submitter: Zuul
Branch: f/centos8

commit b31cd4b01ee0e6c02cc35094b17aef0edd4f6071
Author: Don Penney <email address hidden>
Date: Wed Mar 25 15:19:07 2020 -0400

Update tag for stx-oidc-client to stx.4.0-v1.0.2

    Change-Id: I6deab3e81b2210968c93f82d667003e5805a1136
    Partial-Bug: 1868895
    Signed-off-by: Don Penney <email address hidden>

commit bd0d9015fa095948443e29f40a701bc3f81c328f
Author: Don Penney <email address hidden>
Date: Wed Mar 18 10:56:08 2020 -0400

Add managed tag for stx-platformclients image

    Change-Id: Id37f2eaf5bacaad941a7abaf835d3c7204d7ca97
    Story: 2006711
    Task: 39096
    Signed-off-by: Don Penney <email address hidden>

commit f5985aa83ad6c5275f97db53db9a095a1f4a1bb1
Author: Davlet Panech <email address hidden>
Date: Wed Mar 18 16:29:40 2020 -0400

Use private yum cache dir in build scripts

    YUM & friends use a cache directory named /tmp/yum-$USER-xxx or
    similar, even when used concurrently e.g. when bulding unrelated
    source trees. This causes interference between CI pipelines.

This patch sets TMPDIR=$MY_WORKSPACE/tmp when calling repoquery,
etc.

    Change-Id: Ieeb4d6fd9447f1c2988380c3427975893be365a5
    Closes-Bug: 1867817
    Signed-off-by: Davlet Panech <email address hidden>

commit 7dc6b23b517d4d14cad38f29d47af64e66e105b9
Author: Stefan Dinescu <email address hidden>
Date: Wed Mar 18 16:23:59 2020 +0200

Support custom docker registries for remote cli

    The build script for remote cli was receiving as parameters
    only the desired tag for the clients docker images; the docker
    images were always pulled from docker.io/starlinx.

    Nowthe build script take the full link of the docker image
    as parameter, so we can use any docker registry as source
    for the client images.

    Change-Id: Ibc4584a9401c1ff30f8e657ca27d31b35c4e94d5
    Depends-On: https://review.opendev.org/#/c/713665/
    Story: 2006711
    Task: 39095
    Signed-off-by: Stefan Dinescu <email address hidden>

commit 3b0e2dd4ea92b4fe5b53b1fd4b38a5e673e9c92c
Author: Davlet Panech <email address hidden>
Date: Mon Mar 9 14:54:10 2020 -0400

Fixed broken setuptools import

    pyScss tries to import a module called "Feature" from setuptools,
    which was removed from setuptools 46.0.0 (mar 8 2020). This causes
    compilation erros. The fix updates "pip install" in Dockerfile
    to install setuptools < 46.0.0

    Change-Id: Ib5c00aafb934efaf1413e72ede30638f2dc35230
    Signed-off-by: Davlet Panech <email address hidden>
    Closes-Bug: 1866699

commit 3a95ab163f4cb4371fa4c6a94f5f64e27b1980f2
Author: Don Penney <email address hidden>
Date: Wed Mar 4 10:59:37 2020 -0500

Add tag for rvmc image

Update tag management file to publish a static tag for the rvmc image.

    Change-Id: I459e4f676ffa0a28c2c1fca5d61df16f499ded93
    Story: 2006980
    Task: 38935
    Signed-off-by: Don Penney <don.penney...

Reviewed:  https://review.opendev.org/716131
Committed: https://git.openstack.org/cgit/starlingx/root/commit/?id=e5faf8bd5cebfa275823de8cc10be1abbdc2f8f8
Submitter: Zuul
Branch:    f/centos8

commit b31cd4b01ee0e6c02cc35094b17aef0edd4f6071
Author: Don Penney <don.penney@windriver.com>
Date:   Wed Mar 25 15:19:07 2020 -0400

Update tag for stx-oidc-client to stx.4.0-v1.0.2
    
    Change-Id: I6deab3e81b2210968c93f82d667003e5805a1136
    Partial-Bug: 1868895
    Signed-off-by: Don Penney <don.penney@windriver.com>

commit bd0d9015fa095948443e29f40a701bc3f81c328f
Author: Don Penney <don.penney@windriver.com>
Date:   Wed Mar 18 10:56:08 2020 -0400

Add managed tag for stx-platformclients image
    
    Change-Id: Id37f2eaf5bacaad941a7abaf835d3c7204d7ca97
    Story: 2006711
    Task: 39096
    Signed-off-by: Don Penney <don.penney@windriver.com>

commit f5985aa83ad6c5275f97db53db9a095a1f4a1bb1
Author: Davlet Panech <davlet.panech@windriver.com>
Date:   Wed Mar 18 16:29:40 2020 -0400

Use private yum cache dir in build scripts
    
    YUM & friends use a cache directory named /tmp/yum-$USER-xxx or
    similar, even when used concurrently e.g. when bulding unrelated
    source trees. This causes interference between CI pipelines.
    
    This patch sets TMPDIR=$MY_WORKSPACE/tmp when calling repoquery,
    etc.
    
    Change-Id: Ieeb4d6fd9447f1c2988380c3427975893be365a5
    Closes-Bug: 1867817
    Signed-off-by: Davlet Panech <davlet.panech@windriver.com>

commit 7dc6b23b517d4d14cad38f29d47af64e66e105b9
Author: Stefan Dinescu <stefan.dinescu@windriver.com>
Date:   Wed Mar 18 16:23:59 2020 +0200

Support custom docker registries for remote cli
    
    The build script for remote cli was receiving as parameters
    only the desired tag for the clients docker images; the docker
    images were always pulled from docker.io/starlinx.
    
    Nowthe build script take the full link of the docker image
    as parameter, so we can use any docker registry as source
    for the client images.
    
    Change-Id: Ibc4584a9401c1ff30f8e657ca27d31b35c4e94d5
    Depends-On: https://review.opendev.org/#/c/713665/
    Story: 2006711
    Task: 39095
    Signed-off-by: Stefan Dinescu <stefan.dinescu@windriver.com>

commit 3b0e2dd4ea92b4fe5b53b1fd4b38a5e673e9c92c
Author: Davlet Panech <davlet.panech@windriver.com>
Date:   Mon Mar 9 14:54:10 2020 -0400

Fixed broken setuptools import
    
    pyScss tries to import a module called "Feature" from setuptools,
    which was removed from setuptools 46.0.0 (mar 8 2020). This causes
    compilation erros. The fix updates "pip install" in Dockerfile
    to install setuptools < 46.0.0
    
    Change-Id: Ib5c00aafb934efaf1413e72ede30638f2dc35230
    Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
    Closes-Bug: 1866699

commit 3a95ab163f4cb4371fa4c6a94f5f64e27b1980f2
Author: Don Penney <don.penney@windriver.com>
Date:   Wed Mar 4 10:59:37 2020 -0500

Add tag for rvmc image
    
    Update tag management file to publish a static tag for the rvmc image.
    
    Change-Id: I459e4f676ffa0a28c2c1fca5d61df16f499ded93
    Story: 2006980
    Task: 38935
    Signed-off-by: Don Penney <don.penney@windriver.com>

commit 11ae91f2d85cd7cd9ab07d6595261b249df94855
Author: Don Penney <don.penney@windriver.com>
Date:   Sat Feb 22 08:33:35 2020 -0500

Update image tags for stx-oidc-client and n3000-opae
    
    Update image tags for the stx-oidc-client and n3000-opae images to
    point to new image versions.
    
    Change-Id: I15c8854c335911567c2ad5f38eedb95c0ba2ee91
    Story: 2006711
    Task: 38850
    Signed-off-by: Don Penney <don.penney@windriver.com>

commit 9b77e2e1b346c7f2f6fe617bc9c48e3c9c39d6de
Author: Scott Little <scott.little@windriver.com>
Date:   Mon Oct 28 19:22:57 2019 -0400

Build avoidance fixes
    
    1) Build avoidance fails on a clean workspace because directory
    $MY_WORKSPACE/tmp doesn't exist when mktemp needs to
    create a forth subdir under that path.
    
    Add a 'mkdir -p $MY_WORKSPACE/tmp' to ensure it exists
    prior to mktemp.
    
    2) Failed to rebuild kmods after a kernel change and
    building with build-avoidance enabled.  The problem was
    a missing dependency cache that would have listed
    the kmods as having dependencies on the kernel.
    Reusing kmods from a prior build in conjunction with
    a rebuilt kernel results int load with unusable kmods
    due to a key mismatch.
    
    Add the dependancy-cache to the list of artifacts to
    syncronize as part of build avoidance.  A primed
    dependency cache will allow build-pkgs to determine
    that kmods must be rebuilt when the kernel has changed.
    
    3) Make sure $MY_WORKSPACE/$BUILD_TYPE/repo
    symlink exists after a build avoidance build.
    If no packages require a rebuild, it was missing.
    
    4) Fix an error message that was puzzling while debugging
    the above issues.
    
    Closes-Bug: 1864110
    Change-Id: Ia912b2e3e110efbe69004d7978109ebb089d684b
    Signed-off-by: Scott Little <scott.little@windriver.com>

commit b1532a41dacaa4b08fbf0d378adf51e17705c32c
Author: Don Penney <don.penney@windriver.com>
Date:   Wed Feb 19 18:44:04 2020 -0500

Lock down upper-constraints for wheels
    
    Currently, the images built via loci are python2-based. Given that
    support for python2 has ended, most new module versions in pypi are
    python3-only. As a result, these new versions will fail to install in
    the image during the build.
    
    A recent change to the upstream upper-constraints upversioned the
    ovsdbapp module to a python3-only version, which caused the stx-nova
    and stx-neutron image builds to fail. As a temporary workaround until
    we can move these images to python3, this update locks down the
    upper-constraints and global-requirements files to a version prior to
    this change.
    
    Change-Id: Iaf0e9c76c96b21c398345dfc6b0b8715e978e879
    Closes-Bug: 1863957
    Signed-off-by: Don Penney <don.penney@windriver.com>

commit 921cb526ffd9fe9d6dddcf7253bf0812d7299350
Author: Don Penney <don.penney@windriver.com>
Date:   Wed Feb 12 14:51:52 2020 -0500

Add tag-management utility
    
    This commit introduces a tag management utility for images to allow
    retagging of specific images. This allows the StarlingX ansible
    playbooks and packaged charts to use a static image tag rather than the
    moving stable-latest tag.
    
    This utility can only be run by a StarlingX build team member with
    publishing capability on the docker hub, to allow for updated tags to
    be available publicly, or as part of a CENGN build job.
    
    Change-Id: I7659695c701ccf3bfe8255b24ad0b1896b59cc23
    Partial-Bug: 1854869
    Signed-off-by: Don Penney <don.penney@windriver.com>

commit d02fd0c39e6c9ee12afffd7fee2a8ebfbaf23e03
Author: Don Penney <don.penney@windriver.com>
Date:   Wed Feb 12 14:38:59 2020 -0500

Fix patch-iso for StarlingX repos
    
    The patch-iso utility is currently unable to find necessary files
    within the StarlingX repos. This commits corrects the location,
    allowing the utility to run. It also does a little cleanup to add
    robustness to the utility.
    
    Depends-On: https://review.opendev.org/707459
    Change-Id: Ibacdfc7f06a464c7ef62e0a681f4f99e634057f2
    Closes-Bug: 1863011
    Signed-off-by: Don Penney <don.penney@windriver.com>

commit d184c69b7c6c0f2cf93facf888182670211460ad
Author: Don Penney <don.penney@windriver.com>
Date:   Wed Jan 15 14:22:45 2020 -0500

Update tox.ini bashate to test all bash scripts
    
    Currently, the tox.ini file is only running bashate against files that
    end in a .sh extension. However, many of the build tools do not have
    an extension, and therefore don't have bashate executed against them.
    This commit enhances the tox command to search for shell scripts
    regardless of file extension.
    
    The bashate ignore list has been updated to ignore all existing
    warnings. Future updates can perform cleanup to address the
    outstanding issues and remove items from the ignore list one by one.
    
    The original check of .sh is maintained with a smaller ignore list, to
    ensure new issues are not introduced.
    
    Change-Id: I07e8ac83c6ed3d81e7bfad19da1eee1747d60802
    Signed-off-by: Don Penney <don.penney@windriver.com>

commit b2286535c9270ca728a0ed5e053392e2961b9373
Author: Scott Little <scott.little@windriver.com>
Date:   Tue Dec 31 15:46:45 2019 -0500

Update to build tools to support Build layering.
    
    1: Building rpms now produces an rpm.lst file.  This file serves to
    list all rpms produced by the build.  A seperate rpm.lst file
    is producede for each build type (std, rt, installer). The file is
    co-resident with the matching repodata directory.
    
    The rpm.lst files will need to be published by cengn for each layer build.
    
    The download tools in stx-tools will need to find the rpm.lst files
    of lower layer builds, and use it to direct the download of rpms
    from the lower layer.
    
    2: Building rpms now produces an image.inc file.  This file serves
    to list all rpms that the layer recommends be built into an iso.
    The file is stored under the $MY_WORKSPACE/<build-type> subdirectory,
    although it has identical content for any build-type.
    
    The image.inc file will need to be published by cengn for each layer
    build.
    
    The download tools in stx-tools will need to download the per-layer
    image.inc files to the $MY_REPO/cgcs-centos-repo/layer_image_inc/
    sub-directory, renaming the file in some layer specific way.
    
    The build-iso tool will process any *.inc files found under
    $MY_REPO/cgcs-centos-repo/layer_image_inc/ ,
    adding those rpms to the iso.
    
    3) Add a mechanism for layer specific mock.cfg prototypes.
    This will allow special handling of the real-time (aka 'rt'),
    repositories for layer builds that need to do a 'rt' build.
    
    4) Add support for a $MY_REPO/cgcs-centos-repo/rt subdirectory,
    a place to build a repo of real-time rpms originating from
    lower layer builds.
    
    The download tools in stx-tools will need to populate the new
    rt repos.
    
    As of this writing, non-rt rpms remain in $MY_REPO/cgcs-centos-repo/.
    i.e. there is not a new $MY_REPO/cgcs-centos-repo/std/ directory.
    
    5) Some changes to make us more flexible about where we find
    realease and bsp files.
    
    6) Found that kernel mudules were not reliably building against
    the hearnel-headers of our modified kernel.  Found that adding '--update'
    to our mock build command was not working.  Does mock expect '--update'
    to only be used independently of a build command?  It does work when
    used in that manner, so that's what we will do.
    
    7) The build-pkgs, build-srpms, build-rpms family of commands
    can take a layer argument and/or will read the LAYER environment
    variable.  Current the only use of this variable is to modify
    the build-info.  It does NOT limit the compile to packages
    for a specific layer.
    
    Story: 2006166
    Task: 37094
    
    Depends-On: https://review.opendev.org/698756
    Depends-On: https://review.opendev.org/700819
    Change-Id: I817e08a19cdabe08b3fcc47dee63a36b461c13c0
    Co-Authored-by: Martin Chen <haochuan.z.chen@intel.com>
    Signed-off-by: Scott Little <scott.little@windriver.com>

commit 508b15d3c42e95e3ce60320960f40b0f0c643ed7
Author: Bin Qian <bin.qian@windriver.com>
Date:   Thu Feb 6 15:00:45 2020 -0500

Adding job to upload commits to GitHub
    
    Add job to publish containers repo to GitHub
    
    Change-Id: I6b500881aaaddddf548f9121166ecd42cd3d5f61
    Story: 2007252
    Task: 38692
    Signed-off-by: Bin Qian <bin.qian@windriver.com>

tags:

added: in-f-centos8

Revision history for this message

Bill Zvonar (billzvonar) wrote on 2020-08-13:

Hi Don - this looks like it needs to be cherry-picked to r/stx.3.0 - can you do, or say why it's not required.

tags:

added: stx.cherrypickneeded

Revision history for this message

Don Penney (dpenney) wrote on 2020-08-14:

I rebased a r/stx.3.0 clone and did a full build. I then did a build of the stx-centos base, built the wheels tarball, and then successfully built the images

The following images were built:
dpenney/k8s-cni-sriov:dpenneytest3.0-centos-stable-build
dpenney/k8s-plugins-sriov-network-device:dpenneytest3.0-centos-stable-build
dpenney/stx-aodh:dpenneytest3.0-centos-stable-build
dpenney/stx-barbican:dpenneytest3.0-centos-stable-build
dpenney/stx-ceilometer:dpenneytest3.0-centos-stable-build
dpenney/stx-cinder:dpenneytest3.0-centos-stable-build
dpenney/stx-fm-rest-api:dpenneytest3.0-centos-stable-build
dpenney/stx-glance:dpenneytest3.0-centos-stable-build
dpenney/stx-gnocchi:dpenneytest3.0-centos-stable-build
dpenney/stx-heat:dpenneytest3.0-centos-stable-build
dpenney/stx-horizon:dpenneytest3.0-centos-stable-build
dpenney/stx-ironic:dpenneytest3.0-centos-stable-build
dpenney/stx-keystone-api-proxy:dpenneytest3.0-centos-stable-build
dpenney/stx-keystone:dpenneytest3.0-centos-stable-build
dpenney/stx-libvirt:dpenneytest3.0-centos-stable-build
dpenney/stx-mariadb:dpenneytest3.0-centos-stable-build
dpenney/stx-neutron:dpenneytest3.0-centos-stable-build
dpenney/stx-nova-api-proxy:dpenneytest3.0-centos-stable-build
dpenney/stx-nova:dpenneytest3.0-centos-stable-build
dpenney/stx-openstackclients:dpenneytest3.0-centos-stable-build
dpenney/stx-ovs:dpenneytest3.0-centos-stable-build
dpenney/stx-panko:dpenneytest3.0-centos-stable-build
dpenney/stx-placement:dpenneytest3.0-centos-stable-build
dpenney/stx-platformclients:dpenneytest3.0-centos-stable-build

We had already locked down all wheels for r/stx.3.0, insulating the images from upstream changes to python3 modules:
https://bugs.launchpad.net/starlingx/+bug/1887178
https://opendev.org/starlingx/root/commit/e31f5b26f95c43254fd9e593f097c4c7d4b7c6b6

Revision history for this message

Bill Zvonar (billzvonar) wrote on 2020-08-14:

Sounds good to me, thanks.

tags:

removed: stx.cherrypickneeded

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.