Container build failures Jan 21 2022

[Scott Little]

Brief Description
-----------------
Five containers failed to build during Jan 21, 2022 build

stx-fm-rest-api
stx-keystone-api-proxy
stx-nova-api-proxy
stx-pci-irq-affinity-agent
stx-platformclients

Suspect cgtsclient was upversioned, but a matching change to the constrain files of the clients was missed.

Severity
--------
Critical

Steps to Reproduce
------------------
build-stx-images.sh --prefix master --latest-prefix master --os centos --stream stable --version 20220121T145521Z --base docker.io/starlingx/stx-centos:master-stable-20220121T145521Z --wheels http://mirror.starlingx.cengn.ca:80//mirror/starlingx/master/centos/containers/20220121T145521Z/outputs/wheels/stx-centos-stable-wheels.tar --user starlingx --registry docker.io

Expected Behavior
------------------
All containers build

Actual Behavior
----------------
The following container fail to build:
stx-fm-rest-api
stx-keystone-api-proxy
stx-nova-api-proxy
stx-pci-irq-affinity-agent
stx-platformclients

Reproducibility
---------------
100%

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
2022-01-21 14:55:21Z

Last Pass
---------
2022-01-17

Timestamp/Logs
--------------
pip install --no-cache-dir --only-binary :all: --no-compile -c /tmp/wheels/upper-constraints.txt --find-links /tmp/wheels/ bindep cgtsclient fm fm_api fm_core httplib2 keystone keystonemiddleware oslo.db oslo.log oslo.middleware oslo.policy oslo.versionedobjects PyMySQL python_neutronclient pecan tsconfig WSME

Looking in links: /tmp/wheels/
Collecting bindep
  Downloading bindep-2.10.1-py2.py3-none-any.whl (33 kB)
ERROR: Cannot install cgtsclient because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested cgtsclient
    The user requested (constraint) cgtsclient===1.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
WARNING: You are using pip version 21.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/var/lib/openstack/bin/python -m pip install --upgrade pip' command.
The command '/bin/sh -c /opt/loci/scripts/install.sh' returned a non-zero code: 1

additional logs are at ...

http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/20220121T145521Z/logs/

Test Activity
-------------
Build

Workaround
----------
N/A

[Davlet Panech (dpanech)]

Looks like pip & setuptools changed since last successful build.

[Al Bailey (albailey1974)]

Comparing two builds
 - a previous build which failed differently
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/20220120T050921Z/logs/jenkins-STX_build_docker_flock_images-484.log.html

and the one this bug is raised for:
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/20220121T145521Z/logs/jenkins-STX_build_docker_flock_images-485.log.html

The Jan 20 fail does not report the 'conflicts' but clones a much older version of loci.
See this in the logs:
Cloning into 'loci'...
Note: checking out 'f022ecba553903df3df72d3668e143e9eb9ceded'.

that version of loci is a commit dated April 2020
https://github.com/openstack/loci/commit/f022ecba553903df3df72d3668e143e9eb9ceded

The load this is tagged against has the 'conflicts' and uses a newer loci.
Cloning into 'loci'...
Note: checking out 'efccd0a853879ac6af6066eda09792d0d3afe9c0'.

that loci is dated Jan 12 2022
https://github.com/openstack/loci/commit/efccd0a853879ac6af6066eda09792d0d3afe9c0

I do not know why loci is now moved to 'HEAD' instead of whatever that old version was locked to.
There are supposed to be fixes for virtualenv, pip, setuptools already locked in loci, but apparently we stayed locked to the old version up until now.

[Davlet Panech (dpanech)]

Loci was upversioned here: https://review.opendev.org/c/starlingx/root/+/823578

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/utilities/+/826136

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/upstream/+/826137

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/nfv/+/826138

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/fault/+/826139

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/distcloud/+/826140

[Thiago Paiva Brito (outbrito)]

This seems related to the fact that since https://github.com/openstack/loci/commit/a2a88b8ffbc85683bd5be71ebb9de55289800d7a (that was merged prior to the upversion), the default python version for LOCI is python3, and these images doesn't specify the python version, relying on the old behavior . I'm doing some manual builds to make sure we can build all those images with the fixes above.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/root/+/826155

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/root/+/826155
Committed: https://opendev.org/starlingx/root/commit/dba441d29df7ef82ac3dad47d49d2af9096fec77
Submitter: "Zuul (22348)"
Branch: master

commit dba441d29df7ef82ac3dad47d49d2af9096fec77
Author: Thiago Brito <email address hidden>
Date: Mon Jan 24 17:34:42 2022 -0300

    Fixing default behavior of PYTHON3 for LOCI call

    Recently, LOCI changed its default so images are python3 if nothing is
    specified at the *docker_image files. With the recent upversion of LOCI
    in the build tools [1], we need to explictly set PYTHON3=no to images
    that are still using python2. As such, we need to pass WHEELS when no
    PYTHON3 is provided and when PYTHON3 == "yes" (or != "no").

    TEST PLAN
    PASS Build image with PYTHON3=no (stx-pci-irq-affinity-agent)
    PASS Build image with PYTHON3=yes (stx-cinder, modified manually)
    PASS Build image with no PYTHON3 (stx-nova)

    [1] https://opendev.org/starlingx/root/commit/00bd632a68b8172f5d4baec2b1e4f8a08b9d5be5

    Closes-Bug: #1958696
    Signed-off-by: Thiago Brito <email address hidden>
    Change-Id: Iec52b57e36da968d6076361b1b434234ef55d7e8

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/upstream/+/826137
Committed: https://opendev.org/starlingx/upstream/commit/f1757b9f11f1d78135420bb8e8e1a0fb40b83e24
Submitter: "Zuul (22348)"
Branch: master

commit f1757b9f11f1d78135420bb8e8e1a0fb40b83e24
Author: Thiago Brito <email address hidden>
Date: Mon Jan 24 14:49:20 2022 -0300

    Setting PYTHON3=no for LOCI images

    Recently, LOCI changed its default so images are python3 if nothing is
    specified at the *docker_image files. With the recent upversion of LOCI
    in the build tools [1], we need to explictly set PYTHON3=no to images
    that are still using python2. This commit fixes the StarlingX images to
    use this parameter.

    This will also remove all PYTHON3=yes since it's the detault now to
    reduce uneeded LOCs and ease maintenance.

    TEST PLAN
    PASS build the stx-nova on the stable branch
    PASS build the stx-openstackclients on the stable branch
    PASS build the stx-platformclients on the stable branch

    [1] https://opendev.org/starlingx/root/commit/00bd632a68b8172f5d4baec2b1e4f8a08b9d5be5

    Closes-Bug: #1958696
    Depends-On: https://review.opendev.org/c/starlingx/root/+/826155
    Signed-off-by: Thiago Brito <email address hidden>
    Change-Id: Idffec6b57371326e8667ac4b16ac9b11f22afe1a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/utilities/+/826136
Committed: https://opendev.org/starlingx/utilities/commit/150809907357ef70dc710e8ff3d12637db661e86
Submitter: "Zuul (22348)"
Branch: master

commit 150809907357ef70dc710e8ff3d12637db661e86
Author: Thiago Brito <email address hidden>
Date: Mon Jan 24 14:44:07 2022 -0300

    Setting PYTHON3=no for LOCI images

    Recently, LOCI changed its default so images are python3 if nothing is
    specified at the *docker_image files. With the recent upversion of LOCI
    in the build tools [1], we need to explictly set PYTHON3=no to images
    that are still using python2. This commit fixes the StarlingX images to
    use this parameter

    TEST PLAN
    PASS build the stx-pci-irq-affinity-agent on the stable branch

    [1] https://opendev.org/starlingx/root/commit/00bd632a68b8172f5d4baec2b1e4f8a08b9d5be5

    Closes-Bug: #1958696
    Signed-off-by: Thiago Brito <email address hidden>
    Change-Id: I9faae4ae3fa52b42e0a7392a793296d912507f8e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/fault/+/826139
Committed: https://opendev.org/starlingx/fault/commit/e0fba90bdfcae3e613922c6b45cc0eeacd71d7d7
Submitter: "Zuul (22348)"
Branch: master

commit e0fba90bdfcae3e613922c6b45cc0eeacd71d7d7
Author: Thiago Brito <email address hidden>
Date: Mon Jan 24 14:54:47 2022 -0300

    Setting PYTHON3=no for LOCI images

    Recently, LOCI changed its default so images are python3 if nothing is
    specified at the *docker_image files. With the recent upversion of LOCI
    in the build tools [1], we need to explictly set PYTHON3=no to images
    that are still using python2. This commit fixes the StarlingX images to
    use this parameter

    TEST PLAN
    PASS build the stx-fm-rest-api image on the stable branch

    [1] https://opendev.org/starlingx/root/commit/00bd632a68b8172f5d4baec2b1e4f8a08b9d5be5

    Closes-Bug: #1958696

    Signed-off-by: Thiago Brito <email address hidden>
    Change-Id: Icc80f5ca3f2031537e45ab8ff35420250522cb34

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/distcloud/+/826140
Committed: https://opendev.org/starlingx/distcloud/commit/59e1ae68ac8adc6ec3e967561f31f9cd0430e1bf
Submitter: "Zuul (22348)"
Branch: master

commit 59e1ae68ac8adc6ec3e967561f31f9cd0430e1bf
Author: Thiago Brito <email address hidden>
Date: Mon Jan 24 14:55:45 2022 -0300

    Setting PYTHON3=no for LOCI images

    Recently, LOCI changed its default so images are python3 if nothing is
    specified at the *docker_image files. With the recent upversion of LOCI
    in the build tools [1], we need to explictly set PYTHON3=no to images
    that are still using python2. This commit fixes the StarlingX images to
    use this parameter

    TEST PLAN
    PASS build the stx-keystone-api-proxy on the stable branch

    [1] https://opendev.org/starlingx/root/commit/00bd632a68b8172f5d4baec2b1e4f8a08b9d5be5

    Closes-Bug: #1958696

    Signed-off-by: Thiago Brito <email address hidden>
    Change-Id: I7847b4e6c4e8868aaf56b73bc6f056e858f92784

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/nfv/+/826138
Committed: https://opendev.org/starlingx/nfv/commit/cb69ac65900642f96065a7dbba4804657e2fe8ca
Submitter: "Zuul (22348)"
Branch: master

commit cb69ac65900642f96065a7dbba4804657e2fe8ca
Author: Thiago Brito <email address hidden>
Date: Mon Jan 24 14:52:17 2022 -0300

    Setting PYTHON3=no for LOCI images

    Recently, LOCI changed its default so images are python3 if nothing is
    specified at the *docker_image files. With the recent upversion of LOCI
    in the build tools [1], we need to explictly set PYTHON3=no to images
    that are still using python2. This commit fixes the StarlingX images to
    use this parameter

    TEST PLAN
    PASS build the stx-nova-api-proxy image on the stable branch

    [1] https://opendev.org/starlingx/root/commit/00bd632a68b8172f5d4baec2b1e4f8a08b9d5be5

    Closes-Bug: #1958696

    Signed-off-by: Thiago Brito <email address hidden>
    Change-Id: I3b0a1e1d4b64527ce03192fb0f38bb7eba0d05c0