periodic master container build fails ImportError: No module named queue

[Marios Andreou]

This blocks the entire centos7 master promotion chain at [1] and [2] the container build fails with

        2020-02-20 08:22:40 | Requirement already up-to-date: decorator in ./venv_build/lib/python2.7/site-packages (4.4.1)
        2020-02-20 08:22:44 | Traceback (most recent call last):
        2020-02-20 08:22:44 | File "/home/zuul/workspace/venv_build/bin/kolla-build", line 5, in <module>
        2020-02-20 08:22:44 | from kolla.cmd.build import main
        2020-02-20 08:22:44 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/kolla/cmd/build.py", line 28, in <module>
        2020-02-20 08:22:44 | from kolla.image import build # noqa
        2020-02-20 08:22:44 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/kolla/image/build.py", line 21, in <module>
        2020-02-20 08:22:44 | import queue
        2020-02-20 08:22:44 | ImportError: No module named queue

NOT SURE IF RELATED: An earlier run failed on a slightly different error not sure if that is related or a different bug/transient... see [3] ("Could not find a version that satisfies the requirement python-heatclient===2.0.0")

[1] https://logserver.rdoproject.org/80/25080/1/check/periodic-tripleo-centos-7-master-containers-build-push/0318cfe/logs/build.log
[2] https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build-push/e2ac434/logs/build.log
[3] https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build-push/66f34b3/job-output.txt

[Marios Andreou (marios-b)]

that merged https://review.opendev.org/#/c/691316/ is why so we have the import error on queue... in py3 this is now queue (previously Queue, I know right?!) - see [1]

We could pin kolla for now posted that https://review.rdoproject.org/r/25083 but ideally it should be handled kolla side. Even more ideally we should be using python3...

****
[1]
[m@192 ~]$ python2
Python 2.7.17 (default, Oct 20 2019, 00:00:00)
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import queue
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named queue
>>> import Queue
>>> exit()
[m@192 ~]$ python3
Python 3.7.6 (default, Jan 30 2020, 09:44:41)
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import queue
>>> import Queue
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'Queue'
>>> exit()

[Marios Andreou (marios-b)]

12:01 < ykarel> marios|ruck, but ^^ will not help in jobs, in container build jobs, kolla is used from source
can't pin kolla

[Marios Andreou (marios-b)]

Pin kolla venv and package install workaround py2 queue bug https://review.opendev.org/708846
Pin kolla to prevent import queue error until py3 https://review.rdoproject.org/r/25083

Test Run periodic-tripleo-centos-7-master-containers-build-push * https://review.rdoproject.org/r/#/c/25080

[Marios Andreou (marios-b)]

new plan, described there https://review.opendev.org/#/c/708846/2/roles/build-containers/tasks/venv_setup.yml

Trying those instead

1 Pin kolla venv and package install workaround py2 queue bug https://review.opendev.org/708846

2 Dummy commit for pinned kolla version for py2 importerror queue https://review.rdoproject.org/r/25100

3 Add buildcontainers_venv: false master centos7 container build https://review.rdoproject.org/r/25101

[Alex Schultz (alex-schultz)]

or drop centos-7 build job from ussuri+

[Marios Andreou (marios-b)]

OK... update again.. new new plan ... the things we needed here were:

1. (rdoinfo pin kolla to good commit) Pin kolla to prevent import queue error until py3 https://review.rdoproject.org/r/#/c/25083/

2. (config) Add buildcontainers_venv: false master centos7 container build https://review.rdoproject.org/r/#/c/25101/

The above should clear the master centos7 containers build.

chandan posted the revert for Revert "Pin kolla to prevent import queue error until py3" https://review.rdoproject.org/r/#/c/25114/ but we are currently discussing this not clear if something further will be needed if we revert that.

[Rabi Mishra (rabi)]

upstream master centos7 container build jobs are now failing with the same issue.

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b09/709044/1/check/tripleo-build-containers-centos-7-buildah/b093294/logs/build.log

[Marios Andreou (marios-b)]

so for periodic this is now handled however we are hitting the same bug in check

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b09/709044/1/check/tripleo-build-containers-centos-7-buildah/b093294/logs/build.log

more work will be needed here

[Marios Andreou (marios-b)]

(periodic some green runs there https://review.rdoproject.org/zuul/build/cf3c06a76381400f8bbd684460d130f0 https://review.rdoproject.org/zuul/build/28297a5a84354f93bde6179ba69e9fca )

[Marios Andreou (marios-b)]

proposed

Try using packages for centos7 containers build jobs https://review.opendev.org/#/c/709069/

uses the same approach as the periodic fix.. now we have one good build for centos7 we can switch the check jobs to use packages for kolla/tripleoclient rather than source.

[Marios Andreou (marios-b)]

so instead we removed the job from master check with https://review.opendev.org/708893

abaondoned https://review.opendev.org/#/c/709069/

we r done here I believe moving fix-released move back if you disagree thanks

[Russell Tweed (russell-tweed)]

Doesn't seem entirely fixed. Check out last job run for:

https://review.opendev.org/#/c/709087

First example is the pep8 job, which was passing a couple of days ago but now fails with:

ERROR: Could not find a version that satisfies the requirement gitdb2===4.0.1 (from -c /home/zuul/src/opendev.org/openstack/requirements/upper-constraints.txt (line 776)) (from versions: 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.0.6, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.3.post1, 4.0.2)
ERROR: No matching distribution found for gitdb2===4.0.1 (from -c /home/zuul/src/opendev.org/openstack/requirements/upper-constraints.txt (line 776))