lastest virtenv workarounds broke native tripleo-quickstart install

[wes hayutin]

03:51:18 + ./quickstart.sh --working-dir /home/jenkins/workspace/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/ --no-clone --bootstrap --requirements requirements.txt --requirements quickstart-extras-requirements.txt --requirements ci-scripts/ci-base-requirements.txt --playbook noop.yml 127.0.0.2
03:51:18 Print out the commit hash of the git repo
03:51:18 commit 7545358631493c44537672451579a36c474f47e5
03:51:18 Merge: 221773d 71ed15f
03:51:18 Author: Zuul <email address hidden>
03:51:18 Date: Sun Feb 16 12:23:31 2020 +0000
03:51:18
03:51:18 Merge "Add topology_map to 1ctrl 1compute nodes file"
03:51:18
03:51:18 Python Command is:
03:51:18 python2
03:51:18 Running install_deps
03:51:18 sudo-1.8.23-4.el7_7.1.x86_64
03:51:18 sudo: a password is required
03:51:18 WARNING: SUDO user is not passwordless.
03:51:18 export USER_OVERRIDE_SUDO_CHECK=1
03:51:18 to be prompted for sudo password.
03:51:18 checking python modules
03:51:18 pip:10.0.0 virtualenv:15.2.0 setuptools:0.9.8
03:51:18 SUCCESS: install-deps succeeded.
03:51:18 Creating virtualenv at /home/jenkins/workspace/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal
03:51:18
03:51:18 We trust you have received the usual lecture from the local System
03:51:18 Administrator. It usually boils down to these three things:
03:51:18
03:51:18 #1) Respect the privacy of others.
03:51:18 #2) Think before you type.
03:51:18 #3) With great power comes great responsibility.
03:51:18
03:51:18 sudo: no tty present and no askpass program specified
03:51:21 Build step 'Execute shell' marked build as failure

https://ci.centos.org/job/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/673/console

All ci.centos jobs are broken, and user installs are broken.

[Sorin Sbarnea (ssbarnea)]

FYI, newer versions of virtualenv no longer need bumping six, so feel free to remove the workarounds.

[Marios Andreou (marios-b)]

trying on vm right now... following https://ci.centos.org/job/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/673/console ... just install requirements fail on pip:

mkdir WORKSPACE
mv tripleo-quickstart/ WORKSPACE/
set -eux
export WORKSPACE=/home/centos/WORKSPACE/
pushd $WORKSPACE/tripleo-quickstart
./quickstart.sh --working-dir $WORKSPACE/ --no-clone --bootstrap --requirements requirements.txt --requirements quickstart-extras-requirements.txt --requirements ci-scripts/ci-base-requirements.txt --playbook noop.yml 127.0.0.2

No package python-pip available.
Nothing to do
/bin/python2: No module named pip
Connection to fe.fi.fo.fum closed.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/708380

[Marios Andreou (marios-b)]

ignore above after getting epel & then python2-pip & that completed for requirements at least.

just had a call with weshay about that . looks like https://github.com/pypa/virtualenv/pull/1606 should fix us

am looking at reverting our fixes and make sure we get newer https://virtualenv.pypa.io/en/latest/changelog.html#bugfixes-20-0-4 fixed virtualenv > 20.0.4

posted just now openstack/tripleo-quickstart master: Install virtualenv >= 20.0.4 & bump molecule remove six workaround https://review.opendev.org/708380

[Marios Andreou (marios-b)]

18:16 < weshay|ruck> marios|ruck|call, I don't think the ci.centos job is pulling the change https://review.rdoproject.org/r/#/c/25015/

[Marios Andreou (marios-b)]

ensure sudo is not called in ci.centos https://review.opendev.org/#/c/708264/

[Marios Andreou (marios-b)]

spent some time digging (again) here but the bug doesn't have much info so here is a summary:

relevant reviews currently are [1] and depends-on a [2]

in [1] we are effectively reverting those changes [3][4]. The fix at [1] seems to get passed the error here however it causes the tripleo-ci-centos-7-containers-multinode-queens to fail, which is why we have [2].

The issue at [5] may be relevant to the latest issues we are hitting in the queens job

----
[1] https://review.opendev.org/#/c/708264/ ensure sudo is not called in ci.centos

[2] https://review.opendev.org/#/c/708791/ pip install zipp to fix contextlib2 error

[3] https://github.com/openstack/tripleo-quickstart/commit/cd0b737930ad706d090550cc03a90bd80d683cba

[4] https://github.com/openstack/tripleo-quickstart/commit/65ab3aa516e336a50ceee9efde1e302f8443eeb3

[5] https://github.com/pypa/virtualenv/issues/1630

[Marios Andreou (marios-b)]

so we finally got some breakthrough today on this. Summary:

We had a very old version of virtualenv on that queens node.. so old that the error we were getting (after adding debug) was like that:

stderr': u"/usr/bin/python2: cannot import name suppress; 'virtualenv' is a package and cannot be directly executed

suggesting (per zbr) that 'virtualenv is ancient' there

Final solution is to make sure we have contextlib2 (system not in the virtualenv) and that seems to pull in new enough version of virtualenv without causing additional deps issues. You can trace the things we tried by following the change history and comments of https://review.opendev.org/#/c/708791/

So we need these things:

1 https://review.opendev.org/#/c/708264/ Revert sudo and six workarounds for virtualenv 20.0.1 issues

2 https://review.opendev.org/#/c/708791/ Add explicit contextlib2 0.5<0.6 for queens ancient virtualenv

3 https://review.rdoproject.org/r/#/c/25539/ Add missing repo to quickstart gate job

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/708791
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=260689e24d4584396406032f3d9a44f19cf5ca91
Submitter: Zuul
Branch: master

commit 260689e24d4584396406032f3d9a44f19cf5ca91
Author: Wes Hayutin <email address hidden>
Date: Wed Feb 19 20:13:11 2020 -0700

    Add explicit contextlib2 0.5<0.6 for queens ancient virtualenv

    As described in related bug we need to install a particular version
    of contextlib2 to fix an issue with very old virtualenv installed.
    Installing contextlib2 in this way seems to pull in a new enough
    version of virtualenv without causing other dependency issues.

    Also add tags: notest to skip molecule testing which makes
    tox-molecule job fail, ideally this task shouldn't run except queens
    release.

    Related-Bug: 1863632
    Change-Id: Iff5ed3f7f5196016b6cd807ee9408c204b349498

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/708264
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=a949829391254d5b8b59f65ed8350a7bf3d2fcd0
Submitter: Zuul
Branch: master

commit a949829391254d5b8b59f65ed8350a7bf3d2fcd0
Author: Wes Hayutin <email address hidden>
Date: Mon Feb 17 14:57:47 2020 -0700

    Revert sudo and six workarounds for virtualenv 20.0.1 issues

    Reverts the workarounds added for the Partial-Bug below.
    This is to fix issues described in Related-Bug.

    * ensure sudo is not called in ci.centos
    * remove patches to fix lp#1862724
    * add python-virtualenv for queens branch centos-7

    Partial-Bug: #1862724
    Related-Bug: #1863632
    Depends-On: https://review.opendev.org/708791
    Change-Id: I3d9bd1aa3b498890a2a773a838df78b53d2c30e6

[Marios Andreou (marios-b)]

things from https://bugs.launchpad.net/tripleo/+bug/1863632/comments/8 are now all merged but I still don't see a green run in the https://ci.centos.org/job/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/

latest right now is failing in unrelated way i think:

        * 12:04:09 error: failed to get pool 'oooq_pool'
12:04:09 error: Storage pool not found: no storage pool with matching name 'oooq_pool'
        * 12:04:31 error: failed to get vol 'undercloud.qcow2'
12:04:31 error: Storage volume not found: no storage vol with matching path 'undercloud.qcow2'
        * 12:18:44 guestfsd: error:
        * 12:18:44 echo "ERROR: Unknown filesystem, cannot resize."

But lets see green runs on this job before declaring done I think

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sorin Sbarnea (<email address hidden>) on branch: master
Review: https://review.opendev.org/708380
Reason: I think all changes are obsoleted.