Tempest jobs broken on stable branches due to requirements neutron-lib upgrade (the EOLing python2 drama)

[Radosław Piliszek]

Updating neutron-lib to 2.0.0 (py3-only release) in upper constraints on master [1] killed neutron tempest rocky jobs with:
2020-01-16 19:07:29.088781 | controller | Processing /opt/stack/neutron-tempest-plugin
2020-01-16 19:07:29.825378 | controller | Requirement already satisfied: pbr===5.4.4 in ./.tox/tempest/lib/python2.7/site-packages (from -c u-c-m.txt (line 58)) (5.4.4)
2020-01-16 19:07:29.869691 | controller | Collecting neutron-lib===2.0.0 (from -c u-c-m.txt (line 79))
2020-01-16 19:07:30.019373 | controller | Could not find a version that satisfies the requirement neutron-lib===2.0.0 (from -c u-c-m.txt (line 79)) (from versions: 0.0.1, 0.0.2, 0.0.3, 0.1.0, 0.2.0, 0.3.0, 0.4.0, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.9.0, 1.9.1, 1.9.2, 1.10.0, 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.16.0, 1.17.0, 1.18.0, 1.19.0, 1.20.0, 1.21.0, 1.22.0, 1.23.0, 1.24.0, 1.25.0, 1.26.0, 1.27.0, 1.28.0, 1.29.0, 1.29.1, 1.30.0, 1.31.0)
2020-01-16 19:07:30.033128 | controller | No matching distribution found for neutron-lib===2.0.0 (from -c u-c-m.txt (line 79))

see [2] for an example log.

These jobs run on Ubuntu 16.04 with Python 2.

[1] https://review.opendev.org/702288
[2] https://f6bb8d2d9ae7bba883a1-1b1c7df97f8efc0930cf31e3404d1843.ssl.cf2.rackcdn.com/702946/1/check/neutron-tempest-plugin-api-rocky/9043fec/job-output.txt

[Brian Haley (brian-haley)]

I think I can fix this, give me a minute...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/702979

[Sean McGinnis (sean-mcginnis)]

Something is not right here. The requirements set in master should in no way impact stable/rocky. If rocky is picking up newer package versions than what was the upper constraint at the time, then something is wrong with a configuration somewhere.

[Sean McGinnis (sean-mcginnis)]

Yeah, taking a look at those failing jobs, it looks like there is an issue with how the neutron-tempest-plugin-* jobs are defined that is resulting in master things being pulled in for stable testing.

[Ghanshyam Mann (ghanshyammann)]

All tempest plugin uses the master u-c for stable branch testing which is the valid way because master Tempest plugin is being used to test the stable branches which need u-c from master itself.

These failed jobs also used master u-c[1] which is trying to install the latest neutron-lib and failing.

Tempest plugin python2 jobs on a stable branch can not install the new neutron-lib which is maintained for python3 only with metadata 'python-requires = >=3.6'.

[1] https://zuul.opendev.org/t/openstack/build/fb8a928ed3614e09a9a3cf4637f2f6c2/log/job-output.txt#33040

To solve this we need to separate out the testing of python2 jobs with python2 supported version of Tempest plugins and with respective u-c. For example, test all python2 job with tempest plugin train version (or any latest version which support py2) and use u-c from stable/train.

second option is to install the tempest and plugins in py3 env on oy2 jobs also. This should be easy and prefered way. let me try.

[Radosław Piliszek (yoctozepto)]

Still failing due to py3 being 3.5 on those nodes.

[Radosław Piliszek (yoctozepto)]

    name: neutron-tempest-plugin-api-rocky
    nodeset: openstack-single-node-xenial

[Radosław Piliszek (yoctozepto)]

We would also need to bump that to bionic to get py3.6
Or switch to other source of py3.6

The first option has the consequence that neutron rocky becomes tested on bionic.

Another option would be to rewrite the job so that it uses two nodes -> one for tempest with current stuff and one for devstack with rocky stuff. More workload but future-proof. Another downside is greater usage of limited CI resources (as it goes 2x to many jobs...).

[Ghanshyam Mann (ghanshyammann)]

Yeah, we need py3.6. Doing with two ndoe might be more complex than we can think as of now.

Switching stable branches to bionic might be one option but still need to be check if that is option as per testing runtime etc.

[Radosław Piliszek (yoctozepto)]

It came to my mind we could get away with 2nd node by using some container for tempest (looking at you, kolla).

[OpenStack Infra (hudson-openstack)]

Change abandoned by Brian Haley (<email address hidden>) on branch: stable/rocky
Review: https://review.opendev.org/702979

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/703476

[Radosław Piliszek (yoctozepto)]

Hmm, it does not allow me to restore assignee to gmann, meh.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/704188

[Radosław Piliszek (yoctozepto)]

plot thickens by zipp>=2 dropping support for py3.5 (as used on xenial) too

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/703476
Committed: https://git.openstack.org/cgit/openstack/devstack/commit/?id=0b596470c8f7196a5e8e42faaa02db0ecefcf80d
Submitter: Zuul
Branch: stable/rocky

commit 0b596470c8f7196a5e8e42faaa02db0ecefcf80d
Author: Ghanshyam <email address hidden>
Date: Mon Jan 20 12:11:44 2020 -0600

    Install python3.6 on ubuntu xenial for Tempest venv

    Stable/Rocky and Stable/Queens use ubuntu xenial for
    gate which does not have python3.6 interpreter.

    As OpenStack is dropping the python2 support, many dependencies
    of Tempest and its plugins need py3.6 interpreter.
    For Example: neutron-lib latest release Bug#1860033.

    This installs python3.6 and makes it available in Tempest venv
    so that Tempest master can keep testing the stable branch.
    py3.6 provided by ppa.

    Stein onwards distro ver (bionic) has py3.6 so no issue there.

    Closes-Bug: 1860033
    Co-authored-by: Radosław Piliszek <email address hidden>
    Change-Id: I143b5d9b5cbe9a3c3f0d749badf2b6332a753fb5

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/703011
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=1c680fdb728c24a4c9a1507ad8319f0a505cef9c
Submitter: Zuul
Branch: master

commit 1c680fdb728c24a4c9a1507ad8319f0a505cef9c
Author: Ghanshyam <email address hidden>
Date: Mon Jan 20 11:43:29 2020 -0600

    Define python3 as basepython for Tempest tox env

    While OpenStack is in transition to drop py2, many
    lib or dependency of Tempest or its plugins cannot
    be installed on python 2.7.

    To install Tempest on py3 evnv, let's change the basepython
    as python 3 for tox env.

    Defining basepython as python3.6 for all venv using
    .tox/tempest.

    Depends-On: https://review.opendev.org/#/c/703476/
    Depends-On: https://review.opendev.org/#/c/703679/

    Closes-Bug: 1860033
    Change-Id: If3ff71dbe636ef98072b15f6599e6e641c222f30

[Tom Stappaerts (tstappae)]

Can we also backport all these fixes to stable/queens?

[Ghanshyam Mann (ghanshyammann)]

It is not required to backport on stable/queens as such.

stable/queens has been pinned with compatible Tempest tag - https://review.opendev.org/#/c/703679/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Radosław Piliszek (<email address hidden>) on branch: stable/rocky
Review: https://review.opendev.org/704188
Reason: no longer relevant, we handle this with fixes to u-c atm and await rocky em

[Slawek Kaplonski (slaweq)]

I think it's now fixed on neutron's side as all jobs are working.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/pike
Review: https://review.opendev.org/c/openstack/devstack/+/703272
Reason: stable/pike of devstack is EOL'd so patches need to be abandoned to be able to delete the branch.