stable/stein tempest-full job fails with "tempest requires Python '>=3.6' but the running Python is 2.7.17"

[Bernard Cafarelli]

Only seen in stable/stein, tempest-full (and also non-voting neutron-tempest-dvr-ha-multinode-full) job fails with:
Obtaining file:///opt/stack/tempest
tempest requires Python '>=3.6' but the running Python is 2.7.17

Example failure on https://review.opendev.org/#/c/717336/2:
https://zuul.opendev.org/t/openstack/build/f05569c475f44327bff7b7ec58faef8c
https://zuul.opendev.org/t/openstack/build/651ca00e67ab42fd814ec5edad437997

While backport on rocky passed both:
https://review.opendev.org/#/c/717337/2
https://zuul.opendev.org/t/openstack/build/c9c0139cda4f45cd825e169765e6854c
https://zuul.opendev.org/t/openstack/build/6f318c4897ea4864b7cd2691dc2a36ab

and on train:
https://review.opendev.org/#/c/717335/2
https://zuul.opendev.org/t/openstack/build/f84209f049f2459eabd453058ad11ccf

For neutron-tempest-dvr-ha-multinode-full parent in stein is tempest-multinode-full so it looks like common issue in tempest-full job definition for this branch?

[Lajos Katona (lajos-katona)]

As I see these patches introduced this issue:
https://review.opendev.org/#/c/704096/7/setup.cfg
It was useless, but the fix last week was merged:
https://review.opendev.org/715513

As I see we have to cap tempest to <=23.0.0 for these branches, as that is the last tag that has no exact python3 statement in setup.cfg

[Ghanshyam Mann (ghanshyammann)]

All py2 jobs also should use py3 venv to run tempest and stein use Bionic node so py3.6 is there. Let me check more logs and find what happended.

[Ghanshyam Mann (ghanshyammann)]

ok here is the issue,

Devstack installed Tempest in venv of py3 which is what we expect
- https://zuul.opendev.org/t/openstack/build/f05569c475f44327bff7b7ec58faef8c/log/controller/logs/devstacklog.txt#11385

But it try to install Tempest on system wide also which try to use py2 env because jobs is py2 and fail.
- https://zuul.opendev.org/t/openstack/build/f05569c475f44327bff7b7ec58faef8c/log/controller/logs/devstacklog.txt#30654

We need to set INSTALL_TEMPEST flag to false on all stable branch to avoid such issue which we did not do since pike (I will make note of this in Devstack branch cut TODO)
- https://github.com/openstack/devstack/blob/6067e41a5cc44164e34faab2fd9741dbb788f42f/lib/tempest#L60

[Bernard Cafarelli (bcafarel)]

Closing on neutron side, fix as mentioned is going in devstack by gmann:
https://review.opendev.org/#/q/I60949fb735c82959fb2cfcb6aeef9e33fb0445b6

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/718132
Committed: https://git.openstack.org/cgit/openstack/devstack/commit/?id=8cd4e9a9076e05974ed1242ccab3391c55c7070c
Submitter: Zuul
Branch: stable/rocky

commit 8cd4e9a9076e05974ed1242ccab3391c55c7070c
Author: Ghanshyam Mann <email address hidden>
Date: Tue Apr 7 09:32:06 2020 -0500

    Fix Tempest installtion on system wide for stable branch

    INSTALL_TEMPEST flag enable to install Tempest by default on
    devstack env which is meant for using on master gate only and has
    to be false for stable branch. On stable branch master Tempest
    on system wide can fail to install for various reason like constraint
    etc. That is why we install Tempest on venv always.

    This started failing tempest-full on stein gate[1] where
    devstack try to install Tempest on system wide also which try to
    use py2 env because jobs is py2 and fail.
    - https://zuul.opendev.org/t/openstack/build/f05569c475f44327bff7b7ec58faef8c/log/controller/logs/devstacklog.txt#30654

    We need to set INSTALL_TEMPEST flag to false on all stable branch to
    avoid such issue which we did not do since pike.
    (I will make note of this in Devstack branch cut TODO list)
    - https://github.com/openstack/devstack/blob/6067e41a5cc44164e34faab2fd9741dbb788f42f/lib/tempest#L60

    [1] https://zuul.opendev.org/t/openstack/build/f05569c475f44327bff7b7ec58faef8c

    Change-Id: I60949fb735c82959fb2cfcb6aeef9e33fb0445b6
    Closes-Bug: #1871327

[Nautik (nautik)]

Hello,

I still have the same issue using stable/stein devstack, except it says the running Python is 3.5.2 and not 2.7:

```
tempest requires Python '>=3.6' but the running Python is 3.5.2
```

Running ubuntu xenial, which is I believe the version used for stable/stein.

See the failure on the `devstack-platform-xenial` zull job: https://zuul.opendev.org/t/openstack/build/2e8ea601750f430095990194631cbd2e

(displayed on the cherry-picked change on stable/stein: https://review.opendev.org/#/c/718131/1)

[Nautik (nautik)]

Was able to workaround the devstack installation step by adding this to my .localrc:
`TEMPEST_BRANCH=22.1.0`

[Ann Taraday (akamyshnikova)]

As @nautik pointed out stable\stein is still broken https://review.opendev.org/#/c/714855/
I've got constant failures on my change.

[Dr. Martin Bobak (martin.bobak)]

Yes, it looks like stable\stein isn't fixed yet. I'm getting the bug on a clean Bionic installation.

[Bernard Cafarelli (bcafarel)]

2 different failures I think here:
* failure on xenial nodes (octavia) - not sure stein+xenial is a supported upstream combination, from these links nodes should be bionic (which has py3.6):
https://review.opendev.org/#/c/649097/
https://ghanshyammann.com/openstack/openstack-ci-cd-migration-from-ubuntu-xenial-bionic-ubuntu-lts-18-04/
 failure on clean Bionic install: check you have devstack checked out on matching stable branch too, and check INSTALL_TEMPEST is set to False - in your logs you should not have anymore the line "lib/tempest:configure_tempest[...]setup_develop /opt/stack/tempest"

[Dr. Martin Bobak (martin.bobak)]

@Stein + Bionic: Yes, you're right. If I cancel the installation of Tempest, the problem is solved. However, I like the idea to have a basic validation of OpenStack deployment which also makes a basic configuration for further testing.

Thus is there a possibility to prepare a basic networking, images, etc. automatically as Tempest does?