neutron-rally-task fails 100% on stable branches

[Bernard Cafarelli]

Since March 23, the job started to fail on train:
https://zuul.opendev.org/t/openstack/build/0bea3158833944c185b5101f823af9c1
https://zuul.opendev.org/t/openstack/build/0b24c107a4e44d91addda71bf193ccfa
https://zuul.opendev.org/t/openstack/build/20b752ab736a45c09ac25921ad19c7d9

recheck does not help, jobs fail with:
++ /opt/stack/rally-openstack/devstack/lib/rally:init_rally:142 : rally --config-file /etc/rally/rally.conf deployment create --name devstack --filename /tmp/tmp.U7S1L63idY
2020-03-23 23:47:41.570 31668 WARNING rally.common.plugin.discover [-] Failed to load plugins from module 'rally_openstack' (package: 'rally-openstack 1.7.1.dev9'): (kubernetes 10.0.1 (/usr/local/lib/python3.6/dist-packages), Requirement.parse('kubernetes>=11.0.0')): pkg_resources.VersionConflict: (kubernetes 10.0.1 (/usr/local/lib/python3.6/dist-packages), Requirement.parse('kubernetes>=11.0.0'))
Env manager got invalid spec:
['There is no Platform plugin `existing` in openstack platform.']
+ /opt/stack/rally-openstack/devstack/lib/rally:init_rally:1 : exit_trap

[Bernard Cafarelli (bcafarel)]

This is the rally-openstack commit that broke gate:
https://opendev.org/openstack/rally-openstack/commit/5776e015f16321fdb489ebb482d984ccd3cd2903
(apparently rally is branchless too)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/714590

[Bernard Cafarelli (bcafarel)]

As seen in DNM patch, stein is broken too

[Bernard Cafarelli (bcafarel)]

There is also an upcoming patch to rally-openstack to drop python < 3.6 support, plus other task related to fixing deprecated stuff.

Adding venv support in its devstack plugin should fix the issue but is currently low priority task, so we should apply a neutron workaround similar to bug 1777506 and pin rally-openstack version

Affected versions are rocky to train, older as mentioned are already pinned on rally version

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/714661

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/714663

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/714665

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/714665
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=2a8af5f1659e8fd64923943a9db4698f94746e0a
Submitter: Zuul
Branch: stable/train

commit 2a8af5f1659e8fd64923943a9db4698f94746e0a
Author: Bernard Cafarelli <email address hidden>
Date: Tue Mar 24 15:41:53 2020 +0100

    Use rally-openstack 1.7.0 for stable/train

    Master branch now has requirements incompatible with stable branches
    constraints [0], so this switches to last tag compatible.

    Longer term fix will be to run rally in a virtualenv, but this is lower
    priority than other tasks. This can be revisited at the time, though for
    older branches we will probably stick with pinned version

    [0] https://opendev.org/openstack/rally-openstack/commit/5776e015f16321fdb489ebb482d984ccd3cd2903

    Change-Id: Iba1da7b4214d98bc1500ea57ed1faade423adf31
    Closes-Bug: #1868691

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/714661
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b38dc02d748b0d222b98fe5d587057cfaa26dbdc
Submitter: Zuul
Branch: stable/train

commit b38dc02d748b0d222b98fe5d587057cfaa26dbdc
Author: Bernard Cafarelli <email address hidden>
Date: Tue Mar 24 15:29:15 2020 +0100

    Use rally-openstack 1.7.0 for stable/train

    Master branch now has requirements incompatible with stable branches
    constraints [0], so this switches to last tag compatible.

    Longer term fix will be to run rally in a virtualenv, but this is lower
    priority than other tasks. This can be revisited at the time, though for
    older branches we will probably stick with pinned version

    [0] https://opendev.org/openstack/rally-openstack/commit/5776e015f16321fdb489ebb482d984ccd3cd2903

    Change-Id: I24fad07d3a15c62d4de711a957f19d375862c28d
    Closes-Bug: #1868691

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/714590
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f1ef0d23374917a6f7067ac1b98d355f9c97f747
Submitter: Zuul
Branch: stable/stein

commit f1ef0d23374917a6f7067ac1b98d355f9c97f747
Author: Bernard Cafarelli <email address hidden>
Date: Tue Mar 24 10:37:17 2020 +0100

    Use rally-openstack 1.7.0 for stable/stein

    Master branch now has requirements incompatible with stable branches
    constraints [0], so this switches to last tag compatible.

    Longer term fix will be to run rally in a virtualenv, but this is lower
    priority than other tasks. This can be revisited at the time, though for
    older branches we will probably stick with pinned version

    [0] https://opendev.org/openstack/rally-openstack/commit/5776e015f16321fdb489ebb482d984ccd3cd2903

    Change-Id: Iad7434f6d75e067feba31882a6417368b1915294
    Closes-Bug: #1868691

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/714663
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f25c5a6c95ad694f68c7f0d31144604156d411a4
Submitter: Zuul
Branch: stable/rocky

commit f25c5a6c95ad694f68c7f0d31144604156d411a4
Author: Bernard Cafarelli <email address hidden>
Date: Tue Mar 24 15:29:15 2020 +0100

    Use rally-openstack 1.7.0 for stable/rocky

    Master branch now has requirements incompatible with stable branches
    constraints [0], so this switches to last tag compatible.

    Longer term fix will be to run rally in a virtualenv, but this is lower
    priority than other tasks. This can be revisited at the time, though for
    older branches we will probably stick with pinned version

    [0] https://opendev.org/openstack/rally-openstack/commit/5776e015f16321fdb489ebb482d984ccd3cd2903

    Change-Id: I9ec8331bae54d191955a843f9d3f8c5537dc37f6
    Closes-Bug: #1868691

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/715430

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/715433

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/715434

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/715430
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=52a4ec13693c89d7b0ae65bb2702c45776a60507
Submitter: Zuul
Branch: stable/stein

commit 52a4ec13693c89d7b0ae65bb2702c45776a60507
Author: Bernard Cafarelli <email address hidden>
Date: Tue Mar 24 15:41:53 2020 +0100

    Use rally-openstack 1.7.0 for stable/stein

    Master branch now has requirements incompatible with stable branches
    constraints [0], so this switches to last tag compatible.

    Longer term fix will be to run rally in a virtualenv, but this is lower
    priority than other tasks. This can be revisited at the time, though for
    older branches we will probably stick with pinned version

    [0] https://opendev.org/openstack/rally-openstack/commit/5776e015f16321fdb489ebb482d984ccd3cd2903

    Change-Id: Iba1da7b4214d98bc1500ea57ed1faade423adf31
    Closes-Bug: #1868691
    (cherry picked from commit 2a8af5f1659e8fd64923943a9db4698f94746e0a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/715433
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=4991184eca3da12893af1c453512c7118bbac6be
Submitter: Zuul
Branch: stable/rocky

commit 4991184eca3da12893af1c453512c7118bbac6be
Author: Bernard Cafarelli <email address hidden>
Date: Fri Mar 27 15:26:11 2020 +0100

    Use rally-openstack 1.7.0 for stable/rocky

    Master branch now has requirements incompatible with stable branches
    constraints [0], so this switches to last tag compatible.

    Longer term fix will be to run rally in a virtualenv, but this is lower
    priority than other tasks. This can be revisited at the time, though for
    older branches we will probably stick with pinned version

    Also pin neutron-tempest-plugin to latest version that supports python 2

    [0] https://opendev.org/openstack/rally-openstack/commit/5776e015f16321fdb489ebb482d984ccd3cd2903

    Change-Id: I54fb021dec49724f17e7b5a330e09bcc638310d6
    Closes-Bug: #1868691

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/715434
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=ba80793644338ca992281bd180e85244f28645d4
Submitter: Zuul
Branch: stable/queens

commit ba80793644338ca992281bd180e85244f28645d4
Author: Bernard Cafarelli <email address hidden>
Date: Fri Mar 27 15:26:11 2020 +0100

    Use rally-openstack 1.7.0 for stable/queens

    Master branch now has requirements incompatible with stable branches
    constraints [0], so this switches to last tag compatible.

    Longer term fix will be to run rally in a virtualenv, but this is lower
    priority than other tasks. This can be revisited at the time, though for
    older branches we will probably stick with pinned version

    [0] https://opendev.org/openstack/rally-openstack/commit/5776e015f16321fdb489ebb482d984ccd3cd2903

    Change-Id: I54fb021dec49724f17e7b5a330e09bcc638310d6
    Closes-Bug: #1868691

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn rocky-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron rocky-eol release.