Tempest plugins jobs using devstack are not tested with Tempest master version.

[Ghanshyam Mann]

In CI/CD jobs with any Tempest plugins enabled on master is not tested with master Tempest even devstack TEMPEST_BRANCH is master or tempest is in required_project of job definition.

This is because Tempest and its plugins are installed in virtual env which first install the master Tempest but in the next step when it installs the tempest plugin using pip install, it pulls the PyPI latest release version of Tempest because tempest is one of the deps of tempest plugins (mentioned in requirements.txt file)

Example: https://zuul.opendev.org/t/openstack/build/b7e576393f904b7eafeb488bdbec37d5/log/job-output.txt#61925

[Ghanshyam Mann (ghanshyammann)]

and the Tempest is restricted with the what we have in upper-constraints. Having Tempest which is brachless and master version is supposed to be work/used in master + stable branch testing (except EM stable).

We should not pin Tempest in u-c where Tempest master is used for all the testing.

[Ghanshyam Mann (ghanshyammann)]

fixed by https://review.opendev.org/c/openstack/requirements/+/787621

[Dmitriy Rabotyagov (noonedeadpunk)]

I do not agree with this solution, as it historically breaks deployments. Master tempest is not always compatible with stable branches.

Here is simple example with Xena, where master tempest just broke neutron: https://zuul.opendev.org/t/openstack/build/7c0215c10e14437e80fc0dd5af778a6c/log/job-output.txt#17237

As log link will gone soon, copy/paste here:

tempest run -l:
=========================
Failures during discovery
=========================
--- import errors ---
Failed to import test module: neutron_tempest_plugin.api.test_qos
Traceback (most recent call last):
  File "/usr/lib/python3.7/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/openstack/venvs/tempest-24.0.1.dev4/lib/python3.7/site-packages/neutron_tempest_plugin/api/test_qos.py", line 1379, in <module>
    class QosMinimumPpsRuleTestJSON(base.BaseAdminNetworkTest):
  File "/openstack/venvs/tempest-24.0.1.dev4/lib/python3.7/site-packages/neutron_tempest_plugin/api/test_qos.py", line 1380, in QosMinimumPpsRuleTestJSON
    RULE_NAME = qos_consts.RULE_TYPE_MINIMUM_PACKET_RATE + "_rule"
AttributeError: module 'neutron_lib.services.qos.constants' has no attribute 'RULE_TYPE_MINIMUM_PACKET_RATE'

================================================================================
The above traceback was encountered during test discovery which imports all the found test modules in the specified test_path.

So you CAN NOT run master tempest against stable branches, even very recent ones.

At very least, we have neutron-lib still in U-C: https://opendev.org/openstack/requirements/src/branch/stable/xena/upper-constraints.txt#L65

[Dmitriy Rabotyagov (noonedeadpunk)]

Also we can't play with u-c by adding/removing stuff from there, as you might know new PIP resolver doesn't allow to have conflicting constraints with requirements.
So once projects start handling constraints on their own, adding them to u-c is breaking deployments.