Incompatible requirements break tempest (at least) on stable/pike

[Luigi Toscano]

A native zuul v3 tempest job on stable/pike fails with "AttributeError: 'tuple' object has no attribute 'lower'"

Full backtrace:

++ lib/tempest:install_tempest_plugins:649 : echo 'Checking installed Tempest plugins:'
Checking installed Tempest plugins:
++ lib/tempest:install_tempest_plugins:650 : tox -evenv-tempest -- tempest list-plugins
/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/setuptools/dist.py:47: DistDeprecationWarning: Do not call this function
  warnings.warn("Do not call this function", DistDeprecationWarning)
venv-tempest develop-inst-noop: /opt/stack/tempest
venv-tempest installed: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.,appdirs==1.4.3,asn1crypto==0.22.0,Babel==2.3.4,bcrypt==3.1.3,certifi==2018.11.29,cffi==1.10.0,chardet==3.0.4,cliff==2.8.2,cmd2==0.7.5,contextlib2==0.5.5,cryptography==2.0.2,debtcollector==1.17.1,decorator==4.3.2,dogpile.cache==0.7.1,enum34==1.1.6,extras==1.0.0,fasteners==0.14.1,fixtures==3.0.0,funcsigs==1.0.2,functools32==3.2.3.post2,future==0.17.1,futures==3.2.0,idna==2.5,ipaddress==1.0.18,iso8601==0.1.11,jmespath==0.9.3,jsonpatch==1.23,jsonpointer==2.0,jsonschema==2.6.0,keystoneauth1==3.11.2,linecache2==1.0.0,Mako==1.0.7,MarkupSafe==1.1.0,mock==2.0.0,monotonic==1.3,mox3==0.26.0,msgpack==0.6.1,msgpack-python==0.4.8,munch==2.3.2,netaddr==0.7.19,netifaces==0.10.6,openstacksdk==0.24.0,os-client-config==1.31.2,os-service-types==1.5.0,osc-lib==1.12.0,oslo.concurrency==3.21.1,oslo.config==4.11.1,oslo.context==2.22.0,oslo.i18n==3.17.1,oslo.log==3.42.2,oslo.serialization==2.20.2,oslo.utils==3.28.3,oslotest==3.7.0,paramiko==2.2.1,pbr==3.1.1,positional==1.1.2,prettytable==0.7.2,pyasn1==0.3.1,pycparser==2.18,pyinotify==0.9.6,PyNaCl==1.1.2,pyOpenSSL==19.0.0,pyparsing==2.2.0,pyperclip==1.5.27,python-cinderclient==4.1.0,python-dateutil==2.8.0,python-glanceclient==2.15.0,python-keystoneclient==3.18.0,python-mimeparse==1.6.0,python-neutronclient==6.11.0,python-novaclient==11.1.0,python-openstackclient==3.17.0,python-saharaclient==2.1.0,python-subunit==1.2.0,python-swiftclient==3.6.0,pytz==2017.2,PyYAML==3.12,requests==2.21.0,requestsexceptions==1.4.0,rfc3986==1.1.0,sahara-tests==0.7.1.dev13,simplejson==3.16.0,six==1.10.0,stestr==2.0.0,stevedore==1.25.1,subprocess32==3.5.3,# Editable Git install with no remote (tempest==19.0.1.dev307),-e /opt/stack/tempest,testtools==2.3.0,traceback2==1.4.0,unicodecsv==0.14.1,unittest2==1.1.0,urllib3==1.22,voluptuous==0.10.5,warlock==1.3.0,wcwidth==0.1.7,wrapt==1.10.10
venv-tempest run-test-pre: PYTHONHASHSEED='3515866893'
venv-tempest runtests: commands[0] | tempest list-plugins
Traceback (most recent call last):
  File "/opt/stack/tempest/.tox/tempest/bin/tempest", line 6, in <module>
    from tempest.cmd.main import main
  File "/opt/stack/tempest/tempest/cmd/main.py", line 18, in <module>
    from oslo_log import log as logging
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_log/log.py", line 48, in <module>
    from oslo_log import _options
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_log/_options.py", line 143, in <module>
    help='Log rotation type.')
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 1261, in __init__
    max_length=max_length),
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_config/types.py", line 115, in __init__
    self.lower_case_choices = [c.lower() for c in choices]
AttributeError: 'tuple' object has no attribute 'lower'
ERROR: InvocationError for command '/opt/stack/tempest/.tox/tempest/bin/tempest list-plugins' (exited with code 1)

== What happens:
When venv-tempest is first deployed in the native tempest Zuul v3 job, upper-constraints.txt from master is used. This means that a recent version of oslo.config (6.8.0) and oslo.log (3.42.2) is installed.

For example:
2019-02-11 18:59:56.888 | ++ lib/tempest:configure_tempest:543 : tox -revenv-tempest --notest
2019-02-11 18:59:57.573 | venv-tempest recreate: /opt/stack/tempest/.tox/tempest
2019-02-11 19:00:01.355 | venv-tempest installdeps: -chttps://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt, -r/opt/stack/tempest/requirements.txt
2019-02-11 19:00:15.350 | venv-tempest develop-inst: /opt/stack/tempest
2019-02-11 19:00:21.258 | venv-tempest installed: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.,asn1crypto==0.24.0,Babel==2.6.0,bcrypt==3.1.6,certifi==2018.11.29,cffi==1.11.5,chardet==3.0.4,cliff==2.14.0,cmd2==0.8.9,contextlib2==0.5.5,cryptography==2.5,debtcollector==1.20.0,enum34==1.1.6,extras==1.0.0,fasteners==0.14.1,fixtures==3.0.0,funcsigs==1.0.2,functools32==3.2.3.post2,future==0.17.1,idna==2.8,ipaddress==1.0.22,iso8601==0.1.12,jsonschema==2.6.0,linecache2==1.0.0,monotonic==1.5,msgpack==0.6.1,netaddr==0.7.19,netifaces==0.10.9,oslo.concurrency==3.29.0,oslo.config==6.8.0,oslo.context==2.22.0,oslo.i18n==3.23.0,oslo.log==3.42.2,oslo.serialization==2.28.1,oslo.utils==3.40.2,paramiko==2.4.2,pbr==5.1.2,prettytable==0.7.2,pyasn1==0.4.5,pycparser==2.19,pyinotify==0.9.6,PyNaCl==1.3.0,pyparsing==2.3.1,pyperclip==1.7.0,python-dateutil==2.8.0,python-mimeparse==1.6.0,python-subunit==1.3.0,pytz==2018.9,PyYAML==3.13,requests==2.21.0,rfc3986==1.2.0,six==1.12.0,stestr==2.2.0,stevedore==1.30.0,subprocess32==3.5.3,# Editable Git install with no remote (tempest==19.0.1.dev307),-e /opt/stack/tempest,testtools==2.3.0,traceback2==1.4.0,unicodecsv==0.14.1,unittest2==1.1.0,urllib3==1.24.1,voluptuous==0.11.5,wcwidth==0.1.7,wrapt==1.11.1

Then the local upper-constraint.txt file is used (from stable/pike):

2019-02-11 19:00:21.282 | ++ lib/tempest:configure_tempest:545 : tox -evenv-tempest -- pip install -c /opt/stack/requirements/upper-constraints.txt -r requirements.txt

The expected result is the downgrade of both oslo.log and oslo.config, among others.
It is important because the newer oslo.log expects oslo.config to handle a configuration key (log_rotation_type) as a list of tuple. But unfortunately, due to the way the dependencies are handled, oslo.config is downgraded to the version in pike, but the version of oslo.log is not:

 tempest 19.0.1.dev307 has requirement oslo.log>=3.36.0, but you'll have oslo-log 3.30.2 which is incompatible.

This means that the newer oslo.log tries to use a specific feature from a version of oslo.config that does not provide it, and the tempest command fails to run.

[Luigi Toscano (ltoscano)]

It looks like the following bug and patch may be related:

https://bugs.launchpad.net/devstack/+bug/1706009
https://review.openstack.org/#/c/473525/

But now we are no more synchronizing the constraints, bumping the minimum. Maybe a possible solution would be to lower the tempest requirements so that it can be installed on every supported branch using the requirements of that branch?

[Luigi Toscano (ltoscano)]

The reason may be traced back to the lack of the following devstack patch in stable/pike:

https://review.openstack.org/#/c/560946/

I proposed a backport and I'm going to test if it really fixes the issue:
https://review.openstack.org/#/c/563617/

[Luigi Toscano (ltoscano)]

The backport has been already proposed last year but it got stuck due to a flaky test; now it's time recheck it.

[Luigi Toscano (ltoscano)]

The backport was merged, so now tempest list-plugins is correctly executed.

But the problem is that tempest-17.0 is installed instead of the master version, probably because the constraints are enforced. So now the execution fails because the --black-regex= option is not recognized:

/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/setuptools/dist.py:47: DistDeprecationWarning: Do not call this function
  warnings.warn("Do not call this function", DistDeprecationWarning)
all develop-inst-noop: /opt/stack/tempest
all installed: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.,appdirs==1.4.3,asn1crypto==0.22.0,Babel==2.3.4,bcrypt==3.1.3,certifi==2017.4.17,cffi==1.10.0,chardet==3.0.4,cliff==2.8.2,cmd2==0.7.5,contextlib2==0.5.5,cryptography==2.0.2,debtcollector==1.17.1,deprecation==1.0.1,enum34==1.1.6,extras==1.0.0,fasteners==0.14.1,fixtures==3.0.0,funcsigs==1.0.2,functools32==3.2.3.post2,future==0.17.1,futures==3.1.1,idna==2.5,ipaddress==1.0.18,iso8601==0.1.11,jsonpatch==1.16,jsonpointer==1.10,jsonschema==2.6.0,keystoneauth1==3.1.0,linecache2==1.0.0,Mako==1.0.7,MarkupSafe==1.0,mock==2.0.0,monotonic==1.3,mox3==0.23.0,msgpack==0.6.1,msgpack-python==0.4.8,netaddr==0.7.19,netifaces==0.10.6,openstacksdk==0.9.17,os-client-config==1.28.0,os-testr==1.0.0,osc-lib==1.7.0,oslo.concurrency==3.21.1,oslo.config==4.11.1,oslo.context==2.17.1,oslo.i18n==3.17.1,oslo.log==3.30.2,oslo.serialization==2.20.2,oslo.utils==3.28.3,oslotest==2.17.1,paramiko==2.2.1,pbr==3.1.1,positional==1.1.2,prettytable==0.7.2,pyasn1==0.3.1,pycparser==2.18,pyinotify==0.9.6,PyNaCl==1.1.2,pyOpenSSL==17.2.0,pyparsing==2.2.0,pyperclip==1.5.27,python-cinderclient==3.1.1,python-dateutil==2.6.1,python-glanceclient==2.8.0,python-keystoneclient==3.13.0,python-mimeparse==1.6.0,python-neutronclient==6.5.0,python-novaclient==9.1.2,python-openstackclient==3.12.1,python-saharaclient==1.3.0,python-subunit==1.2.0,python-swiftclient==3.4.0,pytz==2017.2,PyYAML==3.12,requests==2.18.2,requestsexceptions==1.3.0,rfc3986==1.1.0,sahara-tests==0.7.1.dev19,simplejson==3.11.1,six==1.10.0,stestr==2.0.0,stevedore==1.25.1,subprocess32==3.5.3,# Editable Git install with no remote (tempest==19.0.1.dev319),-e /opt/stack/tempest,testrepository==0.0.20,testtools==2.3.0,traceback2==1.4.0,unicodecsv==0.14.1,unittest2==1.1.0,urllib3==1.22,voluptuous==0.10.5,warlock==1.2.0,wcwidth==0.1.7,wrapt==1.10.10
all run-test-pre: PYTHONHASHSEED='417735469'
all runtests: commands[0] | find . -type f -name '*.pyc' -delete
all runtests: commands[1] | tempest run --regex '^(sahara_tempest_plugin.tests.)' --concurrency=4 --black-regex=
usage: tempest run [-h] [--workspace WORKSPACE]
                   [--workspace-path WORKSPACE_PATH]
                   [--config-file CONFIG_FILE] [--smoke | --regex REGEX]
                   [--whitelist-file WHITELIST_FILE | --blacklist-file BLACKLIST_FILE | --load-list LOAD_LIST]
                   [--list-tests] [--concurrency CONCURRENCY]
                   [--parallel | --serial] [--save-state] [--subunit]
               ...

[Ghanshyam Mann (ghanshyammann)]

Let me understand it clearly.

>The expected result is the downgrade of both oslo.log and oslo.config, among others.
>It is important because the newer oslo.log expects oslo.config to handle a configuration key >(log_rotation_type) as a list of tuple. But unfortunately, due to the way the dependencies are >handled, oslo.config is downgraded to the version in pike, but the version of oslo.log is not:

> tempest 19.0.1.dev307 has requirement oslo.log>=3.36.0, but you'll have oslo-log 3.30.2 which is >incompatible.

As you mentioned this in the description that oslo.config option is downgraded as per upper_constraint of stable/pike but not oslo.log. But I cannot find where is oslo.log not downgraded. I can see its downgraded version in below log.

- http://logs.openstack.org/99/636199/1/check/sahara-tests-tempest/74b35b2/controller/logs/devstacklog.txt.gz#_2019-02-11_19_00_35_827

Second about

[Ghanshyam Mann (ghanshyammann)]

RE on your latest comment:

>The backport was merged, so now tempest list-plugins is correctly executed.

>But the problem is that tempest-17.0 is installed instead of the master version, probably because >the constraints are enforced.

This seems a problem. we should not install tempest-17.0. It should be from the master. In all stable branch job testing, we install tempest from the master and apply the constraint from the master only.

This is the flow:
1. install tempest form master (until you explicitly change TEMPEST_BRANCH which is default to master in all stable branch). It applies the upper_constraint from the stable branch but that will be overridden in step2
2. configure tempest, here the created venv will install all dependency with master's upper_constraint. This change is not backported to pike. This is one backport needed.
3. install tempest plugins in same venv created above. Now tempest plugin use the stable branch upper_constraint [1]. This is problem.

We should install the tempest plugin also with master upper_constraint same as Tempest otherwise we will have version compatibility issue which is nothing but this bug.

[1] https://github.com/openstack-dev/devstack/blob/72f632222f6d90d3545b5d7ca48297da4218e2ea/lib/tempest#L676

[Luigi Toscano (ltoscano)]

But also tempest itself is installed using the requirements from the branch. See the content of install_tempest():

http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/tempest#n667

# install_tempest() - Collect source and prepare
function install_tempest {
    git_clone $TEMPEST_REPO $TEMPEST_DIR $TEMPEST_BRANCH
    pip_install 'tox!=2.8.0'
    pushd $TEMPEST_DIR
    tox -r --notest -efull
    # NOTE(mtreinish) Respect constraints in the tempest full venv, things that
    # are using a tox job other than full will not be respecting constraints but
    # running pip install -U on tempest requirements
    $TEMPEST_DIR/.tox/tempest/bin/pip install -c $REQUIREMENTS_DIR/upper-constraints.txt -r requirements.txt
    PROJECT_VENV["tempest"]=${TEMPEST_DIR}/.tox/tempest
    popd
}

Even if it's a different tempest environment, the directory is shared (~/.tox/tempest).

[Ghanshyam Mann (ghanshyammann)]

yeah, that is overridden by configre_tempest where we used the upper_constarint from master. We can make install_tempest also use master one to avoid duplication of uninstall/install.

I tested the fix of doing master's upper_constraint and it is working fine now - https://review.openstack.org/#/c/639530/

[Ghanshyam Mann (ghanshyammann)]

need this series to be merged for this bug - https://review.openstack.org/#/q/topic:bug/1816022+(status:open+OR+status:merged)

[Martin Kopec (mkopec)]

The patches have been merged, the issue is no longer occurring.