A native zuul v3 tempest job on stable/pike fails with "AttributeError: 'tuple' object has no attribute 'lower'"
Full backtrace:
++ lib/tempest:install_tempest_plugins:649 : echo 'Checking installed Tempest plugins:'
Checking installed Tempest plugins:
++ lib/tempest:install_tempest_plugins:650 : tox -evenv-tempest -- tempest list-plugins
/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/setuptools/dist.py:47: DistDeprecationWarning: Do not call this function
warnings.warn("Do not call this function", DistDeprecationWarning)
venv-tempest develop-inst-noop: /opt/stack/tempest
venv-tempest installed: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.,appdirs==1.4.3,asn1crypto==0.22.0,Babel==2.3.4,bcrypt==3.1.3,certifi==2018.11.29,cffi==1.10.0,chardet==3.0.4,cliff==2.8.2,cmd2==0.7.5,contextlib2==0.5.5,cryptography==2.0.2,debtcollector==1.17.1,decorator==4.3.2,dogpile.cache==0.7.1,enum34==1.1.6,extras==1.0.0,fasteners==0.14.1,fixtures==3.0.0,funcsigs==1.0.2,functools32==3.2.3.post2,future==0.17.1,futures==3.2.0,idna==2.5,ipaddress==1.0.18,iso8601==0.1.11,jmespath==0.9.3,jsonpatch==1.23,jsonpointer==2.0,jsonschema==2.6.0,keystoneauth1==3.11.2,linecache2==1.0.0,Mako==1.0.7,MarkupSafe==1.1.0,mock==2.0.0,monotonic==1.3,mox3==0.26.0,msgpack==0.6.1,msgpack-python==0.4.8,munch==2.3.2,netaddr==0.7.19,netifaces==0.10.6,openstacksdk==0.24.0,os-client-config==1.31.2,os-service-types==1.5.0,osc-lib==1.12.0,oslo.concurrency==3.21.1,oslo.config==4.11.1,oslo.context==2.22.0,oslo.i18n==3.17.1,oslo.log==3.42.2,oslo.serialization==2.20.2,oslo.utils==3.28.3,oslotest==3.7.0,paramiko==2.2.1,pbr==3.1.1,positional==1.1.2,prettytable==0.7.2,pyasn1==0.3.1,pycparser==2.18,pyinotify==0.9.6,PyNaCl==1.1.2,pyOpenSSL==19.0.0,pyparsing==2.2.0,pyperclip==1.5.27,python-cinderclient==4.1.0,python-dateutil==2.8.0,python-glanceclient==2.15.0,python-keystoneclient==3.18.0,python-mimeparse==1.6.0,python-neutronclient==6.11.0,python-novaclient==11.1.0,python-openstackclient==3.17.0,python-saharaclient==2.1.0,python-subunit==1.2.0,python-swiftclient==3.6.0,pytz==2017.2,PyYAML==3.12,requests==2.21.0,requestsexceptions==1.4.0,rfc3986==1.1.0,sahara-tests==0.7.1.dev13,simplejson==3.16.0,six==1.10.0,stestr==2.0.0,stevedore==1.25.1,subprocess32==3.5.3,# Editable Git install with no remote (tempest==19.0.1.dev307),-e /opt/stack/tempest,testtools==2.3.0,traceback2==1.4.0,unicodecsv==0.14.1,unittest2==1.1.0,urllib3==1.22,voluptuous==0.10.5,warlock==1.3.0,wcwidth==0.1.7,wrapt==1.10.10
venv-tempest run-test-pre: PYTHONHASHSEED='3515866893'
venv-tempest runtests: commands[0] | tempest list-plugins
Traceback (most recent call last):
File "/opt/stack/tempest/.tox/tempest/bin/tempest", line 6, in <module>
from tempest.cmd.main import main
File "/opt/stack/tempest/tempest/cmd/main.py", line 18, in <module>
from oslo_log import log as logging
File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_log/log.py", line 48, in <module>
from oslo_log import _options
File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_log/_options.py", line 143, in <module>
help='Log rotation type.')
File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 1261, in __init__
max_length=max_length),
File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_config/types.py", line 115, in __init__
self.lower_case_choices = [c.lower() for c in choices]
AttributeError: 'tuple' object has no attribute 'lower'
ERROR: InvocationError for command '/opt/stack/tempest/.tox/tempest/bin/tempest list-plugins' (exited with code 1)
== What happens:
When venv-tempest is first deployed in the native tempest Zuul v3 job, upper-constraints.txt from master is used. This means that a recent version of oslo.config (6.8.0) and oslo.log (3.42.2) is installed.
For example:
2019-02-11 18:59:56.888 | ++ lib/tempest:configure_tempest:543 : tox -revenv-tempest --notest
2019-02-11 18:59:57.573 | venv-tempest recreate: /opt/stack/tempest/.tox/tempest
2019-02-11 19:00:01.355 | venv-tempest installdeps: -chttps://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt, -r/opt/stack/tempest/requirements.txt
2019-02-11 19:00:15.350 | venv-tempest develop-inst: /opt/stack/tempest
2019-02-11 19:00:21.258 | venv-tempest installed: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.,asn1crypto==0.24.0,Babel==2.6.0,bcrypt==3.1.6,certifi==2018.11.29,cffi==1.11.5,chardet==3.0.4,cliff==2.14.0,cmd2==0.8.9,contextlib2==0.5.5,cryptography==2.5,debtcollector==1.20.0,enum34==1.1.6,extras==1.0.0,fasteners==0.14.1,fixtures==3.0.0,funcsigs==1.0.2,functools32==3.2.3.post2,future==0.17.1,idna==2.8,ipaddress==1.0.22,iso8601==0.1.12,jsonschema==2.6.0,linecache2==1.0.0,monotonic==1.5,msgpack==0.6.1,netaddr==0.7.19,netifaces==0.10.9,oslo.concurrency==3.29.0,oslo.config==6.8.0,oslo.context==2.22.0,oslo.i18n==3.23.0,oslo.log==3.42.2,oslo.serialization==2.28.1,oslo.utils==3.40.2,paramiko==2.4.2,pbr==5.1.2,prettytable==0.7.2,pyasn1==0.4.5,pycparser==2.19,pyinotify==0.9.6,PyNaCl==1.3.0,pyparsing==2.3.1,pyperclip==1.7.0,python-dateutil==2.8.0,python-mimeparse==1.6.0,python-subunit==1.3.0,pytz==2018.9,PyYAML==3.13,requests==2.21.0,rfc3986==1.2.0,six==1.12.0,stestr==2.2.0,stevedore==1.30.0,subprocess32==3.5.3,# Editable Git install with no remote (tempest==19.0.1.dev307),-e /opt/stack/tempest,testtools==2.3.0,traceback2==1.4.0,unicodecsv==0.14.1,unittest2==1.1.0,urllib3==1.24.1,voluptuous==0.11.5,wcwidth==0.1.7,wrapt==1.11.1
Then the local upper-constraint.txt file is used (from stable/pike):
2019-02-11 19:00:21.282 | ++ lib/tempest:configure_tempest:545 : tox -evenv-tempest -- pip install -c /opt/stack/requirements/upper-constraints.txt -r requirements.txt
The expected result is the downgrade of both oslo.log and oslo.config, among others.
It is important because the newer oslo.log expects oslo.config to handle a configuration key (log_rotation_type) as a list of tuple. But unfortunately, due to the way the dependencies are handled, oslo.config is downgraded to the version in pike, but the version of oslo.log is not:
tempest 19.0.1.dev307 has requirement oslo.log>=3.36.0, but you'll have oslo-log 3.30.2 which is incompatible.
This means that the newer oslo.log tries to use a specific feature from a version of oslo.config that does not provide it, and the tempest command fails to run.
It looks like the following bug and patch may be related:
https:/ /bugs.launchpad .net/devstack/ +bug/1706009 /review. openstack. org/#/c/ 473525/
https:/
But now we are no more synchronizing the constraints, bumping the minimum. Maybe a possible solution would be to lower the tempest requirements so that it can be installed on every supported branch using the requirements of that branch?