tox library v2.2.1 (2015-11-16) failing on existing configurations

[Ilya Kutukov]

2015-11-16 new tox library version 2.2.1 was released that can't use our existing config due this bug:
https://bitbucket.org/hpk42/tox/issues/285/tox-220-breaks-some-toxini-config-files

Steps:

1. clone e.g. python-fuelclient
$ git clone https://github.com/openstack/python-fuelclient.git

2. Update tox to the latest version or install it
$ pip install --upgrade tox
>Successfully installed tox-2.2.1 virtualenv-13.1.2

3. Run tox
$ cd python-fuelclient && tox

Result:
...
File "/usr/local/lib/python2.7/site-packages/tox/config.py", line 993, in _replace
    return RE_ITEM_REF.sub(self._replace_match, x)
  File "/usr/local/lib/python2.7/site-packages/tox/config.py", line 989, in _replace_match
    return handler(match)
  File "/usr/local/lib/python2.7/site-packages/tox/config.py", line 913, in _replace_env
    env_list = self.getdict('setenv')
  File "/usr/local/lib/python2.7/site-packages/tox/config.py", line 839, in getdict
    s = self.getstring(name, None)
  File "/usr/local/lib/python2.7/site-packages/tox/config.py", line 893, in getstring
    x = self._replace(x)
  File "/usr/local/lib/python2.7/site-packages/tox/config.py", line 993, in _replace
    return RE_ITEM_REF.sub(self._replace_match, x)
  File "/usr/local/lib/python2.7/site-packages/tox/config.py", line 989, in _replace_match
    return handler(match)
  File "/usr/local/lib/python2.7/site-packages/tox/config.py", line 962, in _replace_substitution
    return str(val)
RuntimeError: maximum recursion depth exceeded

Expected:
Project tests and validation is running normally

Examples of failed CI runs:
https://review.openstack.org/#/c/244633/
http://logs.openstack.org/33/244633/5/check/gate-python-fuelclient-python34/4b8f964/console.html
https://review.openstack.org/#/c/222613/
http://logs.openstack.org/13/222613/15/check/gate-python-fuelclient-python26/f0936db/console.html

P.S. Don't forget to clean .tox cache folder if you are trying to reproduce bug with fresh version of tox on project folder where tox was ran.

[Ilya Kutukov (ikutukov)]

The problem is that tox is real have blocker that failing to parse self-referencing constructions in tox.ini like this:

[testenv]
usedevelop = True
install_command = pip install --allow-external -U {opts} {packages}
whitelist_externals = bash
                      oslo_debug_helper

setenv = VIRTUAL_ENV={envdir}
         ARTIFACTS={toxinidir}/{env:ARTIFACTS:test_run} <<<<<<<

At least python-fuelclient testing is disabled by 2.2.1 version

For now tox author is sweating out solution for this degradation (that was caused by inattentive merge from upstream):
https://bitbucket.org/hpk42/tox/branch/issue285#diff

[Dmitry Pyzhov (dpyzhov)]

What changes are required from Fuel side?

[Sylwester Brzeczkowski (sbrzeczkowski)]

I propose to temporary go back to tox version 2.1

[Roman Prykhodchenko (romcheg)]

tox is installed on OpenStack-CI nodes so the bug has nothing to do with Fuel.

[Roman Prykhodchenko (romcheg)]

We will discuss the issue with tox in #openstack-infra, if there's no fast way to downgrade tox on infra nodes, we will have to do a temporary workaround.

[Roman Prykhodchenko (romcheg)]

Let's do a fast workaround to unlock the CI and let the infra team to fix the issue and re-generate node images in a normal way.

[Roman Prykhodchenko (romcheg)]

The fix is here https://review.openstack.org/#/c/247452

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/247452
Committed: https://git.openstack.org/cgit/openstack/python-fuelclient/commit/?id=3e7738fd3fb18a2d5f53b1ecc9706dc53b65a511
Submitter: Jenkins
Branch: master

commit 3e7738fd3fb18a2d5f53b1ecc9706dc53b65a511
Author: Roman Prykhodchenko <email address hidden>
Date: Thu Nov 19 12:50:54 2015 +0100

    Don't set environment variables in tox.ini

    Because of the bug [1] in tox 2.2.1 tox there's a need
    to hard-code all environment variables except
    the ones which are not set by Fuel CI.

    This patch will be reverted as soon as [1] is fixed.

    1. https://bitbucket.org/hpk42/tox/issues/285/tox-220-breaks-some-toxini-config-files

    Closes-bug: #1517427
    Change-Id: I32903c1fd5545ca56410789add7d9c2a8bbacba7

[Vasily Gorin (vgorin)]

Verified.

Log here http://paste.openstack.org/show/480145/