tox library v2.2.1 (2015-11-16) failing on existing configurations

Bug #1517427 reported by Ilya Kutukov on 2015-11-18
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Critical
Roman Prykhodchenko

Bug Description

2015-11-16 new tox library version 2.2.1 was released that can't use our existing config due this bug:
https://bitbucket.org/hpk42/tox/issues/285/tox-220-breaks-some-toxini-config-files

Steps:

1. clone e.g. python-fuelclient
$ git clone https://github.com/openstack/python-fuelclient.git

2. Update tox to the latest version or install it
$ pip install --upgrade tox
>Successfully installed tox-2.2.1 virtualenv-13.1.2

3. Run tox
$ cd python-fuelclient && tox

Result:
...
File "/usr/local/lib/python2.7/site-packages/tox/config.py", line 993, in _replace
    return RE_ITEM_REF.sub(self._replace_match, x)
  File "/usr/local/lib/python2.7/site-packages/tox/config.py", line 989, in _replace_match
    return handler(match)
  File "/usr/local/lib/python2.7/site-packages/tox/config.py", line 913, in _replace_env
    env_list = self.getdict('setenv')
  File "/usr/local/lib/python2.7/site-packages/tox/config.py", line 839, in getdict
    s = self.getstring(name, None)
  File "/usr/local/lib/python2.7/site-packages/tox/config.py", line 893, in getstring
    x = self._replace(x)
  File "/usr/local/lib/python2.7/site-packages/tox/config.py", line 993, in _replace
    return RE_ITEM_REF.sub(self._replace_match, x)
  File "/usr/local/lib/python2.7/site-packages/tox/config.py", line 989, in _replace_match
    return handler(match)
  File "/usr/local/lib/python2.7/site-packages/tox/config.py", line 962, in _replace_substitution
    return str(val)
RuntimeError: maximum recursion depth exceeded

Expected:
Project tests and validation is running normally

Examples of failed CI runs:
https://review.openstack.org/#/c/244633/
http://logs.openstack.org/33/244633/5/check/gate-python-fuelclient-python34/4b8f964/console.html
https://review.openstack.org/#/c/222613/
http://logs.openstack.org/13/222613/15/check/gate-python-fuelclient-python26/f0936db/console.html

P.S. Don't forget to clean .tox cache folder if you are trying to reproduce bug with fresh version of tox on project folder where tox was ran.

Ilya Kutukov (ikutukov) on 2015-11-18
Changed in fuel:
status: New → Confirmed
tags: added: area-ci
tags: removed: area-ci
Dmitry Pyzhov (dpyzhov) on 2015-11-18
tags: added: tech-debt
Ilya Kutukov (ikutukov) wrote :

The problem is that tox is real have blocker that failing to parse self-referencing constructions in tox.ini like this:

[testenv]
usedevelop = True
install_command = pip install --allow-external -U {opts} {packages}
whitelist_externals = bash
                      oslo_debug_helper

setenv = VIRTUAL_ENV={envdir}
         ARTIFACTS={toxinidir}/{env:ARTIFACTS:test_run} <<<<<<<

At least python-fuelclient testing is disabled by 2.2.1 version

For now tox author is sweating out solution for this degradation (that was caused by inattentive merge from upstream):
https://bitbucket.org/hpk42/tox/branch/issue285#diff

Dmitry Pyzhov (dpyzhov) wrote :

What changes are required from Fuel side?

Changed in fuel:
status: Confirmed → Incomplete

I propose to temporary go back to tox version 2.1

Roman Prykhodchenko (romcheg) wrote :

tox is installed on OpenStack-CI nodes so the bug has nothing to do with Fuel.

Changed in fuel:
status: Incomplete → Invalid
Roman Prykhodchenko (romcheg) wrote :

We will discuss the issue with tox in #openstack-infra, if there's no fast way to downgrade tox on infra nodes, we will have to do a temporary workaround.

Roman Prykhodchenko (romcheg) wrote :

Let's do a fast workaround to unlock the CI and let the infra team to fix the issue and re-generate node images in a normal way.

Changed in fuel:
status: Invalid → Confirmed
importance: High → Critical
assignee: Fuel Python Team (fuel-python) → Roman Prykhodchenko (romcheg)
tags: removed: tech-debt
Dmitry Pyzhov (dpyzhov) on 2015-11-19
tags: added: tech-debt
Changed in fuel:
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/247452
Committed: https://git.openstack.org/cgit/openstack/python-fuelclient/commit/?id=3e7738fd3fb18a2d5f53b1ecc9706dc53b65a511
Submitter: Jenkins
Branch: master

commit 3e7738fd3fb18a2d5f53b1ecc9706dc53b65a511
Author: Roman Prykhodchenko <email address hidden>
Date: Thu Nov 19 12:50:54 2015 +0100

    Don't set environment variables in tox.ini

    Because of the bug [1] in tox 2.2.1 tox there's a need
    to hard-code all environment variables except
    the ones which are not set by Fuel CI.

    This patch will be reverted as soon as [1] is fixed.

    1. https://bitbucket.org/hpk42/tox/issues/285/tox-220-breaks-some-toxini-config-files

    Closes-bug: #1517427
    Change-Id: I32903c1fd5545ca56410789add7d9c2a8bbacba7

Changed in fuel:
status: In Progress → Fix Committed
tags: added: on-verification
Vasily Gorin (vgorin) wrote :
Changed in fuel:
status: Fix Committed → Fix Released
tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.