infrastructure-wide issues with tox 1.7.0

[Jeremy Stanley]

A new tox release (1.7.0) appeared today depending on virtualenv 1.11.2 (which in turn bundles pip 1.5.2). Signatures of this in tests include lines in the console log like:

tox.ConfigError: ConfigError: substitution key "posargs" not found

pkg_resources.DistributionNotFound: virtualenv>=1.11.2

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/69872

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/69872
Committed: https://git.openstack.org/cgit/openstack-infra/config/commit/?id=7b029c87d19781b45c18a71f706421ec8aa136c5
Submitter: Jeremy Stanley (<email address hidden>)
Branch: master

commit 7b029c87d19781b45c18a71f706421ec8aa136c5
Author: Jeremy Stanley <email address hidden>
Date: Wed Jan 29 14:23:02 2014 +0000

    Temporarily pin tox<=1.6.1

    The release of tox 1.7.0 happened today, and we're clearly not ready
    for it yet. Stop the bleeding by sticking with 1.6.1 while we try to
    sort out the issues.

    Change-Id: I3289828ea9a0edadcf40c06212b0925957b99f73
    Related-Bug: #1274135

[Jeremy Stanley (fungi)]

After merging the above patch, downgrading tox and virtualenv on the slaves was somewhat messy since pip uninstall was not cleaning up after itsef. The downgrade was eventually successfully accomplished with:

sudo salt '*.slave.openstack.org' cmd.run '
    sudo pip uninstall --yes tox virtualenv
    sudo rm -rf /tmp/pip-build-* /usr/local/lib/python2.7/dist-packages/tox \
        /usr/local/lib/python2.7/dist-packages/virtualenv \
        /usr/local/bin/tox /usr/local/bin/virtualenv
    sudo pip install -U tox==1.6.1 virtualenv==1.10.1'

(Well, almost... I special-cased the precisepy3k slaves to s/pip/pip3/ and s/2.7/3.3/ but otherwise the same.)

[Jeremy Stanley (fungi)]

(...and s/2.7.2.6/ for the centos6 slaves too.)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/70041

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/70041
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=3235ba587de55ea3bdfd1d2bbd29fa0835dbb3e1
Submitter: Jenkins
Branch: master

commit 3235ba587de55ea3bdfd1d2bbd29fa0835dbb3e1
Author: Joe Gordon <email address hidden>
Date: Wed Jan 29 18:13:38 2014 -0800

    Add fingerprint for bug 1274135

    bug 1274135 is related to incompatibilities with pip 1.7

    Change-Id: I159927ec16217f347b2dde523ced0bac97a1d44b
    Related-Bug: #1274135

[Clark Boylan (cboylan)]

Upstream docs (http://tox.readthedocs.org/en/latest/example/general.html#interactively-passing-positional-arguments) indicate that posargs are still a thing. May need to debug this on a case by case basis :/

[Edward Hope-Morley (hopem)]

I've lodged a bug against tox for this - https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror

[Dolph Mathews (dolph)]

As a workaround for anyone running into this, install tox 1.6.x:

  $ pip install --upgrade "tox>=1.6,<1.7"

[Clark Boylan (cboylan)]

I have been working with the tox upstream to sort this out. Long story short is tox 1.7.0 introduced backward incompat changes to posargs parsing which means we can't just fix it without backward compatibility issues. Current suggestion from hpk is we might use a new substitution variable that gives us the behavior we want (the old behavior) so that they can preserve the new behavior with {posargs}.

You can follow along at:
https://bitbucket.org/hpk42/tox/pull-request/85/fix-command-expansion-and-parsing/diff and https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror

[Joe Gordon (jogo)]

No hits in last two weeks

[Marc Abramowitz (msabramo)]

I think the tox 1.7.0 behavior is a bug, though upstream doesn't seem to be convinced of that yet.

I just commented to reinforce @cboylan's point that this is a tox bug and it should be fixed.

https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror#comment-9481536

[Spencer Krum (nibalizer)]

I just hit this working with storyboard.

[Mohammad Hanif (mhanif)]

I just hit with this bug when using tox 1.7.1. I get the following error:

Traceback (most recent call last):
  File "/usr/local/bin/tox", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 2603, in <module>
    working_set.require(__requires__)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 666, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 565, in resolve
    raise DistributionNotFound(req) # XXX put more info here
pkg_resources.DistributionNotFound: virtualenv>=1.11.2

[Jeremy Stanley (fungi)]

This has been fixed upstream, but is slated for the 1.8.x release train so you'll need to use 1.6.x until there is a 1.8.x release of tox.

[Andrew Woodward (xarses)]

Fixed in upstream 1.7.2

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/108515

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/108515
Committed: https://git.openstack.org/cgit/openstack-infra/config/commit/?id=334957d1ede40f02dbd477f2e982df6c6fc12734
Submitter: Jenkins
Branch: master

commit 334957d1ede40f02dbd477f2e982df6c6fc12734
Author: Clark Boylan <email address hidden>
Date: Tue Aug 19 11:39:24 2014 -0700

    Tox fixed the openstack breaking bug use latest

    Hurray upstream tox merged a fix for the issue that had us stuck on
    1.6.1. They have now released a new version so we can consume that fix.
    Start using latest tox again to pull that in.

    See https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror
    for all the details.

    Change-Id: Ia2c7cb07c98c9942113309054b4c40af23596d04
    Fixes-bug: 1274135