infrastructure-wide issues with tox 1.7.0

Bug #1274135 reported by Jeremy Stanley
124
This bug affects 26 people
Affects Status Importance Assigned to Milestone
OpenStack Core Infrastructure
Fix Released
Critical
Clark Boylan

Bug Description

A new tox release (1.7.0) appeared today depending on virtualenv 1.11.2 (which in turn bundles pip 1.5.2). Signatures of this in tests include lines in the console log like:

tox.ConfigError: ConfigError: substitution key "posargs" not found

pkg_resources.DistributionNotFound: virtualenv>=1.11.2

Tags: gate-failure
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to config (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/69872

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to config (master)

Reviewed: https://review.openstack.org/69872
Committed: https://git.openstack.org/cgit/openstack-infra/config/commit/?id=7b029c87d19781b45c18a71f706421ec8aa136c5
Submitter: Jeremy Stanley (<email address hidden>)
Branch: master

commit 7b029c87d19781b45c18a71f706421ec8aa136c5
Author: Jeremy Stanley <email address hidden>
Date: Wed Jan 29 14:23:02 2014 +0000

    Temporarily pin tox<=1.6.1

    The release of tox 1.7.0 happened today, and we're clearly not ready
    for it yet. Stop the bleeding by sticking with 1.6.1 while we try to
    sort out the issues.

    Change-Id: I3289828ea9a0edadcf40c06212b0925957b99f73
    Related-Bug: #1274135

description: updated
Revision history for this message
Jeremy Stanley (fungi) wrote :

After merging the above patch, downgrading tox and virtualenv on the slaves was somewhat messy since pip uninstall was not cleaning up after itsef. The downgrade was eventually successfully accomplished with:

sudo salt '*.slave.openstack.org' cmd.run '
    sudo pip uninstall --yes tox virtualenv
    sudo rm -rf /tmp/pip-build-* /usr/local/lib/python2.7/dist-packages/tox \
        /usr/local/lib/python2.7/dist-packages/virtualenv \
        /usr/local/bin/tox /usr/local/bin/virtualenv
    sudo pip install -U tox==1.6.1 virtualenv==1.10.1'

(Well, almost... I special-cased the precisepy3k slaves to s/pip/pip3/ and s/2.7/3.3/ but otherwise the same.)

Revision history for this message
Jeremy Stanley (fungi) wrote :

(...and s/2.7.2.6/ for the centos6 slaves too.)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to elastic-recheck (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/70041

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to elastic-recheck (master)

Reviewed: https://review.openstack.org/70041
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=3235ba587de55ea3bdfd1d2bbd29fa0835dbb3e1
Submitter: Jenkins
Branch: master

commit 3235ba587de55ea3bdfd1d2bbd29fa0835dbb3e1
Author: Joe Gordon <email address hidden>
Date: Wed Jan 29 18:13:38 2014 -0800

    Add fingerprint for bug 1274135

    bug 1274135 is related to incompatibilities with pip 1.7

    Change-Id: I159927ec16217f347b2dde523ced0bac97a1d44b
    Related-Bug: #1274135

Revision history for this message
Clark Boylan (cboylan) wrote :

Upstream docs (http://tox.readthedocs.org/en/latest/example/general.html#interactively-passing-positional-arguments) indicate that posargs are still a thing. May need to debug this on a case by case basis :/

Revision history for this message
Edward Hope-Morley (hopem) wrote :

I've lodged a bug against tox for this - https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror

Jeremy Stanley (fungi)
Changed in openstack-ci:
assignee: Jeremy Stanley (fungi) → Clark Boylan (cboylan)
Revision history for this message
Dolph Mathews (dolph) wrote :

As a workaround for anyone running into this, install tox 1.6.x:

  $ pip install --upgrade "tox>=1.6,<1.7"

Revision history for this message
Clark Boylan (cboylan) wrote :

I have been working with the tox upstream to sort this out. Long story short is tox 1.7.0 introduced backward incompat changes to posargs parsing which means we can't just fix it without backward compatibility issues. Current suggestion from hpk is we might use a new substitution variable that gives us the behavior we want (the old behavior) so that they can preserve the new behavior with {posargs}.

You can follow along at:
https://bitbucket.org/hpk42/tox/pull-request/85/fix-command-expansion-and-parsing/diff and https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror

Revision history for this message
Joe Gordon (jogo) wrote :

No hits in last two weeks

Revision history for this message
Marc Abramowitz (msabramo) wrote :

I think the tox 1.7.0 behavior is a bug, though upstream doesn't seem to be convinced of that yet.

I just commented to reinforce @cboylan's point that this is a tox bug and it should be fixed.

https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror#comment-9481536

Revision history for this message
Spencer Krum (nibalizer) wrote :

I just hit this working with storyboard.

Revision history for this message
Mohammad Hanif (mhanif) wrote :

I just hit with this bug when using tox 1.7.1. I get the following error:

Traceback (most recent call last):
  File "/usr/local/bin/tox", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 2603, in <module>
    working_set.require(__requires__)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 666, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 565, in resolve
    raise DistributionNotFound(req) # XXX put more info here
pkg_resources.DistributionNotFound: virtualenv>=1.11.2

Revision history for this message
Jeremy Stanley (fungi) wrote :

This has been fixed upstream, but is slated for the 1.8.x release train so you'll need to use 1.6.x until there is a 1.8.x release of tox.

Revision history for this message
Andrew Woodward (xarses) wrote :

Fixed in upstream 1.7.2

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (master)

Fix proposed to branch: master
Review: https://review.openstack.org/108515

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (master)

Reviewed: https://review.openstack.org/108515
Committed: https://git.openstack.org/cgit/openstack-infra/config/commit/?id=334957d1ede40f02dbd477f2e982df6c6fc12734
Submitter: Jenkins
Branch: master

commit 334957d1ede40f02dbd477f2e982df6c6fc12734
Author: Clark Boylan <email address hidden>
Date: Tue Aug 19 11:39:24 2014 -0700

    Tox fixed the openstack breaking bug use latest

    Hurray upstream tox merged a fix for the issue that had us stuck on
    1.6.1. They have now released a new version so we can consume that fix.
    Start using latest tox again to pull that in.

    See https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror
    for all the details.

    Change-Id: Ia2c7cb07c98c9942113309054b4c40af23596d04
    Fixes-bug: 1274135

Changed in openstack-ci:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.