tox doesn't work under proxy

Bug #1465086 reported by Yuiko Takada on 2015-06-15
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Bandit
Undecided
Pradeep Chandrasekar
Barbican
Won't Fix
Undecided
Yuiko Takada
Ceilometer
Undecided
Yuiko Takada
Designate
Undecided
sonu
Gnocchi
Won't Fix
Undecided
Ken'ichi Ohmichi
Magnum
Fix Released
Medium
Yuiko Takada
cloudkitty
Undecided
lvdongbing
python-cueclient
Undecided
sonu
python-designateclient
Undecided
sonu
python-magnumclient
Low
yuanying
senlin
Medium
lvdongbing

Bug Description

When a development environment is under a proxy, tox fails like this:
(even if environment variables of the proxy are set.)

$ tox -epep8
pep8 create: /home/system/magnum/.tox/pep8
pep8 installdeps: -r/home/system/magnum/requirements.txt, -r/home/system/magnum/test-requirements.txt
ERROR: invocation failed (exit code 1), logfile: /home/system/magnum/.tox/pep8/log/pep8-1.log
ERROR: actionid: pep8
msg: getenv
cmdargs: [local('/home/system/magnum/.tox/pep8/bin/pip'), 'install', '-U', '-r/home/system/magnum/requirements.txt', '-r/home/system/magnum/test-requirements.txt']
env: {'PATH': '/home/system/magnum/.tox/pep8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games', 'VIRTUAL_ENV': '/home/system/magnum/.tox/pep8', 'PYTHONHASHSEED': '2857363521'}

Collecting Babel>=1.3 (from -r /home/system/magnum/requirements.txt (line 1))
  Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Could not find a version that satisfies the requirement Babel>=1.3 (from -r /home/system/magnum/requirements.txt (line 1)) (from versions: )
No matching distribution found for Babel>=1.3 (from -r /home/system/magnum/requirements.txt (line 1))

ERROR: could not install deps [-r/home/system/magnum/requirements.txt, -r/home/system/magnum/test-requirements.txt]; v = InvocationError('/home/system/magnum/.tox/pep8/bin/pip install -U -r/home/system/magnum/requirements.txt -r/home/system/magnum/test-requirements.txt (see /home/system/magnum/.tox/pep8/log/pep8-1.log)', 1)
___________________________________ summary ____________________________________
ERROR: pep8: could not install deps [-r/home/system/magnum/requirements.txt, -r/home/system/magnum/test-requirements.txt]; v = InvocationError('/home/system/magnum/.tox/pep8/bin/pip install -U -r/home/system/magnum/requirements.txt -r/home/system/magnum/test-requirements.txt (see /home/system/magnum/.tox/pep8/log/pep8-1.log)', 1)

yuanying (yuanying) on 2015-06-15
Changed in magnum:
milestone: none → liberty-1
importance: Undecided → Medium
assignee: nobody → Yuiko Takada (takada-yuiko)
status: New → In Progress
Changed in python-magnumclient:
importance: Undecided → Low
assignee: nobody → yuanying (ootsuka)

Fix proposed to branch: master
Review: https://review.openstack.org/191626

Changed in python-magnumclient:
status: New → In Progress

Reviewed: https://review.openstack.org/191600
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=eab8c8aa63bbd08b3e246bb9f41bf12f6eafc4c6
Submitter: Jenkins
Branch: master

commit eab8c8aa63bbd08b3e246bb9f41bf12f6eafc4c6
Author: Yuiko Takada <email address hidden>
Date: Mon Jun 15 10:03:21 2015 +0900

    Pass environment variables of proxy to tox

    When a development environment is under a proxy, tox is failed even if
    environment variables of the proxy are set.

    This patch fix this problem.

    See:
    https://review.openstack.org/#/c/189569

    Closes-Bug: #1465086

    Change-Id: I76727e598ebb77dc146370b576d6992db3ca1ec9

Changed in magnum:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/191626
Committed: https://git.openstack.org/cgit/openstack/python-magnumclient/commit/?id=b9ca1d534456aecfd4fd5caac053d4b1b49e231d
Submitter: Jenkins
Branch: master

commit b9ca1d534456aecfd4fd5caac053d4b1b49e231d
Author: Motohiro OTSUKA <email address hidden>
Date: Mon Jun 15 12:54:41 2015 +0900

    Pass environment variables of proxy to tox

    When a development environment is under a proxy, tox is failed even if
    environment variables of the proxy are set.
    This patch fixes this problem.

    See Also: https://review.openstack.org/#/c/191600

    Change-Id: I0045cb2253a4d102196fdab277462891a886fe3a
    Co-Authored-By: Yuiko Takada <email address hidden>
    Closes-Bug: #1465086

Changed in python-magnumclient:
status: In Progress → Fix Committed
Adrian Otto (aotto) on 2015-11-24
Changed in python-magnumclient:
status: Fix Committed → Fix Released
Adrian Otto (aotto) on 2015-11-24
Changed in magnum:
status: Fix Committed → Fix Released
Changed in barbican:
assignee: nobody → Yuiko Takada (takada-yuiko)

Fix proposed to branch: master
Review: https://review.openstack.org/252120

Changed in gnocchi:
assignee: nobody → Ken'ichi Ohmichi (oomichi)
status: New → In Progress
Changed in ceilometer:
assignee: nobody → Yuiko Takada (takada-yuiko)
Kenji Yasui (k-yasui) on 2015-12-02
Changed in neutron:
assignee: nobody → Kenji Yasui (k-yasui)

Fix proposed to branch: master
Review: https://review.openstack.org/252123

Changed in barbican:
status: New → In Progress
Changed in ceilometer:
status: New → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/252127

Changed in neutron:
status: New → In Progress
lvdongbing (dbcocle) on 2015-12-02
Changed in senlin:
assignee: nobody → lvdongbing (dbcocle)

Fix proposed to branch: master
Review: https://review.openstack.org/252137

Changed in senlin:
status: New → In Progress

Reviewed: https://review.openstack.org/252137
Committed: https://git.openstack.org/cgit/openstack/senlin/commit/?id=cec689f64de112bb7629fa7379ace3c8700d473b
Submitter: Jenkins
Branch: master

commit cec689f64de112bb7629fa7379ace3c8700d473b
Author: lvdongbing <email address hidden>
Date: Tue Dec 1 20:50:21 2015 -0500

    Pass environment variables of proxy to tox

    When a development environment is under a proxy, tox is failed even if
    environment variables of the proxy are set.
    This patch fixes this problem.
    See:
    https://review.openstack.org/#/c/189569
    Closes-Bug: #1465086
    Change-Id: Ic227796e201f47db7db7bbe7085ed3fe409e1b13

Changed in senlin:
status: In Progress → Fix Committed
Qiming Teng (tengqim) on 2015-12-02
Changed in senlin:
importance: Undecided → Medium
milestone: none → mitaka-1

Change abandoned by Ken'ichi Ohmichi (<email address hidden>) on branch: master
Review: https://review.openstack.org/252120

lvdongbing (dbcocle) on 2015-12-02
Changed in cloudkitty:
assignee: nobody → lvdongbing (dbcocle)
lvdongbing (dbcocle) on 2015-12-02
no longer affects: tempest

Fix proposed to branch: master
Review: https://review.openstack.org/252241

Changed in cloudkitty:
status: New → In Progress
Mehdi Abaakouk (sileht) wrote :

proxy this is not "required" by unittest, just use TOX_TESTENV_PASSENV if you want to add other stuff "not required"

Changed in gnocchi:
status: In Progress → Won't Fix
Changed in ceilometer:
status: In Progress → Won't Fix
Changed in cloudkitty:
status: In Progress → Invalid
Changed in cloudkitty:
status: Invalid → Won't Fix
Mehdi Abaakouk (sileht) wrote :

Also a proxy can be setup only for pip by putting in your .pip/pip.conf:

[global]
proxy = [user:passwd@]proxy.server:port

This solution is safer than using passenv

Ken'ichi Ohmichi (oomichi) wrote :

The discussion on IRC is the following:

14:58 (sileht) oomichi, have you seen my comment that suggest to configure your .pip.conf
14:58 (sileht) I already use proxy everywhere and I just change this file and that works
14:59 (oomichi) sileht: yeah, I saw
14:59 (sileht) it's universall for all pip based python project
14:59 (oomichi) sileht: but http_proxy is more common on linux
15:00 (sileht) using passenv it is dangerous, a badly configured http_proxy can break unittest
15:00 (oomichi) sileht: what use case for bad configuration?
15:01 (oomichi) sileht: this is just for unit tests meaning development
15:01 (sileht) oomichi, some project setup http server and use request to connect to it
15:02 (sileht) oomichi, if http_proxy is set but not no_proxy with all hosts used in tests that will pass all requests to the proxy
15:03 (oomichi) sileht: yeah, that is very common thing on the combination of http_proxy and no_proxy
15:05 (oomichi) and if that is a problem, the other applications on the env also would be a problem
15:10 (sileht) Anyways if you use a proxy you should known how to configure app to use it, I have proposed solutions that doesn't risk to break tests and change code tree. That just my 2cents I'm not the only developer here.

I still am not sure what is the safe point of that even if blocking its usability.

Ghanshyam Mann (ghanshyammann) wrote :

@Mehdi, sheeprine - right, actually that can be done with TOX_TESTENV_PASSENV but here whole point is to automate those thing via tox.ini.

So that we can avoid tox specific knowledge and manual effort as much as possible.

sonu (sonu-bhumca11) on 2015-12-04
Changed in designate:
assignee: nobody → sonu (sonu-bhumca11)

Fix proposed to branch: master
Review: https://review.openstack.org/253320

Changed in designate:
status: New → In Progress
sonu (sonu-bhumca11) on 2015-12-04
Changed in python-designateclient:
assignee: nobody → sonu (sonu-bhumca11)

Fix proposed to branch: master
Review: https://review.openstack.org/253324

Changed in python-designateclient:
status: New → In Progress
Akihiro Motoki (amotoki) wrote :

While this change might be useful, I am afraid that introducing this settings to some projects will brings asymmetry configurations among various openstack projects. We have many many openstack projects and tox setting inconsistency brings inconsistent developer experience. I am not sure why these projects are selected. it doesn't cover all defcore projects for example.
For example, if neutron has proxy settings and nova does not, when a neutron developer tries to fix some nova fix, he/she will feel different experience and might be confused.

Instead, I would suggest to share common knowledge in one place and encourage new developers to read it.

Reviewed: https://review.openstack.org/253324
Committed: https://git.openstack.org/cgit/openstack/python-designateclient/commit/?id=729301a224a8e3f870092595ca195ba7845054f2
Submitter: Jenkins
Branch: master

commit 729301a224a8e3f870092595ca195ba7845054f2
Author: sonu.kumar <email address hidden>
Date: Fri Dec 4 10:49:54 2015 +0530

    Pass environment variables of proxy to tox

    When a development environment is under a proxy, tox is failed even if
    environment variables of the proxy are set.
    This patch fixes this problem.

    Change-Id: I9bc55bd2ebb546c783df38751c2cb16dbffcb7a5
    Closes-Bug: #1465086

Changed in python-designateclient:
status: In Progress → Fix Committed
Changed in designate:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/253320
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=cfd59d44dbe95c1639302412f7c36c8dfebcb132
Submitter: Jenkins
Branch: master

commit cfd59d44dbe95c1639302412f7c36c8dfebcb132
Author: sonu.kumar <email address hidden>
Date: Fri Dec 4 10:41:03 2015 +0530

    pass environment variables of proxy to tox

    When a development environment is under a proxy, tox is failed even if
    environment variables of the proxy are set.

    This patch fix this problem.

    Change-Id: I25eb1d42ca0e3e3d8a028c22799da383ececcac5
    Closes-Bug: #1465086

Akihiro Motoki (amotoki) wrote :

we don't need to have environment specific settings in our repo. TOX_TESTENV_PASSENV can solve it.
In addition, having this settings in neutron only does not help much in the neutron stadium. There are various neutron related repositories and developers have inconsistent experience. Setting TOX_TESTENV_PASSENV gives us more consistent developer experience.

Changed in neutron:
status: In Progress → Won't Fix

Change abandoned by Kenji Yasui (<email address hidden>) on branch: master
Review: https://review.openstack.org/252127

no longer affects: neutron

Change abandoned by lvdongbing (<email address hidden>) on branch: master
Review: https://review.openstack.org/252241

sonu (sonu-bhumca11) on 2015-12-09
Changed in oslo.service:
assignee: nobody → sonu (sonu-bhumca11)
status: New → In Progress
sonu (sonu-bhumca11) on 2015-12-11
Changed in python-cueclient:
assignee: nobody → sonu (sonu-bhumca11)
status: New → In Progress
Qiming Teng (tengqim) on 2015-12-14
Changed in senlin:
status: Fix Committed → Fix Released
Julien Danjou (jdanjou) wrote :

Please stop spamming projects with patches for this bug which is not a bug.

Reviewed: https://review.openstack.org/256202
Committed: https://git.openstack.org/cgit/openstack/python-cueclient/commit/?id=50c5360d57417e1979209d55475fd38c8c6764c4
Submitter: Jenkins
Branch: master

commit 50c5360d57417e1979209d55475fd38c8c6764c4
Author: sonu.kumar <email address hidden>
Date: Fri Dec 11 09:37:16 2015 +0530

    Pass environment variables of proxy to tox

    Change-Id: I894ee696a3bb987a7faf584c3acba56b05ab738f
    Closes-Bug: #1465086

Changed in python-cueclient:
status: In Progress → Fix Released
Doug Hellmann (doug-hellmann) wrote :

It looks like you can work around this problem by setting an environment variable on your dev system. See https://tox.readthedocs.org/en/latest/config.html#confval-passenv=SPACE-SEPARATED-GLOBNAMES

Change abandoned by gordon chung (<email address hidden>) on branch: master
Review: https://review.openstack.org/252126

This issue was fixed in the openstack/designate 2.0.0.0b2 development milestone.

This issue was fixed in the openstack/senlin 1.0.0.0b2 development milestone.

This issue was fixed in the openstack/python-designateclient 2.0.0 release.

no longer affects: oslo.service
no longer affects: oslo.context
Changed in bandit:
assignee: nobody → Pradeep Chandrasekar (pradeepcsekar)
Changed in bandit:
status: New → In Progress

Reviewed: https://review.openstack.org/286177
Committed: https://git.openstack.org/cgit/openstack/bandit/commit/?id=78c87e5385ccff5d20a5e654b0466257b0e486f2
Submitter: Jenkins
Branch: master

commit 78c87e5385ccff5d20a5e654b0466257b0e486f2
Author: pradeepcsekar <email address hidden>
Date: Mon Feb 29 09:37:21 2016 -0800

    Pass environment variables of proxy to tox

    When a development environment is under a proxy, tox fails even
    when the environment variables are set.

    This patch fixes the problem by passing them on to the env.

    Change-Id: I903a023918d48e4cd9625576522c4df7d118949b
    Closes-Bug: #1465086

Changed in bandit:
status: In Progress → Fix Released

Change abandoned by Kaitlin Farr (<email address hidden>) on branch: master
Review: https://review.openstack.org/252123
Reason: To clean up the Barbican review queue, I am abandoning this patch because it has negative reviews and has not been updated since the start of the current development cycle (Newton). Please feel free to restore the patch and address any comments or issues if you'd still like to contribute this patch.

Changed in designate:
status: Fix Committed → Fix Released
Changed in barbican:
status: In Progress → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers