tox doesn't work under proxy

[Yuiko Takada]

When a development environment is under a proxy, tox fails like this:
(even if environment variables of the proxy are set.)

$ tox -epep8
pep8 create: /home/system/magnum/.tox/pep8
pep8 installdeps: -r/home/system/magnum/requirements.txt, -r/home/system/magnum/test-requirements.txt
ERROR: invocation failed (exit code 1), logfile: /home/system/magnum/.tox/pep8/log/pep8-1.log
ERROR: actionid: pep8
msg: getenv
cmdargs: [local('/home/system/magnum/.tox/pep8/bin/pip'), 'install', '-U', '-r/home/system/magnum/requirements.txt', '-r/home/system/magnum/test-requirements.txt']
env: {'PATH': '/home/system/magnum/.tox/pep8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games', 'VIRTUAL_ENV': '/home/system/magnum/.tox/pep8', 'PYTHONHASHSEED': '2857363521'}

Collecting Babel>=1.3 (from -r /home/system/magnum/requirements.txt (line 1))
  Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/babel/
  Could not find a version that satisfies the requirement Babel>=1.3 (from -r /home/system/magnum/requirements.txt (line 1)) (from versions: )
No matching distribution found for Babel>=1.3 (from -r /home/system/magnum/requirements.txt (line 1))

ERROR: could not install deps [-r/home/system/magnum/requirements.txt, -r/home/system/magnum/test-requirements.txt]; v = InvocationError('/home/system/magnum/.tox/pep8/bin/pip install -U -r/home/system/magnum/requirements.txt -r/home/system/magnum/test-requirements.txt (see /home/system/magnum/.tox/pep8/log/pep8-1.log)', 1)
___________________________________ summary ____________________________________
ERROR: pep8: could not install deps [-r/home/system/magnum/requirements.txt, -r/home/system/magnum/test-requirements.txt]; v = InvocationError('/home/system/magnum/.tox/pep8/bin/pip install -U -r/home/system/magnum/requirements.txt -r/home/system/magnum/test-requirements.txt (see /home/system/magnum/.tox/pep8/log/pep8-1.log)', 1)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/191626

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/191600
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=eab8c8aa63bbd08b3e246bb9f41bf12f6eafc4c6
Submitter: Jenkins
Branch: master

commit eab8c8aa63bbd08b3e246bb9f41bf12f6eafc4c6
Author: Yuiko Takada <email address hidden>
Date: Mon Jun 15 10:03:21 2015 +0900

    Pass environment variables of proxy to tox

    When a development environment is under a proxy, tox is failed even if
    environment variables of the proxy are set.

    This patch fix this problem.

    See:
    https://review.openstack.org/#/c/189569

    Closes-Bug: #1465086

    Change-Id: I76727e598ebb77dc146370b576d6992db3ca1ec9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/191626
Committed: https://git.openstack.org/cgit/openstack/python-magnumclient/commit/?id=b9ca1d534456aecfd4fd5caac053d4b1b49e231d
Submitter: Jenkins
Branch: master

commit b9ca1d534456aecfd4fd5caac053d4b1b49e231d
Author: Motohiro OTSUKA <email address hidden>
Date: Mon Jun 15 12:54:41 2015 +0900

    Pass environment variables of proxy to tox

    When a development environment is under a proxy, tox is failed even if
    environment variables of the proxy are set.
    This patch fixes this problem.

    See Also: https://review.openstack.org/#/c/191600

    Change-Id: I0045cb2253a4d102196fdab277462891a886fe3a
    Co-Authored-By: Yuiko Takada <email address hidden>
    Closes-Bug: #1465086

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/252120

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/252123

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/252126

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/252127

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/252137

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/252137
Committed: https://git.openstack.org/cgit/openstack/senlin/commit/?id=cec689f64de112bb7629fa7379ace3c8700d473b
Submitter: Jenkins
Branch: master

commit cec689f64de112bb7629fa7379ace3c8700d473b
Author: lvdongbing <email address hidden>
Date: Tue Dec 1 20:50:21 2015 -0500

    Pass environment variables of proxy to tox

    When a development environment is under a proxy, tox is failed even if
    environment variables of the proxy are set.
    This patch fixes this problem.
    See:
    https://review.openstack.org/#/c/189569
    Closes-Bug: #1465086
    Change-Id: Ic227796e201f47db7db7bbe7085ed3fe409e1b13

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ken'ichi Ohmichi (<email address hidden>) on branch: master
Review: https://review.openstack.org/252120

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/252241

[Mehdi Abaakouk (sileht)]

proxy this is not "required" by unittest, just use TOX_TESTENV_PASSENV if you want to add other stuff "not required"

[Mehdi Abaakouk (sileht)]

Also a proxy can be setup only for pip by putting in your .pip/pip.conf:

[global]
proxy = [user:passwd@]proxy.server:port

This solution is safer than using passenv

[Ken'ichi Ohmichi (oomichi)]

The discussion on IRC is the following:

14:58 (sileht) oomichi, have you seen my comment that suggest to configure your .pip.conf
14:58 (sileht) I already use proxy everywhere and I just change this file and that works
14:59 (oomichi) sileht: yeah, I saw
14:59 (sileht) it's universall for all pip based python project
14:59 (oomichi) sileht: but http_proxy is more common on linux
15:00 (sileht) using passenv it is dangerous, a badly configured http_proxy can break unittest
15:00 (oomichi) sileht: what use case for bad configuration?
15:01 (oomichi) sileht: this is just for unit tests meaning development
15:01 (sileht) oomichi, some project setup http server and use request to connect to it
15:02 (sileht) oomichi, if http_proxy is set but not no_proxy with all hosts used in tests that will pass all requests to the proxy
15:03 (oomichi) sileht: yeah, that is very common thing on the combination of http_proxy and no_proxy
15:05 (oomichi) and if that is a problem, the other applications on the env also would be a problem
15:10 (sileht) Anyways if you use a proxy you should known how to configure app to use it, I have proposed solutions that doesn't risk to break tests and change code tree. That just my 2cents I'm not the only developer here.

I still am not sure what is the safe point of that even if blocking its usability.

[Ghanshyam Mann (ghanshyammann)]

@Mehdi, sheeprine - right, actually that can be done with TOX_TESTENV_PASSENV but here whole point is to automate those thing via tox.ini.

So that we can avoid tox specific knowledge and manual effort as much as possible.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/253320

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/253324

[Akihiro Motoki (amotoki)]

While this change might be useful, I am afraid that introducing this settings to some projects will brings asymmetry configurations among various openstack projects. We have many many openstack projects and tox setting inconsistency brings inconsistent developer experience. I am not sure why these projects are selected. it doesn't cover all defcore projects for example.
For example, if neutron has proxy settings and nova does not, when a neutron developer tries to fix some nova fix, he/she will feel different experience and might be confused.

Instead, I would suggest to share common knowledge in one place and encourage new developers to read it.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/253324
Committed: https://git.openstack.org/cgit/openstack/python-designateclient/commit/?id=729301a224a8e3f870092595ca195ba7845054f2
Submitter: Jenkins
Branch: master

commit 729301a224a8e3f870092595ca195ba7845054f2
Author: sonu.kumar <email address hidden>
Date: Fri Dec 4 10:49:54 2015 +0530

    Pass environment variables of proxy to tox

    When a development environment is under a proxy, tox is failed even if
    environment variables of the proxy are set.
    This patch fixes this problem.

    Change-Id: I9bc55bd2ebb546c783df38751c2cb16dbffcb7a5
    Closes-Bug: #1465086

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/253320
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=cfd59d44dbe95c1639302412f7c36c8dfebcb132
Submitter: Jenkins
Branch: master

commit cfd59d44dbe95c1639302412f7c36c8dfebcb132
Author: sonu.kumar <email address hidden>
Date: Fri Dec 4 10:41:03 2015 +0530

    pass environment variables of proxy to tox

    When a development environment is under a proxy, tox is failed even if
    environment variables of the proxy are set.

    This patch fix this problem.

    Change-Id: I25eb1d42ca0e3e3d8a028c22799da383ececcac5
    Closes-Bug: #1465086

[Akihiro Motoki (amotoki)]

we don't need to have environment specific settings in our repo. TOX_TESTENV_PASSENV can solve it.
In addition, having this settings in neutron only does not help much in the neutron stadium. There are various neutron related repositories and developers have inconsistent experience. Setting TOX_TESTENV_PASSENV gives us more consistent developer experience.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kenji Yasui (<email address hidden>) on branch: master
Review: https://review.openstack.org/252127

[OpenStack Infra (hudson-openstack)]

Change abandoned by lvdongbing (<email address hidden>) on branch: master
Review: https://review.openstack.org/252241

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/256202

[Julien Danjou (jdanjou)]

Please stop spamming projects with patches for this bug which is not a bug.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/256202
Committed: https://git.openstack.org/cgit/openstack/python-cueclient/commit/?id=50c5360d57417e1979209d55475fd38c8c6764c4
Submitter: Jenkins
Branch: master

commit 50c5360d57417e1979209d55475fd38c8c6764c4
Author: sonu.kumar <email address hidden>
Date: Fri Dec 11 09:37:16 2015 +0530

    Pass environment variables of proxy to tox

    Change-Id: I894ee696a3bb987a7faf584c3acba56b05ab738f
    Closes-Bug: #1465086

[Doug Hellmann (doug-hellmann)]

It looks like you can work around this problem by setting an environment variable on your dev system. See https://tox.readthedocs.org/en/latest/config.html#confval-passenv=SPACE-SEPARATED-GLOBNAMES

[OpenStack Infra (hudson-openstack)]

Change abandoned by gordon chung (<email address hidden>) on branch: master
Review: https://review.openstack.org/252126

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/designate 2.0.0.0b2 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/senlin 1.0.0.0b2 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/python-designateclient 2.0.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/286177

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/286177
Committed: https://git.openstack.org/cgit/openstack/bandit/commit/?id=78c87e5385ccff5d20a5e654b0466257b0e486f2
Submitter: Jenkins
Branch: master

commit 78c87e5385ccff5d20a5e654b0466257b0e486f2
Author: pradeepcsekar <email address hidden>
Date: Mon Feb 29 09:37:21 2016 -0800

    Pass environment variables of proxy to tox

    When a development environment is under a proxy, tox fails even
    when the environment variables are set.

    This patch fixes the problem by passing them on to the env.

    Change-Id: I903a023918d48e4cd9625576522c4df7d118949b
    Closes-Bug: #1465086

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kaitlin Farr (<email address hidden>) on branch: master
Review: https://review.openstack.org/252123
Reason: To clean up the Barbican review queue, I am abandoning this patch because it has negative reviews and has not been updated since the start of the current development cycle (Newton). Please feel free to restore the patch and address any comments or issues if you'd still like to contribute this patch.