Some Python requirements are not hosted on PyPI

[Jeremy Stanley]

Pip 1.5 (released January 2nd, 2014) will by default refuse to download packages which are linked from PyPI but not hosted on pypi.python.org. The workaround is to whitelist these package names individually with both the --allow-external and --allow-insecure options.

These options are new in pip 1.4, so encoding them will break for people trying to use pip 1.3.x or earlier. Those earlier versions of pip are not secure anyway since they don't connect via HTTPS with host certificate validation, so we should be encouraging people to use 1.4 and later anyway.

The --allow-insecure option is transitioning to a clearer --allow-unverified option name starting with 1.5, but the new form does not work with pip before 1.5 so we should use the old version for now to allow people to transition gracefully. The --allow-insecure form won't be removed until at least pip 1.7 according to comments in the source code.

Virtualenv 1.11 (released the same day) bundles pip 1.5 by default, and so requires these workarounds when using requirements external to PyPI. Be aware that 1.11 is broken for projects using sitepackages=True in their tox.ini. The fix is https://github.com/pypa/virtualenv/commit/a6ca6f4 which is slated to appear in 1.11.1 (no ETA available). We've worked around it on our test infrastructure with https://git.openstack.org/cgit/openstack-infra/config/commit/?id=20cd18a for now, but that is hiding the external-packages issue since we're currently running all tests with pip 1.4.1 as a result.

This bug will also be invisible in our test infrastructure for projects listed as having the PyPI mirror enforced in openstack/requirements (except for jobs which bypass the mirror, such as those for requirements changes), since our update jobs will pull in and mirror external packages and pip sees the mirror as being PyPI itself in that situation.

We'll use this bug to track necessary whitelist updates to tox.ini and test scripts.

[Jeremy Stanley (fungi)]

We're tracking general work for this under item #3 in https://etherpad.openstack.org/p/pip1.5Upgrade but the current changes in flight so far are...

openstack/nova (netaddr):
https://review.openstack.org/65019 (master)
https://review.openstack.org/65020 (stable/havana)
https://review.openstack.org/65021 (stable/grizzly)

openstack/swift (netifaces):
https://review.openstack.org/65023 (master)
https://review.openstack.org/65024 (stable/havana)
https://review.openstack.org/65025 (stable/grizzly)

openstack-infra/reviewday (lazr.authentication): https://review.openstack.org/65026

openstack-infra/elastic-recheck (lazr.authentication): https://review.openstack.org/64898

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/65149

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/65166

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/65166
Committed: https://git.openstack.org/cgit/openstack/tripleo-image-elements/commit/?id=d9cec3c1aec633b027076c01ab750b2e4542b8c3
Submitter: Jenkins
Branch: master

commit d9cec3c1aec633b027076c01ab750b2e4542b8c3
Author: Clint Byrum <email address hidden>
Date: Mon Jan 6 14:34:17 2014 -0800

    Pin virtualenv to 1.10.1 to work around pip bug

    If 1.11 is used with --system-site-packages pip won't be installed in
    the virtualenv, leading to an activated virtualenv not using pip and
    thus not installing anything into the virtualenv.

    Change-Id: Ic9907438b17ca9a32fbeccc4ca806b8b43f41cd1
    Partial-Bug: #1266513

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/65208

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/65209

[Akihiro Motoki (amotoki)]

According to my investigation on neutron and horizon,

- horizon (netaddr)
-- master : affected
-- havana, grizzly : not affected. netaddr 0.7.3 (from PyPI) works well.
- neutron
-- master (netaddr>=0.7.6, psutil) : affected
-- havana (netaddr) : not affected. netaddr 0.7.3 (from PyPI) works well.
-- grizzlly (netifaces) : affected

Do we need to update tox.ini for "non affected" stable branches?

[Akihiro Motoki (amotoki)]

I updated etherpad too.

[Mike Spreitzer (mike-spreitzer)]

It is not sufficient to fix tox.ini; in nova, at least, run_tests.sh invokes tools/install_venv.py --- which has it's own copy of the problem. As noted in Matt's rejection of https://review.openstack.org/#/c/65148/ the fix needs to come through oslo-incubator.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/65149
Committed: https://git.openstack.org/cgit/openstack/python-keystoneclient/commit/?id=01dc81aa148adaa127281f197047636cf2d2c898
Submitter: Jenkins
Branch: master

commit 01dc81aa148adaa127281f197047636cf2d2c898
Author: Brant Knudson <email address hidden>
Date: Mon Jan 6 14:50:45 2014 -0600

    Whitelist external netaddr requirement

    * tox.ini(testenv.install_command): Use the --allow-external and
    --allow-insecure options so that pip 1.5 and later will assent to
    retrieve the netaddr package even though it's not hosted on PyPI.
    The --allow-insecure option is aliased to a clearer
    --allow-unverified wording in 1.5, but the old form is being used to
    avoid breaking users of 1.4.x and will be valid at least through
    1.6.x according to comments in the pip source.

    Change-Id: I7c29467ebdddee7a6c68ec3fd13579e4f0c40d30
    Closes-Bug: #1266513
    Co-Authored-By: Jeremy Stanley

[Jeremy Stanley (fungi)]

Akihiro: thanks! I agree it wasn't immediately obvious to me on the initia patches that havana and grizzly were okay with old enough netaddr to make do with the versions still on PyPI. I'll revert my nova backports of those. I need to check whether the situation with netifaces in swift is the same, though I think it is not.

[Clint Byrum (clint-fewbar)]

Note that netaddr 0.7.10 was uploaded to pypi today.

Excerpts from Jeremy Stanley's message of 2014-01-08 03:24:52 UTC:
> Akihiro: thanks! I agree it wasn't immediately obvious to me on the
> initia patches that havana and grizzly were okay with old enough netaddr
> to make do with the versions still on PyPI. I'll revert my nova
> backports of those. I need to check whether the situation with netifaces
> in swift is the same, though I think it is not.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1266513
>
> Title:
> Some Python requirements are not hosted on PyPI
>
> Status in OpenStack Dashboard (Horizon):
> In Progress
> Status in OpenStack Neutron (virtual network service):
> In Progress
> Status in OpenStack Compute (Nova):
> In Progress
> Status in OpenStack Core Infrastructure:
> In Progress
> Status in Python client library for Keystone:
> Fix Committed
> Status in OpenStack Object Storage (Swift):
> In Progress
> Status in tripleo - openstack on openstack:
> In Progress
>
> Bug description:
> Pip 1.5 (released January 2nd, 2014) will by default refuse to
> download packages which are linked from PyPI but not hosted on
> pypi.python.org. The workaround is to whitelist these package names
> individually with both the --allow-external and --allow-insecure
> options.
>
> These options are new in pip 1.4, so encoding them will break for
> people trying to use pip 1.3.x or earlier. Those earlier versions of
> pip are not secure anyway since they don't connect via HTTPS with host
> certificate validation, so we should be encouraging people to use 1.4
> and later anyway.
>
> The --allow-insecure option is transitioning to a clearer --allow-
> unverified option name starting with 1.5, but the new form does not
> work with pip before 1.5 so we should use the old version for now to
> allow people to transition gracefully. The --allow-insecure form won't
> be removed until at least pip 1.7 according to comments in the source
> code.
>
> Virtualenv 1.11 (released the same day) bundles pip 1.5 by default,
> and so requires these workarounds when using requirements external to
> PyPI. Be aware that 1.11 is broken for projects using
> sitepackages=True in their tox.ini. The fix is
> https://github.com/pypa/virtualenv/commit/a6ca6f4 which is slated to
> appear in 1.11.1 (no ETA available). We've worked around it on our
> test infrastructure with https://git.openstack.org/cgit/openstack-
> infra/config/commit/?id=20cd18a for now, but that is hiding the
> external-packages issue since we're currently running all tests with
> pip 1.4.1 as a result.
>
> This bug will also be invisible in our test infrastructure for
> projects listed as having the PyPI mirror enforced in
> openstack/requirements (except for jobs which bypass the mirror, such
> as those for requirements changes), since our update jobs will pull in
> and mirror external packages and pip sees the mirror as being PyPI
> itself in that situation.
>
> We'll use this bug to track necessary whitelist updates to tox.ini and
> test scripts.
>...

[Jeremy Stanley (fungi)]

https://github.com/drkjam/netaddr/issues/57#issuecomment-31796111

So at this point we can revert the netaddr-specific workarounds unless there are issues requiring us to pin to a version of it which isn't on PyPI.

Unfortunately, at the moment we still need workarounds in place for netifaces, lazr.restful and others.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/65535

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/65019
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=de59c55e77ead72df42c0445c09cf4d63131f6bd
Submitter: Jenkins
Branch: master

commit de59c55e77ead72df42c0445c09cf4d63131f6bd
Author: Jeremy Stanley <email address hidden>
Date: Mon Jan 6 03:06:01 2014 +0000

    Whitelist external netaddr requirement

    * tox.ini(testenv.install_command): Use the --allow-external and
    --allow-insecure options so that pip 1.5 and later will assent to
    retrieve the netaddr package even though it's not hosted on PyPI.
    The --allow-insecure option is aliased to a clearer
    --allow-unverified wording in 1.5, but the old form is being used to
    avoid breaking users of 1.4.x and will be valid at least through
    1.6.x according to comments in the pip source.

    Partial-Bug: #1266513

    Change-Id: I005e752c8cb9bca78b1ebe777eb5230e9bf05bd8

[Jeremy Stanley (fungi)]

Revert of https://review.openstack.org/65019 is proposed as https://review.openstack.org/65579 now that nova should no longer need it.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/65208
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=459a0003905af66e15eca033409e1f425cf35a77
Submitter: Jenkins
Branch: master

commit 459a0003905af66e15eca033409e1f425cf35a77
Author: Zhenguo Niu <email address hidden>
Date: Tue Jan 7 13:24:05 2014 +0800

    Whitelist external netaddr requirement

    * tox.ini(testenv.install_command): Use the --allow-external and
    --allow-insecure options so that pip 1.5 and later will assent to
    retrieve the netaddr package even though it's not hosted on PyPI.
    The --allow-insecure option is aliased to a clearer
    --allow-unverified wording in 1.5, but the old form is being used to
    avoid breaking users of 1.4.x and will be valid at least through
    1.6.x according to comments in the pip source.

    Change-Id: I291684fa65279f0cb86f84e0d8e69bfa4b4087c0
    Partial-Bug: #1266513

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/65535
Committed: https://git.openstack.org/cgit/openstack/python-keystoneclient/commit/?id=9d13d5929fd957bc471dde32bb3b1742f449dbb1
Submitter: Jenkins
Branch: master

commit 9d13d5929fd957bc471dde32bb3b1742f449dbb1
Author: Brant Knudson <email address hidden>
Date: Wed Jan 8 14:41:01 2014 -0600

    Revert "Whitelist external netaddr requirement"

    This reverts commit 01dc81aa148adaa127281f197047636cf2d2c898.

    The netaddr package is now hosted on PyPI, so this workaround is
    not needed anymore.

    Change-Id: I76957aa5e276036921696f27e18efb5494b33120
    Closes-Bug: #1266513

[Alan Pevec (apevec)]

Fix merged to keystone (master): https://review.openstack.org/65835

[Clint Byrum (clint-fewbar)]

Dropping TripleO to HIgh. Builds are succeeding now, but we may be missing some install problems, still need to verify.

[Henry Gessau (gessau)]

Resolved by https://review.openstack.org/68993

[Clint Byrum (clint-fewbar)]

Was fixed in tripleo by this:

commit 09e1abdfdc2f8828074e1c44484def2915d478c4
Author: Michael Kerrin <email address hidden>
Date: Thu Feb 20 15:23:40 2014 +0000

    Install latest version of pip and virtualenv

    We install pip with get-pip.py and use this to install virtualenv. This
    is done as part of the new pip-and-virtualenv element. All elements that
    need pip or virtualenv should use this element to install them.

    Our motivation is that we need to talk to a pypi mirror generated with
    a recent http://github.com/openstack-infra/pypi-mirror This mirror
    caches some 'wheels' that the previous version of pip & virtualenv
    can't install.

    virtualenv contains its own version of pip that is used for all
    virtualenv environments.

    Change-Id: I282fd8fffc8d5707a078f78f2b4571138d7266f3

Which was released first in tripleo-image-elements 0.6.4

[Louis Taylor (kragniz)]

This was fixed in glance in https://review.openstack.org/#/c/59699/

[Derek Higgins (derekh)]

Fixed in tripleo here
https://review.openstack.org/#/c/75071/
Change-Id: I282fd8fffc8d5707a078f78f2b4571138d7266f3