pypi-mirror does not pickup all necessary python packages

[Steven DuChene]

Following all of the directions at:

https://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/pypi/README.md

Once I went to use the resulting mirror referenced from another system with a fresh checkout of the tripleo-incubator sources, the resulting devtest.sh run would error out complaining there was no suitable pip installable source.

Downloading/unpacking pystache (from os-apply-config)
  Running setup.py egg_info for package pystache
    pystache: using: version '3.3' of <module 'setuptools' from '/root/.cache/tripleo/tripleo-incubator/openstack-tools/local/lib/python2.7/site-packages/setuptools/__init__.pyc'>

Could not find any downloads that satisfy the requirement pip>=1.0 in ./tripleo-incubator/openstack-tools/lib/python2.7/site-packages (from pbr>=0.5.21,<1.0->python-cinderclient)
Downloading/unpacking pip>=1.0 (from pbr>=0.5.21,<1.0->python-cinderclient)
Cleaning up...
No distributions at all found for pip>=1.0 in ./tripleo-incubator/openstack-tools/lib/python2.7/site-packages (from pbr>=0.5.21,<1.0->python-cinderclient)

I looked around and found the only files under /mirror/pip/ was an index.html file and a pip-1.5.4-py2.py3-none-any.whl file.

To solve the immediate problem I went out to https://pypi.python.org/simple/ and downloaded pip-1.5.4.tar.gz, put it in the directory, did an md5 sum of it, and then added the appropriate line to the index.html file in the same directory.

In short if run-mirror was doing the right thing then I would not have had to do this manual step of adding pip-1.5.4.tar.gz to the mirror.

I am including the $HOME/.pip/pip.log file where the error occurance is shown.

[Clark Boylan (cboylan)]

I think the problem here is pip and friends are special cased in pypi-mirror. In that special case they do not use the --no-use-wheel flag to pip so you only get wheels for these packages.

https://git.openstack.org/cgit/openstack-infra/pypi-mirror/tree/pypi_mirror/cmd/run_mirror.py#n260 and https://git.openstack.org/cgit/openstack-infra/pypi-mirror/tree/pypi_mirror/cmd/run_mirror.py#n310 should probably have --no-use-wheel passed to pip as well. Though that second run on line 310 doesn't have a corresponding pip wheel so you may need to add that to the second run too.

[Manuel Desbonnet (manuel-desbonnet)]

I don't think adding --no-use-wheel to the 'upgrade_format' command (lines 260/310 above) will fix this.
If the particular package (specifically pip) is already at the latest available version in the virtualenv, pip does nothing.

I have been using the following patch locally to get around this issue: https://review.openstack.org/84081

[Clark Boylan (cboylan)]

I think 84081 fixes this as a side effect or pip being a transitive dependency for projects (pbr depends on it). The explicit installs of pip, virtualenv, wheel and friends does not use the pip command, they use the upgrade command. We should make sure that the explicit installs of pip and friends do the correct thing instead of relying on the convenient but potentially unreliable behavior 84041 exploits.

Not only that but we don't actually want to reinstall every package. Mirror builds are much quicker the second time around because it takes advantage of precached packages. If we must reinstall pip and friends we should only reinstall pip and friends and not the others.

[Manuel Desbonnet (manuel-desbonnet)]

Good point about the caching.

(I'm not sure I quite understand your point about pip being a transitive dependency - it seems to me that if its listed as a requirement it should be added to the mirror regardless of why it's there - but I guess that's moot in light of what --force-reinstall does with caching).

Is the solution to add the --no-use-wheel and --force-reinstall flags to the 'upgrade_format' template ?

[Clark Boylan (cboylan)]

The openstack infra team is no longer using this tool. I am marking the bug wontfix as we have no intention of continuing to support and feed this tool. Potential future maintainers are more than welcome to setup a bug tracker for this project and suck this bug in though.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/105650

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/105650
Committed: https://git.openstack.org/cgit/openstack-infra/pypi-mirror/commit/?id=fb16c5dc6845374b06b2ffa60f0d49038499c2dc
Submitter: Jenkins
Branch: master

commit fb16c5dc6845374b06b2ffa60f0d49038499c2dc
Author: Gregory Haynes <email address hidden>
Date: Tue Jul 8 21:12:35 2014 -0700

    Force --no-use-mirror installaions

    Some packages require this in order to get an sdist installation.

    Change-Id: I8e6fca71f13835b5019366c34535c64c3c58c6ea
    Closes-Bug: #1294381