pip isn't case-insensitive when using a file:///-based index-url

[Haiwei Xu]

When running the devtest_seed.sh, I got this error:

Downloading/unpacking markupsafe (from Jinja2->-r /opt/stack/nova/requirements.txt (line 8))
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/pip/req.py", line 1177, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/pip/index.py", line 194, in find_requirement
    page = self._get_page(main_index_url, req)
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/pip/index.py", line 568, in _get_page
    session=self.session,
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/pip/index.py", line 670, in get_page
    resp = session.get(url, headers={"Accept": "text/html"})
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/pip/download.py", line 237, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/pip/download.py", line 194, in send
    stats = os.stat(pathname)
OSError: [Errno 2] No such file or directory: '/tmp/pypi/markupsafe/'

Storing debug log for failure in /root/.pip/pip.log

I'm using mirror, and I can find MarkupSafe in ~/.cache/image-create/pypi/mirror.

[Robert Collins (lifeless)]

This is important to fix because running a mirror without apache is kindof important ;)

[James Polley (tchaypo)]

Further logs are at http://paste.openstack.org/show/74795/

[James Polley (tchaypo)]

Notes from IRC conversation:

lifeless verified that there was a MarkupSafe directory under /tmp/pypi.

xuhaiwei had been using pip 1.0-1build1; but applying http://paste.ubuntu.com/7193216/ (to update this to the latest version) did not make the problem go away.

I can reproduce the error by setting:

DIB_NOPYPI_PIP=1
DIB_COMMON_ELEMENTS=stackuser pip-cache use-ephemeral pypi

[James Polley (tchaypo)]

The referenced /root/.pip/pip.log is very slightly illuminating:

 root@bill:/tmp/pypi# cat /root/.pip/pip.log
------------------------------------------------------------
/usr/local/bin/pip run on Wed Apr 2 20:56:35 2014
Downloading/unpacking markupsafe
  Getting page file:///tmp/pypi/markupsafe/
Cleaning up...
  Removing temporary dir /tmp/pip_build_root...
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1177, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 194, in find_requirement
    page = self._get_page(main_index_url, req)
  File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 568, in _get_page
    session=self.session,
  File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 670, in get_page
    resp = session.get(url, headers={"Accept": "text/html"})
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 237, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 194, in send
    stats = os.stat(pathname)
OSError: [Errno 2] No such file or directory: '/tmp/pypi/markupsafe/'
root@bill:/tmp/pypi#

[James Polley (tchaypo)]

https://github.com/jamezpolley/pip/compare/pypa:develop...develop has been submitted to pip for the PyPA's consideration as a way to solve this problem.

I'm working on an element that will monkey-patch this onto pip inside the chroot when an image is being built.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/84946

[James Polley (tchaypo)]

While we wait for my fix to land in pip, we've been working on a workaround.

We were hoping to be able to patch the pip used in the virtualenvs to have my fix; but it turns out that due to https://bugs.launchpad.net/openstack-ci/+bug/1294381 only the wheel exists in the pypi-mirror cache.

[James Polley (tchaypo)]

The simplest solution I'm aware of right now would be to find the elements which are setting up virtualenvs and have them manually "pip install -r" a requirements.txt (specific to that element would be nice) which contains a list of case-mangled elements. clarkb has a suggested fix to pypi-mirror on https://bugs.launchpad.net/openstack-ci/+bug/1294381 which would give us the sdist for pip, but we'd still have to patch it for each virtualenv.

[Haiwei Xu (xu-haiwei)]

I met a similar error when running devtest_cloud.sh.

Downloading/unpacking pastedeploy>=1.3.3 (from -r /opt/stack/swift/requirements.txt (line 5))
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/opt/stack/venvs/swift/local/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/opt/stack/venvs/swift/local/lib/python2.7/site-packages/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/opt/stack/venvs/swift/local/lib/python2.7/site-packages/pip/req.py", line 1177, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "/opt/stack/venvs/swift/local/lib/python2.7/site-packages/pip/index.py", line 194, in find_requirement
    page = self._get_page(main_index_url, req)
  File "/opt/stack/venvs/swift/local/lib/python2.7/site-packages/pip/index.py", line 568, in _get_page
    session=self.session,
  File "/opt/stack/venvs/swift/local/lib/python2.7/site-packages/pip/index.py", line 670, in get_page
    resp = session.get(url, headers={"Accept": "text/html"})
  File "/opt/stack/venvs/swift/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/opt/stack/venvs/swift/local/lib/python2.7/site-packages/pip/download.py", line 237, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/opt/stack/venvs/swift/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/stack/venvs/swift/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/opt/stack/venvs/swift/local/lib/python2.7/site-packages/pip/download.py", line 194, in send
    stats = os.stat(pathname)
OSError: [Errno 2] No such file or directory: '/tmp/pypi/pastedeploy/'

I add a line 'pip install PasteDeploy' to elements/os-svc-install/bin/os-svc-install, the same fix like what you pointed out in your patch
https://review.openstack.org/84946

But it doesn't work this time.

I can see PasteDeploy has been installed from the log.

Downloading/unpacking PasteDeploy>=1.5.0 (from -r /opt/stack/glance/requirements.txt (line 12))
Downloading/unpacking Routes>=1.12.3 (from -r /opt/stack/glance/requirements.txt (line 13))

[James Polley (tchaypo)]

I'm not certain, but I believe my patch was published as part of pip-1.5.6

If we can verify this we should be able to close this bug.

[Haiwei Xu (xu-haiwei)]

James Polley

Thank you for your job, in fact, I handled this problem by running ' export PIP_INDEX_URL=http://pypi.openstack.org/openstack' before running the script.

I'm using pip-1.0 should I update pip and check it out?

[Derek Higgins (derekh)]

Moving to Fix Released

    - Upgrade the virtualenv installed pip to the latest released pip.
      This is in preperation for the release of pip-1.5.7 which has
      fixes for related-bug: #1301220

    Change-Id: I6441f6f78bfb111f7446362d97a2eddc3c51a99f