pypi-mirror barfs when wheels fail to build (e.g. httppretty fails with pip 1.5.4)

[Steven DuChene]

Following all of the directions at:

https://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/pypi/README.md

once I get to the run-mirror command of:

run-mirror -b remotes/origin/master --verbose -c mirror.yaml

Each time the process reachs the point where it is going to build the httpretty module, it fails with an error of:

Building wheels for collected packages: httpretty
  Running setup.py bdist_wheel for httpretty
  Destination directory: /home/opsauto/.cache/image-create/pypi/download/wheelhouse
  Complete output from command /tmp/tmpdFUgSI/venv/bin/python -c "import setuptools;__file__='/tmp/tmpdFUgSI/venv/build/httpretty/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /home/opsauto/.cache/image-create/pypi/download/wheelhouse:
  running bdist_wheel

running build

running build_py

creating build

creating build/lib.linux-x86_64-2.7

creating build/lib.linux-x86_64-2.7/httpretty

copying httpretty/__init__.py -> build/lib.linux-x86_64-2.7/httpretty

copying httpretty/compat.py -> build/lib.linux-x86_64-2.7/httpretty

copying httpretty/core.py -> build/lib.linux-x86_64-2.7/httpretty

copying httpretty/errors.py -> build/lib.linux-x86_64-2.7/httpretty

copying httpretty/http.py -> build/lib.linux-x86_64-2.7/httpretty

copying httpretty/utils.py -> build/lib.linux-x86_64-2.7/httpretty

installing to build/bdist.linux-x86_64/wheel

running install

running install_lib

creating build/bdist.linux-x86_64

creating build/bdist.linux-x86_64/wheel

creating build/bdist.linux-x86_64/wheel/httpretty

copying build/lib.linux-x86_64-2.7/httpretty/__init__.py -> build/bdist.linux-x86_64/wheel/httpretty

copying build/lib.linux-x86_64-2.7/httpretty/compat.py -> build/bdist.linux-x86_64/wheel/httpretty

copying build/lib.linux-x86_64-2.7/httpretty/core.py -> build/bdist.linux-x86_64/wheel/httpretty

copying build/lib.linux-x86_64-2.7/httpretty/errors.py -> build/bdist.linux-x86_64/wheel/httpretty

copying build/lib.linux-x86_64-2.7/httpretty/http.py -> build/bdist.linux-x86_64/wheel/httpretty

copying build/lib.linux-x86_64-2.7/httpretty/utils.py -> build/bdist.linux-x86_64/wheel/httpretty

running install_egg_info

running egg_info

writing dependency_links to httpretty.egg-info/dependency_links.txt

writing requirements to httpretty.egg-info/requires.txt

writing httpretty.egg-info/PKG-INFO

writing top-level names to httpretty.egg-info/top_level.txt

warning: manifest_maker: standard file '-c' not found

reading manifest file 'httpretty.egg-info/SOURCES.txt'

reading manifest template 'MANIFEST.in'

writing manifest file 'httpretty.egg-info/SOURCES.txt'

Copying httpretty.egg-info to build/bdist.linux-x86_64/wheel/httpretty-0.8.0-py2.7.egg-info

running install_scripts

Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File "/tmp/tmpdFUgSI/venv/build/httpretty/setup.py", line 60, in <module>

    "Topic :: Software Development :: Testing"],

  File "/usr/lib/python2.7/distutils/core.py", line 152, in setup

    dist.run_commands()

  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands

    self.run_command(cmd)

  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command

    cmd_obj.run()

  File "/tmp/tmpdFUgSI/venv/local/lib/python2.7/site-packages/wheel/bdist_wheel.py", line 222, in run

    self.distinfo_dir)

  File "/tmp/tmpdFUgSI/venv/local/lib/python2.7/site-packages/wheel/bdist_wheel.py", line 387, in egg2dist

    distribution=self.distribution)

  File "/tmp/tmpdFUgSI/venv/local/lib/python2.7/site-packages/wheel/metadata.py", line 160, in pkginfo_to_dict

    new_requirements = list(convert_requirements(requirements))

  File "/tmp/tmpdFUgSI/venv/local/lib/python2.7/site-packages/wheel/metadata.py", line 215, in convert_requirements

    parsed_requirement = pkg_resources.Requirement.parse(req)

  File "/tmp/tmpdFUgSI/venv/local/lib/python2.7/site-packages/pkg_resources.py", line 2630, in parse

    raise ValueError("No requirements found", s)

ValueError: ('No requirements found', '')

----------------------------------------
  Failed building wheel for httpretty
Failed to build httpretty

I have attached the complete output from the whole run-mirrror command. I have also attached my mirror.yaml file as well.

BTW, right after the build of httypretty fails the next build command takes a VERY long time to complete:

Run: /tmp/tmpPU0kUd/venv/bin/pip install -U --exists-action=w --download-cache=/home/opsauto/.cache/image-create/pypi/download/pip/openstack --build /tmp/tmpPU0kUd/build -f /home/opsauto/.cache/image-create/pypi/download/wheelhouse --allow-external netifaces --allow-insecure netifaces --allow-external psutil --allow-insecure psutil --allow-external pysendfile --allow-insecure pysendfile --allow-external pytidylib --allow-insecure pytidylib --no-use-wheel -r /tmp/tmpogLRB6

[Steven DuChene (steven-a-duchene)]

BTW, I also tried removing the downloaded httpretty-0.8.0.tar.gz from the cache to see if by forcing it to download again it would be alright but it still errored out in the same manner.

[Clark Boylan (cboylan)]

What versions of python, pip, and virtualenv are in use? Also, can you reproduce this bug without pypi-mirror? eg `pip wheel httpretty` fail in the same way? It would be good to get a smaller reproduceable failure case if possible.

[Clark Boylan (cboylan)]

I have tested `pip wheel httpretty` and it fails for me locally using pip==1.5.4, setuptools==2.2 in a virtualenv created by virtualenv==1.11.4. I think this is a bug outside of pypi-mirror. pypi-mirror should handle failures gracefully and if it doesn't handle this particular failure we should fix that.

[Clark Boylan (cboylan)]

After digging into this pypi-mirror makes a best effort with wheels. It should still get you tarball packages for things even if wheel builds fail. Not sure pypi-mirror can do anything more to handle cases where upstream packages are broken for wheels.