Pbr cannot include unreleased dependencies from requirements

[Darragh Bailey]

pbr is unable to support retrieving dependencies directly from git repositories based on the SHA1

Looking at a fork of the openstack zuul project as hosted at https://github.com/BonnyCI/zuul, it was noticed that they do not specify the actual required Github3.py dependency which is currently only available from a GitHub SHA1 as it is as yet unreleased.

E.g. for a project using pbr and a requirements.txt containing the follow dependency:
-e git+https://github.com/sigmavirus24/github3.py.git@8e9ca0056b8fed956b66dafb5398757cd8d8bed9#egg=Github3.py

And using the following command to install in the source directory:

 pip install -U .

Will fail to install the version as specified by the SHA1 and will instead install the last released version from PyPi.

Turning on the verbose mode (pip -v ...) produces output containing the following:

1 location(s) to search for versions of Github3.py:
* https://pypi.python.org/simple/github3-py/
Getting page https://pypi.python.org/simple/github3-py/
Looking up "https://pypi.python.org/simple/github3-py/" in the cache

However switching to using the following command:

 pip -v install -U -r requirements.txt .

Will produce output contain the following message:

Obtaining Github3.py from git+https://github.com/sigmavirus24/github3.py.git@8e9ca0056b8fed956b66dafb5398757cd8d8bed9#egg=Github3.py (from -r requirements.txt (line 6))
  Running command git config --get-regexp remote\..*\.url
  remote.origin.url https://github.com/sigmavirus24/github3.py.git
  Clone in ./.tox/test-pip/src/github3.py exists, and has correct URL (https://github.com/sigmavirus24/github3.py.git)
  Running command git rev-parse HEAD
  8e9ca0056b8fed956b66dafb5398757cd8d8bed9
...

It seems that the requirements after being parsed by pbr and presented to setuptools for pip to process do not contain sufficient information for it to know that it should go to github instead for this dependency.

In an attempt to understand what was going on, I added some 'logger.info' calls around the following block of code in pip (installed in a virtualenv) https://github.com/pypa/pip/blob/b9f70f206138d9a11158b411faf0329deb8d1d8b/pip/req/req_set.py#L368-L377

I used this to dump the contents of `req.__dict__` to see what differences existed:

When using pbr purely req.__dict__ for the Github3.py requirement contained:
{'comes_from': '-r requirements.txt (line 6)', 'req': <Requirement('Github3.py')>, 'target_dir': None, '_wheel_cache': <pip.wheel.WheelCache object at 0x7f855e052f90>, 'use_user_site': None, 'original_link': <Link git+https://github.com/sigmavirus24/github3.py.git@8e9ca0056b8fed956b66dafb5398757cd8d8bed9#egg=Github3.py>, 'satisfied_by': None, '_egg_info_path': None, 'isolated': False, 'markers': None, 'source_dir': None, 'nothing_to_uninstall': False, 'uninstalled': None, 'as_egg': None, '_ideal_build_dir': None, 'conflicts_with': None, '_temp_build_dir': None, 'editable': False, 'update': True, 'prepared': False, 'link': <Link git+https://github.com/sigmavirus24/github3.py.git@8e9ca0056b8fed956b66dafb5398757cd8d8bed9#egg=Github3.py>, 'pycompile': True, 'install_succeeded': None, 'constraint': False, 'is_direct': True, 'extras': set([]), 'options': {}}

When passing the requirements.txt file to pip req.__dict__ for the Github3.py requirement contained:
{'comes_from': <InstallRequirement object: zuul==2.5.2.dev117 from file:///home/baileybd/git/bonnyci-zuul editable=False>, 'req': <Requirement('Github3.py')>, 'target_dir': None, '_wheel_cache': <pip.wheel.WheelCache object at 0x7f2679b18f90>, 'use_user_site': None, 'original_link': None, 'satisfied_by': None, '_egg_info_path': None, 'isolated': False, 'markers': None, 'source_dir': None, 'nothing_to_uninstall': False, 'uninstalled': None, 'as_egg': None, '_ideal_build_dir': None, 'conflicts_with': None, '_temp_build_dir': None, 'editable': False, 'update': True, 'prepared': False, 'link': None, 'pycompile': True, 'install_succeeded': None, 'constraint': False, 'is_direct': False, 'extras': set([]), 'options': {}}

At a guess it appears that the dependency link information indicating to use GitHub does not get passed through to pip in the same expected fashion when having pbr reflect the contents of requirements.txt, as opposed to when pip processes requirements files itself.

Changing the zuul requirements.txt to use the following
---
pbr>=1.8.0

# We need https://github.com/sigmavirus24/github3.py/pull/671 which is not yet
# in a release. This is only in testing because pip install -U . does not
# respect pulling from git and so is installed manually in deployment.
-e git+https://github.com/sigmavirus24/github3.py.git@8e9ca0056b8fed956b66dafb5398757cd8d8bed9#egg=Github3.py-2.0.0beta

PyYAML>=3.1.0
Paste<2.0
WebOb>=1.2.3
paramiko>=1.8.0,<2.0.0
GitPython>=0.3.3,<2.1.2
ordereddict
python-daemon>=2.0.4,<2.1.0
extras
statsd>=1.0.0,<3.0
voluptuous>=0.7
gear>=0.5.7,<1.0.0
apscheduler>=3.0
PrettyTable>=0.6,<0.8
babel>=1.0
six>=1.6.0
pyjwt
cryptography
iso8601
jenkins-job-builder>1.6.0,<2.0
---

And adding the option '--process-dependency-links' to the pip install command to run using the following will work around this issue:

  pip install --process-dependency-links -U .

This seems wrong, 'pip install -U .' and using pbr to provide the requirements automatically or using 'pip install -U -r requirements.txt .' should result in the same outcome and currently it does not.

Package versions:
pbr version: 2.0.0
pip version: 9.0.1