pip fails to install package from source when extra-index-url returns 404

[Dr. Jens Harbott]

[Impact]
Using `--extra-index-url` with pip results in `HTTPError: 404 Client Error: NOT FOUND`

[Test Case]
cat > /etc/pip.conf <<EOF
[global]
timeout = 60
index-url = http://mirror.dfw.rax.openstack.org/pypi/simple
trusted-host = mirror.dfw.rax.openstack.org
extra-index-url = http://mirror.dfw.rax.openstack.org/wheel/ubuntu-18.04-x86_64
EOF
virtualenv --no-download venv
./venv/bin/pip2 install -U hvac

[Regression Potential]
Minimal; this bug is mainly due to the way the distro pip version does not vendor dependencies - the patches to resolve simply deal with this alternative reality.

[Original Bug Report]
The upstream issue for this is https://github.com/pypa/pip/issues/4195 , the issue only happens when pip is installed from the Ubuntu python-pip-whl package, it disappears when replacing it with the same version from pypi.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: python-pip-whl 9.0.1-2.3~ubuntu1
ProcVersionSignature: User Name 4.15.0-46.49-generic 4.15.18
Uname: Linux 4.15.0-46-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
Date: Tue Apr 2 15:31:57 2019
Ec2AMI: ami-00000036
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: zone01
Ec2InstanceType: blc.8-24
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
SourcePackage: python-pip
UpgradeStatus: No upgrade log present (probably fresh install)

[Dr. Jens Harbott (j-harbott)]

Debian bug for this is https://bugs.debian.org/837764, the small fix should be able to be applied against the Bionic version, too.

[James Page (james-page)]

Sync from Debian unstable fixes in Ubuntu Disco:

python-pip (18.1-5) unstable; urgency=medium

  * Team upload.
  * Refresh hands-off-system-packages.patch.
  * Add Properly_catch_requests_HTTPError_in_index.py.patch: this fixes 404
    when using --extra-index-url due to the way we de-bundle python-requests
    from pip. Thanks to Fabio Natali <email address hidden> for the bug report,
    Clark Boylan for the patch, and all of the OpenStack infra team for their
    help and involving me for this fix (Closes: #837764).

 -- Thomas Goirand <email address hidden> Sat, 30 Mar 2019 21:10:13 +0100

[James Page (james-page)]

package updates for bionic and cosmic uploaded for SRU team review.

[James Page (james-page)]

For those impacted:

  https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3690/+packages

[Timo Aaltonen (tjaalton)]

Hello Dr., or anyone else affected,

Accepted python-pip into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-pip/9.0.1-2.3ubuntu0.18.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Dr., or anyone else affected,

Accepted python-pip into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-pip/9.0.1-2.3~ubuntu1.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Jesse Pretorius (jesse-pretorius)]

I have tested and verified that this works as expected with the package from bionic-proposed. Note that the python-pip package change results in the change in python-pip-whl which is used for virtualenvs.

root@aio1-utility-container-f98ba613:/# apt-cache policy python-pip-whl
python-pip-whl:
  Installed: 9.0.1-2.3~ubuntu1.18.04.1
  Candidate: 9.0.1-2.3~ubuntu1.18.04.1
  Version table:
 *** 9.0.1-2.3~ubuntu1.18.04.1 500
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     9.0.1-2.3~ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages
     9.0.1-2 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

root@aio1-utility-container-f98ba613:/# apt-cache policy python-virtualenv
python-virtualenv:
  Installed: 15.1.0+ds-1.1
  Candidate: 15.1.0+ds-1.1
  Version table:
 *** 15.1.0+ds-1.1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
        100 /var/lib/dpkg/status

root@aio1-utility-container-f98ba613:/# apt-cache policy virtualenv
virtualenv:
  Installed: 15.1.0+ds-1.1
  Candidate: 15.1.0+ds-1.1
  Version table:
 *** 15.1.0+ds-1.1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
        100 /var/lib/dpkg/status

root@aio1-utility-container-f98ba613:/# virtualenv --no-download /tmp/test1
Running virtualenv with interpreter /usr/bin/python2
New python executable in /tmp/test1/bin/python2
Also creating executable in /tmp/test1/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.

root@aio1-utility-container-f98ba613:/# /tmp/test1/bin/pip2 install -U --extra-index-url=http://mirror.dfw.rax.openstack.org/wheel/ubuntu-18.04-x86_64 --trusted-host=mirror.dfw.rax.openstack.org pip setuptools wheel
Collecting pip
  Downloading https://files.pythonhosted.org/packages/d8/f3/413bab4ff08e1fc4828dfc59996d721917df8e8583ea85385d51125dceff/pip-19.0.3-py2.py3-none-any.whl (1.4MB)
    100% |████████████████████████████████| 1.4MB 720kB/s
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/44/56/75e64a8fbbe9e0bd30cfdd58ca1856bc0dc15a43e41504a58d8373f34213/setuptools-40.9.0-py2.py3-none-any.whl (575kB)
    100% |████████████████████████████████| 583kB 1.4MB/s
Collecting wheel
  Downloading https://files.pythonhosted.org/packages/96/ba/a4702cbb6a3a485239fbe9525443446203f00771af9ac000fa3ef2788201/wheel-0.33.1-py2.py3-none-any.whl
Installing collected packages: pip, setuptools, wheel
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
  Found existing installation: setuptools 39.0.1
    Uninstalling setuptools-39.0.1:
      Successfully uninstalled setuptools-39.0.1
  Found existing installation: wheel 0.30.0
    Uninstalling wheel-0.30.0:
      Successfully uninstalled wheel-0.30.0
Successfully installed pip-19.0.3 setuptools-40.9.0 wheel-0.33.1

[James Page (james-page)]

Cosmic verfication:

root@concise-fly:~# virtualenv --no-download venv
Running virtualenv with interpreter /usr/bin/python2
New python executable in /root/venv/bin/python2
Also creating executable in /root/venv/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
root@concise-fly:~# ./venv/bin/pip2 install -U hvac
Collecting hvac
  Downloading http://mirror.dfw.rax.openstack.org/pypifiles/packages/97/96/ee2d10b985bb756cbcc8f177bb4eb5cb780a749fb15cff443f9a33751de5/hvac-0.8.2-py2.py3-none-any.whl (91kB)
    100% |████████████████████████████████| 92kB 224kB/s
Collecting requests>=2.21.0 (from hvac)
  Downloading http://mirror.dfw.rax.openstack.org/pypifiles/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl (57kB)
    100% |████████████████████████████████| 61kB 2.6MB/s
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.21.0->hvac)
  Downloading http://mirror.dfw.rax.openstack.org/pypifiles/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 915kB/s
Collecting idna<2.9,>=2.5 (from requests>=2.21.0->hvac)
  Downloading http://mirror.dfw.rax.openstack.org/pypifiles/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 51.9MB/s
Collecting certifi>=2017.4.17 (from requests>=2.21.0->hvac)
  Downloading http://mirror.dfw.rax.openstack.org/pypifiles/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl (158kB)
    100% |████████████████████████████████| 163kB 1.6MB/s
Collecting urllib3<1.25,>=1.21.1 (from requests>=2.21.0->hvac)
  Downloading http://mirror.dfw.rax.openstack.org/pypifiles/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
    100% |████████████████████████████████| 122kB 7.5MB/s
Installing collected packages: chardet, idna, certifi, urllib3, requests, hvac
Successfully installed certifi-2019.3.9 chardet-3.0.4 hvac-0.8.2 idna-2.8 requests-2.21.0 urllib3-1.24.1
root@concise-fly:~# apt-cache policy python-pip-whly
N: Unable to locate package python-pip-whly
root@concise-fly:~# apt-cache policy python-pip-whl
python-pip-whl:
  Installed: 9.0.1-2.3ubuntu0.18.10.1
  Candidate: 9.0.1-2.3ubuntu0.18.10.1
  Version table:
 *** 9.0.1-2.3ubuntu0.18.10.1 500
        500 http://archive.ubuntu.com/ubuntu cosmic-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     9.0.1-2.3 500
        500 http://archive.ubuntu.com/ubuntu cosmic/universe amd64 Packages

[Łukasz Zemczak (sil2100)]

Thanks for the verification! The upload seems to report some autopkgtest regressions, could you take a look at those?

https://people.canonical.com/~ubuntu-archive/pending-sru.html

[Robie Basak (racb)]

Incomplete pending dep8 regression resolution.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-pip - 9.0.1-2.3~ubuntu1.18.04.1

---------------
python-pip (9.0.1-2.3~ubuntu1.18.04.1) bionic; urgency=medium

  * d/p/bug1822842.patch: Pick fix to resolve issues when using
    --{extra-}-index-urls due to de-bundling of dependencies in
    distro versions of pip (LP: #1822842).

 -- James Page <email address hidden> Wed, 03 Apr 2019 10:23:03 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for python-pip has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Daniel Andersson (drandersson)]

It seems like `9.0.1-2.3~ubuntu1.18.04.1` breaks the `--extra-index-url`
functionality of PIP.

In my understanding, the idea is for `--extra-index-url` to provide a
PyPI-compliant repository that offers a few additional packages, allowing PIP
to fall back onto the instance configured as `--index-url` (defaults to
upstream PyPI). This has been the case with earlier versions, allowing us to
host an internal PyPI repository containing a subset of packages, as well as a
generic local caching PyPI mirror.

In `9.0.1-2.3~ubuntu1.18.04.1`, it seems like the fallback mechanisms do not
work properly, instead failing if _either_ the `--index-url` _or_
`--extra-index-url` instances lack the package.

---

With `~/.config/pip/pip.conf` as:

    [global]
    index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
    extra-index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_extra/simple

With `python3-pip` version `9.0.1-2.3~ubuntu1.18.04.1`, I get a session like:

    ~$ python3 -m venv env/pip-latest
    ~$ . env/pip-latest/bin/activate
    (pip-latest) ~$ pip --version
    pip 9.0.1 from /home/dandersson/env/pip-latest/lib/python3.6/site-packages (python 3.6)
    (pip-latest) ~$ pip install hpt
    Collecting hpt
    Exception:
    Traceback (most recent call last):
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
        status = self.run(options, args)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
        wb.build(autobuilding=True)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
        self.requirement_set.prepare_files(self.finder)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
        ignore_dependencies=self.ignore_dependencies))
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
        require_hashes
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
        self.link = finder.find_requirement(self, upgrade)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
        all_candidates = self.find_all_candidates(req.name)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
        for page in self._get_pages(url_locations, project_name):
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
        page = self._get_page(location)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
        return HTMLPage.get_page(link, session=self.session)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
        resp.raise_for_status()
      File "/home/dandersson/env/pip-latest/share/py...

[Daniel Andersson (drandersson)]

I opened a new bug with the information in the previous comment: https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1833229 .