pip fails to install package from source when extra-index-url returns 404

Bug #1822842 reported by Dr. Jens Harbott on 2019-04-02
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
python-pip (Ubuntu)
Medium
Unassigned
Bionic
Medium
Unassigned
Cosmic
Medium
Unassigned
Disco
Medium
Unassigned

Bug Description

[Impact]
Using `--extra-index-url` with pip results in `HTTPError: 404 Client Error: NOT FOUND`

[Test Case]
cat > /etc/pip.conf <<EOF
[global]
timeout = 60
index-url = http://mirror.dfw.rax.openstack.org/pypi/simple
trusted-host = mirror.dfw.rax.openstack.org
extra-index-url = http://mirror.dfw.rax.openstack.org/wheel/ubuntu-18.04-x86_64
EOF
virtualenv --no-download venv
./venv/bin/pip2 install -U hvac

[Regression Potential]
Minimal; this bug is mainly due to the way the distro pip version does not vendor dependencies - the patches to resolve simply deal with this alternative reality.

[Original Bug Report]
The upstream issue for this is https://github.com/pypa/pip/issues/4195 , the issue only happens when pip is installed from the Ubuntu python-pip-whl package, it disappears when replacing it with the same version from pypi.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: python-pip-whl 9.0.1-2.3~ubuntu1
ProcVersionSignature: User Name 4.15.0-46.49-generic 4.15.18
Uname: Linux 4.15.0-46-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
Date: Tue Apr 2 15:31:57 2019
Ec2AMI: ami-00000036
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: zone01
Ec2InstanceType: blc.8-24
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
SourcePackage: python-pip
UpgradeStatus: No upgrade log present (probably fresh install)

Dr. Jens Harbott (j-harbott) wrote :
Dr. Jens Harbott (j-harbott) wrote :

Debian bug for this is https://bugs.debian.org/837764, the small fix should be able to be applied against the Bionic version, too.

James Page (james-page) wrote :

Sync from Debian unstable fixes in Ubuntu Disco:

python-pip (18.1-5) unstable; urgency=medium

  * Team upload.
  * Refresh hands-off-system-packages.patch.
  * Add Properly_catch_requests_HTTPError_in_index.py.patch: this fixes 404
    when using --extra-index-url due to the way we de-bundle python-requests
    from pip. Thanks to Fabio Natali <email address hidden> for the bug report,
    Clark Boylan for the patch, and all of the OpenStack infra team for their
    help and involving me for this fix (Closes: #837764).

 -- Thomas Goirand <email address hidden> Sat, 30 Mar 2019 21:10:13 +0100

Changed in python-pip (Ubuntu Disco):
status: New → Fix Released
importance: Undecided → Medium
Changed in python-pip (Ubuntu Cosmic):
importance: Undecided → Medium
Changed in python-pip (Ubuntu Bionic):
importance: Undecided → Medium
Changed in python-pip (Ubuntu Cosmic):
status: New → Triaged
Changed in python-pip (Ubuntu Bionic):
status: New → Triaged
James Page (james-page) on 2019-04-03
description: updated
James Page (james-page) wrote :

package updates for bionic and cosmic uploaded for SRU team review.

Hello Dr., or anyone else affected,

Accepted python-pip into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-pip/9.0.1-2.3ubuntu0.18.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python-pip (Ubuntu Cosmic):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Changed in python-pip (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed-bionic
Timo Aaltonen (tjaalton) wrote :

Hello Dr., or anyone else affected,

Accepted python-pip into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-pip/9.0.1-2.3~ubuntu1.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

I have tested and verified that this works as expected with the package from bionic-proposed. Note that the python-pip package change results in the change in python-pip-whl which is used for virtualenvs.

root@aio1-utility-container-f98ba613:/# apt-cache policy python-pip-whl
python-pip-whl:
  Installed: 9.0.1-2.3~ubuntu1.18.04.1
  Candidate: 9.0.1-2.3~ubuntu1.18.04.1
  Version table:
 *** 9.0.1-2.3~ubuntu1.18.04.1 500
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     9.0.1-2.3~ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages
     9.0.1-2 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

root@aio1-utility-container-f98ba613:/# apt-cache policy python-virtualenv
python-virtualenv:
  Installed: 15.1.0+ds-1.1
  Candidate: 15.1.0+ds-1.1
  Version table:
 *** 15.1.0+ds-1.1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
        100 /var/lib/dpkg/status

root@aio1-utility-container-f98ba613:/# apt-cache policy virtualenv
virtualenv:
  Installed: 15.1.0+ds-1.1
  Candidate: 15.1.0+ds-1.1
  Version table:
 *** 15.1.0+ds-1.1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
        100 /var/lib/dpkg/status

root@aio1-utility-container-f98ba613:/# virtualenv --no-download /tmp/test1
Running virtualenv with interpreter /usr/bin/python2
New python executable in /tmp/test1/bin/python2
Also creating executable in /tmp/test1/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.

root@aio1-utility-container-f98ba613:/# /tmp/test1/bin/pip2 install -U --extra-index-url=http://mirror.dfw.rax.openstack.org/wheel/ubuntu-18.04-x86_64 --trusted-host=mirror.dfw.rax.openstack.org pip setuptools wheel
Collecting pip
  Downloading https://files.pythonhosted.org/packages/d8/f3/413bab4ff08e1fc4828dfc59996d721917df8e8583ea85385d51125dceff/pip-19.0.3-py2.py3-none-any.whl (1.4MB)
    100% |████████████████████████████████| 1.4MB 720kB/s
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/44/56/75e64a8fbbe9e0bd30cfdd58ca1856bc0dc15a43e41504a58d8373f34213/setuptools-40.9.0-py2.py3-none-any.whl (575kB)
    100% |████████████████████████████████| 583kB 1.4MB/s
Collecting wheel
  Downloading https://files.pythonhosted.org/packages/96/ba/a4702cbb6a3a485239fbe9525443446203f00771af9ac000fa3ef2788201/wheel-0.33.1-py2.py3-none-any.whl
Installing collected packages: pip, setuptools, wheel
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
  Found existing installation: setuptools 39.0.1
    Uninstalling setuptools-39.0.1:
      Successfully uninstalled setuptools-39.0.1
  Found existing installation: wheel 0.30.0
    Uninstalling wheel-0.30.0:
      Successfully uninstalled wheel-0.30.0
Successfully installed pip-19.0.3 setuptools-40.9.0 wheel-0.33.1

tags: added: verification-done-bionic
removed: verification-needed-bionic
James Page (james-page) wrote :

Cosmic verfication:

root@concise-fly:~# virtualenv --no-download venv
Running virtualenv with interpreter /usr/bin/python2
New python executable in /root/venv/bin/python2
Also creating executable in /root/venv/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
root@concise-fly:~# ./venv/bin/pip2 install -U hvac
Collecting hvac
  Downloading http://mirror.dfw.rax.openstack.org/pypifiles/packages/97/96/ee2d10b985bb756cbcc8f177bb4eb5cb780a749fb15cff443f9a33751de5/hvac-0.8.2-py2.py3-none-any.whl (91kB)
    100% |████████████████████████████████| 92kB 224kB/s
Collecting requests>=2.21.0 (from hvac)
  Downloading http://mirror.dfw.rax.openstack.org/pypifiles/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl (57kB)
    100% |████████████████████████████████| 61kB 2.6MB/s
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.21.0->hvac)
  Downloading http://mirror.dfw.rax.openstack.org/pypifiles/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 915kB/s
Collecting idna<2.9,>=2.5 (from requests>=2.21.0->hvac)
  Downloading http://mirror.dfw.rax.openstack.org/pypifiles/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 51.9MB/s
Collecting certifi>=2017.4.17 (from requests>=2.21.0->hvac)
  Downloading http://mirror.dfw.rax.openstack.org/pypifiles/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl (158kB)
    100% |████████████████████████████████| 163kB 1.6MB/s
Collecting urllib3<1.25,>=1.21.1 (from requests>=2.21.0->hvac)
  Downloading http://mirror.dfw.rax.openstack.org/pypifiles/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
    100% |████████████████████████████████| 122kB 7.5MB/s
Installing collected packages: chardet, idna, certifi, urllib3, requests, hvac
Successfully installed certifi-2019.3.9 chardet-3.0.4 hvac-0.8.2 idna-2.8 requests-2.21.0 urllib3-1.24.1
root@concise-fly:~# apt-cache policy python-pip-whly
N: Unable to locate package python-pip-whly
root@concise-fly:~# apt-cache policy python-pip-whl
python-pip-whl:
  Installed: 9.0.1-2.3ubuntu0.18.10.1
  Candidate: 9.0.1-2.3ubuntu0.18.10.1
  Version table:
 *** 9.0.1-2.3ubuntu0.18.10.1 500
        500 http://archive.ubuntu.com/ubuntu cosmic-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     9.0.1-2.3 500
        500 http://archive.ubuntu.com/ubuntu cosmic/universe amd64 Packages

tags: added: verification-done verification-done-cosmic
removed: verification-needed verification-needed-cosmic
Łukasz Zemczak (sil2100) wrote :

Thanks for the verification! The upload seems to report some autopkgtest regressions, could you take a look at those?

https://people.canonical.com/~ubuntu-archive/pending-sru.html

Robie Basak (racb) on 2019-05-08
Changed in python-pip (Ubuntu Cosmic):
status: Fix Committed → Incomplete
Robie Basak (racb) wrote :

Incomplete pending dep8 regression resolution.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-pip - 9.0.1-2.3~ubuntu1.18.04.1

---------------
python-pip (9.0.1-2.3~ubuntu1.18.04.1) bionic; urgency=medium

  * d/p/bug1822842.patch: Pick fix to resolve issues when using
    --{extra-}-index-urls due to de-bundling of dependencies in
    distro versions of pip (LP: #1822842).

 -- James Page <email address hidden> Wed, 03 Apr 2019 10:23:03 +0100

Changed in python-pip (Ubuntu Bionic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for python-pip has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Daniel Andersson (drandersson) wrote :
Download full text (18.0 KiB)

It seems like `9.0.1-2.3~ubuntu1.18.04.1` breaks the `--extra-index-url`
functionality of PIP.

In my understanding, the idea is for `--extra-index-url` to provide a
PyPI-compliant repository that offers a few additional packages, allowing PIP
to fall back onto the instance configured as `--index-url` (defaults to
upstream PyPI). This has been the case with earlier versions, allowing us to
host an internal PyPI repository containing a subset of packages, as well as a
generic local caching PyPI mirror.

In `9.0.1-2.3~ubuntu1.18.04.1`, it seems like the fallback mechanisms do not
work properly, instead failing if _either_ the `--index-url` _or_
`--extra-index-url` instances lack the package.

---

With `~/.config/pip/pip.conf` as:

    [global]
    index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
    extra-index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_extra/simple

With `python3-pip` version `9.0.1-2.3~ubuntu1.18.04.1`, I get a session like:

    ~$ python3 -m venv env/pip-latest
    ~$ . env/pip-latest/bin/activate
    (pip-latest) ~$ pip --version
    pip 9.0.1 from /home/dandersson/env/pip-latest/lib/python3.6/site-packages (python 3.6)
    (pip-latest) ~$ pip install hpt
    Collecting hpt
    Exception:
    Traceback (most recent call last):
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
        status = self.run(options, args)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
        wb.build(autobuilding=True)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
        self.requirement_set.prepare_files(self.finder)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
        ignore_dependencies=self.ignore_dependencies))
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
        require_hashes
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
        self.link = finder.find_requirement(self, upgrade)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
        all_candidates = self.find_all_candidates(req.name)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
        for page in self._get_pages(url_locations, project_name):
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
        page = self._get_page(location)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
        return HTMLPage.get_page(link, session=self.session)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
        resp.raise_for_status()
      File "/home/dandersson/env/pip-latest/share/py...

Daniel Andersson (drandersson) wrote :

I opened a new bug with the information in the previous comment: https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1833229 .

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.