`--extra-index-url` not working for PIP
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
python-pip (Ubuntu) |
Fix Released
|
High
|
James Page | ||
Bionic |
Fix Released
|
Undecided
|
Stefano Rivera |
Bug Description
[Impact]
* The --extra-index-url feature is not working when an index doesn't contain all the packages in the dependency set.
[Test Plan]
# apt install python3-venv python3-dev libglib2.0-dev libcairo-dev libgirepository
# python3 -m venv /tmp/test3env
# /tmp/test3env/
# /tmp/test3env/
A 404 error is the failure.
A successful install is success.
[Where problems could occur]
* Changes were cherry-picked from upstream trunk, where they are still present.
* It's entirely possible that other de-bundling bugs will be fixed by the same changes.
[Other Info]
* https:/
[Original Bug Report]
I originally wrote this as a comment on
https:/
guess it makes sense to open a new separate bug since the other report was
already closed. Pasting my comment contents below.
~$ lsb_release -rd
Description: Ubuntu 18.04.2 LTS
Release: 18.04
~$ apt-cache policy python3-pip
python3-pip:
Installed: 9.0.1-2.
Candidate: 9.0.1-2.
Version table:
*** 9.0.1-2.
500 http://
500 http://
100 /var/lib/
9.0.1-2 500
500 http://
500 http://
---
It seems like `9.0.1-
functionality of PIP.
In my understanding, the idea is for `--extra-index-url` to provide a
PyPI-compliant repository that offers a few additional packages, allowing PIP
to fall back onto the instance configured as `--index-url` (defaults to
upstream PyPI). This has been the case with earlier versions, allowing us to
host an internal PyPI repository containing a subset of packages, as well as a
generic local caching PyPI mirror.
In `9.0.1-
work properly, instead failing if _either_ the `--index-url` _or_
`--extra-index-url` instances lack the package.
---
With `~/.config/
[global]
index-url = https://<user>:
extra-index-url = https://<user>:
With `python3-pip` version `9.0.1-
~$ python3 -m venv env/pip-latest
~$ . env/pip-
(pip-latest) ~$ pip --version
pip 9.0.1 from /home/dandersso
(pip-latest) ~$ pip install hpt
Collecting hpt
Exception:
Traceback (most recent call last):
File "/home/
status = self.run(options, args)
File "/home/
File "/home/
File "/home/
File "/home/
File "/home/
self.link = finder.
File "/home/
File "/home/
for page in self._get_
File "/home/
page = self._get_
File "/home/
return HTMLPage.
File "/home/
File "/home/
raise HTTPError(
requests.
PIP fails with a traceback due to not finding `hpt` on the `--index-url`
instance -- but the point of giving `--extra-index-url` is that `hpt` resides
on that instance.
Trying to install a package that should be present on the `--index-url`
instance:
(pip-latest) ~$ pip install requests
Collecting requests
Exception:
Traceback (most recent call last):
File "/home/
status = self.run(options, args)
File "/home/
File "/home/
File "/home/
File "/home/
File "/home/
self.link = finder.
File "/home/
File "/home/
for page in self._get_
File "/home/
page = self._get_
File "/home/
return HTMLPage.
File "/home/
File "/home/
raise HTTPError(
requests.
Now it fails with a traceback since `requests` does not exist on the
`--extra-index-url` instance, but the purpose of that instance is to provide a
small subset of extra packages, not to be a full mirror.
Changing `~/.config/
[global]
index-url = https://<user>:
removing the `--extra-index-url` instance altogether, it behaves as expected,
not finding the `hpt` package, but being able to install `requests`:
(pip-latest) ~$ pip install hpt
Collecting hpt
Exception:
Traceback (most recent call last):
File "/home/
status = self.run(options, args)
File "/home/
File "/home/
File "/home/
File "/home/
File "/home/
self.link = finder.
File "/home/
File "/home/
for page in self._get_
File "/home/
page = self._get_
File "/home/
return HTMLPage.
File "/home/
File "/home/
raise HTTPError(
requests.
It is expected to not be able to install `hpt`, but PIP should not fail with a
traceback, but just with a diagnostic message like `No matching distribution
found for hpt`.
Installing a common package will correctly use the `--index-url` instance:
(pip-latest) ~$ pip install requests
Collecting requests
Using cached https:/
Collecting idna<2.9,>=2.5 (from requests)
Using cached https:/
Collecting certifi>=2017.4.17 (from requests)
Using cached https:/
Collecting urllib3!
Using cached https:/
Collecting chardet<
Using cached https:/
Installing collected packages: idna, certifi, urllib3, chardet, requests
Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
---
Downgrading `python3-pip` to `9.0.1-2`:
~$ sudo apt install python3-pip=9.0.1-2 python-
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be DOWNGRADED:
python-
0 upgraded, 0 newly installed, 2 downgraded, 0 to remove and 5 not upgraded.
Need to get 1 493 kB of archives.
After this operation, 226 kB disk space will be freed.
Do you want to continue? [Y/n]
Get:1 http://
Get:2 http://
Fetched 1 493 kB in 0s (4 177 kB/s)
dpkg: warning: downgrading python3-pip from 9.0.1-2.
(Reading database ... 196581 files and directories currently installed.)
Preparing to unpack .../python3-
Unpacking python3-pip (9.0.1-2) over (9.0.1-
dpkg: warning: downgrading python-pip-whl from 9.0.1-2.
Preparing to unpack .../python-
Unpacking python-pip-whl (9.0.1-2) over (9.0.1-
Setting up python-pip-whl (9.0.1-2) ...
Setting up python3-pip (9.0.1-2) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
and retrying the first attempt with `~/.config/
[global]
index-url = https://<user>:
extra-index-url = https://<user>:
yields a session like:
~$ python3 -m venv env/pip-previous
~$ . env/pip-
(pip-previous) ~$ pip --version
pip 9.0.1 from /home/dandersso
(pip-previous) ~$ pip install hpt
Collecting hpt
Using cached https:/
Collecting argh (from hpt)
Using cached https:/
Collecting pyyaml==5.1 (from hpt)
Using cached https:/
Collecting argcomplete (from hpt)
Using cached https:/
Collecting distro (from hpt)
Using cached https:/
Collecting requests (from hpt)
Using cached https:/
Collecting termcolor (from hpt)
Using cached https:/
Collecting cilib==v0.1.397 (from hpt)
Using cached https:/
Collecting idna<2.9,>=2.5 (from requests->hpt)
Using cached https:/
Collecting chardet<
Using cached https:/
Collecting certifi>=2017.4.17 (from requests->hpt)
Using cached https:/
Collecting urllib3!
Using cached https:/
Collecting hvac (from cilib==
Using cached https:/
Collecting psycopg2-binary (from cilib==
Using cached https:/
[...SNIP...]
Successfully installed argcomplete-1.10.0 argh-0.26.2 certifi-2019.6.16 chardet-3.0.4 cilib-0.1.397 distro-1.4.0 hpt-0.1.397 hvac-0.9.2 idna-2.8 psycopg2-
PIP now correctly resolves the respective packages to their respective
instances, using `--extra-index-url` for the packages that are _only_ present
there, and falling back to `--index-url` for everything else.
Changing `~/.config/
[global]
index-url = https://<user>:
cleaning out the virtual environment and trying again:
(pip-previous) ~$ deactivate
~$ rm -rf env/pip-previous/
~$ python3 -m venv env/pip-previous
~$ . env/pip-
(pip-previous) ~$ pip --version
pip 9.0.1 from /home/dandersso
(pip-previous) ~$ pip install hpt
Collecting hpt
Could not find a version that satisfies the requirement hpt (from versions: )
No matching distribution found for hpt
(pip-previous) ~$ pip install requests
Collecting requests
Using cached https:/
Collecting chardet<
Using cached https:/
Collecting urllib3!
Using cached https:/
Collecting certifi>=2017.4.17 (from requests)
Using cached https:/
Collecting idna<2.9,>=2.5 (from requests)
Using cached https:/
Installing collected packages: chardet, urllib3, certifi, idna, requests
Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
---
In summary, from my point-of-view, the `9.0.1-
breaks all functionality related to `--extra-
the behaviour when a package is not found when using `--index-url`.
I will be happy to provide any additional information that can help.
information type: | Public → Public Security |
information type: | Public Security → Private Security |
information type: | Private Security → Public |
tags: | added: regression-update |
Changed in python-pip (Ubuntu): | |
status: | Confirmed → Incomplete |
Changed in python-pip (Ubuntu): | |
status: | Incomplete → New |
importance: | Undecided → High |
Changed in python-pip (Ubuntu): | |
status: | New → Incomplete |
assignee: | nobody → James Page (james-page) |
Changed in python-pip (Ubuntu): | |
status: | Incomplete → Confirmed |
tags: |
added: verification-done verification-done-bionic removed: verification-needed verification-needed-bionic |
description: | updated |
Status changed to 'Confirmed' because the bug affects multiple users.