| 2019-06-18 12:23:01 |
Daniel Andersson |
bug |
|
|
added bug |
| 2019-06-18 19:38:34 |
Launchpad Janitor |
python-pip (Ubuntu): status |
New |
Confirmed |
|
| 2019-06-20 08:57:51 |
Leon Wright |
bug |
|
|
added subscriber Leon Wright |
| 2019-06-20 19:04:12 |
Rodrigo Menezes |
bug |
|
|
added subscriber Rodrigo Menezes |
| 2019-06-22 13:54:44 |
Mauricio Villegas |
information type |
Public |
Public Security |
|
| 2019-06-22 13:54:47 |
Mauricio Villegas |
information type |
Public Security |
Private Security |
|
| 2019-06-22 13:55:08 |
Mauricio Villegas |
information type |
Private Security |
Public |
|
| 2019-06-29 00:08:33 |
Michael Knight |
bug |
|
|
added subscriber Michael Knight |
| 2019-07-01 08:55:18 |
Dr. Jens Harbott |
bug |
|
|
added subscriber Dr. Jens Harbott |
| 2019-07-02 15:05:25 |
Valtteri Mäkelä |
bug |
|
|
added subscriber Valtteri Mäkelä |
| 2019-07-10 10:24:36 |
Matheus |
bug |
|
|
added subscriber Matheus |
| 2019-09-06 21:54:58 |
Seth Arnold |
tags |
|
regression-update |
|
| 2019-09-16 02:13:09 |
Johnny |
bug |
|
|
added subscriber Johnny |
| 2019-09-24 15:41:06 |
James Page |
python-pip (Ubuntu): status |
Confirmed |
Incomplete |
|
| 2019-09-25 12:39:31 |
James Page |
python-pip (Ubuntu): status |
Incomplete |
New |
|
| 2019-09-25 12:39:33 |
James Page |
python-pip (Ubuntu): importance |
Undecided |
High |
|
| 2019-09-26 08:14:34 |
James Page |
python-pip (Ubuntu): status |
New |
Incomplete |
|
| 2019-09-26 08:14:37 |
James Page |
python-pip (Ubuntu): assignee |
|
James Page (james-page) |
|
| 2019-10-23 17:09:49 |
Robert Sigler |
bug |
|
|
added subscriber Robert Sigler |
| 2019-12-09 13:35:03 |
chbrosso |
bug watch added |
|
https://github.com/pypa/pip/issues/6799 |
|
| 2019-12-09 13:35:26 |
chbrosso |
bug |
|
|
added subscriber chbrosso |
| 2019-12-16 17:08:59 |
Akkana Peck |
bug |
|
|
added subscriber Akkana Peck |
| 2019-12-16 17:23:40 |
Akkana Peck |
bug watch added |
|
https://github.com/pypa/pip/issues/7486 |
|
| 2020-01-14 10:15:32 |
chbrosso |
python-pip (Ubuntu): status |
Incomplete |
Confirmed |
|
| 2020-01-14 14:45:35 |
chbrosso |
attachment added |
|
Fix-1833229.patch https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1833229/+attachment/5320130/+files/Fix-1833229.patch |
|
| 2020-01-14 16:24:10 |
Ubuntu Foundations Team Bug Bot |
tags |
regression-update |
patch regression-update |
|
| 2020-01-14 16:24:17 |
Ubuntu Foundations Team Bug Bot |
bug |
|
|
added subscriber Ubuntu Review Team |
| 2020-04-24 10:05:16 |
Gerard Weatherby |
bug |
|
|
added subscriber Gerard Weatherby |
| 2020-05-05 18:08:16 |
Scott Kitterman |
python-pip (Ubuntu): status |
Confirmed |
Fix Released |
|
| 2020-08-30 03:49:34 |
Kai Kasurinen |
bug |
|
|
added subscriber Kai Kasurinen |
| 2021-04-30 18:51:08 |
Stefano Rivera |
nominated for series |
|
Ubuntu Bionic |
|
| 2021-04-30 18:51:08 |
Stefano Rivera |
bug task added |
|
python-pip (Ubuntu Bionic) |
|
| 2021-04-30 18:55:40 |
Stefano Rivera |
description |
I originally wrote this as a comment on
https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1822842 , but I
guess it makes sense to open a new separate bug since the other report was
already closed. Pasting my comment contents below.
~$ lsb_release -rd
Description: Ubuntu 18.04.2 LTS
Release: 18.04
~$ apt-cache policy python3-pip
python3-pip:
Installed: 9.0.1-2.3~ubuntu1.18.04.1
Candidate: 9.0.1-2.3~ubuntu1.18.04.1
Version table:
*** 9.0.1-2.3~ubuntu1.19.04.1 500
500 http://se.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 packages
500 http://se.archive.ubuntu.com/ubuntu bionic-updates/universe i386 packages
100 /var/lib/dpkg/status
9.0.1-2 500
500 http://se.archive.ubuntu.com/ubuntu bionic/universe amd64 packages
500 http://se.archive.ubuntu.com/ubuntu bionic/universe i386 packages
---
It seems like `9.0.1-2.3~ubuntu1.18.04.1` breaks the `--extra-index-url`
functionality of PIP.
In my understanding, the idea is for `--extra-index-url` to provide a
PyPI-compliant repository that offers a few additional packages, allowing PIP
to fall back onto the instance configured as `--index-url` (defaults to
upstream PyPI). This has been the case with earlier versions, allowing us to
host an internal PyPI repository containing a subset of packages, as well as a
generic local caching PyPI mirror.
In `9.0.1-2.3~ubuntu1.18.04.1`, it seems like the fallback mechanisms do not
work properly, instead failing if _either_ the `--index-url` _or_
`--extra-index-url` instances lack the package.
---
With `~/.config/pip/pip.conf` as:
[global]
index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
extra-index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_extra/simple
With `python3-pip` version `9.0.1-2.3~ubuntu1.18.04.1`, I get a session like:
~$ python3 -m venv env/pip-latest
~$ . env/pip-latest/bin/activate
(pip-latest) ~$ pip --version
pip 9.0.1 from /home/dandersson/env/pip-latest/lib/python3.6/site-packages (python 3.6)
(pip-latest) ~$ pip install hpt
Collecting hpt
Exception:
Traceback (most recent call last):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
wb.build(autobuilding=True)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
require_hashes
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
page = self._get_page(location)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
resp.raise_for_status()
File "/home/dandersson/env/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple/hpt/
PIP fails with a traceback due to not finding `hpt` on the `--index-url`
instance -- but the point of giving `--extra-index-url` is that `hpt` resides
on that instance.
Trying to install a package that should be present on the `--index-url`
instance:
(pip-latest) ~$ pip install requests
Collecting requests
Exception:
Traceback (most recent call last):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
wb.build(autobuilding=True)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
require_hashes
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
page = self._get_page(location)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
resp.raise_for_status()
File "/home/dandersson/env/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://artifactory.internal/artifactory/api/pypi/PyPI_extra/simple/requests/
Now it fails with a traceback since `requests` does not exist on the
`--extra-index-url` instance, but the purpose of that instance is to provide a
small subset of extra packages, not to be a full mirror.
Changing `~/.config/pip/pip.conf` to:
[global]
index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
removing the `--extra-index-url` instance altogether, it behaves as expected,
not finding the `hpt` package, but being able to install `requests`:
(pip-latest) ~$ pip install hpt
Collecting hpt
Exception:
Traceback (most recent call last):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
wb.build(autobuilding=True)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
require_hashes
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
page = self._get_page(location)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
resp.raise_for_status()
File "/home/dandersson/env/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple/hpt/
It is expected to not be able to install `hpt`, but PIP should not fail with a
traceback, but just with a diagnostic message like `No matching distribution
found for hpt`.
Installing a common package will correctly use the `--index-url` instance:
(pip-latest) ~$ pip install requests
Collecting requests
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Installing collected packages: idna, certifi, urllib3, chardet, requests
Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
---
Downgrading `python3-pip` to `9.0.1-2`:
~$ sudo apt install python3-pip=9.0.1-2 python-pip-whl=9.0.1-2
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be DOWNGRADED:
python-pip-whl python3-pip
0 upgraded, 0 newly installed, 2 downgraded, 0 to remove and 5 not upgraded.
Need to get 1 493 kB of archives.
After this operation, 226 kB disk space will be freed.
Do you want to continue? [Y/n]
Get:1 http://se.archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pip all 9.0.1-2 [114 kB]
Get:2 http://se.archive.ubuntu.com/ubuntu bionic/universe amd64 python-pip-whl all 9.0.1-2 [1 379 kB]
Fetched 1 493 kB in 0s (4 177 kB/s)
dpkg: warning: downgrading python3-pip from 9.0.1-2.3~ubuntu1.18.04.1 to 9.0.1-2
(Reading database ... 196581 files and directories currently installed.)
Preparing to unpack .../python3-pip_9.0.1-2_all.deb ...
Unpacking python3-pip (9.0.1-2) over (9.0.1-2.3~ubuntu1.18.04.1) ...
dpkg: warning: downgrading python-pip-whl from 9.0.1-2.3~ubuntu1.18.04.1 to 9.0.1-2
Preparing to unpack .../python-pip-whl_9.0.1-2_all.deb ...
Unpacking python-pip-whl (9.0.1-2) over (9.0.1-2.3~ubuntu1.18.04.1) ...
Setting up python-pip-whl (9.0.1-2) ...
Setting up python3-pip (9.0.1-2) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
and retrying the first attempt with `~/.config/pip/pip.conf` as:
[global]
index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
extra-index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_extra/simple
yields a session like:
~$ python3 -m venv env/pip-previous
~$ . env/pip-previous/bin/activate
(pip-previous) ~$ pip --version
pip 9.0.1 from /home/dandersson/env/pip-previous/lib/python3.6/site-packages (python 3.6)
(pip-previous) ~$ pip install hpt
Collecting hpt
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_extra/packages/hpt/0.1.397/hpt-0.1.397.tar.gz
Collecting argh (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/06/1c/e667a7126f0b84aaa1c56844337bf0ac12445d1beb9c8a6199a7314944bf/argh-0.26.2-py2.py3-none-any.whl
Collecting pyyaml==5.1 (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/9f/2c/9417b5c774792634834e730932745bc09a7d36754ca00acf1ccd1ac2594d/PyYAML-5.1.tar.gz
Collecting argcomplete (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/4d/82/f44c9661e479207348a979b1f6f063625d11dc4ca6256af053719bbb0124/argcomplete-1.10.0-py2.py3-none-any.whl
Collecting distro (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/ea/35/82f79b92fa4d937146c660a6482cee4f3dfa1f97ff3d2a6f3ecba33e712e/distro-1.4.0-py2.py3-none-any.whl
Collecting requests (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting termcolor (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz
Collecting cilib==v0.1.397 (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_extra/packages/cilib/0.1.397/cilib-0.1.397.tar.gz
Collecting idna<2.9,>=2.5 (from requests->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting hvac (from cilib==v0.1.397->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/7b/2c/b95b551769459406ca440788c32d699820f4c875020ff95c6e3844eee6d3/hvac-0.9.2-py2.py3-none-any.whl
Collecting psycopg2-binary (from cilib==v0.1.397->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/b1/35/75c9c2d9cfc073ab6c42b2d8e91ff58c9b99f4ed7ed56b36647642e6080e/psycopg2_binary-2.8.3-cp36-cp36m-manylinux1_x86_64.whl
[...SNIP...]
Successfully installed argcomplete-1.10.0 argh-0.26.2 certifi-2019.6.16 chardet-3.0.4 cilib-0.1.397 distro-1.4.0 hpt-0.1.397 hvac-0.9.2 idna-2.8 psycopg2-binary-2.8.3 pyyaml-5.1 requests-2.22.0 termcolor-1.1.0 urllib3-1.25.3
PIP now correctly resolves the respective packages to their respective
instances, using `--extra-index-url` for the packages that are _only_ present
there, and falling back to `--index-url` for everything else.
Changing `~/.config/pip/pip.conf` to:
[global]
index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
cleaning out the virtual environment and trying again:
(pip-previous) ~$ deactivate
~$ rm -rf env/pip-previous/
~$ python3 -m venv env/pip-previous
~$ . env/pip-previous/bin/activate
(pip-previous) ~$ pip --version
pip 9.0.1 from /home/dandersson/env/pip-previous/lib/python3.6/site-packages (python 3.6)
(pip-previous) ~$ pip install hpt
Collecting hpt
Could not find a version that satisfies the requirement hpt (from versions: )
No matching distribution found for hpt
(pip-previous) ~$ pip install requests
Collecting requests
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Installing collected packages: chardet, urllib3, certifi, idna, requests
Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
---
In summary, from my point-of-view, the `9.0.1-2.3~ubuntu1.18.04.1` version
breaks all functionality related to `--extra-index-url`, as well as regresses
the behaviour when a package is not found when using `--index-url`.
I will be happy to provide any additional information that can help. |
[Impact]
* The --extra-index-url feature is not working when an index doesn't contain all the packages in the dependency set.
[Test Plan]
# apt install python3-venv
# python3 -m venv /tmp/test3env
# /tmp/test3env/bin/pip install wheel
# /tmp/test3env/bin/pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pytopo==1.6.1
A 404 error is the failure.
A successful install is success.
[Where problems could occur]
* Changes were cherry-picked from upstream trunk, where they are still present.
* It's entirely possible that other de-bundling bugs will be fixed by the same changes.
[Other Info]
* https://github.com/pypa/pip/pull/6113 has a good explanation of the bug.
[Original Bug Report]
I originally wrote this as a comment on
https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1822842 , but I
guess it makes sense to open a new separate bug since the other report was
already closed. Pasting my comment contents below.
~$ lsb_release -rd
Description: Ubuntu 18.04.2 LTS
Release: 18.04
~$ apt-cache policy python3-pip
python3-pip:
Installed: 9.0.1-2.3~ubuntu1.18.04.1
Candidate: 9.0.1-2.3~ubuntu1.18.04.1
Version table:
*** 9.0.1-2.3~ubuntu1.19.04.1 500
500 http://se.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 packages
500 http://se.archive.ubuntu.com/ubuntu bionic-updates/universe i386 packages
100 /var/lib/dpkg/status
9.0.1-2 500
500 http://se.archive.ubuntu.com/ubuntu bionic/universe amd64 packages
500 http://se.archive.ubuntu.com/ubuntu bionic/universe i386 packages
---
It seems like `9.0.1-2.3~ubuntu1.18.04.1` breaks the `--extra-index-url`
functionality of PIP.
In my understanding, the idea is for `--extra-index-url` to provide a
PyPI-compliant repository that offers a few additional packages, allowing PIP
to fall back onto the instance configured as `--index-url` (defaults to
upstream PyPI). This has been the case with earlier versions, allowing us to
host an internal PyPI repository containing a subset of packages, as well as a
generic local caching PyPI mirror.
In `9.0.1-2.3~ubuntu1.18.04.1`, it seems like the fallback mechanisms do not
work properly, instead failing if _either_ the `--index-url` _or_
`--extra-index-url` instances lack the package.
---
With `~/.config/pip/pip.conf` as:
[global]
index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
extra-index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_extra/simple
With `python3-pip` version `9.0.1-2.3~ubuntu1.18.04.1`, I get a session like:
~$ python3 -m venv env/pip-latest
~$ . env/pip-latest/bin/activate
(pip-latest) ~$ pip --version
pip 9.0.1 from /home/dandersson/env/pip-latest/lib/python3.6/site-packages (python 3.6)
(pip-latest) ~$ pip install hpt
Collecting hpt
Exception:
Traceback (most recent call last):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
wb.build(autobuilding=True)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
require_hashes
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
page = self._get_page(location)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
resp.raise_for_status()
File "/home/dandersson/env/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple/hpt/
PIP fails with a traceback due to not finding `hpt` on the `--index-url`
instance -- but the point of giving `--extra-index-url` is that `hpt` resides
on that instance.
Trying to install a package that should be present on the `--index-url`
instance:
(pip-latest) ~$ pip install requests
Collecting requests
Exception:
Traceback (most recent call last):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
wb.build(autobuilding=True)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
require_hashes
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
page = self._get_page(location)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
resp.raise_for_status()
File "/home/dandersson/env/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://artifactory.internal/artifactory/api/pypi/PyPI_extra/simple/requests/
Now it fails with a traceback since `requests` does not exist on the
`--extra-index-url` instance, but the purpose of that instance is to provide a
small subset of extra packages, not to be a full mirror.
Changing `~/.config/pip/pip.conf` to:
[global]
index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
removing the `--extra-index-url` instance altogether, it behaves as expected,
not finding the `hpt` package, but being able to install `requests`:
(pip-latest) ~$ pip install hpt
Collecting hpt
Exception:
Traceback (most recent call last):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
wb.build(autobuilding=True)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
require_hashes
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
page = self._get_page(location)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
resp.raise_for_status()
File "/home/dandersson/env/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple/hpt/
It is expected to not be able to install `hpt`, but PIP should not fail with a
traceback, but just with a diagnostic message like `No matching distribution
found for hpt`.
Installing a common package will correctly use the `--index-url` instance:
(pip-latest) ~$ pip install requests
Collecting requests
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Installing collected packages: idna, certifi, urllib3, chardet, requests
Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
---
Downgrading `python3-pip` to `9.0.1-2`:
~$ sudo apt install python3-pip=9.0.1-2 python-pip-whl=9.0.1-2
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be DOWNGRADED:
python-pip-whl python3-pip
0 upgraded, 0 newly installed, 2 downgraded, 0 to remove and 5 not upgraded.
Need to get 1 493 kB of archives.
After this operation, 226 kB disk space will be freed.
Do you want to continue? [Y/n]
Get:1 http://se.archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pip all 9.0.1-2 [114 kB]
Get:2 http://se.archive.ubuntu.com/ubuntu bionic/universe amd64 python-pip-whl all 9.0.1-2 [1 379 kB]
Fetched 1 493 kB in 0s (4 177 kB/s)
dpkg: warning: downgrading python3-pip from 9.0.1-2.3~ubuntu1.18.04.1 to 9.0.1-2
(Reading database ... 196581 files and directories currently installed.)
Preparing to unpack .../python3-pip_9.0.1-2_all.deb ...
Unpacking python3-pip (9.0.1-2) over (9.0.1-2.3~ubuntu1.18.04.1) ...
dpkg: warning: downgrading python-pip-whl from 9.0.1-2.3~ubuntu1.18.04.1 to 9.0.1-2
Preparing to unpack .../python-pip-whl_9.0.1-2_all.deb ...
Unpacking python-pip-whl (9.0.1-2) over (9.0.1-2.3~ubuntu1.18.04.1) ...
Setting up python-pip-whl (9.0.1-2) ...
Setting up python3-pip (9.0.1-2) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
and retrying the first attempt with `~/.config/pip/pip.conf` as:
[global]
index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
extra-index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_extra/simple
yields a session like:
~$ python3 -m venv env/pip-previous
~$ . env/pip-previous/bin/activate
(pip-previous) ~$ pip --version
pip 9.0.1 from /home/dandersson/env/pip-previous/lib/python3.6/site-packages (python 3.6)
(pip-previous) ~$ pip install hpt
Collecting hpt
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_extra/packages/hpt/0.1.397/hpt-0.1.397.tar.gz
Collecting argh (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/06/1c/e667a7126f0b84aaa1c56844337bf0ac12445d1beb9c8a6199a7314944bf/argh-0.26.2-py2.py3-none-any.whl
Collecting pyyaml==5.1 (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/9f/2c/9417b5c774792634834e730932745bc09a7d36754ca00acf1ccd1ac2594d/PyYAML-5.1.tar.gz
Collecting argcomplete (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/4d/82/f44c9661e479207348a979b1f6f063625d11dc4ca6256af053719bbb0124/argcomplete-1.10.0-py2.py3-none-any.whl
Collecting distro (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/ea/35/82f79b92fa4d937146c660a6482cee4f3dfa1f97ff3d2a6f3ecba33e712e/distro-1.4.0-py2.py3-none-any.whl
Collecting requests (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting termcolor (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz
Collecting cilib==v0.1.397 (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_extra/packages/cilib/0.1.397/cilib-0.1.397.tar.gz
Collecting idna<2.9,>=2.5 (from requests->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting hvac (from cilib==v0.1.397->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/7b/2c/b95b551769459406ca440788c32d699820f4c875020ff95c6e3844eee6d3/hvac-0.9.2-py2.py3-none-any.whl
Collecting psycopg2-binary (from cilib==v0.1.397->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/b1/35/75c9c2d9cfc073ab6c42b2d8e91ff58c9b99f4ed7ed56b36647642e6080e/psycopg2_binary-2.8.3-cp36-cp36m-manylinux1_x86_64.whl
[...SNIP...]
Successfully installed argcomplete-1.10.0 argh-0.26.2 certifi-2019.6.16 chardet-3.0.4 cilib-0.1.397 distro-1.4.0 hpt-0.1.397 hvac-0.9.2 idna-2.8 psycopg2-binary-2.8.3 pyyaml-5.1 requests-2.22.0 termcolor-1.1.0 urllib3-1.25.3
PIP now correctly resolves the respective packages to their respective
instances, using `--extra-index-url` for the packages that are _only_ present
there, and falling back to `--index-url` for everything else.
Changing `~/.config/pip/pip.conf` to:
[global]
index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
cleaning out the virtual environment and trying again:
(pip-previous) ~$ deactivate
~$ rm -rf env/pip-previous/
~$ python3 -m venv env/pip-previous
~$ . env/pip-previous/bin/activate
(pip-previous) ~$ pip --version
pip 9.0.1 from /home/dandersson/env/pip-previous/lib/python3.6/site-packages (python 3.6)
(pip-previous) ~$ pip install hpt
Collecting hpt
Could not find a version that satisfies the requirement hpt (from versions: )
No matching distribution found for hpt
(pip-previous) ~$ pip install requests
Collecting requests
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Installing collected packages: chardet, urllib3, certifi, idna, requests
Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
---
In summary, from my point-of-view, the `9.0.1-2.3~ubuntu1.18.04.1` version
breaks all functionality related to `--extra-index-url`, as well as regresses
the behaviour when a package is not found when using `--index-url`.
I will be happy to provide any additional information that can help. |
|
| 2021-04-30 18:56:44 |
Stefano Rivera |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2021-04-30 18:56:46 |
Stefano Rivera |
bug |
|
|
added subscriber Stefano Rivera |
| 2021-04-30 18:57:01 |
Stefano Rivera |
attachment added |
|
python-pip_9.0.1-2.3~ubuntu1.18.04.5.debdiff https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1833229/+attachment/5493956/+files/python-pip_9.0.1-2.3~ubuntu1.18.04.5.debdiff |
|
| 2021-04-30 18:59:04 |
Stefano Rivera |
python-pip (Ubuntu Bionic): status |
New |
In Progress |
|
| 2021-04-30 18:59:05 |
Stefano Rivera |
python-pip (Ubuntu Bionic): assignee |
|
Stefano Rivera (stefanor) |
|
| 2021-05-26 12:01:02 |
Robie Basak |
python-pip (Ubuntu Bionic): status |
In Progress |
Fix Committed |
|
| 2021-05-26 12:01:06 |
Robie Basak |
bug |
|
|
added subscriber SRU Verification |
| 2021-05-26 12:01:17 |
Robie Basak |
tags |
patch regression-update |
patch regression-update verification-needed verification-needed-bionic |
|
| 2021-05-27 00:56:34 |
Stefano Rivera |
tags |
patch regression-update verification-needed verification-needed-bionic |
patch regression-update verification-done verification-done-bionic |
|
| 2021-05-27 00:56:54 |
Stefano Rivera |
description |
[Impact]
* The --extra-index-url feature is not working when an index doesn't contain all the packages in the dependency set.
[Test Plan]
# apt install python3-venv
# python3 -m venv /tmp/test3env
# /tmp/test3env/bin/pip install wheel
# /tmp/test3env/bin/pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pytopo==1.6.1
A 404 error is the failure.
A successful install is success.
[Where problems could occur]
* Changes were cherry-picked from upstream trunk, where they are still present.
* It's entirely possible that other de-bundling bugs will be fixed by the same changes.
[Other Info]
* https://github.com/pypa/pip/pull/6113 has a good explanation of the bug.
[Original Bug Report]
I originally wrote this as a comment on
https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1822842 , but I
guess it makes sense to open a new separate bug since the other report was
already closed. Pasting my comment contents below.
~$ lsb_release -rd
Description: Ubuntu 18.04.2 LTS
Release: 18.04
~$ apt-cache policy python3-pip
python3-pip:
Installed: 9.0.1-2.3~ubuntu1.18.04.1
Candidate: 9.0.1-2.3~ubuntu1.18.04.1
Version table:
*** 9.0.1-2.3~ubuntu1.19.04.1 500
500 http://se.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 packages
500 http://se.archive.ubuntu.com/ubuntu bionic-updates/universe i386 packages
100 /var/lib/dpkg/status
9.0.1-2 500
500 http://se.archive.ubuntu.com/ubuntu bionic/universe amd64 packages
500 http://se.archive.ubuntu.com/ubuntu bionic/universe i386 packages
---
It seems like `9.0.1-2.3~ubuntu1.18.04.1` breaks the `--extra-index-url`
functionality of PIP.
In my understanding, the idea is for `--extra-index-url` to provide a
PyPI-compliant repository that offers a few additional packages, allowing PIP
to fall back onto the instance configured as `--index-url` (defaults to
upstream PyPI). This has been the case with earlier versions, allowing us to
host an internal PyPI repository containing a subset of packages, as well as a
generic local caching PyPI mirror.
In `9.0.1-2.3~ubuntu1.18.04.1`, it seems like the fallback mechanisms do not
work properly, instead failing if _either_ the `--index-url` _or_
`--extra-index-url` instances lack the package.
---
With `~/.config/pip/pip.conf` as:
[global]
index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
extra-index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_extra/simple
With `python3-pip` version `9.0.1-2.3~ubuntu1.18.04.1`, I get a session like:
~$ python3 -m venv env/pip-latest
~$ . env/pip-latest/bin/activate
(pip-latest) ~$ pip --version
pip 9.0.1 from /home/dandersson/env/pip-latest/lib/python3.6/site-packages (python 3.6)
(pip-latest) ~$ pip install hpt
Collecting hpt
Exception:
Traceback (most recent call last):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
wb.build(autobuilding=True)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
require_hashes
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
page = self._get_page(location)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
resp.raise_for_status()
File "/home/dandersson/env/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple/hpt/
PIP fails with a traceback due to not finding `hpt` on the `--index-url`
instance -- but the point of giving `--extra-index-url` is that `hpt` resides
on that instance.
Trying to install a package that should be present on the `--index-url`
instance:
(pip-latest) ~$ pip install requests
Collecting requests
Exception:
Traceback (most recent call last):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
wb.build(autobuilding=True)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
require_hashes
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
page = self._get_page(location)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
resp.raise_for_status()
File "/home/dandersson/env/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://artifactory.internal/artifactory/api/pypi/PyPI_extra/simple/requests/
Now it fails with a traceback since `requests` does not exist on the
`--extra-index-url` instance, but the purpose of that instance is to provide a
small subset of extra packages, not to be a full mirror.
Changing `~/.config/pip/pip.conf` to:
[global]
index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
removing the `--extra-index-url` instance altogether, it behaves as expected,
not finding the `hpt` package, but being able to install `requests`:
(pip-latest) ~$ pip install hpt
Collecting hpt
Exception:
Traceback (most recent call last):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
wb.build(autobuilding=True)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
require_hashes
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
page = self._get_page(location)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
resp.raise_for_status()
File "/home/dandersson/env/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple/hpt/
It is expected to not be able to install `hpt`, but PIP should not fail with a
traceback, but just with a diagnostic message like `No matching distribution
found for hpt`.
Installing a common package will correctly use the `--index-url` instance:
(pip-latest) ~$ pip install requests
Collecting requests
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Installing collected packages: idna, certifi, urllib3, chardet, requests
Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
---
Downgrading `python3-pip` to `9.0.1-2`:
~$ sudo apt install python3-pip=9.0.1-2 python-pip-whl=9.0.1-2
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be DOWNGRADED:
python-pip-whl python3-pip
0 upgraded, 0 newly installed, 2 downgraded, 0 to remove and 5 not upgraded.
Need to get 1 493 kB of archives.
After this operation, 226 kB disk space will be freed.
Do you want to continue? [Y/n]
Get:1 http://se.archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pip all 9.0.1-2 [114 kB]
Get:2 http://se.archive.ubuntu.com/ubuntu bionic/universe amd64 python-pip-whl all 9.0.1-2 [1 379 kB]
Fetched 1 493 kB in 0s (4 177 kB/s)
dpkg: warning: downgrading python3-pip from 9.0.1-2.3~ubuntu1.18.04.1 to 9.0.1-2
(Reading database ... 196581 files and directories currently installed.)
Preparing to unpack .../python3-pip_9.0.1-2_all.deb ...
Unpacking python3-pip (9.0.1-2) over (9.0.1-2.3~ubuntu1.18.04.1) ...
dpkg: warning: downgrading python-pip-whl from 9.0.1-2.3~ubuntu1.18.04.1 to 9.0.1-2
Preparing to unpack .../python-pip-whl_9.0.1-2_all.deb ...
Unpacking python-pip-whl (9.0.1-2) over (9.0.1-2.3~ubuntu1.18.04.1) ...
Setting up python-pip-whl (9.0.1-2) ...
Setting up python3-pip (9.0.1-2) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
and retrying the first attempt with `~/.config/pip/pip.conf` as:
[global]
index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
extra-index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_extra/simple
yields a session like:
~$ python3 -m venv env/pip-previous
~$ . env/pip-previous/bin/activate
(pip-previous) ~$ pip --version
pip 9.0.1 from /home/dandersson/env/pip-previous/lib/python3.6/site-packages (python 3.6)
(pip-previous) ~$ pip install hpt
Collecting hpt
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_extra/packages/hpt/0.1.397/hpt-0.1.397.tar.gz
Collecting argh (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/06/1c/e667a7126f0b84aaa1c56844337bf0ac12445d1beb9c8a6199a7314944bf/argh-0.26.2-py2.py3-none-any.whl
Collecting pyyaml==5.1 (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/9f/2c/9417b5c774792634834e730932745bc09a7d36754ca00acf1ccd1ac2594d/PyYAML-5.1.tar.gz
Collecting argcomplete (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/4d/82/f44c9661e479207348a979b1f6f063625d11dc4ca6256af053719bbb0124/argcomplete-1.10.0-py2.py3-none-any.whl
Collecting distro (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/ea/35/82f79b92fa4d937146c660a6482cee4f3dfa1f97ff3d2a6f3ecba33e712e/distro-1.4.0-py2.py3-none-any.whl
Collecting requests (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting termcolor (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz
Collecting cilib==v0.1.397 (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_extra/packages/cilib/0.1.397/cilib-0.1.397.tar.gz
Collecting idna<2.9,>=2.5 (from requests->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting hvac (from cilib==v0.1.397->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/7b/2c/b95b551769459406ca440788c32d699820f4c875020ff95c6e3844eee6d3/hvac-0.9.2-py2.py3-none-any.whl
Collecting psycopg2-binary (from cilib==v0.1.397->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/b1/35/75c9c2d9cfc073ab6c42b2d8e91ff58c9b99f4ed7ed56b36647642e6080e/psycopg2_binary-2.8.3-cp36-cp36m-manylinux1_x86_64.whl
[...SNIP...]
Successfully installed argcomplete-1.10.0 argh-0.26.2 certifi-2019.6.16 chardet-3.0.4 cilib-0.1.397 distro-1.4.0 hpt-0.1.397 hvac-0.9.2 idna-2.8 psycopg2-binary-2.8.3 pyyaml-5.1 requests-2.22.0 termcolor-1.1.0 urllib3-1.25.3
PIP now correctly resolves the respective packages to their respective
instances, using `--extra-index-url` for the packages that are _only_ present
there, and falling back to `--index-url` for everything else.
Changing `~/.config/pip/pip.conf` to:
[global]
index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
cleaning out the virtual environment and trying again:
(pip-previous) ~$ deactivate
~$ rm -rf env/pip-previous/
~$ python3 -m venv env/pip-previous
~$ . env/pip-previous/bin/activate
(pip-previous) ~$ pip --version
pip 9.0.1 from /home/dandersson/env/pip-previous/lib/python3.6/site-packages (python 3.6)
(pip-previous) ~$ pip install hpt
Collecting hpt
Could not find a version that satisfies the requirement hpt (from versions: )
No matching distribution found for hpt
(pip-previous) ~$ pip install requests
Collecting requests
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Installing collected packages: chardet, urllib3, certifi, idna, requests
Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
---
In summary, from my point-of-view, the `9.0.1-2.3~ubuntu1.18.04.1` version
breaks all functionality related to `--extra-index-url`, as well as regresses
the behaviour when a package is not found when using `--index-url`.
I will be happy to provide any additional information that can help. |
[Impact]
* The --extra-index-url feature is not working when an index doesn't contain all the packages in the dependency set.
[Test Plan]
# apt install python3-venv python3-dev libglib2.0-dev libcairo-dev libgirepository1.0-dev build-essential
# python3 -m venv /tmp/test3env
# /tmp/test3env/bin/pip install wheel
# /tmp/test3env/bin/pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pytopo==1.6.1
A 404 error is the failure.
A successful install is success.
[Where problems could occur]
* Changes were cherry-picked from upstream trunk, where they are still present.
* It's entirely possible that other de-bundling bugs will be fixed by the same changes.
[Other Info]
* https://github.com/pypa/pip/pull/6113 has a good explanation of the bug.
[Original Bug Report]
I originally wrote this as a comment on
https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1822842 , but I
guess it makes sense to open a new separate bug since the other report was
already closed. Pasting my comment contents below.
~$ lsb_release -rd
Description: Ubuntu 18.04.2 LTS
Release: 18.04
~$ apt-cache policy python3-pip
python3-pip:
Installed: 9.0.1-2.3~ubuntu1.18.04.1
Candidate: 9.0.1-2.3~ubuntu1.18.04.1
Version table:
*** 9.0.1-2.3~ubuntu1.19.04.1 500
500 http://se.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 packages
500 http://se.archive.ubuntu.com/ubuntu bionic-updates/universe i386 packages
100 /var/lib/dpkg/status
9.0.1-2 500
500 http://se.archive.ubuntu.com/ubuntu bionic/universe amd64 packages
500 http://se.archive.ubuntu.com/ubuntu bionic/universe i386 packages
---
It seems like `9.0.1-2.3~ubuntu1.18.04.1` breaks the `--extra-index-url`
functionality of PIP.
In my understanding, the idea is for `--extra-index-url` to provide a
PyPI-compliant repository that offers a few additional packages, allowing PIP
to fall back onto the instance configured as `--index-url` (defaults to
upstream PyPI). This has been the case with earlier versions, allowing us to
host an internal PyPI repository containing a subset of packages, as well as a
generic local caching PyPI mirror.
In `9.0.1-2.3~ubuntu1.18.04.1`, it seems like the fallback mechanisms do not
work properly, instead failing if _either_ the `--index-url` _or_
`--extra-index-url` instances lack the package.
---
With `~/.config/pip/pip.conf` as:
[global]
index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
extra-index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_extra/simple
With `python3-pip` version `9.0.1-2.3~ubuntu1.18.04.1`, I get a session like:
~$ python3 -m venv env/pip-latest
~$ . env/pip-latest/bin/activate
(pip-latest) ~$ pip --version
pip 9.0.1 from /home/dandersson/env/pip-latest/lib/python3.6/site-packages (python 3.6)
(pip-latest) ~$ pip install hpt
Collecting hpt
Exception:
Traceback (most recent call last):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
wb.build(autobuilding=True)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
require_hashes
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
page = self._get_page(location)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
resp.raise_for_status()
File "/home/dandersson/env/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple/hpt/
PIP fails with a traceback due to not finding `hpt` on the `--index-url`
instance -- but the point of giving `--extra-index-url` is that `hpt` resides
on that instance.
Trying to install a package that should be present on the `--index-url`
instance:
(pip-latest) ~$ pip install requests
Collecting requests
Exception:
Traceback (most recent call last):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
wb.build(autobuilding=True)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
require_hashes
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
page = self._get_page(location)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
resp.raise_for_status()
File "/home/dandersson/env/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://artifactory.internal/artifactory/api/pypi/PyPI_extra/simple/requests/
Now it fails with a traceback since `requests` does not exist on the
`--extra-index-url` instance, but the purpose of that instance is to provide a
small subset of extra packages, not to be a full mirror.
Changing `~/.config/pip/pip.conf` to:
[global]
index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
removing the `--extra-index-url` instance altogether, it behaves as expected,
not finding the `hpt` package, but being able to install `requests`:
(pip-latest) ~$ pip install hpt
Collecting hpt
Exception:
Traceback (most recent call last):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
wb.build(autobuilding=True)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
require_hashes
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
page = self._get_page(location)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
resp.raise_for_status()
File "/home/dandersson/env/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple/hpt/
It is expected to not be able to install `hpt`, but PIP should not fail with a
traceback, but just with a diagnostic message like `No matching distribution
found for hpt`.
Installing a common package will correctly use the `--index-url` instance:
(pip-latest) ~$ pip install requests
Collecting requests
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Installing collected packages: idna, certifi, urllib3, chardet, requests
Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
---
Downgrading `python3-pip` to `9.0.1-2`:
~$ sudo apt install python3-pip=9.0.1-2 python-pip-whl=9.0.1-2
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be DOWNGRADED:
python-pip-whl python3-pip
0 upgraded, 0 newly installed, 2 downgraded, 0 to remove and 5 not upgraded.
Need to get 1 493 kB of archives.
After this operation, 226 kB disk space will be freed.
Do you want to continue? [Y/n]
Get:1 http://se.archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pip all 9.0.1-2 [114 kB]
Get:2 http://se.archive.ubuntu.com/ubuntu bionic/universe amd64 python-pip-whl all 9.0.1-2 [1 379 kB]
Fetched 1 493 kB in 0s (4 177 kB/s)
dpkg: warning: downgrading python3-pip from 9.0.1-2.3~ubuntu1.18.04.1 to 9.0.1-2
(Reading database ... 196581 files and directories currently installed.)
Preparing to unpack .../python3-pip_9.0.1-2_all.deb ...
Unpacking python3-pip (9.0.1-2) over (9.0.1-2.3~ubuntu1.18.04.1) ...
dpkg: warning: downgrading python-pip-whl from 9.0.1-2.3~ubuntu1.18.04.1 to 9.0.1-2
Preparing to unpack .../python-pip-whl_9.0.1-2_all.deb ...
Unpacking python-pip-whl (9.0.1-2) over (9.0.1-2.3~ubuntu1.18.04.1) ...
Setting up python-pip-whl (9.0.1-2) ...
Setting up python3-pip (9.0.1-2) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
and retrying the first attempt with `~/.config/pip/pip.conf` as:
[global]
index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
extra-index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_extra/simple
yields a session like:
~$ python3 -m venv env/pip-previous
~$ . env/pip-previous/bin/activate
(pip-previous) ~$ pip --version
pip 9.0.1 from /home/dandersson/env/pip-previous/lib/python3.6/site-packages (python 3.6)
(pip-previous) ~$ pip install hpt
Collecting hpt
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_extra/packages/hpt/0.1.397/hpt-0.1.397.tar.gz
Collecting argh (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/06/1c/e667a7126f0b84aaa1c56844337bf0ac12445d1beb9c8a6199a7314944bf/argh-0.26.2-py2.py3-none-any.whl
Collecting pyyaml==5.1 (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/9f/2c/9417b5c774792634834e730932745bc09a7d36754ca00acf1ccd1ac2594d/PyYAML-5.1.tar.gz
Collecting argcomplete (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/4d/82/f44c9661e479207348a979b1f6f063625d11dc4ca6256af053719bbb0124/argcomplete-1.10.0-py2.py3-none-any.whl
Collecting distro (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/ea/35/82f79b92fa4d937146c660a6482cee4f3dfa1f97ff3d2a6f3ecba33e712e/distro-1.4.0-py2.py3-none-any.whl
Collecting requests (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting termcolor (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz
Collecting cilib==v0.1.397 (from hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_extra/packages/cilib/0.1.397/cilib-0.1.397.tar.gz
Collecting idna<2.9,>=2.5 (from requests->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting hvac (from cilib==v0.1.397->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/7b/2c/b95b551769459406ca440788c32d699820f4c875020ff95c6e3844eee6d3/hvac-0.9.2-py2.py3-none-any.whl
Collecting psycopg2-binary (from cilib==v0.1.397->hpt)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/b1/35/75c9c2d9cfc073ab6c42b2d8e91ff58c9b99f4ed7ed56b36647642e6080e/psycopg2_binary-2.8.3-cp36-cp36m-manylinux1_x86_64.whl
[...SNIP...]
Successfully installed argcomplete-1.10.0 argh-0.26.2 certifi-2019.6.16 chardet-3.0.4 cilib-0.1.397 distro-1.4.0 hpt-0.1.397 hvac-0.9.2 idna-2.8 psycopg2-binary-2.8.3 pyyaml-5.1 requests-2.22.0 termcolor-1.1.0 urllib3-1.25.3
PIP now correctly resolves the respective packages to their respective
instances, using `--extra-index-url` for the packages that are _only_ present
there, and falling back to `--index-url` for everything else.
Changing `~/.config/pip/pip.conf` to:
[global]
index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
cleaning out the virtual environment and trying again:
(pip-previous) ~$ deactivate
~$ rm -rf env/pip-previous/
~$ python3 -m venv env/pip-previous
~$ . env/pip-previous/bin/activate
(pip-previous) ~$ pip --version
pip 9.0.1 from /home/dandersson/env/pip-previous/lib/python3.6/site-packages (python 3.6)
(pip-previous) ~$ pip install hpt
Collecting hpt
Could not find a version that satisfies the requirement hpt (from versions: )
No matching distribution found for hpt
(pip-previous) ~$ pip install requests
Collecting requests
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Installing collected packages: chardet, urllib3, certifi, idna, requests
Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
---
In summary, from my point-of-view, the `9.0.1-2.3~ubuntu1.18.04.1` version
breaks all functionality related to `--extra-index-url`, as well as regresses
the behaviour when a package is not found when using `--index-url`.
I will be happy to provide any additional information that can help. |
|
| 2021-05-28 08:14:57 |
Mathew Hodson |
bug watch removed |
https://github.com/pypa/pip/issues/6799 |
|
|
| 2021-05-28 08:17:00 |
Mathew Hodson |
bug watch removed |
https://github.com/pypa/pip/issues/7486 |
|
|
| 2021-06-08 17:26:40 |
Launchpad Janitor |
python-pip (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2021-06-08 17:26:44 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
