`--extra-index-url` not working for PIP

[Daniel Andersson]

[Impact]

 * The --extra-index-url feature is not working when an index doesn't contain all the packages in the dependency set.

[Test Plan]
# apt install python3-venv python3-dev libglib2.0-dev libcairo-dev libgirepository1.0-dev build-essential
# python3 -m venv /tmp/test3env
# /tmp/test3env/bin/pip install wheel
# /tmp/test3env/bin/pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pytopo==1.6.1

A 404 error is the failure.
A successful install is success.

[Where problems could occur]

 * Changes were cherry-picked from upstream trunk, where they are still present.
 * It's entirely possible that other de-bundling bugs will be fixed by the same changes.

[Other Info]

* https://github.com/pypa/pip/pull/6113 has a good explanation of the bug.

[Original Bug Report]

I originally wrote this as a comment on
https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1822842 , but I
guess it makes sense to open a new separate bug since the other report was
already closed. Pasting my comment contents below.

    ~$ lsb_release -rd
    Description: Ubuntu 18.04.2 LTS
    Release: 18.04

    ~$ apt-cache policy python3-pip
    python3-pip:
      Installed: 9.0.1-2.3~ubuntu1.18.04.1
      Candidate: 9.0.1-2.3~ubuntu1.18.04.1
      Version table:
     *** 9.0.1-2.3~ubuntu1.19.04.1 500
            500 http://se.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 packages
            500 http://se.archive.ubuntu.com/ubuntu bionic-updates/universe i386 packages
            100 /var/lib/dpkg/status
         9.0.1-2 500
            500 http://se.archive.ubuntu.com/ubuntu bionic/universe amd64 packages
            500 http://se.archive.ubuntu.com/ubuntu bionic/universe i386 packages

---

It seems like `9.0.1-2.3~ubuntu1.18.04.1` breaks the `--extra-index-url`
functionality of PIP.

In my understanding, the idea is for `--extra-index-url` to provide a
PyPI-compliant repository that offers a few additional packages, allowing PIP
to fall back onto the instance configured as `--index-url` (defaults to
upstream PyPI). This has been the case with earlier versions, allowing us to
host an internal PyPI repository containing a subset of packages, as well as a
generic local caching PyPI mirror.

In `9.0.1-2.3~ubuntu1.18.04.1`, it seems like the fallback mechanisms do not
work properly, instead failing if _either_ the `--index-url` _or_
`--extra-index-url` instances lack the package.

---

With `~/.config/pip/pip.conf` as:

    [global]
    index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
    extra-index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_extra/simple

With `python3-pip` version `9.0.1-2.3~ubuntu1.18.04.1`, I get a session like:

    ~$ python3 -m venv env/pip-latest
    ~$ . env/pip-latest/bin/activate
    (pip-latest) ~$ pip --version
    pip 9.0.1 from /home/dandersson/env/pip-latest/lib/python3.6/site-packages (python 3.6)
    (pip-latest) ~$ pip install hpt
    Collecting hpt
    Exception:
    Traceback (most recent call last):
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
        status = self.run(options, args)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
        wb.build(autobuilding=True)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
        self.requirement_set.prepare_files(self.finder)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
        ignore_dependencies=self.ignore_dependencies))
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
        require_hashes
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
        self.link = finder.find_requirement(self, upgrade)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
        all_candidates = self.find_all_candidates(req.name)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
        for page in self._get_pages(url_locations, project_name):
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
        page = self._get_page(location)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
        return HTMLPage.get_page(link, session=self.session)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
        resp.raise_for_status()
      File "/home/dandersson/env/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
        raise HTTPError(http_error_msg, response=self)
    requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple/hpt/

PIP fails with a traceback due to not finding `hpt` on the `--index-url`
instance -- but the point of giving `--extra-index-url` is that `hpt` resides
on that instance.

Trying to install a package that should be present on the `--index-url`
instance:

    (pip-latest) ~$ pip install requests
    Collecting requests
    Exception:
    Traceback (most recent call last):
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
        status = self.run(options, args)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
        wb.build(autobuilding=True)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
        self.requirement_set.prepare_files(self.finder)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
        ignore_dependencies=self.ignore_dependencies))
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
        require_hashes
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
        self.link = finder.find_requirement(self, upgrade)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
        all_candidates = self.find_all_candidates(req.name)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
        for page in self._get_pages(url_locations, project_name):
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
        page = self._get_page(location)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
        return HTMLPage.get_page(link, session=self.session)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
        resp.raise_for_status()
      File "/home/dandersson/env/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
        raise HTTPError(http_error_msg, response=self)
    requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://artifactory.internal/artifactory/api/pypi/PyPI_extra/simple/requests/

Now it fails with a traceback since `requests` does not exist on the
`--extra-index-url` instance, but the purpose of that instance is to provide a
small subset of extra packages, not to be a full mirror.

Changing `~/.config/pip/pip.conf` to:

    [global]
    index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple

removing the `--extra-index-url` instance altogether, it behaves as expected,
not finding the `hpt` package, but being able to install `requests`:

    (pip-latest) ~$ pip install hpt
    Collecting hpt
    Exception:
    Traceback (most recent call last):
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
        status = self.run(options, args)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
        wb.build(autobuilding=True)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
        self.requirement_set.prepare_files(self.finder)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
        ignore_dependencies=self.ignore_dependencies))
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
        require_hashes
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
        self.link = finder.find_requirement(self, upgrade)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
        all_candidates = self.find_all_candidates(req.name)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
        for page in self._get_pages(url_locations, project_name):
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
        page = self._get_page(location)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
        return HTMLPage.get_page(link, session=self.session)
      File "/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
        resp.raise_for_status()
      File "/home/dandersson/env/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
        raise HTTPError(http_error_msg, response=self)
    requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple/hpt/

It is expected to not be able to install `hpt`, but PIP should not fail with a
traceback, but just with a diagnostic message like `No matching distribution
found for hpt`.

Installing a common package will correctly use the `--index-url` instance:

    (pip-latest) ~$ pip install requests
    Collecting requests
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
    Collecting idna<2.9,>=2.5 (from requests)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
    Collecting certifi>=2017.4.17 (from requests)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
    Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
    Collecting chardet<3.1.0,>=3.0.2 (from requests)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
    Installing collected packages: idna, certifi, urllib3, chardet, requests
    Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3

---

Downgrading `python3-pip` to `9.0.1-2`:

    ~$ sudo apt install python3-pip=9.0.1-2 python-pip-whl=9.0.1-2
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following packages will be DOWNGRADED:
      python-pip-whl python3-pip
    0 upgraded, 0 newly installed, 2 downgraded, 0 to remove and 5 not upgraded.
    Need to get 1 493 kB of archives.
    After this operation, 226 kB disk space will be freed.
    Do you want to continue? [Y/n]
    Get:1 http://se.archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pip all 9.0.1-2 [114 kB]
    Get:2 http://se.archive.ubuntu.com/ubuntu bionic/universe amd64 python-pip-whl all 9.0.1-2 [1 379 kB]
    Fetched 1 493 kB in 0s (4 177 kB/s)
    dpkg: warning: downgrading python3-pip from 9.0.1-2.3~ubuntu1.18.04.1 to 9.0.1-2
    (Reading database ... 196581 files and directories currently installed.)
    Preparing to unpack .../python3-pip_9.0.1-2_all.deb ...
    Unpacking python3-pip (9.0.1-2) over (9.0.1-2.3~ubuntu1.18.04.1) ...
    dpkg: warning: downgrading python-pip-whl from 9.0.1-2.3~ubuntu1.18.04.1 to 9.0.1-2
    Preparing to unpack .../python-pip-whl_9.0.1-2_all.deb ...
    Unpacking python-pip-whl (9.0.1-2) over (9.0.1-2.3~ubuntu1.18.04.1) ...
    Setting up python-pip-whl (9.0.1-2) ...
    Setting up python3-pip (9.0.1-2) ...
    Processing triggers for man-db (2.8.3-2ubuntu0.1) ...

and retrying the first attempt with `~/.config/pip/pip.conf` as:

    [global]
    index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
    extra-index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_extra/simple

yields a session like:

    ~$ python3 -m venv env/pip-previous
    ~$ . env/pip-previous/bin/activate
    (pip-previous) ~$ pip --version
    pip 9.0.1 from /home/dandersson/env/pip-previous/lib/python3.6/site-packages (python 3.6)
    (pip-previous) ~$ pip install hpt
    Collecting hpt
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_extra/packages/hpt/0.1.397/hpt-0.1.397.tar.gz
    Collecting argh (from hpt)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/06/1c/e667a7126f0b84aaa1c56844337bf0ac12445d1beb9c8a6199a7314944bf/argh-0.26.2-py2.py3-none-any.whl
    Collecting pyyaml==5.1 (from hpt)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/9f/2c/9417b5c774792634834e730932745bc09a7d36754ca00acf1ccd1ac2594d/PyYAML-5.1.tar.gz
    Collecting argcomplete (from hpt)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/4d/82/f44c9661e479207348a979b1f6f063625d11dc4ca6256af053719bbb0124/argcomplete-1.10.0-py2.py3-none-any.whl
    Collecting distro (from hpt)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/ea/35/82f79b92fa4d937146c660a6482cee4f3dfa1f97ff3d2a6f3ecba33e712e/distro-1.4.0-py2.py3-none-any.whl
    Collecting requests (from hpt)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
    Collecting termcolor (from hpt)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz
    Collecting cilib==v0.1.397 (from hpt)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_extra/packages/cilib/0.1.397/cilib-0.1.397.tar.gz
    Collecting idna<2.9,>=2.5 (from requests->hpt)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
    Collecting chardet<3.1.0,>=3.0.2 (from requests->hpt)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
    Collecting certifi>=2017.4.17 (from requests->hpt)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
    Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests->hpt)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
    Collecting hvac (from cilib==v0.1.397->hpt)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/7b/2c/b95b551769459406ca440788c32d699820f4c875020ff95c6e3844eee6d3/hvac-0.9.2-py2.py3-none-any.whl
    Collecting psycopg2-binary (from cilib==v0.1.397->hpt)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/b1/35/75c9c2d9cfc073ab6c42b2d8e91ff58c9b99f4ed7ed56b36647642e6080e/psycopg2_binary-2.8.3-cp36-cp36m-manylinux1_x86_64.whl
    [...SNIP...]
    Successfully installed argcomplete-1.10.0 argh-0.26.2 certifi-2019.6.16 chardet-3.0.4 cilib-0.1.397 distro-1.4.0 hpt-0.1.397 hvac-0.9.2 idna-2.8 psycopg2-binary-2.8.3 pyyaml-5.1 requests-2.22.0 termcolor-1.1.0 urllib3-1.25.3

PIP now correctly resolves the respective packages to their respective
instances, using `--extra-index-url` for the packages that are _only_ present
there, and falling back to `--index-url` for everything else.

Changing `~/.config/pip/pip.conf` to:

    [global]
    index-url = https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple

cleaning out the virtual environment and trying again:

    (pip-previous) ~$ deactivate
    ~$ rm -rf env/pip-previous/
    ~$ python3 -m venv env/pip-previous
    ~$ . env/pip-previous/bin/activate
    (pip-previous) ~$ pip --version
    pip 9.0.1 from /home/dandersson/env/pip-previous/lib/python3.6/site-packages (python 3.6)
    (pip-previous) ~$ pip install hpt
    Collecting hpt
      Could not find a version that satisfies the requirement hpt (from versions: )
    No matching distribution found for hpt
    (pip-previous) ~$ pip install requests
    Collecting requests
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
    Collecting chardet<3.1.0,>=3.0.2 (from requests)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
    Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
    Collecting certifi>=2017.4.17 (from requests)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
    Collecting idna<2.9,>=2.5 (from requests)
      Using cached https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
    Installing collected packages: chardet, urllib3, certifi, idna, requests
    Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3

---

In summary, from my point-of-view, the `9.0.1-2.3~ubuntu1.18.04.1` version
breaks all functionality related to `--extra-index-url`, as well as regresses
the behaviour when a package is not found when using `--index-url`.

I will be happy to provide any additional information that can help.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Valtteri Mäkelä (vltrrr)]

Confirmed also with Mint 19.1 with latest packages, if using our internal pypi server. Creating a plain py3 virtualenv fails:

$ virtualenv testenv --python python3

Fails with:

  File "/home/user/testenv/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pypi.internal.com/simple/setuptools/

Disabling the internal pypi server from ~/.pip/pip.conf works, but of course removes access to internal packages.

Downgrading to previous version as suggested by Daniel Anderson resolved the issue:

sudo apt install python3-pip=9.0.1-2 python-pip-whl=9.0.1-2

[Marc Mercer (daemoen)]

Can confirm here as well. I also tested the pip/whl builds from universal which are at 18.x -- these unfortunately have the exact same issue. Was hoping maybe they had moved past it, which suggests that the issue is with the unvendored requests, as opposed to pip itself. https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1842120 can be marked as a duplicate of this.

[Johnny (jairoalonso25)]

Can Confirm. Using a private PyPI repo in my company.

pip install --extra-index-url=https://pypi.pipelineservices.alertlogic.com alps
Collecting alps
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 353, in run
    wb.build(autobuilding=True)
  File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 465, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 423, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 568, in _get_pages
    page = self._get_page(location)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 683, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 795, in get_page
    resp.raise_for_status()
  File "/usr/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 404 Client Error: Not Found for url: https://pypi.org/simple/alps/
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 465, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 423, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/lib/python3/dist-packages/pip/index.py", line 568, in _get_pages
    page = self._get_page(location)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 683, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 795, in get_page
    resp.raise_for_status()
  File "/usr/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pypi.org/simple/alps/

lsb_release -rd
Description: Ubuntu 18.04.3 LTS
Release: 18.04

Using WLS form Windows 10

[James Page (james-page)]

I'm slightly baffled by this as the original bug report and associated fix resolved an issue where:

  requests.exceptions.HTTPError

was not the same as requests.HTTPError in the distro packaging - but the exception raised in this bug report infers that something is still wonky in the debundling of the vendored requests...

[James Page (james-page)]

Struggling to reproduce this issue; I've setup a test instance with a extra-index-url which points to an empty index; this throws a 404 when accessed for the modules being requested but the exception handler is correctly catching the exception, falling back to the main package index.

[James Page (james-page)]

404 Client Error: Not Found for url: http://mirror.dfw.rax.openstack.org/wheel/urllib3/
Traceback (most recent call last):
  File "/root/pip-latest/lib/python3.6/site-packages/pip/index.py", line 796, in get_page
    resp.raise_for_status()
  File "/root/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
pip._vendor.requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://mirror.dfw.rax.openstack.org/wheel/urllib3/

As you can see the correct/vendor path-ed exception is being raised and caught.

[Marc Mercer (daemoen)]

@james-page I think the thing being overlooked is what it is doing, not wether or not the exception is correct. In our case, we have repositories that have some artifacts, and repositories that may have other artifacts... (and once in a rare while, overlap with different versions).. If we request a package that does *not* exist in the custom urls, then it will fail completely, instead of passing with the only failure being on that one repository. By default, it should check each of the repositories, see if they have it, etc.. It seems to be checking them all, but if *any* of them fails to have an artifact being requested, the entire pip operation implodes, even if the package *is* available in a different repository successfully prior to the fetching of the failed repo.

[James Page (james-page)]

The exception drives the behaviour so the behaviour and the incorrect exception class are linked - the pip index.py code is mismatching on the type HTTPError resulting in an unhandled exception and the behaviour you are seeing.

I can't reproduce this at the moment so there must be something we're missing.

[James Page (james-page)]

https://github.com/pypa/pip/commit/e4936da2e1b62c33f0ed9034f5c984ae17e38e86 might be related - I'll prepare a test package to see if that drives the correct behaviour in all cases.

[James Page (james-page)]

I believe this is something todo with missing vendor pathing for requests.exceptions.

I've picked the commit in #10 into the bionic package and uploaded to:

  https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3690

as I'm unable to reproduce the issue it would be helpful if one of the impacted bug participants could verify it resolves the issue.

[Marc Mercer (daemoen)]

@james-page Tested -- not fixed

root@vagrant:/etc# dpkg -l | grep pip
ii libpipeline1:amd64 1.5.0-1 amd64 pipeline manipulation library
ii python-pip 9.0.1-2.3~ubuntu1.18.04.2~ubuntu18.04.1~ppa201909251252 all Python package installer
ii python-pip-whl 9.0.1-2.3~ubuntu1.18.04.2~ubuntu18.04.1~ppa201909251252 all Python package installer
ii python3-pip 9.0.1-2.3~ubuntu1.18.04.2~ubuntu18.04.1~ppa201909251252 all Python package installer
root@vagrant:/etc# pip install ansible==2.8.0
Collecting ansible==2.8.0
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 353, in run
    wb.build(autobuilding=True)
  File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 465, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 423, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 568, in _get_pages
    page = self._get_page(location)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 683, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 795, in get_page
    resp.raise_for_status()
  File "/usr/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 404 Client Error: Not Found for url: https://artifacts.ourdomain.com/repository/pypi-release/simple/ansible/

[Marc Mercer (daemoen)]

@james-page -- Please ping me on freenode (Daemoen) on the ubuntu channel. I have created a user for you to test against our repository host, as hopefully that should simplify this a bit. I can provide you all of the details for the configuration needed, but don't want to provide them here as this is a public medium. I just tested the credentials and ensured that it will allow you to replicate the issue as the rest of us see it.

[chbrosso (chbrosso)]

It looks like the upstream issue is here, and it mentions that it's a vendoring issue:

https://github.com/pypa/pip/issues/6799

[Akkana Peck (akkzilla)]

This might be a simpler way to reproduce it. I uploaded a package to test.pypi and wanted to use --extra-index-url to verify that its dependencies are pulled in (it has dependencies that are in pypi but not in test.pypi). Here are my steps to reproduce it:

python3 -m venv /tmp/test3env
source /tmp/test3env/bin/activate
pip install wheel
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pytopo==1.6.1

which gives the error:

Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://test.pypi.org/simple/pycairo/

Sometimes it errors on a different package, not always pycairo, but the rest of the error is the same. I filed https://github.com/pypa/pip/issues/7486 upstream and I suspect they're right that it's the same issue as this Ubuntu bug (I'm on 19.10).

[chbrosso (chbrosso)]

A simple command to test for the issue is pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple simplejson==3.17

@james-page I tested your proposed PPA with this command, and the issue is still here.

I gave a shot to this, and generated a patch from the upstream pull request https://github.com/pypa/pip/pull/6367. I could test it by getting source package, patching it, and building it back, I can confirm that the issue is fixed this way.

I'm unable to further propose a new package as I'm not used to contributing fixes, but would be happy to if necessary.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Fix-1833229.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[chbrosso (chbrosso)]

@james-page (or any other maintainer), did you have a chance to look at the patch? I'm pretty confident it solves the issue.

[James Page (james-page)]

sorry - I missed this update - looking now!

[James Page (james-page)]

pip right through to Ubuntu development appears to be missing part of the fix for this problem so I've picked fixes for bionic/eoan/focal into:

  https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3690

I'm still not able to reproduce this issue but please test and see if the proposed version resolves this issue with the test case in #16.

[James Page (james-page)]

Right - I was able to reproduce this issue on focal using:

python3 -m venv /tmp/test3env
source /tmp/test3env/bin/activate
pip install wheel
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pytopo==1.6.1

(thanks for that)

However with the patched package in the PPA, I now hit a completely different problem with the test case which looks related to the patch:

Traceback (most recent call last):
  File "/usr/lib/python3.8/ensurepip/__init__.py", line 54, in _run_pip
    import pip._internal
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpzye4xja3/pip-18.1-py2.py3-none-any.whl/pip/_internal/__init__.py", line 40, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpzye4xja3/pip-18.1-py2.py3-none-any.whl/pip/_internal/cli/autocompletion.py", line 8, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpzye4xja3/pip-18.1-py2.py3-none-any.whl/pip/_internal/cli/main_parser.py", line 8, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpzye4xja3/pip-18.1-py2.py3-none-any.whl/pip/_internal/cli/cmdoptions.py", line 22, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpzye4xja3/pip-18.1-py2.py3-none-any.whl/pip/_internal/utils/ui.py", line 15, in <module>
ModuleNotFoundError: No module named 'pip._vendor.progress.helpers'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.8/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
  File "/usr/lib/python3.8/ensurepip/__init__.py", line 263, in _main
    return _bootstrap(
  File "/usr/lib/python3.8/ensurepip/__init__.py", line 181, in _bootstrap
    return _run_pip(args + _PROJECTS, additional_paths)
  File "/usr/lib/python3.8/ensurepip/__init__.py", line 58, in _run_pip
    return pip.main(args)
AttributeError: module 'pip' has no attribute 'main'

[unknown (charstar)]

Applying https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3690 fixed the issues I was having.

Before:

$ pip3 install --trusted-host packages.[redacted].com --extra-index-url http://packages.[redacted].com/software/python --extra-index-url https://[redacted].com/pypi/simple/ talos-mrt
Collecting talos-mrt
Exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 353, in run
    wb.build(autobuilding=True)
  File "/usr/lib/python3/dist-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 465, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 423, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/lib/python3/dist-packages/pip/index.py", line 568, in _get_pages
    page = self._get_page(location)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 683, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 795, in get_page
    resp.raise_for_status()
  File "/usr/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pypi.org/simple/talos-mrt/

After:

$ pip3 install --trusted-host packages.[redacted].com --extra-index-url http://packages.[redacted].com/software/python --extra-index-url https://[redacted].com/pypi/simple/ talos-mrt
Collecting talos-mrt
  Downloading http://packages.[redacted].com/software/python/talos-mrt/talos-mrt-1.0.1.tar.gz (40kB)
...
Building wheels for collected packages: talos-mrt, ...
...
Successfully built talos-mrt ...

Thanks!

[Gerard Weatherby (gweatherby)]

To make this copy-and-paste obvious for the next person to come by:

add-apt-repository ppa:ci-train-ppa-service/3690
apt update
apt install python3-pip

[Akkana Peck (akkzilla)]

When I use the python3-pip from ppa:ci-train-ppa-service/3690 and run:
/usr/bin/pip3 install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple metapho
in a virtualenv that's brand new except that I've installed wheel, I get:

Looking in indexes: https://test.pypi.org/simple/, https://pypi.org/simple
Requirement already satisfied: metapho in /home/akkana/.local/lib/python3.7/site-packages (1.0)
Requirement already satisfied: pycairo in /usr/lib/python3/dist-packages (from metapho) (1.16.2)
Requirement already satisfied: PyGObject in /usr/lib/python3/dist-packages (from metapho) (3.34.0)

Is that what it's supposed to do? I created the virtualenv with python3 -m venv /tmp/testpythonenv, no --system-site-packages, and what I want it to do is pull the requirements from pypi.org, not find them in /usr/lib/python3. (It also shouldn't be putting anything in .local when I'm using a venv, but that's a different issue.)

Does anyone know if this is thought to be fixed in Fossa?

[Scott Kitterman (kitterman)]

The fix was included in pip 20.0.2 brought in from Debian:

 python-pip | 20.0.2-5ubuntu1 | focal/universe | source
 python-pip | 20.0.2-5ubuntu1 | groovy/universe | source

[andrei (akela1101)]

I'm not sure how the LTS system works, but for me this doesn't look Fixed if bug is still present in bionic.

I guess, a quick workaround is to put pip2 >=20.0.2 whl to the extra-index-url repo and use it.

[Gerard Weatherby (gweatherby)]

It appears the ppa is no longer "just working" because the mainline bionic Ubuntu version of python3-pip has incremented. That is to say, even after adding the ppa and doing apt update, apt-get install pulls the buggy one from the mainline repo.

I was able to get the right version by using:
apt install python3-pip=9.0.1-2.3~ubuntu1.18.04.2~ubuntu18.04.1~ppa202002141134 python-pip-whl=9.0.1-2.3~ubuntu1.18.04.2~ubuntu18.04.1~ppa202002141134

[Gerard Weatherby (gweatherby)]

To stop automatic updates from overwriting the bug fix version of python3-pip, create file
/etc/apt/preferences.d/citrain-pin-700 with the following content:
Package: python3-pip
Pin: origin "ppa.launchpad.net"
Pin-Priority: 700

[Gerard Weatherby (gweatherby)]

Yesterday's (10-13) update of the PPA has broken the fix.

[Robie Basak (racb)]

Hello Daniel, or anyone else affected,

Accepted python-pip into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-pip/9.0.1-2.3~ubuntu1.18.04.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted python-pip (9.0.1-2.3~ubuntu1.18.04.5) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

python3.6/unknown (arm64)
python-pip/unknown (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#python-pip

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Stefano Rivera (stefanor)]

Those autopkgtest failures were just transient issues with the test environment, retrying fixed them.

[Łukasz Zemczak (sil2100)]

Can you just quickly mention which package version have you used for testing? For formality. Thanks! ;)

[Stefano Rivera (stefanor)]

Oops. Tested the version in proposed, and verified that it fixed the issue.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-pip - 9.0.1-2.3~ubuntu1.18.04.5

---------------
python-pip (9.0.1-2.3~ubuntu1.18.04.5) bionic; urgency=medium

  * Fix handling of requests exceptions when dependencies are debundled.
    (LP: #1833229)

 -- Stefano Rivera <email address hidden> Fri, 30 Apr 2021 14:11:12 -0400

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for python-pip has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.