pypy virtualenv has broken pip on focal

[Ben Bariteau]

[Impact]

 * pypy and pypy3 virtualenvs created with virtualenv have an unuseable pip.

[Test Plan]

# apt install pypy virtualenv
$ virtualenv -p pypy testve --clear-app-data
$ testve/bin/python -m pip

Failure:
A traceback ending in:
ModuleNotFoundError: No module named 'pip._vendor.six'

Success:
--help output

[Where problems could occur]

 * This change has been carried in Debian and Ubuntu since 20.1-2, so it's well tested. It's very simple, and should have been included in the Bug #1904945 SRU

[Original Bug report]

I reproduced this in a docker image (ubuntu:20.04)

Reproduction steps:
1) install `virtualenv` and `pypy`
2) run `virtualenv --python pypy venv`
3) run `venv/bin/pip`
This fails with this error output:
Traceback (most recent call last):
  File "venv/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
  File "/venv/site-packages/pip/_internal/cli/main.py", line 10, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/venv/site-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/venv/site-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
    from pip._internal.cli import cmdoptions
  File "/venv/site-packages/pip/_internal/cli/cmdoptions.py", line 24, in <module>
    from pip._internal.exceptions import CommandError
  File "/venv/site-packages/pip/_internal/exceptions.py", line 10, in <module>
    from pip._vendor.six import iteritems
ImportError: No module named pip._vendor.six

This should have printed out the usage information for pip.

I tried working around this by running `virtualenv --python pypy --download venv`. This allows running just `venv/bin/pip` to succeed, but a typical real usage of it fails like this:
# venv/bin/pip install requests
DEPRECATION: pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting requests
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
     |################################| 61 kB 8.8 MB/s
Collecting chardet<5,>=3.0.2
  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
     |################################| 178 kB 26.8 MB/s
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
     |################################| 58 kB 9.8 MB/s
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
     |################################| 138 kB 43.6 MB/s
Collecting certifi>=2017.4.17
  Downloading certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
     |################################| 145 kB 48.1 MB/s
Installing collected packages: chardet, idna, urllib3, certifi, requests
ERROR: Exception:
Traceback (most recent call last):
  File "/venv/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
    status = self.run(options, args)
  File "/venv/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
    return func(self, options, args)
  File "/venv/site-packages/pip/_internal/commands/install.py", line 404, in run
    pycompile=options.compile,
  File "/venv/site-packages/pip/_internal/req/__init__.py", line 90, in install_given_reqs
    pycompile=pycompile,
  File "/venv/site-packages/pip/_internal/req/req_install.py", line 824, in install
    requested=self.user_supplied,
  File "/venv/site-packages/pip/_internal/operations/install/wheel.py", line 845, in install_wheel
    requested=requested,
  File "/venv/site-packages/pip/_internal/operations/install/wheel.py", line 719, in _install_wheel
    assert os.path.exists(pyc_path)
AssertionError

This should have installed the requests package, instead of failing.

It's worth noting this seems to work with cpython. I specifically tried with python2.7 and the default python3 (python3.8). Using pypy3 fails in the same way.

Release info:
# lsb_release -rd
Description: Ubuntu 20.04.2 LTS
Release: 20.04

[Stefano Rivera (stefanor)]

This was fixed in https://salsa.debian.org/python-team/packages/python-pip/-/commit/752d2a38f88d5c80b6d93ce91992057f5c75cafc

I should have included that in 20.0.2-5ubuntu1.2

[Brian Murray (brian-murray)]

Hello Ben, or anyone else affected,

Accepted python-pip into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-pip/20.0.2-5ubuntu1.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Stefano Rivera (stefanor)]

Tests successfully for me, although I have to clear the virtualenv cached eggs with --clear-app-data.

We could upload virtualenv to bump the version again, like https://launchpad.net/ubuntu/+source/python-virtualenv/20.0.17-1ubuntu0.3 to make this automatic, and avoid users who have hit the bug, needing to --clear-app-data, to resolve it.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted python-pip (20.0.2-5ubuntu1.6) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

python3.9/3.9.5-3~20.04.1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#python-pip

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ben Bariteau (benbariteau)]

I used the proposed version of python-pip-whl, 20.0.2-5ubuntu1.6 and it solved the bugs I was dealing with. I followed the reproduction steps above but got the proper help output. Also, I went ahead and installed some packages, specifically a different version of pip and the pip-custom-platform package and it seems to work as expected.

[Stefano Rivera (stefanor)]

> The following regressions have been reported in tests triggered by the package:

> python3.9/3.9.5-3~20.04.1 (amd64)

This was a transient failure (a timeout), retrying it worked.

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for python-pip has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-pip - 20.0.2-5ubuntu1.6

---------------
python-pip (20.0.2-5ubuntu1.6) focal; urgency=medium

  * Update debundle.patch so pypy uses base_prefix as well, fixing pypy
    virtualenvs. (LP: #1935882)

 -- Stefano Rivera <email address hidden> Mon, 12 Jul 2021 23:11:28 -0400