New virtualenvs contain unwanted libraries

[mimosomal]

[Impact]

 * Virtualenvs created on Ubuntu 20.04 are not as empty as they are
   expected to be. They are expected to contain pip and setuptools.
   But the way pip's vendored dependencies were de-vendored resulted
   in the libraries being exposed in the new virtualenv.
   This was a side-effect of packaging the (rewritten) virtualenv 20.
   It should have been resolved pre-release, but wasn't.

 * The behaviour is radically different to upstream virtualenv and
   not what was expected by Python developers.

[Test Case]

# apt install python3 virtualenv
$ virtualenv -p python3 foo
$ foo/bin/python -m pip freeze
Should output nothing. In Ubuntu 20.04, it lists 24 packages.

[Where problems could occur]

 * Anyone who has discovered this change and now expects it will be in
   for a surprise. I'd expect this to be rare, as there are workarounds
   instead (e.g. venv or upstream virtualenv).
 * The patch could have issues too. It is close to what is currently in
   Debian unstable & later Ubuntu releases, and seems to be working
   correctly.

[Other Info]

 * See also LP: #1880749 which is fixed in the same patch.

[Original Report]

Since updating to ubuntu 20.04, whenever I create a virtual environment with `virtualenv` it contains many unwanted libraries. Unsetting PYTHONPATH, setting the `--python` flag to a different version, etc does not fix the issue. I could run it with `--no-seed` but then the new venv doesn't even have pip.

I found another user who has the exact same issue posting about it on stackoverflow and the list of packages which were installed by default is the same, this leads me to believe that something is wrong with the package itself and not a local config issue.
`https://stackoverflow.com/q/62991007/5953826`

This is the output from creating a virtualenv:
```
➜ virtualenv --python=/usr/bin/python3.8 ~/projects/environments/test
created virtual environment CPython3.8.5.final.0-64 in 156ms
  creator CPython3Posix(dest=/home/redacted/projects/environments/test, clear=False, global=False)
  seeder FromAppData(download=False, urllib3=latest, contextlib2=latest, idna=latest, ipaddr=latest, colorama=latest, pkg_resources=latest, webencodings=latest, pip=latest, progress=latest, setuptools=latest, retrying=latest, certifi=latest, chardet=latest, appdirs=latest, msgpack=latest, html5lib=latest, pytoml=latest, pyparsing=latest, distro=latest, CacheControl=latest, distlib=latest, pep517=latest, lockfile=latest, requests=latest, packaging=latest, six=latest, wheel=latest, via=copy, app_data_dir=/home/redacted/.local/share/virtualenv/seed-app-data/v1.0.1.debian)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
```

And these exact package versions are what get installed by default.
```
appdirs==1.4.3
CacheControl==0.12.6
certifi==2019.11.28
chardet==3.0.4
colorama==0.4.3
contextlib2==0.6.0
distlib==0.3.0
distro==1.4.0
html5lib==1.0.1
idna==2.8
ipaddr==2.2.0
lockfile==0.12.2
msgpack==0.6.2
packaging==20.3
pep517==0.8.2
progress==1.5
pyparsing==2.4.6
pytoml==0.1.21
requests==2.22.0
retrying==1.3.3
six==1.14.0
urllib3==1.25.8
webencodings==0.5.1
```

All of the packages above are in the folder `~/.local/share/virtualenv/seed-app-data/v1.0.1.debian/3.8/wheels`, but removing ~/.local/share/virtualenv/seed-app-data does not fix the issue. It is recreated with the same packages.

Creating a venv with `python3 -m venv path/to/venv` gives me a clean environment with no extra packages which is my current workaround.

System Info:

lsb_release -rd
Description: Ubuntu 20.04.1 LTS
Release: 20.04

apt-cache policy python3-virtualenv
python3-virtualenv:
  Installed: 20.0.17-1
  Candidate: 20.0.17-1
  Version table:
 *** 20.0.17-1 500
        500 http://us.archive.ubuntu.com/ubuntu focal/universe amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu focal/universe i386 Packages
        100 /var/lib/dpkg/status

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: python3-virtualenv 20.0.17-1
ProcVersionSignature: Ubuntu 5.4.0-53.59-generic 5.4.65
Uname: Linux 5.4.0-53-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair nvidia_modeset nvidia
ApportVersion: 2.20.11-0ubuntu27.12
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: ubuntu:GNOME
Date: Thu Nov 19 18:54:36 2020
InstallationDate: Installed on 2020-05-11 (192 days ago)
InstallationMedia: Ubuntu 20.04 LTS "Focal Fossa" - Release amd64 (20200423)
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/usr/bin/zsh
SourcePackage: python-virtualenv
UpgradeStatus: No upgrade log present (probably fresh install)

[Stefano Rivera (stefanor)]

The issue is a result of devendoring the pip dependencies. See the discussion in #1880749. It was improved in Ubuntu 20.10, but too late for Focal.

[Stefano Rivera (stefanor)]

Let's use the right syntax for a link: LP: #1880749

[Łukasz Zemczak (sil2100)]

Hello mimosomal, or anyone else affected,

Accepted python-pip into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-pip/20.0.2-5ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Ok, so looking at the python3-virtualenv change for this SRU - this now adds a completely new dependency to the python3-virtualenv package. From what I see it never depended on python3-pip before. I know it is a dependency for groovy+, but is it really required for every venv package installation?

[Stefano Rivera (stefanor)]

Whoops, looks like a typo, that should have been python-pip-whl, not python3-pip

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted python-pip (20.0.2-5ubuntu1.2) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

python3.8/3.8.5-1~20.04 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#python-pip

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Stefano Rivera (stefanor)]

Corrected the dependency

[Łukasz Zemczak (sil2100)]

Hello mimosomal, or anyone else affected,

Accepted python-virtualenv into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-virtualenv/20.0.17-1ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello mimosomal, or anyone else affected,

Accepted python-pip into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-pip/20.0.2-5ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Stefano Rivera (stefanor)]

LGTM, verification-done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-pip - 20.0.2-5ubuntu1.3

---------------
python-pip (20.0.2-5ubuntu1.3) focal; urgency=medium

  * Add Breaks: python3-virtualenv (<< 20.0.17-1ubuntu0.3) to python-pip-whl,
    which bumps the seed-app-data version. Our LP: #1880749 change broke
    existing seed-app-data caches.

python-pip (20.0.2-5ubuntu1.2) focal; urgency=medium

  * Switch from vendoring pytoml to toml, following pep517's dependency.
    (LP: #1880749)
  * Use sys.base_prefix instead of sys.prefix in debundle.patch. Back-ported
    from 20.1-1 to allow virtualenvs to avoid needing to install pip's
    dependencies in new venvs. (LP: #1904945)

 -- Stefano Rivera <email address hidden> Fri, 26 Feb 2021 18:38:56 -0800

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for python-pip has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Chris Patterson (cjp256)]

The change to use sys.base_prefix has caused a regression in at least one python snap (Charmcraft). The effect of this change caused core20 python snaps to now revert to using '/usr' base_prefix when looking for wheels rather than the virtual environment root, effectively $SNAP.

For example:
>>> sys.prefix
'/snap/charmcraft/x1'
>>> sys.base_prefix
'/usr'

This prevented wheels from inside the snap from being loaded. We worked around this using a snap layout in a strict snap to put the python wheels in the newly expected location. For example:

```
layout:
  /usr/share/python-wheels:
    bind: $SNAP/share/python-wheels
```

I wonder if it really should even be loading the host's wheels when inside a virtual env? With `python3 -m venv` the wheels appear to be copied into the virtual env. `virtualenv` seems to ship its own embedded variants.

[Stefano Rivera (stefanor)]

Sorry I missed that regression, it's tracked in bug 1935882.