Ubuntu 20.04: virtualenv: pep517 requires toml, but it isn't bundled in venvs

Bug #1880749 reported by Rick Elrod on 2020-05-26
106
This bug affects 21 people
Affects Status Importance Assigned to Milestone
pip
Fix Released
Unknown
python-pip (Ubuntu)
Low
Unassigned
Focal
Low
Unassigned
python-virtualenv (Ubuntu)
Low
Unassigned
Focal
Low
Unassigned

Bug Description

[Impact]

 * The pep517 module vendored in pip is missing its toml dependency.
 * No specific examples of what that breaks, but it seems worth fixing
   while we deal with LP: #1912248.
 * This upload backports upstream's 20.1 patch, replacing pytoml with
   toml, following pep517.

[Test Case]

# apt install python3 virtualenv
# virtualenv --clear-app-data -p python3 foo
# foo/bin/python -m pep517.build

Note:
ModuleNotFoundError: No module named 'toml'

Ideally the virtualenv wouldn't even contain pep517, it would be internal to pip. See LP: #1904945

[Where problems could occur]

 * Anybody who was expecting pytoml to be installed in Ubuntu Focal
   virtualenvs will have their expectation broken.
   They really shouldn't be expecting that, though.

[Original Bug Report]

On a clean 20.04 machine (or container), observe the following:

apt-get update
apt-get install -y python3-virtualenv
python3 -m virtualenv foo && source foo/bin/activate
pip list

You will notice there are a plethora of extra packages in the virtual environment that should not normally be there, resulting in a dirty virtual environment.

The packages listed here are those that are bundled with pip: https://github.com/pypa/pip/tree/master/src/pip/_vendor

To make matters worse, the latest release of pip bundles incompatible versions of libraries. The net result is that `pip install pep517` will show that it is already installed, and but `import pep517` will result in an ImportError.

This problem has been fixed in the Debian Testing/Unstable python-virtualenv package. Could the Ubuntu package backport these fixes?

This is blocking Ansible supporting 20.04 officially, since the dirty virtualenvs are causing our tests to fail. https://github.com/ansible/ansible/issues/69203

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in python-virtualenv (Ubuntu):
status: New → Confirmed
Blade Coates (romeoblade) wrote :

Please backport the fixes, this affects how we install our toolset at my company.

Jeff Geerling (geerlingguy) wrote :

It would be great to get this fixed as more people are upgrading servers and CI environments to Ubuntu 20.04 and starting to find weird edge cases related to this bug (besides not being able to install many popular packages from PPAs that haven't been updated like Ansible's).

Kai Kasurinen (kai-kasurinen) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better. However, I am closing it because the bug has been fixed in the latest development version of Ubuntu.

If you need a fix for the bug in previous versions of Ubuntu, please perform as much as possible of the SRU Procedure [1] to bring the need to a developer's attention.

[1]: https://wiki.ubuntu.com/StableReleaseUpdates#Procedure

Changed in python-virtualenv (Ubuntu):
status: Confirmed → Fix Released
Per Lundberg (perlun) wrote :

For the record: This is still broken in Ubuntu 20.04 (Focal Fossa). It breaks Ansible being able to support 20.04 in their official PPA. Tracking ticket on the Ansible side: https://github.com/ansible/ansible/issues/69203

It would be awesome if someone with the time and energy would do the work suggested by Kai to get this fix into a Focal Fossa update as well, so we can get these packages working there. Otherwise, people on Focal Fossa are bound to run the repo-provided Ansible packages (i.e. some older version).

Francois (fswanepoel) wrote :

20.04 is an LTS release, so it's the next step in upgrading from 18.04 and all new server builds.
Sadly, this fix did not make it into 20.04.1, according to the Release Notes.

As mentioned by Kai, does "latest development version of Ubuntu." mean 20.10+ releases?

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in python-virtualenv (Ubuntu Focal):
status: New → Confirmed
Roger Lehmann (roger-lehmann-u) wrote :

Almost half a year now and the LTS version is still not fixed?

This is sad, because it keeps Ansible from bringing out their PPA for focal.
I guess most users switching to pip installations will never automatically apply released security patches for it.

Julian Alarcon (julian-alarcon) wrote :

Still waiting for this bug to be resolve in Ubuntu 20.04.1 Focal Fossa.

Stefano Rivera (stefanor) wrote :
Download full text (3.4 KiB)

> You will notice there are a plethora of extra packages in the virtual environment that should not normally be there, resulting in a dirty virtual environment.

Looking at if this can be changed for Ubuntu 20.04. I'm afraid it may be complex. I think we may need to patch pip to be able to apply these changes to virtualenv.

> To make matters worse, the latest release of pip bundles incompatible versions of libraries. The net result is that `pip install pep517` will show that it is already installed, and but `import pep517` will result in an ImportError.

I can't reproduce that:

1. pep517 imports successfully.
2. pip can upgrade it, without error.

root@actual-krill:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04
Codename: focal
root@actual-krill:~# python3 -m virtualenv foo && source foo/bin/activate
created virtual environment CPython3.8.5.final.0-64 in 590ms
  creator CPython3Posix(dest=/root/foo, clear=False, global=False)
  seeder FromAppData(download=False, CacheControl=latest, appdirs=latest, certifi=latest, chardet=latest, colorama=latest, contextlib2=latest, distlib=latest, distro=latest, html5lib=latest, idna=latest, ipaddr=latest, lockfile=latest, msgpack=latest, packaging=latest, pep517=latest, pip=latest, pkg_resources=latest, progress=latest, pyparsing=latest, pytoml=latest, requests=latest, retrying=latest, setuptools=latest, six=latest, urllib3=latest, webencodings=latest, wheel=latest, via=copy, app_data_dir=/root/.local/share/virtualenv/seed-app-data/v1.0.1.debian)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
(foo) root@actual-krill:~# pip list
Package Version
------------- ----------
appdirs 1.4.3
CacheControl 0.12.6
certifi 2019.11.28
chardet 3.0.4
colorama 0.4.3
contextlib2 0.6.0
distlib 0.3.0
distro 1.4.0
html5lib 1.0.1
idna 2.8
ipaddr 2.2.0
lockfile 0.12.2
msgpack 0.6.2
packaging 20.3
pep517 0.8.2
pip 20.0.2
pkg-resources 0.0.0
progress 1.5
pyparsing 2.4.6
pytoml 0.1.21
requests 2.22.0
retrying 1.3.3
setuptools 44.0.0
six 1.14.0
urllib3 1.25.8
webencodings 0.5.1
wheel 0.34.2
(foo) root@actual-krill:~# python
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pep517
>>>
(foo) root@actual-krill:~# pip install pep517
Requirement already satisfied: pep517 in ./foo/lib/python3.8/site-packages (0.8.2)
(foo) root@actual-krill:~# pip install -U pep517
Collecting pep517
  Downloading pep517-0.9.1-py2.py3-none-any.whl (18 kB)
Collecting toml
  Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
Installing collected packages: toml, pep517
  Attempting uninstall: pep517
    Found existing installation: pep517 0.8.2
    Uninstalling pep517-0.8.2:
      Successfully uninstalled pep517-0.8.2
Successfully installed pep517-0.9.1 t...

Read more...

Changed in python-virtualenv (Ubuntu Focal):
status: Confirmed → Incomplete

Stefano,

For the pep517 issue, try importing pep517.check or pep517.build:

(foo) root@3edb3d8dca37:/# python3 -m pep517.build
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/foo/lib/python3.8/site-packages/pep517/build.py", line 6, in <module>
    import toml
ModuleNotFoundError: No module named 'toml'

Upgrading it does indeed work fine, the main issue is that it's reported as being installed already but the installed version doesn't work. This is because it's bundled with pip (and gets injected into the virtualenv even though it shouldn't (the main reason for this bug report)), and this version of pip bundled conflicting versions of libraries. They bundled a pep517 version that wants toml, but bundled pytoml alongside it instead of toml.

Rick Elrod (relrod) wrote :

Sorry, posted that from the wrong account, that comment was by me (the original bug reporter).

Stefano Rivera (stefanor) wrote :

OK, so sounds like we're missing a toml wheel in python-pip-whl.

What is the impact of this? It seems to happily install PEP 517 packages as-is.

Rick Elrod (relrod) wrote :

The impact of the pep517 issue isn't too big of an issue for us- we can work around it (installing toml).

The "impact" is that we have a test scenario which runs `pip install pep517` (reports already installed) and then tries to do `python -m pep517.build --binary --out-dir dist .` (which fails due to the toml issue).

As mentioned, we can work around this by installing toml. I'm more worried about the broader issue in the report, that virtualenvs are dirty in the first place, this limits their utility :(

Stefano Rivera (stefanor) wrote :

I think we can get the missing toml resolved.

Agreed that that's really ugly that the virtualenvs include all of the pip deps. That was an issue in the early virtualenv 20 days, that took a while to get the necessary upstream support for cleaner de-vendoring. But messyness on its own can't justify broader changes to a stable release. The process for these changes is a Stable Release Update (SRU), see the policy linked below.
We need to avoid breaking things for users of stable releases, once they are released. The change has to be proven to be worth the risk. I'll need some more concrete examples of the impact, to push through an SRU. And to show that the SRU resolves the issues.

https://wiki.ubuntu.com/StableReleaseUpdates

Stefano Rivera (stefanor) wrote :

Retargetting this bug at just the missing toml. Use LP: #1904945 for the dirty virtualenvs.

summary: - python3 virtualenvs include (broken) bundled pip deps
+ Ubuntu 20.04: virtualenv: pep517 requires toml, but it isn't bundled in
+ venvs
Changed in virtualenv:
status: Unknown → Fix Released
Changed in python-pip (Ubuntu Groovy):
status: New → Fix Released
Changed in python-pip (Ubuntu):
status: New → Fix Released
description: updated
Stefano Rivera (stefanor) wrote :

@Rick Elrod: It would be useful if you could describe something that's concretely broken by toml being missing? I haven't been able to trigger any issue, installing packages with pip.

Stefano Rivera (stefanor) wrote :
description: updated
Łukasz Zemczak (sil2100) wrote :

It's always a bit tricky reviewing SRUs without a visible user impact. But I would say that since the vendoring of pytoml instead of toml *is* a bug, let's get it included in the SRU for fixing the dirty venvs. I can't think of any use-cases this could break. Let's see how it goes.

Changed in python-pip (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed verification-needed-focal

Hello Rick, or anyone else affected,

Accepted python-pip into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-pip/20.0.2-5ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python-virtualenv (Ubuntu Focal):
status: Incomplete → Invalid

All autopkgtests for the newly accepted python-pip (20.0.2-5ubuntu1.2) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

python3.8/3.8.5-1~20.04 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#python-pip

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Stefano Rivera (stefanor) wrote :

Re-tried the armhf autopkgtest and it passed.

Rick Elrod (relrod) wrote :

> What is the impact of this? It seems to happily install PEP 517 packages as-is.

@Stefano sorry for the delay here. Installing PEP517 packages is fine without the patch, but building them (building a wheel) is where the issue comes in.

If you need a concrete example, download the .tar.gz of any project that is normally capable of producing wheels, such as 'sampleproject' from pypi (https://pypi.org/project/sampleproject/#files).
Activate a fresh virtualenv, untar it the .tar.gz, cd into the project, and run:

  python -m pep517.build --binary --out-dir dist .

Before the patch mentioned above, it will fail with: ModuleNotFoundError: No module named 'toml'

The .deb's from the SRU above get us a step closer but seem to still have an issue/traceback (see attachment which includes the commands to reproduce). It looks to be due to the vendored pytoml being listed in `lib/python3.8/site-packages/pip/_vendor/__init__.py` in the virtualenv.

Whereas doing similar steps in a ubuntu:20.10 container (and installing pep517 explicitly in the virtualenv) produces a successful result.

Mathew Hodson (mhodson) on 2021-02-15
no longer affects: python-virtualenv (Ubuntu)
no longer affects: python-virtualenv (Ubuntu Focal)
no longer affects: python-virtualenv (Ubuntu Groovy)
affects: virtualenv → pip
Stefano Rivera (stefanor) wrote :

@Rick: Thanks for the response.

A few things:
1. I don't think you're using the current wheels.
   Try with --clear-app-data (added to the SRU instructions in the description).
2. What this still doesn't tell me is how *pip* is impacted by this breakage.
   LP: #1904945 means that we won't expose pep517 to the virtualenv outside of pip, any more.
   So, pep517 being broken outside of pip wouldn't be an issue.

description: updated
James Page (james-page) wrote :

We have -proposed enabled in one of our focal OpenStack deployments and this update appears to have regressed the setup of virtualenv's for reactive charms (which ship with a wheelhouse of python module dependencies including pip) - we see this error as new charms attempt to setup a venv using pip:

ImportError: cannot import name 'pytoml' from 'pip._vendor'

Downgrading the python3-pip + hard versioned depends resolves this issue for us.

tags: added: verification-failed verification-failed-focal
removed: verification-needed verification-needed-focal
James Page (james-page) wrote :

I've marked the verification tags as 'failed' as releasing this update to -updates will have a major impact on charm deployments on focal.

James Page (james-page) wrote :
James Page (james-page) wrote :

Full error in attached logfile

Łukasz Zemczak (sil2100) wrote :

Thank you for flagging this regression. @Stefano, can you take a look at these issues? I think we might need to drop this update from -proposed.

Stefano Rivera (stefanor) wrote :

After Rick's failed verification comment I had been wondering if we'd run into trouble with stale seed-app-data, and yes we have 2 different regressions:

1. The new python-pip-whl breaks new virtualenv creation if you have existing seed-app-data.
   This was James's issue. Traceback ends with:
   File "lib/python3.8/site-packages/pip/_internal/pyproject.py", line 8, in <module>
     from pip._vendor import pytoml, six
   ImportError: cannot import name 'pytoml' from 'pip._vendor' (lib/python3.8/site-packages/pip/_vendor/__init__.py)

2. The new virtualenv breaks new virtualenv creation if you have existing seed-app-data.
   File "lib/python3.8/site-packages/pip/_internal/exceptions.py", line 10, in <module>
     from pip._vendor.six import iteritems
   ModuleNotFoundError: No module named 'pip._vendor.six'

Attached is my script to reproduce.

This is all pretty horrible. I'm tempted to patch out seed-app-data entirely for our use-case. But
I see the whole seed-app-data mechanism was re-done in 20.0.24, hopefully it will be more friendly to patching...

My solution is:

1. Bump the seed-app-data version in python3-virtualenv, to cache-bust.
2. Add a Breaks: python3-virtualenv (<< 20.0.17-1ubuntu0.3) (the version in 1) to the new python-pip-whl.

Stefano Rivera (stefanor) wrote :
Changed in python-virtualenv (Ubuntu):
status: New → Fix Released
Mathew Hodson (mhodson) on 2021-03-01
no longer affects: python-pip (Ubuntu Groovy)
no longer affects: python-virtualenv (Ubuntu)
no longer affects: python-pip (Ubuntu Focal)
Changed in python-pip (Ubuntu Focal):
status: New → Fix Committed
Changed in python-virtualenv (Ubuntu):
status: New → Fix Released
Changed in python-pip (Ubuntu):
importance: Undecided → Low
Changed in python-pip (Ubuntu Focal):
importance: Undecided → Low
Changed in python-virtualenv (Ubuntu):
importance: Undecided → Low
Changed in python-virtualenv (Ubuntu Focal):
importance: Undecided → Low

Hello Rick, or anyone else affected,

Accepted python-pip into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-pip/20.0.2-5ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-needed verification-needed-focal
removed: verification-failed verification-failed-focal
Łukasz Zemczak (sil2100) wrote :

Hello Rick, or anyone else affected,

Accepted python-virtualenv into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-virtualenv/20.0.17-1ubuntu0.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python-virtualenv (Ubuntu Focal):
status: New → Fix Committed
Stefano Rivera (stefanor) wrote :

Nobody has reported any more issues, and the tests we've got pass for me, so marking verification-done

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-pip - 20.0.2-5ubuntu1.3

---------------
python-pip (20.0.2-5ubuntu1.3) focal; urgency=medium

  * Add Breaks: python3-virtualenv (<< 20.0.17-1ubuntu0.3) to python-pip-whl,
    which bumps the seed-app-data version. Our LP: #1880749 change broke
    existing seed-app-data caches.

python-pip (20.0.2-5ubuntu1.2) focal; urgency=medium

  * Switch from vendoring pytoml to toml, following pep517's dependency.
    (LP: #1880749)
  * Use sys.base_prefix instead of sys.prefix in debundle.patch. Back-ported
    from 20.1-1 to allow virtualenvs to avoid needing to install pip's
    dependencies in new venvs. (LP: #1904945)

 -- Stefano Rivera <email address hidden> Fri, 26 Feb 2021 18:38:56 -0800

Changed in python-pip (Ubuntu Focal):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for python-pip has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-virtualenv - 20.0.17-1ubuntu0.3

---------------
python-virtualenv (20.0.17-1ubuntu0.3) focal; urgency=medium

  * Bump the seed-app-data version, because we changed the bundled wheels in
    LP: #1880749.

 -- Stefano Rivera <email address hidden> Fri, 26 Feb 2021 18:09:22 -0800

Changed in python-virtualenv (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.