isolated-builds aren't isolated when run with system dist-packages

Bug #2018288 reported by Abou Samra Jean
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
pip
Fix Released
Unknown
python-pip (Ubuntu)
Fix Released
Undecided
Unassigned
Jammy
Confirmed
Undecided
Unassigned

Bug Description

[ Impact ]

pip is unable to perform isolated builds (that are truly isolated) when run in a --system-site-packages virtualenv, or on the bare system.

This means one can't depend on newer setuptools features, for example

[ Test Plan ]
# apt install python3-venv
$ python3 -m venv --system-site-packages /tmp/ve
$ mkdir /tmp/test-pkg
$ cd /tmp/test-pkg
$ cat > pyproject.toml <<EOF
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "tmp"
version = "0"
EOF
$ cat > tmp.py <<EOF
print("Hi")
EOF
$ cat > setup.py <<EOF
import sys
from setuptools import setup

assert "/usr/lib/python3/dist-packages" not in sys.path

setup()
EOF
$ /tmp/ve/bin/python -m pip install .
...
Successfully installed tmp-0

Or, in the case of this bug, a failure due to the AssertionError

[ Where problems could occur ]

Upstream went through quite a meandering path to fix this series of bugs in their isolated build implementation. Since Debian got involved in reviewing the patches, we got down to a relatively simple approach, that this SRU can apply.

The current approach, applied here, shipped in 23.1, and we haven't been aware of any issues with it.

[ Other Info ]
https://github.com/pypa/pip/issues/6264
https://github.com/pypa/pip/issues/11539
https://github.com/pypa/pip/pull/11466
https://github.com/pypa/pip/pull/11598
https://github.com/pypa/pip/pull/11623
https://github.com/pypa/pip/pull/11740

== Original Description ==
This happens on Ubuntu Jammy. I have not tested other versions.

To reproduce this bug, first create a dummy Python package. In a directory called `tmp`, put these files:

```
jean@ubuntu22:~/tmp$ cat pyproject.toml
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "tmp"
version = "0"
jean@ubuntu22:~/tmp$ cat tmp.py
print("Hi")
```

Now run

```
pip install --verbose .
```

to install this package.

```
jean@ubuntu22:~/tmp$ pip install --verbose .
Using pip 22.0.2 from /home/jean/.local/lib/python3.10/site-packages/pip (python 3.10)
Defaulting to user installation because normal site-packages is not writeable
Processing /home/jean/tmp
  Running command pip subprocess to install build dependencies
  Collecting setuptools
    Using cached setuptools-67.7.2-py3-none-any.whl (1.1 MB)
  Installing collected packages: setuptools
  Successfully installed setuptools-67.7.2
  WARNING: You are using pip version 22.0.2; however, version 23.1.2 is available.
  You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  running egg_info
  writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
  Getting requirements to build wheel ... done
  Running command pip subprocess to install backend dependencies
  Collecting wheel
    Using cached wheel-0.40.0-py3-none-any.whl (64 kB)
  Installing collected packages: wheel
  Successfully installed wheel-0.40.0
  WARNING: You are using pip version 22.0.2; however, version 23.1.2 is available.
  You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
  Installing backend dependencies ... done
  Running command Preparing metadata (pyproject.toml)
  running dist_info
  creating /tmp/pip-modern-metadata-txaro9id/UNKNOWN.egg-info
  writing manifest file '/tmp/pip-modern-metadata-txaro9id/UNKNOWN.egg-info/SOURCES.txt'
  writing manifest file '/tmp/pip-modern-metadata-txaro9id/UNKNOWN.egg-info/SOURCES.txt'
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: UNKNOWN
  Running command Building wheel for UNKNOWN (pyproject.toml)
  running bdist_wheel
  running build
  running install
  running install_egg_info
  running egg_info
  writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
  Copying UNKNOWN.egg-info to build/bdist.linux-x86_64/wheel/UNKNOWN-0.0.0.egg-info
  running install_scripts
  Building wheel for UNKNOWN (pyproject.toml) ... done
  Created wheel for UNKNOWN: filename=UNKNOWN-0.0.0-py3-none-any.whl size=961 sha256=b830a03d9373fbb21b32a5af7c67e0ca7152b2a1df065a6ff4b3b0cff50d2025
  Stored in directory: /tmp/pip-ephem-wheel-cache-nvdj0gls/wheels/35/9b/89/cfafbe958c42b0384d04f2116451bbb21b671e85b591550ef1
Successfully built UNKNOWN
Installing collected packages: UNKNOWN
Successfully installed UNKNOWN-0.0.0
WARNING: You are using pip version 22.0.2; however, version 23.1.2 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
```

Note how the package ends up called "unknown". This is a known problem with older setuptools versions. However, pip is building a wheel for the package in an isolated environment, where it has installed setuptools 67.7.2, which is the latest version.

Another hint at the problem:

```
$ pip install --verbose --editable .
Using pip 22.0.2 from /home/jean/.local/lib/python3.10/site-packages/pip (python 3.10)
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/jean/tmp
  Running command pip subprocess to install build dependencies
  Collecting setuptools
    Using cached setuptools-67.7.2-py3-none-any.whl (1.1 MB)
  Installing collected packages: setuptools
  Successfully installed setuptools-67.7.2
  WARNING: You are using pip version 22.0.2; however, version 23.1.2 is available.
  You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
  Installing build dependencies ... done
  Running command Checking if build backend supports build_editable
  Checking if build backend supports build_editable ... done
ERROR: Project file:///home/jean/tmp has a 'pyproject.toml' and its build backend is missing the 'build_editable' hook. Since it does not have a 'setup.py' nor a 'setup.cfg', it cannot be installed in editable mode. Consider using a build backend that supports PEP 660.
WARNING: You are using pip version 22.0.2; however, version 23.1.2 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
```

The latest setuptools version definitely supports editable builds.

This indicates that within the isolated build environment, setuptools from the system package python3-setuptools was used instead of the temporary copy of setuptools.

This problem does not occur on Fedora 38. I verified this by editing `/usr/lib/python3.11/site-packages/setuptools/__init__.py` to raise SystemExit. The install still completes successfully. This hints at problems in distro patches from Ubuntu or Debian. This is also the opinion of two pip maintainers; see https://discuss.python.org/t/pip-complains-your-backend-does-not-support-editable-installs-with-latest-setuptools/26387

I was not quite sure which package to file this bug against. It does not seem to come from distro patches to pip *itself* (but perhaps distro patches to the Python interpreter, e.g., sysconfig): using pip from PyPI ("pip install --upgrade pip==22.0.2", installing a pip that shadows the system pip) also exhibits the bug. Although "pip install --upgrade pip" to get the latest version does *not* exhibit the bug, so this is probably an interaction between distro patches *and* the older pip version in Jammy.

In my opinion, this bug is quite severe, as it makes the system pip mostly unusable for installing packages from local checkouts, at least when those packages use setuptools as their build backend.

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: python3-pip 22.0.2+dfsg-1ubuntu0.2
ProcVersionSignature: Ubuntu 5.19.0-41.42~22.04.1-generic 5.19.17
Uname: Linux 5.19.0-41-generic x86_64
ApportVersion: 2.20.11-0ubuntu82.4
Architecture: amd64
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
Date: Tue May 2 11:25:42 2023
InstallationDate: Installed on 2023-01-06 (115 days ago)
InstallationMedia: Ubuntu 22.04.1 LTS "Jammy Jellyfish" - Release amd64 (20220809.1)
PackageArchitecture: all
SourcePackage: python-pip
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Abou Samra Jean (jean-abou-samra) wrote :
description: updated
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in python-pip (Ubuntu):
status: New → Confirmed
Revision history for this message
lt (groove646) wrote :

Hi, I followed the installation instructions https://github.com/marph91/joplin-sticky-notes.git and this happened:

...@pop-os:$ pip install git+https://github.com/marph91/joplin-sticky-notes.git
Command 'pip' not found, but can be installed with:
sudo apt install python3-pip
...@pop-os:$ sudo apt install python3-pip
...@pop-os:~$ pip install git+https://github.com/marph91/joplin-sticky-notes.git
Defaulting to user installation because normal site-packages is not writeable
Collecting git+https://github.com/marph91/joplin-sticky-notes.git
Cloning https://github.com/marph91/joplin-sticky-notes.git to /tmp/pip-req-build-w4n_85po
Running command git clone --filter=blob:none --quiet https://github.com/marph91/joplin-sticky-notes.git /tmp/pip-req-build-w4n_85po
Resolved https://github.com/marph91/joplin-sticky-notes.git to commit faa7e99
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: UNKNOWN
Building wheel for UNKNOWN (pyproject.toml) ... done
Created wheel for UNKNOWN: filename=UNKNOWN-0.0.0-py3-none-any.whl size=961 sha256=4b34b9b6973c05a23b906ad485854a441269048f9dec625e49dd76807295348c
Stored in directory: /tmp/pip-ephem-wheel-cache-pa0cdzy6/wheels/7f/60/43/6c2f34e3a73f733374d7421d6371370ae78c99cc786f42488d
Successfully built UNKNOWN
Installing collected packages: UNKNOWN
Successfully installed UNKNOWN-0.0.0

Revision history for this message
Stefano Rivera (stefanor) wrote :

Can't reproduce either of your issues on jammy. You have other things going on, clearly.

# apt install python3-venv git
$ python3 -m venv ve
$ mkdir test-pkg
$ cat > test-pkg/pyproject.toml <<EOF
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "tmp"
version = "0"
EOF
$ cat > test-pkg/tmp.py <<EOF
print("Hi")
EOF
$ cd test-pkg/
$ ../ve/bin/python -m pip install .
Processing /home/testuser/test-pkg
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: tmp
  Building wheel for tmp (pyproject.toml) ... done
  Created wheel for tmp: filename=tmp-0-py3-none-any.whl size=992 sha256=f06ad5416dc5d1e05d2832794a6f998c5db2da537c7f6b985064ddea45230d81
  Stored in directory: /home/testuser/.cache/pip/wheels/51/14/ce/f5f8a5531af6b5bddf4a8db7307e920df305f48175c301a865
Successfully built tmp
Installing collected packages: tmp
Successfully installed tmp-0
$ ../ve/bin/python -m pip install git+https://github.com/marph91/joplin-sticky-notes.git
Collecting git+https://github.com/marph91/joplin-sticky-notes.git
  Cloning https://github.com/marph91/joplin-sticky-notes.git to /tmp/pip-req-build-gmky4g20
  Running command git clone --filter=blob:none --quiet https://github.com/marph91/joplin-sticky-notes.git /tmp/pip-req-build-gmky4g20
  Resolved https://github.com/marph91/joplin-sticky-notes.git to commit faa7e9912c6411afc60d970df9eb32dbc5906fb4
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: joplin-sticky-notes
  Building wheel for joplin-sticky-notes (pyproject.toml) ... done
  Created wheel for joplin-sticky-notes: filename=joplin_sticky_notes-0.0.1-py3-none-any.whl size=22626 sha256=625531c87d913b0bfe3b5951bf1fd8e6e744f4a537d82189b1f0d4a33f99e6b2
  Stored in directory: /tmp/pip-ephem-wheel-cache-pxo67jxq/wheels/7f/60/43/6c2f34e3a73f733374d7421d6371370ae78c99cc786f42488d
Successfully built joplin-sticky-notes
Installing collected packages: joplin-sticky-notes
Successfully installed joplin-sticky-notes-0.0.1

Changed in python-pip (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Abou Samra Jean (jean-abou-samra) wrote :

@stefanor : You created a venv and ran the pip from inside. I ran the global pip (yes, yes, I know there is PEP 668, but I had good reasons).

Changed in python-pip (Ubuntu):
status: Incomplete → Confirmed
Changed in pip:
status: Unknown → Fix Released
Changed in python-pip (Ubuntu):
status: Confirmed → Fix Released
Changed in python-pip (Ubuntu Jammy):
status: New → Confirmed
summary: - pip picks up wrong setuptools version during isolated build
+ isolated-builds aren't isolated when run with system dist-packages
Revision history for this message
Stefano Rivera (stefanor) wrote :
description: updated
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

stefanor: did you build the source on Debian? source.changes doesn't refer to this bug

Revision history for this message
Stefano Rivera (stefanor) wrote :

> did you build the source on Debian? source.changes doesn't refer to this bug

I always do. And sometimes forget to invoke the magic to make it do Ubuntu stuff. Re-uploaded.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.