python-pip >= 21.3.1+dfsg-3 fails pip3-root.sh autopkgtest with python3.10

[Alexandre Ghiti]

pip3-root.sh expects world package to be installed in /usr/local/lib/python3.*/dist-packages whereas it is actually installed into /usr/lib/python3.10/site-packages/.

It even fails to uninstall the package right after installing it:

ubuntu@autopkgtest:~$ sudo python3 -m pip install world
Collecting world
  Using cached world-4.1.1-py3-none-any.whl
Collecting atpublic
  Using cached atpublic-3.0.1-py3-none-any.whl (4.8 kB)
Installing collected packages: atpublic, world
Successfully installed atpublic-3.0.1 world-4.1.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
ubuntu@autopkgtest:~$ sudo python3 -m pip uninstall world
WARNING: Skipping world as it is not installed.
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[Stefano Rivera (stefanor)]

As I understand it, this is because pip expects the installation layouts in sysconfig to be correct since Python 3.10. Previously it used the layouts in distutils.sysconfig.

See:
* LP: #1940705
* LP: #1408092
* https://github.com/pypa/pip/pull/10358

The correct resolution here is to enable sysconfig-debian-schemes.diff in Python 3.10. I'm not sure why this hasn't been enabled, yet.

[Alexandre Ghiti (alexghiti)]

I talked with doko about that, and he told me to give it a try. The patch does not apply directly, I had to rebase it: it seems to fix this issue (i.e.: I can install a package to the right location with my new python 3.10 package here [1]) but then when installed, it fails to build python-pip package with the following error:

make[1]: Leaving directory '/tmp/autopkgtest.hzRvXB/build.kr1/real-tree'
   dh_installsystemduser -O--buildsystem=pybuild
   dh_lintian -O--buildsystem=pybuild
   dh_perl -O--buildsystem=pybuild
   dh_usrlocal -O--buildsystem=pybuild
dh_usrlocal: error: debian/python3-pip/usr/local/bin/pip is not a directory
make: *** [debian/rules:8: binary] Error 255
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
blame: python-pip-22.0.2+dfsg/
badpkg: rules build failed with exit code 2

I attach my rebase that may be wrong if you can take a look.

(I'll send the same info by mail as requested by doko.)

[1] https://launchpad.net/~alexghiti/+archive/ubuntu/riscv/+sourcepub/13261732/+listing-archive-extra

[Stefano Rivera (stefanor)]

OK, that seems to be caused by the default scheme changing from posix_prefix to posix_local, dh-python was making silly assumptions, there.

Your updated sysconfig patch looks good.

[Launchpad Janitor (janitor)]

This bug was fixed in the package dh-python - 5.20220215

---------------
dh-python (5.20220215) unstable; urgency=medium

  * pybuild plugin: Explicitly unpack wheels using the "deb_system" sysconfig
    scheme, allowing python3.10 to change the default sysconfig scheme to
    "posix_local". (LP: #1960608)

 -- Stefano Rivera <email address hidden> Tue, 15 Feb 2022 20:26:39 -0400

[Alexandre Ghiti (alexghiti)]

I'm not sure I understand: do I need to enable the sysconfig patch for python3.10 in addition to your fix?

Thanks

[Stefano Rivera (stefanor)]

Yes, it's required. My fix was resolving the python-pip FTBFS you reported with that patch.

[Matthias Klose (doko)]

python3.10 now in -proposed

[Gianfranco Costamagna (costamagnagianfranco)]

Hello, there are also bugs in python.m4, the same fix as https://github.com/pybind/pybind11/pull/3764/files fixes the issue in m4 too

I patched locally my python.m4 with something like this:

- sitedir = sysconfig.get_path('purelib', vars={'base':'$am_py_prefix'})
- sitedir = sysconfig.get_path('purelib', scheme='posix_prefix', vars={'base':'$am_py_prefix'})

- sitedir = sysconfig.get_path('platlib', vars={'platbase':'$am_py_exec_prefix'})
to
+ sitedir = sysconfig.get_path('platlib', scheme='posix_prefix', vars={'platbase':'$am_py_exec_prefix'})

And firewalld was building correctly again.

[Launchpad Janitor (janitor)]

This bug was fixed in the package automake-1.16 - 1:1.16.5-1.1ubuntu1

---------------
automake-1.16 (1:1.16.5-1.1ubuntu1) jammy; urgency=medium

  * Upload to jammy with a fix for LP: #1960608

 -- Gianfranco Costamagna <email address hidden> Fri, 04 Mar 2022 10:29:43 +0100

[Gianfranco Costamagna (costamagnagianfranco)]

<BTS> Opened #1006784 in src:automake-1.16 1:1.16.5-1.1 by Gianfranco Costamagna (locutusofborg) «automake: fixes for python3.10 distutils changes». https://bugs.debian.org/1006784

Please have a look at it if possible.

[Stefano Rivera (stefanor)]

python3.10 was fixed in 3.10.2-4, which is in the release pocket.

[Stefano Rivera (stefanor)]

pip was fixed by fixing python3.10 and dh-python.

[Anders Kaseorg (andersk)]

This new sysconfig-debian-schemes.diff has broken virtual environments (venv and virtualenv). Please take a look at the fix in https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1962791.