sysconfig.get_path('purelib') points to site-packages directory

[Gökçen Eraslan]

purelib and platlib paths in Python 3 sysconfig module point to site-packages directory:

$ python3 -m sysconfig

Platform: "linux-x86_64"
Python version: "3.4"
Current installation scheme: "posix_prefix"

Paths:
 data = "/usr"
 include = "/usr/include/python3.4m"
 platinclude = "/usr/include/python3.4m"
 platlib = "/usr/lib/python3.4/site-packages"
 platstdlib = "/usr/lib/python3.4"
 purelib = "/usr/lib/python3.4/site-packages"
 scripts = "/usr/bin"
 stdlib = "/usr/lib/python3.4"
...

whereas Python 2 points to dist-packages:

Platform: "linux-x86_64"
Python version: "2.7"
Current installation scheme: "posix_local"

Paths:
 data = "/usr/local"
 include = "/usr/local/include/python2.7"
 platinclude = "/usr/local/include/python2.7"
 platlib = "/usr/local/lib/python2.7/dist-packages"
 platstdlib = "/usr/lib/python2.7"
 purelib = "/usr/local/lib/python2.7/dist-packages"
 scripts = "/usr/local/bin"
 stdlib = "/usr/lib/python2.7"
...

One severe consequence of this is that automake is installing Python modules to a location which is not in sys.path, namely site-packages directory.

When one uses automake to create a Makefile that installs a Python module to the system, it is installed into /usr/local/lib/python3.4/site-packages since automake/aclocal uses sysconfig to determine where the package will be installed.

From /usr/share/aclocal-1.14/python.m4 file:

  AC_CACHE_CHECK([for $am_display_PYTHON script directory],
    [am_cv_python_pythondir],
    [if test "x$prefix" = xNONE
     then
       am_py_prefix=$ac_default_prefix
     else
       am_py_prefix=$prefix
     fi
     am_cv_python_pythondir=`$PYTHON -c "
$am_python_setup_sysconfig
if can_use_sysconfig:
    sitedir = sysconfig.get_path('purelib', vars={'base':'$am_py_prefix'}) <-----------
else:
    from distutils import sysconfig
...

Here, it can be seen that automake uses purelib path defined by the sysconfig module of Python. However, since this path is not included in sys.path, and thus the installed package cannot be used:

$ python3 -c "import sys; print(sys.path)"
['', '/usr/lib/python3.4', '/usr/lib/python3.4/plat-x86_64-linux-gnu', '/usr/lib/python3.4/lib-dynload', '/usr/local/lib/python3.4/dist-packages', '/usr/lib/python3/dist-packages']

Ubuntu version:14.10

$ apt-cache policy python3-minimal
python3-minimal:
  Installed: 3.4.2-1
  Candidate: 3.4.2-1
  Version table:
 *** 3.4.2-1 0
        500 http://mirror.23media.de/ubuntu/ utopic/main amd64 Packages
        100 /var/lib/dpkg/status

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[astro (bernard-godard)]

This bug also affects python 3.5 in Ubuntu 16.04.

Note that automake scripts use also platlib in addition to purelib. platlib is also wrongly pointing to site-packages.

[astro (bernard-godard)]

Additionally if for some reason the sysconfig module cannot be used, automake scripts will fallback to using distutils.sysconfig like this:

distutils.sysconfig.get_python_lib(1,0,prefix=$prefix))

The result of this is (Ubuntu 16.04):
 /usr/local/lib/python3/dist-packages

This is correctly set to dist-packages. However, sys.path contains /usr/local/lib/python3.5/dist-packages not /usr/local/lib/python3/dist-packages
Hence the modules are not installed in the correct path unless a symlink python3->python3.5 is manually made.

[Yury V. Zaytsev (zyv)]

Hi Barry,

Just trying to bring this to your attention...

Thanks!

[Marian (djzielony)]

Please fix this, I can't normally package and install packages I develop because of this.

[Dmitry Eremin-Solenikov (lumag)]

Seconding this bug.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jack Hale (mttl)]

Perhaps it gets lost in this thread I wanted to confirm that the problem highlighted by @bernard-godard comment https://bugs.launchpad.net/ubuntu/+source/python3.5/+bug/1408092/comments/3 is also causing serious issues when using distutils to install into the prefix /usr/local.

[Stephen Sinclair (radarsat1)]

Hello, yes this last issue (distutils.sysconfig) is causing a problem on a project I work on, so I'd like to confirm as well. It appears to have been previously brought up here without reply: https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2016-October/017005.html

I am seeing the issue on Ubuntu 17.04 so it seems this bug has not been fixed.

[Stephen Sinclair (radarsat1)]

In fact looking at /usr/lib/python3.4/distutils/sysconfig.py, it appears the fix is quite easy. In get_python_inc(), the variable `python_dir = python_dir = 'python' + get_python_version() + build_flags` is used to build the path. However, in get_python_lib(), "python3" is a hard-coded string: `return os.path.join(prefix, "lib", "python3", "dist-packages")`

This is a special case for the value of prefix if it's something like /usr/local.. Currently to get around this I am forced to provide a fake argument to `prefix` and then doing a string replace for /usr/local, which is just silly.

[Jeff Larson (jmlarson)]

Is there any motion towards resolving this issue?

[Jeff Larson (jmlarson)]

I just checked and this issue persists in python3.6.2.

[linas (linasvepstas)]

my downstream package suffers from this: https://github.com/opencog/link-grammar/issues/530

[Kevron Rees (tripzero)]

The issue is that /usr/lib/python3/sysconfig.py is hardcoded to 'site-packages' (see _INSTALL_SCHEMES variable in the code). If your code uses sysconfig.get_path() to get the python3 site dir, it will fail on ubuntu.

So either the fix needs to change /usr/lib/python3/sysconfig.py or packages need to be fixed to use distutils

[Gabriel Rauter (raetiacorvus)]

Mesons python module uses the purelib/platlib path when using the function install_sources() to install a python module.

https://mesonbuild.com/Python-module.html

It would be helpful to know if ubuntu is going to fix this or not so corresponding bug reports can be opened in projects depending on this.