Python distutils installs into 'site-packages' instead of 'dist-packages' when a prefix is set

Bug #362570 reported by Oben Sonne on 2009-04-16
72
This bug affects 10 people
Affects Status Importance Assigned to Milestone
python2.6 (Debian)
Fix Released
Unknown
python2.6 (Ubuntu)
Wishlist
Unassigned
python3-defaults (Ubuntu)
Undecided
Unassigned

Bug Description

Binary package hint: python2.6

When running a setup.py script with distutils in Jaunty like this:

$ python setup.py install

modules get installed at '/usr/local/lib/python2.6/dist-packages'.

In contrast, when running

$ python setup.py install --prefix=/usr/local

modules get installed at '/usr/local/lib/python2.6/site-packages'.

In the same way

$ python setup.py install --prefix=/usr

installs modules at '/usr/lib/python2.6/site-packages'.

The problem is that '/usr/[local/]lib/python2.6/site-packages' is not part of the default sys.path .

Maybe this is intended but I would expect that specifying an install prefix like 'usr' or 'usr/local' still installs modules to a place which is in Python's default search path - it worked this way in Ubuntu <= 8.10 .

ProblemType: Bug
Architecture: i386
DistroRelease: Ubuntu 9.04
Package: python2.6 2.6.2~rc1-0ubuntu2
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: python2.6
Uname: Linux 2.6.28-11-generic i686

Oben Sonne (obensonne) wrote :
Matthias Klose (doko) wrote :

yes, this works as designed. --prefix always overrides the default that Debian's system python does set. If you want to install into /usr/local/lib/python2.6/dist-packages, just omit the --prefix.

Changed in python2.6 (Ubuntu):
status: New → Confirmed
Matthias Klose (doko) wrote :

> $ python setup.py install --prefix=/usr
>
> installs modules at '/usr/lib/python2.6/site-packages'.

you should not install into /usr/lib/python2.6 yourself. this has the potential to override system compnents.

see https://lists.ubuntu.com/archives/ubuntu-devel/2009-February/027439.html

Oben Sonne (obensonne) wrote :

> you should not install into /usr/lib/python2.6 yourself. this has the potential to override system compnents.

Indeed, to avoid this I was used to set the prefix to /usr/local and I recommended this also when giving instructions for installing a module - but this recommendation fails now on Jaunty, as /usr/local/lib/python2.6/site-packages is not in the sys.path anymore.

> see https://lists.ubuntu.com/archives/ubuntu-devel/2009-February/027439.html

I see the renaming to dist-packages is pretty useful. To be sure here: /usr/local/lib/python2.6/site-packages is reserved for custom Python installations and will not be part of the default sys.path anymore?

Matthias Klose (doko) wrote :

> To be sure here: /usr/local/lib/python2.6/site-packages is reserved for custom
> Python installations and will not be part of the default sys.path anymore?

yes.

Changed in python2.6 (Ubuntu):
importance: Undecided → Wishlist
Colin Watson (cjwatson) wrote :

Matthias: I think the problem here is that the logic for selecting between dist-packages and site-packages is not consistent. Ignoring --install-layout for a moment, distutils.sysconfig says, in effect:

  is_default_prefix = not prefix or os.path.normpath(prefix) in ('/usr', '/usr/local')
  if is_default_prefix and 'PYTHONUSERBASE' not in os.environ and 'real_prefix' not in sys.__dict__:
      return os.path.join(libpython, "dist-packages")

... while distutils.command.install says, in effect:

  if self.prefix_option or 'real_prefix' in sys.__dict__:
      self.select_scheme("unix_prefix")
  else:
      self.select_scheme("unix_local")

I think that we should be consistent here, and apply the same "is the prefix /usr or /usr/local?" check in distutils.command.install (and perhaps check PYTHONUSERBASE too, though that's a side issue) - if it is, we should use the unix_local scheme and thus install into dist-packages rather than site-packages.

This bug seems more important than wishlist - it breaks third-party build systems that use --prefix=/usr/local in the expectation that it will install things somewhere sensible where the version of python you're using will be able to get at them. (Such build systems can simply remove --prefix=/usr/local as a workaround, but with older packaged versions of python in Debian/Ubuntu that will install into /usr which is obviously undesirable.)

Matthias Klose (doko) wrote :

subscribing barry ...

yes, there is some kind of conflict.

we have to distinguish between a (manual) python installation going to /usr/local by default and using /usr/local/lib/python2.x/site-packages and the system python using /usr/local/lib/python2.x/dist-packages. IMO it should still be possible to install into the former using the system python. how would you do this? otoh we could just ignore this case and require the locally installed python for installations into /usr/local/lib/python2.x/site-packages. in any case, site-packages should be used for all occasions where prefix is not /usr or /usr/local or when stuff like virtualenv or a userbase installation is wanted.

Colin Watson (cjwatson) wrote :

Installing into site-packages using the system python seems fraught with difficulties anyway; what if extensions are involved and the system and manual python installations are extension-ABI-incompatible for some reason? Requiring use of the locally-installed python for this case seems reasonably justifiable to me, unless I'm missing something.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python2.6 - 2.6.2-0ubuntu2

---------------
python2.6 (2.6.2-0ubuntu2) karmic; urgency=low

  * Update to 20090619, taken from the 2.6 release branch.
  * distutils: Always install into `/usr/local/lib/python2.6/dist-packages'
    if an option `--prefix=/usr/local' is present (except for virtualenv
    and PYTHONUSERBASE installations). LP: #362570.

 -- Matthias Klose <email address hidden> Fri, 19 Jun 2009 14:40:09 +0200

Changed in python2.6 (Ubuntu):
status: Confirmed → Fix Released
EdC (edc-cce) wrote :

For me, installing with --prefix=/usr/local still installs into a directory (/usr/local/lib/python2.6/site-packages/). That choice is OK with me, but the problem is that this directory is not in the default search path and so the original bug has not been resolved and this breaks installations of third-party packages that used to work in 8.10.

Here's an example showing that installing a package (rdiff-backup in this case) with the --prefix=/usr/local puts it in a location that is not in the python search path:

/usr/local/src/rdiff-backup-1.2.5 (450) $ dpkg -l python
...
ii python 2.6.2-0ubuntu1 An interactive high-level object-oriented language (default version)

/usr/local/src/rdiff-backup-1.2.5 (453) $ sudo python setup.py install --prefix=/usr/local --record files-installed
running install
...
writing list of installed files to 'files-installed'

/usr/local/src/rdiff-backup-1.2.5 (454) $ grep Main.pyc files-installed
/usr/local/lib/python2.6/site-packages/rdiff_backup/Main.pyc

/usr/local/src/rdiff-backup-1.2.5 (457) $ rdiff-backup --version
Traceback (most recent call last):
  File "/usr/local/bin/rdiff-backup", line 20, in <module>
    import rdiff_backup.Main
ImportError: No module named rdiff_backup.Main

/usr/local/src/rdiff-backup-1.2.5 (462) $ sudo bash -c 'echo "/usr/local/lib/python2.6/site-packages/" >/usr/lib/python2.6/dist-packages/local.pth'

/usr/local/src/rdiff-backup-1.2.5 (464) $ rdiff-backup --version
rdiff-backup 1.2.5

So I think /usr/local/lib/python2.6/site-packages/ needs to be added to the default search path.

Peter Antoniac (pan1nx) wrote :

I didn't check it myself, but the fix is released for python2.6 (2.6.2-0ubuntu2) karmic; urgency=low. You might want to try that out. I see that you have 2.6.2-0ubuntu1...

EdC (edc-cce) wrote :

Thanks for the explanation for why the fix doesn't work for me :). I can't seem to (easily) install the karmic package so I'll have to wait until I upgrade to test the fix.

René Stadler (rene.stadler) wrote :

Is the fix coming to jaunty? What do I tell to users that want to install my modules?

thorwil (t-w-) wrote :

I have the problem described in #10 now with Maverick.

That means I can't offer my software as tarball with just a setup.py, unless I tell people to add export PYTHONPATH=/usr/lib/python2.6/site-packages to their .bashrc.

thorwil (t-w-) wrote :

Eek, forget what I said about PYTHONPATH, that doesn't work even with the right path. Sorry.

Robert Bruce Park (robru) wrote :

I am experiencing exactly what is described in comment #10 on Precise. `setup.py install` is putting things into site-packages directory, and then when I try to run the program it raises ImportError because sys.path does not contain any variation of 'site-packages' whatsoever.

The workaround I've created is ugly beyond belief: https://github.com/robru/gottengeography/commit/54f52fa0a3f97e22fb61e70bc0b5ec77c1a19015

The code all worked fine on Fedora without the above patch, so something is pretty broken with Ubuntu's distutils that it's installing packages to places where python can't even find them.

Changed in python2.6 (Debian):
status: Unknown → Won't Fix
Changed in python2.6 (Debian):
status: Won't Fix → Fix Released
tripzero (tripzero) wrote :

This also needs to be fixed for python3...

On May 02, 2016, at 08:28 PM, tripzero wrote:

>This also needs to be fixed for python3...

What exactly needs fixing? `sudo python3 setup.py install` will install into
/usr/local/.../dist-packages which is exactly where it should install, and is
definitely on the system Python's sys.path.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in python3-defaults (Ubuntu):
status: New → Confirmed
Changed in python3-defaults (Ubuntu):
status: Confirmed → New
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.