distutils not installed

[Stuart Bishop]

distutils is supposed to be part of core python for all current versions.
However, it has been put in the python-dev package.

distutils should be moved to the main package, or python-dev installed by
default on Ubuntu.

[Matthias Klose (doko)]

yes, python-dev should be added to the desktop.

there is no clear line, if distutils should be in -dev or in the core. Building
and installing modules is a developer task.

[Matt Zimmerman (mdz)]

It's not clear to me whether python-dev belongs in desktop. The task at hand is
building and/or installing third-party Python modules, correct? That seems
clearly a development task, and we've set a precedent of omitting development
tools in the desktop install.

[daishi (daishi)]

(In reply to comment #2)
> It's not clear to me whether python-dev belongs in desktop. The task at hand is
> building and/or installing third-party Python modules, correct? That seems
> clearly a development task, and we've set a precedent of omitting development
> tools in the desktop install.

As long as it's made clear what one needs to install
to have what the upstream would consider the standard
installation, this would be okay. What would be useful
to avoid is users going to comp.lang.python and saying
"I installed python and the docs on python.org say that
... blah blah ... distutils ...", and have "installed
python" mean something rather unexpected.

[Stuart Bishop (stub)]

I suppose distutils should not be included unless gcc is also included, as
without gcc it will fail if it tries to install a package with C extensions that
need building.

I don't see any reason for gcc to not be incuded unless it is the footprint, but
I'm sure this has been discussed to death already :-)

[anthony baxter (anthony)]

Please please please - lose the python/python-dev package, or at
least move distutils into the python package.

I'm fairly involved in a whole pile of python mailing lists and
python projects, and debian's stupid decision to put distutils
into it's own package is a constant source of annoyance to users,
and to the people who get the same. question. over. and. over.

Please, please - don't split the stdlib across two packages. It's
a pain in the arse.

[anthony baxter (anthony)]

Ugh. A point I missed, the second time around (bugzilla ate the first time).

distutils is used to _install_ python packages, not just write them or
package them. Unless your distro contains absolutely every package that
anyone's ever going to want to use, and contains the latest versions
straight after they're released, people are going to want to install
their own versions.

[Matt Zimmerman (mdz)]

The right way to propose a packaging change is to start a useful discussion on
the ubuntu-devel mailing list. Calling it "stupid" in Bugzilla (while it may be
cathartic) does not promote progress in this area.

[Mark Shuttleworth (sabdfl)]

Oops, I forgot about the use of distutils to install new libraries. Perhaps that
changes things? We'll have to break the gcc dependency in some way, or give in
on the "users don't need a compiler" argument.

[Matt Zimmerman (mdz)]

python-dev doesn't depend on gcc

[Stuart Bishop (stub)]

Installing a Python package with C extensions and no compiler will fail with a
'cc: command not found' error. As this is what happens if you type 'make' and
gcc or some library isn't installed it is probably acceptible.

My take on the 'no need for CC on a desktop' is that users who don't need it
won't notice, and users who do *do so regularly and loudly*. Security policies
and such that forbid cc don't solve any problem (people can already write
programs with Bash, Python or Perl - they don't even write access to the
filesystem - or just build on any other Linux box and copy the binary over. This
policy might have worked in the 80's but it is a bit silly now. I'd rather force
the miniscule minority of users who actively don't want it to remove gcc, rather
than annoy the much larger group who need it.)

[Matt Zimmerman (mdz)]

(In reply to comment #10)
> My take on the 'no need for CC on a desktop' is that users who don't need it
> won't notice, and users who do *do so regularly and loudly*. Security policies
> and such that forbid cc don't solve any problem (people can already write
> programs with Bash, Python or Perl - they don't even write access to the
> filesystem - or just build on any other Linux box and copy the binary over. This
> policy might have worked in the 80's but it is a bit silly now. I'd rather force
> the miniscule minority of users who actively don't want it to remove gcc, rather
> than annoy the much larger group who need it.)

The compiler-in-desktop discussion happened over a month ago (sounder list,
2004-09-03), and Mark made a final call on it, so the matter has already been
decided.
Rest assured, though, that the decision wasn't made on the basis of questionable
ideas about security, but rather a fluffy desktop philosophy argument. :-)

[Mark Shuttleworth (sabdfl)]

Can we have cc on Warty-without-gcc give an error message that explains that the
user may want apt-get install build-essential? We discussed previously having a
mapping from command-to-package, and this may be a very good place to start.

As for distutils, if the user is installing a package that needs libraries, it
will come with packages for those libraries. So distutils is not required. I
don't see why we should encourage people to use distutils to install software
which apt et al won't know anything about. THey might tread on it in horrible
and unpredictable ways. Python libraries should be packaged and installed via dpkg.

That said, if python-dev is inappropriate, i've no problem for there to be a
python-distutils for people who want that without the rest of -dev. Matthias, if
you think that's reasonable, go ahead for Hoary.

[Stuart Bishop (stub)]

(In reply to comment #12)

> That said, if python-dev is inappropriate, i've no problem for there to be a
> python-distutils for people who want that without the rest of -dev. Matthias, if
> you think that's reasonable, go ahead for Hoary.

NOOOO! The problem people have is that things are already split up too
granularly. The case we see is someone is happily tapping away using Python and
then some horrible error pops up they wern't expecting and they then either give
up or post an FAQ to one of the discussion lists. Which then annoys the
community who spent a lot of time assembling the standard library and
documenting it only to have it arbitarily sliced up and their time wasted
answering problems they didn't create for the sake of a tiny bit of disk space.

[Mark Shuttleworth (sabdfl)]

Does distutils do anything other than install software? We have a perfectly good
way to install software: dpkg. If we encouraged people to install software on
Ubuntu using distutils, we would never be able to use our existing tools to
figure out what software was installed. I think it's great that the Python
community has a tool to do this in a platform-independent way for someone who
has INSTALLED PYTHON FROM SCRATCH. But Ubuntu is going to great lengths to put a
lot of Python support into the distro out of the box. And part of that builds on
dpkg, which gives us excellent dependency tracking, handles conflicts and
versions, etc.

So rather than putting distutils on the desktop, I would rather invest a little
energy in making it easy for developers to package up Python libraries for Ubuntu.

Matthias, as soon as Warty is out, could you start work on a guide for packaging
Python libraries in Ubuntu? We can include that and link to it heavily. We can
even make "import distutils" refer you to that document by default, unless
distutils has been deliberately installed.

[anthony baxter (anthony)]

Mark,

It's all well and good to say that everyone should only install packages via
apt-get, but unfortunately, unless you're going to make the effort to package up
every single piece of python software ever, and do it pretty much straight after
each new version, that's not going to happen.

And it's not realistic to expect authors of packages to make .deb files (and
.rpms, and solaris .pkgs, and ...) for their software - right now, I can get
away with a single .tar.gz (well, ok, and occasionally a windows installer) to
release a piece of software.

The inherited-from-debian distutils-in-python-dev is a real nuisance for those
of us cutting and releasing python software. If Ubuntu is successful (and I hope
it is), it would be massively helpful to those of us cutting and releasing
open-source software in Python if this was fixed.

[Stuart Bishop (stub)]

(In reply to comment #14)

> Does distutils do anything other than install software? We have a perfectly good
> way to install software: dpkg. If we encouraged people to install software on
> Ubuntu using distutils, we would never be able to use our existing tools to
> figure out what software was installed. I think it's great that the Python
> community has a tool to do this in a platform-independent way for someone who
> has INSTALLED PYTHON FROM SCRATCH. But Ubuntu is going to great lengths to put a
> lot of Python support into the distro out of the box. And part of that builds on
> dpkg, which gives us excellent dependency tracking, handles conflicts and
> versions, etc.

distutils is not targeted at people who have installed python from scratch - it
is targeted at every single python installation no matter if it was installed
via RPM, deb packages, windows installer or copied off your brothers computer.
It is the tool used to install Python packages, build distributions of Python
packages and register Python packages with the Python Catalog. In the future it
will be the method used to download and build modules automatically when someone
tries to import something that doesn't exist.

Also, as part of the standard library, there is nothing stopping people
importing parts of it and using it. The following example is perfectly correct
and quite useful, although will fail to run on Ubuntu default:

>>> from distutils.util import get_platform
>>> get_platform()
'linux-i686'

> So rather than putting distutils on the desktop, I would rather invest a little
> energy in making it easy for developers to package up Python libraries for Ubuntu.

The reason Python developers tend to only release as tarballs is that they then
only have to document and test one installation method -- you are not going to
have much success getting developers using Windows, Fedora, Gentoo or whatnot to
sit down and work out dependancies and release a .deb of their product that they
have no way of testing.
I also don't think you can use a .deb to install a private copy of a libary into
your own account space, which is a requirement for webhosting since ISP's
generally don't like to give root access to their boxes and people on multiuser
hosts may need different versions of the same library. In short, despite its
shortcommings distutils addresses a lot of the needs of the Python community and
these needs are not addressed.

It would be quite possible to enhance distutils so that packages installed by it
are automatically added to the Ubuntu package registration database.

If it could be done in a platform independant away, the Python community would
also welcome enhancements or a replacement to distutils that handles
dependancies, a central repository, mirroring and other advanced features -- the
Catalog-SIG is pretty stagnant due to lack of man hours but most people agree it
is needed (athough discussion on this should move out of Buzilla if people want
to pursue it).

I find it quite ironic that Perl's equivalent of distutils (CPAN.pm) is
installed by default and Python's distutils is not.

[Mark Shuttleworth (sabdfl)]

Providing up-to-date Deb's of the latest Python modules is exactly what we
intend to do. You can see the beginnings of that by doing a default Desktop
installation of Ubuntu and checking out the installed packages list, there are
many, many Python packages.

Python is a major feature goal for me with Ubuntu. I want it to be recognised as
the best platform for people coding in Python. So yes, I'm willing to invest the
resources to do this properly. I *don't* believe that means shipping distutils
as it stands. If there's a need for us to provide the lead then I'm perfectly
willing to fund some work to do that.

  - automatic discovery of needed Python libraries
  - infrastructure to support packaging the very latest and best Python modules
  - community building to get people focused on Python as a universal glue
language in Ubuntu

If distutils can be integrated with apt so that a call to distutils looks in the
APT repositories first, and uses existing packages, and failing that puts the
needed libs in a defined place where they will not interfere with formal
packages, that would be fine. Anthony, would you be interested in doing that work?

[anthony baxter (anthony)]

The short answer is 'no'. The longer answer is that the time I spend on
distutils is spent in extending or improving distutils in a way that works on
all platforms. I've not got the time to waste it on work for a single platform,
or worse, a single distro.

I don't have enough time to do the stuff I _have_ to get done, and I certainly
don't have the time to start making packaages up for every random distro. The
point of this bug thread was to try and ask you guys to do the right thing and
make my (and most other people who release python software) easier.

I'm going to shut up after this piece - I will just, once again, ask that
regardless of how you go forward in the future, you _please_ fix this problem in
the simple way of moving distutils to the python package. Right now, there is no
solution available for what you plan, nor is there likely to be, anytime soon.
And in the meantime, I will continue to get email messages from people who want
to install a piece of software, and get "ImportError: no such module
distutils.core". I look forward to seeing CPAN removed from the standard Perl
package and moved into perl-dev, if you genuinely don't want people installing
software that's not been packaged.

[Andrew Bennetts (spiv)]

Some statistics:

The "Python Packages Index" on python.org lists 647 python packages:
    http://www.python.org/pypi?:action=index

We have roughly 182 python packages according to this quick grepping:
    apt-cache search --names-only python | egrep -v python2\\.. | grep -v --
"-doc " | grep -v -- "-dev " | wc -l

We have a lot of work to do to package everything, even accounting for
things like Windows-only modules, and multiple versions of some packages
in PyPI (and occasionally you need a specific version anyway).

The other point that springs to mind is that Python makes a big deal of
it's "Batteries Included" philosophy. By arbitrarily cutting pieces out
of the standard library, we're going against upstream's packaging
philosophy, and also against the expectations of python users from almost
every other system. The official docs on docs.python.org document
distutils as part of the standard library -- because they are -- and I'm
sure that books such as _Python Essential Reference_ and _Python in a
Nutshell_ do too. Even if we fix our python-doc package to not describe
distutils as part of the standard library, we're still going to surprise
Python users from other platforms, and even distributions, or just new
users that are relying on the plethora of high-quality third-party python
documentation in books and on the net.

In my view, the simplest and best course of action is to leave distutils
in the standard library (it's only 668k installed according to a quick ls,
and that includes all the .pyc and .pyo files generated at install time).

Also note that distutils will not trample over packaged software by default.
It defaults to installing to /usr/local/lib/pythonX.Y/site-packages. If we
want to be really conservative about this, we can even make sure that this
path comes right at the end of the default python sys.path. The danger of
users breaking their system with non-packaged python software is very low.

[Christian Reis (kiko)]

My take on the issue here is:

  a) Distutils should probably go into the main python package; I have a strong
suspicion it won't hurt us at all.

  b) Disutils, if we're definitely not installing it on a matter of policy,
could be a stub module that raises a "Get python-dev for distutils"-sort-of-error.

  c) Generating .debs from distutils would be a nice way to reduce the cost of
packaging Python modules for Ubuntu. A bdist_deb `target' for setup.py would do
us a whole lot of good -- `they' already have bdist_rpm, remember.

Not all of this is spot-on-topic here, but it's at least a way to move forward
with the issues pointed out here.

[Matt Zimmerman (mdz)]

I support moving distutils into python2.3-dev for Hoary.

[Andrew Bennetts (spiv)]

(In reply to comment #21)
> I support moving distutils into python2.3-dev for Hoary.

I'm confused. distutils is already in python2.3-dev. Did you
mean "I support moving distutils into *python2.3* for Hoary"?

[Matt Zimmerman (mdz)]

Heh, yes, of course I meant python2.3.

[James Blackwell (jblack)]

As requested by SteveA, I'm adding a comment that I got bit by the lack of
python-dev today.

Its not that python-dev is missing is the problem. Its that there's no clear
indication that that's what's wrong.

[Mark Shuttleworth (sabdfl)]

After further discussion it seems the right way for us is:

  - to include distutils in a desktop install
  - to customise it to look for debs and offer those in preference
    to installing a library in /usr/local if an appropriate deb exists

This last bit is going to be tricky, since there are complex version
issues to be dealt with, the distutils will need a db of known libs
and their mappings to preferred debs.

[Matt Zimmerman (mdz)]

We should definitely include distutils in the core python package, rather than
keeping it in python-dev. python-dev can continue to hold the bits needed for
compiling python extensions written in C.

As I said on IRC, though, I don't think that distutils should interfere when the
user asks it to do something. It might make sense, though, for distutils to be
taught how to create Debian source packages as one of its targets, and if we
integrate them, I think that is the form that it should take.

distutils is basically python's version of make(1). It takes a (very
customizable, and often complex) specification as input, and executes a bunch of
commands which handle building and installing some program code. Generally,
that tends to result in some C source code being compiled, or some python
modules being copied around, but really it can mean anything. It makes no more
sense for distutils to go looking for .debs than it does for autoconf or make.
Instead, distutils is a framework that provides an interface between Debian
packages and Python developers, and we can enhance it in that direction.

[Matt Zimmerman (mdz)]

Now that Warty is finished, please upload this change to Hoary soon.

[Matthias Klose (doko)]

python2.3_2.3.4-13ubuntu1 has been uploaded containing this change.
there are pros and cons for the inclusion. the "stupid" decision
comes from the time when distutils was distributed in it's own source
package.

- completeness of the standard library: If the package doesn't
  depend on other packages not included in the base/desktop section,
  then include it in the python package.

- don't depend on/use external packages without having an explicit
  dependency. therefore we have the -tk, -gdbm, and -mpz packages.
  It doesn't make sense to force the installation of all libraries
  (X11) by default. Even the Windows installer makes this choice.
  Introducing here an implicit dependency on the C compiler is the
  counter argument. Although depending on the compiler alone doesn't
  help much as the apropriate -dev package for the extension module
  is needed for most cases.

- I still consider packages depending on distutils on runtime buggy.
  Mostly things from distutils.sysconfig are used. If these things
  are useful for _running_ packages, they maybe should go to sys.

This change enables the change of all pure python modules, but doesn't
change anything for python extensions. Maybe we can add python-dev to
build-essential for Ubuntu to address extensions which don't depend
on external headers and libraries, but you're still on your own
installing the -dev packages for the "real" extension modules.

[Matt Zimmerman (mdz)]

(In reply to comment #28)
> This change enables the change of all pure python modules, but doesn't
> change anything for python extensions. Maybe we can add python-dev to
> build-essential for Ubuntu to address extensions which don't depend
> on external headers and libraries, but you're still on your own
> installing the -dev packages for the "real" extension modules.

This is fine as-is. The user already must install additional packages in order
to install extension modules, because they must be compiled.

The original issue (distutils) has been addressed, and I think this bug can be
closed. If there is anything further, it can be discussed elsewhere.