Do not add namespace packages to depends

[Gediminas Paulauskas]

Hello,

I would like to change the behavior of van.pydeb so that it does not add namespace packages (like python-zope.app) to Depends. The code that does this has a comment

# this is a pretty ugly way to get __init__.py into the namespace packages
# which seems to be necessary.
# though testing it out on ubuntu gutsy said it wasnt, it was on Debian etch
#
# Perhaps we could remove this a bit later

This behaviour requires more work and unnecessary packages. I am trying to push more packages to Ubuntu, and now I need python-zope.app (LP: #563211), python-z3c, and python-hurry in addition to python-zope and python-zc that already exist. I would like to avoid them. Their names are too general and confusing.

[Gediminas Paulauskas (menesis)]

Here is a proposed change. Don't know if such behaviour change needs a minor version (1.4.0).

I can commit to Zope svn myself if you think this is good. I can't release this to PyPI or Debian, though.

[Brian Sutherland (jinty)]

That comment is pretty old :) The use of virtual namespace packages in Debian turned it into a reasonably elegant solution.

For the patch to be committed, we'll need more proof that not having the __init__.py for a namespace package works properly and doesn't make us vulnerable to setuptools/distribute vagaries and weird setups. Honestly, I'd rather play it safe and depend on the good-old-fashioned python import with the __init__.py. It is surprising that importing without the __init__.py worked at all.

You made a comment about requiring extra packages. I presume you mean packages like python-zope? We resolved that in debian by using a "Virtual" package. The __init__.py file is shipped in python-zope.interface which "Provides:" python-zope.

see: http://packages.debian.org/sid/python-zope

You can use diversions (dpkg-divert) in case you need multiple packages to provide the virtual namespace package.

[Matthias Klose (doko)]

On 17.08.2010 18:59, Brian Sutherland wrote:
> You can use diversions (dpkg-divert) in case you need multiple packages
> to provide the virtual namespace package.

only if you have exactly one package which diverts the file. doesn't work for
more than one.

[Gediminas Paulauskas (menesis)]

When the builder installs the package (setup.py install --single-version-externally-managed, and more specifically, setuptools' install_lib command), __init__.py files are skipped, because they would conflict if installed by multiple packages . Instead, setuptools' install_egg_info command installs a *-nspkg.pth file that ensures that the package exists -- if an __init__.py file exists, it is used, but if not, a module is added to sys.modules, so imports work anyway. Since all zope projects use setuptools, they don't need the __init__.py files.

I can add a virtual python-zope.app to some random package, because I can't think of the most important zope.app.* library like zope.interface is. But I would like to avoid depending on python-zope.app at all, that's why I filed this bug.

[Brian Sutherland (jinty)]

> When the builder installs the package (setup.py install --single-
> version-externally-managed, and more specifically, setuptools'
> install_lib command), __init__.py files are skipped, because they would
> conflict if installed by multiple packages . Instead, setuptools'
> install_egg_info command installs a *-nspkg.pth file that ensures that
> the package exists -- if an __init__.py file exists, it is used, but if
> not, a module is added to sys.modules, so imports work anyway. Since all
> zope projects use setuptools, they don't need the __init__.py files.

Thanks, that info helped me find this:
    http://mail.python.org/pipermail/distutils-sig/2009-May/011730.html

I made a test (attached), it fails without the __init__.py in the
namespace package for me like this:

    jinty@ec2-staging:~/test_ns$ python test.py
    jinty@ec2-staging:~/test_ns$ sudo mv /usr/lib/python2.5/site-packages/zope/__init__.py* tmp/
    jinty@ec2-staging:~/test_ns$ python2.5 test.py
    Traceback (most recent call last):
      File "test.py", line 1, in <module>
        import zope.testpackage
    ImportError: No module named testpackage
    jinty@ec2-staging:~/test_ns$ sudo mv tmp/__init__.py* /usr/lib/python2.5/site-packages/zope/

This test failed Debian Lenny and Squeeze machines for python 2.5 and
2.6. Could you check that the same occurs in Ubuntu? You need to extract
the tarball in your home directory and have python-zope.interface
and python-zope.event installed.

P.S. The test is possibly destructive to the machine it runs on, so
use one you don't care about.

Unless we can get the above to work, I think we should not commit your
patch.

> I can add a virtual python-zope.app to some random package, because I
> can't think of the most important zope.app.* library like zope.interface
> is. But I would like to avoid depending on python-zope.app at all,
> that's why I filed this bug.

zope.app.publisher seems to have no zope.app.* dependencies and is
pretty central.

Failing that, I think there's a good argument for putting the virtual
package for zope.app.* in either python-zope.interface or
python-zope.event.

[Gediminas Paulauskas (menesis)]

Turns out the __init__.py file is needed when trying to use uninstalled zope packages, or virtualenv without --no-site-packages option.

setuptools.find_packages does not recurse into packages that do not have an __init__.py file, so they are not included in results, and that may cause problems, but I do not have a test case to demonstrate that. One problem is that z3c.autoinclude does not find the target package. schooltool uses it to automatically include plugins, but I get a KeyError: 'schooltool' if schooltool/__init__.py is not installed.

A dependency on the namespace package may force the packager to create one.