tests fail with python3.4 as default version

[Martin Pitt]

When running s3ql's tests with python3.4 as default python3 version, all tests fail completely:

  http://people.canonical.com/~pitti/tmp/autopkgtest-py34/fail/py34-s3ql/results/

____________________ ERROR collecting tests/t1_backends.py _____________________
tests/t1_backends.py:11: in <module>
> from s3ql.backends import local, s3, gs, s3c, swift, rackspace
E ImportError: No module named s3ql.backends
______________________ ERROR collecting tests/t1_dump.py _______________________
tests/t1_dump.py:11: in <module>
> import unittest2 as unittest
E ImportError: No module named unittest2
__________________ ERROR collecting tests/t1_ordered_dict.py ___________________
tests/t1_ordered_dict.py:11: in <module>
> import unittest2 as unittest
E ImportError: No module named unittest2
___________________ ERROR collecting tests/t2_block_cache.py ___________________
tests/t2_block_cache.py:12: in <module>
> from s3ql.backends import local
E ImportError: No module named s3ql.backends

and similar. There are a few such failures when merely making python3.4 a supported version (but not the default), as in current trusty-proposed:

  https://jenkins.qa.ubuntu.com/job/trusty-adt-s3ql/36/ARCH=i386,label=adt/

but it is not nearly as bad.

[Nikolaus Rath (nikratio)]

The log on https://jenkins.qa.ubuntu.com/job/trusty-adt-s3ql/36/ARCH=i386,label=adt/console says that the tests were run under Python 3.3.

Did you built with python3.4, but then ran the self tests with python 3.3? This mustfail, because then the C extension (deltadump.so) is not available for python 3.3.

[Nikolaus Rath (nikratio)]

I found a problem that causes the unit tests after (and including) t4_adm.py to fail. The problem is that when forking subprocesses, the subprocesses try to load the modules from the source package rather than the installed package.

This is, however, unrelated to Python 3.4 being installed or not. Do you have an example of a test run from trusty where these tests actually pass?

[Nikolaus Rath (nikratio)]

The subprocess related bug is fixed in https://bitbucket.org/nikratio/s3ql/commits/8772e4b7548d09dbf77bf8b126bd084a1f338a36

Is there a way to reproduce the problems with python3.4 as default python? I'm running a chroot with trusty-proposed, but it still seems to use python3.3 as default.

[Matthias Klose (doko)]

> Is there a way to reproduce the problems with python3.4 as default python?

please use the packages from the ubuntu-toolchain-r/python3 PPA.

[Nikolaus Rath (nikratio)]

I've added the ppa and upgraded the system. Now all the s3ql unit tests fail because none of the dependencies are available for Python 3.4 (e.g. no python-apsw), not really surprising and (as far as I understand) not a bug in S3QL. I'd like to debug this problem, but I'll need some instruction to replicate whatever environment you used to run the tests.

[Matthias Klose (doko)]

afaics, python3-apsw was the only one missing. now uploaded

[Nikolaus Rath (nikratio)]

After another update && dist-upgrade, the testcases now fail because s3ql itself has not been rebuilt with the new default python (as a Python app rather than Python module, the package is build only with the default interpreter).

 Is there some missing metadata that ought to have triggered a rebuilt?

[Matthias Klose (doko)]

now built in the ubuntu-toolchain-r/python3 PPA.

> Is there some missing metadata that ought to have triggered a rebuilt?
no. the change of the default version is not done in the main archive, and we'll have to prepare packages which will require a rebuild with 3.4 as the default in this ppa. planning to do this only on request.

[Nikolaus Rath (nikratio)]

In other words, this bug will resolve itself automatically, and there is nothing that needs to be fixed in the s3ql package?

[Martin Pitt (pitti)]

Ah I'm sorry, unlike in the other bugs I forgot to link to the python3.4 test logs in that one. I now updated the description to include it. So rebuilding s3ql to also include a 3.4 extension will mostly fix this bug. But the original failures that you also see with 3.3 will most likely still happen. If the upstream commit you linked to fixes those, then indeed all is well.

Note that Matthias' no-change rebuild only changed the default module to python 3.4 (https://launchpadlibrarian.net/163787523/buildlog_ubuntu-trusty-amd64.s3ql_2.7-1build1_UPLOADING.txt.gz), but the package actually needs to be fixed to build extensions for *all* supported python3 versions.

[Martin Pitt (pitti)]

FTR, I just ran the autopkgtest again with the PPA enabled, and now s3ql is uninstallable:

  The following packages have unmet dependencies:
     s3ql : Depends: python3 (>= 3.4) but 3.4~b2-0ubuntu1 is to be installed

This seems to be a bug in dh_python3: It shouldn't autogenerate a dependency like "python3 (>= 3.4)" as that will break with pre-release interpreters. At least it should autogenerate it as (>= 3.4~), or put in the version of python3 that you built it with.

[Nikolaus Rath (nikratio)]

Martin, why should the package build extensions for all supported python3 versions? It is an application, not a Python module. So isn't it sufficient if it works with the default python version? At least in Debian, this seems to be the officially blessed way to package Python apps (see eg. https://wiki.debian.org/Python/Packaging, Example 2).

[Martin Pitt (pitti)]

Nikolaus; ah yes, sorry about that; I should have checked before (I looked at too many Python packages that day..) Of course building with the default python3 (only) is the right thing.

[Matthias Klose (doko)]

the dependency is now fixed in the ppa

[Martin Pitt (pitti)]

Thanks Matthias! Now the test failure is back to the single "ImportError: No module named 's3ql.deltadump'" and the "1 != 0" assertion failure, exactly like with 3.3 (https://jenkins.qa.ubuntu.com/job/trusty-adt-s3ql/36/ARCH=i386,label=adt/console). So with the PPA's fixed dh-python and a no-change s3ql rebuild this bug can be closed.

[Matthias Klose (doko)]

s3ql rebuilt in trusty