Add python 3 bindings for librpm

[Jeff Johnson]

tracker

[Dave (dave-redhat-bugs)]

I need to finish up the work done here:
http://lists.rpm.org/pipermail/rpm-maint/2009-October/002528.html

and get this built as a separate subpackage, ideally for Fedora 13.

[Panu (panu-redhat-bugs)]

Note that this is going to require a new major version of rpm to be included in Fedora 13 (that is the plan anyway so it shouldn't be an issue). The python bindings in current rpm.org HEAD require all sorts of things that are simply not available in rpm 4.7.x.

[Dave (dave-redhat-bugs)]

Created attachment 368275
Patch to rpm.spec to add a python3- subpackage

Patch to specfile:
  - adds a patch to configure.ac and Makefile.am
  - at the end of %prep, takes a copy of the "python" subdir as "python3", the same, except using PYTHON3 versions of the autotooled config macros
  - reruns autotools since Makefile.am files and configure.ac have changed
  - adds "--enable-python3" to configuration
  - adds "python3-rpm" subpackage
(missing a %changelog entry for now)

[Dave (dave-redhat-bugs)]

Created attachment 368276
Patch to add python3 configuration detection

Patch:
  - uses AM_PATH_PYTHON3 (from new automake "python3.m4" file, see bug 533920) to add a parallel configuration of a build for python 3. Note that I had to manually clear autoconf cache checks in two places here, to avoid autoconf from erroneously using incorrect cached values when include paths have changed from under it.
  - add "python3" subdirectory to the build

[Bug (bug-redhat-bugs)]

This bug appears to have been reported against 'rawhide' during the Fedora 12 development cycle.
Changing version to '12'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

[Dave (dave-redhat-bugs)]

Created attachment 395412
Updated version of patch to rpm.spec, adding an "rpm-python3" subpackage

[Dave (dave-redhat-bugs)]

Created attachment 395413
Add configuration for Python 3 to build

[Dave (dave-redhat-bugs)]

Created attachment 395414
Definition of an AM_PATH_PYTHON3 macro, for automake

[Dave (dave-redhat-bugs)]

Created attachment 395415
Add rpm-python3 subpackage (this time adding autotool build requirements)

[Dave (dave-redhat-bugs)]

Successful scratch build in Koji here:
http://koji.fedoraproject.org/koji/taskinfo?taskID=2004177

This is with the patch to rpm.spec from attachment 395415, with new source python3.m4 (attachment 395414) and new patch (attachment 395413).

The new source (python3.m4) embedded here eliminates the need for a new automake, so I'm removing bug 533920 as a blocker for this.

How does this look?

[Dave (dave-redhat-bugs)]

Ping? I'm still keen to get this into Fedora 13.

[Dave (dave-redhat-bugs)]

Doh, was assigned to me.

Reassigning to pmatilai; Panu: how does this look?

[Panu (panu-redhat-bugs)]

In one word: ugly :)

I could live with it if it was going to be just a short-lived workaround but as I suspect we (and everybody else so much as thinking of python 2 -> 3 transition) are going to need parallel python 2 and 3 stacks for a long long time, I think we need to figure out something that's sanely upstreamable.

Which probably means the python module build needs to be separated from the rest of the rpm, so that for a setup like this you can
1) configure rpm at top level tree to build without python
2) build + install rpm itself
3) do 'cd python' and then build + install with whatever python versions are wanted

[Fedora (fedora-redhat-bugs)]

This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.

[Dave (dave-redhat-bugs)]

4.10.0 is now in rawhide, and I see in http://rpm.org/wiki/Releases/4.10.0:
> Python bindings can now be alternatively built with Python distutils,
> making it simpler to build the bindings for different Python versions,
> notably 2.x vs 3.x (RhBug:531543)

[Dave (dave-redhat-bugs)]

Created attachment 586747
Adds an rpm-python3 subpackage

The attached patch adds a python3 subpackage. It removes the --with-python configuration option, instead using setup.py to build the bindings twice.

I had to pass in various env vars to get the bindings to build (to locate the not-yet-installed .pc file and the libs).

Successful scratch build here:
http://koji.fedoraproject.org/koji/taskinfo?taskID=4099767

[Panu (panu-redhat-bugs)]

Sorry I had completely forgotten about this thing :-/

The setup.py variant is less ugly than the previous ones, but as you note the setup.py itself is flawed for the purposes of in-tree building. And the otherwise "clean" option of splitting rpm-python out of rpm itself would make rpm version upgrades absolutely hideous.

It'd be better to fix up setup.py to properly support in-tree building instead of hacking around it. Somebody more experienced with python distutils than me might have a better chance of getting it right.

[Bohuslav (bohuslav-redhat-bugs)]

Created attachment 731128
Adds rpm-python3 subpackage with cleaner spec

Hey guys,
seems I'll be taking this over from Dave.
I'm attaching a patch that makes the spec look much cleaner (with some modifications in setup.py.in, which can hopefully be merged upstream if you like them).

[Panu (panu-redhat-bugs)]

The spec-side looks a whole lot nicer now sure, but the py3-fixes and setup.py.in patch seem to have gotten truncated / garbled somehow so can't comment on those yet. Please (re)submit the python-side fixes separately.

[Bohuslav (bohuslav-redhat-bugs)]

(In reply to comment #19)
> The spec-side looks a whole lot nicer now sure, but the py3-fixes and
> setup.py.in patch seem to have gotten truncated / garbled somehow so can't
> comment on those yet. Please (re)submit the python-side fixes separately.

It seems that BZ diff displaying is buggy for diffs containing new files (it doesn't display new files even in Dave's previous patch...). If you look at the raw diff [1], everything is there - I hope that that's enough.

[1] https://bugzilla.redhat.com/attachment.cgi?id=731128

[Panu (panu-redhat-bugs)]

Hum, I could've sworn I already commented on this but apparently I've missed hitting submit :-/

Anyway... yeah, now that you mention it I've seen the bugzilla diff bug before but had forgotten all about it. It's all there alright. The PyString -> PyBytes fix is upstreamed now, that was my mistake to begin with (old habits die hard :)

The latest patch certainly looks nicer and is something I can live with in Fedora for now. For upstreaming, there would seem to be two issues left:

The setup.py.in patch changes it to always assume being built within rpm source tree, which is fine for Fedora purposes but the idea was to technically permit distributing and building the bindings separately too. So for upstreaming that, it'd need some kind of detection whether its being built in rpm source tree or externally. Shouldn't be that big a deal.

The other issue is that this breaks the python tests in the self-test suite. A couple of pesky little details here: the self-test suite (incorrectly) assumes rpm is being built with --enable-python which it shouldn't do as the bindings are an optional add-on thing. The other question is how to enable the tests when the bindings are built outside automake control, optimally testing for both python2 and python3.

[Bohuslav (bohuslav-redhat-bugs)]

(In reply to comment #21)
> Hum, I could've sworn I already commented on this but apparently I've missed
> hitting submit :-/
>
> Anyway... yeah, now that you mention it I've seen the bugzilla diff bug
> before but had forgotten all about it. It's all there alright. The PyString
> -> PyBytes fix is upstreamed now, that was my mistake to begin with (old
> habits die hard :)
>
> The latest patch certainly looks nicer and is something I can live with in
> Fedora for now. For upstreaming, there would seem to be two issues left:
>
> The setup.py.in patch changes it to always assume being built within rpm
> source tree, which is fine for Fedora purposes but the idea was to
> technically permit distributing and building the bindings separately too. So
> for upstreaming that, it'd need some kind of detection whether its being
> built in rpm source tree or externally. Shouldn't be that big a deal.
>

Good point, I'll work on that.

> The other issue is that this breaks the python tests in the self-test suite.
> A couple of pesky little details here: the self-test suite (incorrectly)
> assumes rpm is being built with --enable-python which it shouldn't do as the
> bindings are an optional add-on thing. The other question is how to enable
> the tests when the bindings are built outside automake control, optimally
> testing for both python2 and python3.

I'll try to look into that, too.

[Bohuslav (bohuslav-redhat-bugs)]

Ping @panu could you please build with the patch in Fedora? It'd be great if everyone had the opportunity to test the bindings with Python 3. Thanks.

[Panu (panu-redhat-bugs)]

The "show-stopper" with the current patch is breaking the self-test suite, I'm not very enthusiastic about adding the patch as long as that is the case.

[Bohuslav (bohuslav-redhat-bugs)]

(In reply to Panu Matilainen from comment #24)
> The "show-stopper" with the current patch is breaking the self-test suite,
> I'm not very enthusiastic about adding the patch as long as that is the case.

Hmm, it doesn't break self test suite in my test builds... Would you at least consider accepting the patch downstream for Fedora as you said in comment 21? I'll work on getting the patch acceptable for upstream in the meanwhile.

[Panu (panu-redhat-bugs)]

Heh, I just tried it and noticed the test-suite actually still works. Which is supposed to be impossible because the python bindings dont get installed into the test-root when the bindings are not built through automake.

The trick is that unlike everything else, the python tests dont run through fakechroot as arranging a full python environment inside the root is rather tricky. What happens is that it just falls back to using system rpm-python bindings, which "works" as long as the system bindings are reasonably similar as the one currently being built. So the test-suite *appears* to work when building on eg F19, but would fail on F17.

There doesn't seem to be any sane way to hook the setup.py stuff into automake test-environment creation, but what we can do is leave --enable-python into configure for the test-suite, but nuke the automake-installed bindings from buildroot and then install the bindings with setup.py, ie

make DESTDIR="$RPM_BUILD_ROOT" install

rm -rf $RPM_BUILD_ROOT/%{python_sitearch}
pushd python
%{__python} setup.py install --skip-build --root $RPM_BUILD_ROOT
%{__python3} setup.py install --skip-build --root $RPM_BUILD_ROOT
popd

Of course this isn't optimal since the python2 bindings are getting built twice and the test-suite is now strictly speaking testing a different build than what gets packaged, but meh...

[Panu (panu-redhat-bugs)]

Only took four years to get there *cough* but we finally have python3 bindings in rawhide:
http://koji.fedoraproject.org/koji/taskinfo?taskID=5836322

Oh and thanks to both of you for making it happen.

[Bohuslav (bohuslav-redhat-bugs)]

(In reply to Panu Matilainen from comment #27)
> Only took four years to get there *cough* but we finally have python3
> bindings in rawhide:
> http://koji.fedoraproject.org/koji/taskinfo?taskID=5836322
>
> Oh and thanks to both of you for making it happen.

Thank you so much, this is great :)

[Dave (dave-redhat-bugs)]

 _______
< woot! >
 -------
        \ ^__^
         \ (oo)\_______
            (__)\ )\/\
                ||----w |
                || ||