C extensions placed in wrong directory on karmic

Bug #392355 reported by Robert Collins on 2009-06-26
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Bazaar
High
Martin Pool
bzr (Ubuntu)
Undecided
Unassigned
python2.6 (Ubuntu)
High
Matthias Klose

Bug Description

In Karmic, it seems that distutils build_ext puts compiled extensions into eg 'bzrlib/bzrlib/foo.so' when it should be just 'bzrlib/foo.so', so the extension is not used. This was discovered in Bazaar but reproduced in other places, so seems independent of the program. This is a new regression compared to Jaunty.

----

C extensions are placed in bzrlib/bzrlib and thus are not used.

$make
setup.py build_ext -i
running build_ext
building 'bzrlib._bencode_pyx' extension
creating .../iter-changes-partial-parents/bzrlib/bzrlib
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-x86_64-2.6/bzrlib/_bencode_pyx.o -o .../iter-changes-partial-parents/bzrlib/bzrlib/_bencode_pyx.so
building 'bzrlib._btree_serializer_pyx' extension
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-x86_64-2.6/bzrlib/_btree_serializer_pyx.o -o .../iter-changes-partial-parents/bzrlib/bzrlib/_btree_serializer_pyx.so
...

description: updated
Jonathan Lange (jml) wrote :
Download full text (3.3 KiB)

On karmic, with bzr.dev r4479

jml@truth:~/src/bzr/trunk$ make
building extension modules.
python setup.py build_ext -i
running build_ext
building 'bzrlib._bencode_pyx' extension
creating build
creating build/temp.linux-x86_64-2.6
creating build/temp.linux-x86_64-2.6/bzrlib
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c bzrlib/_bencode_pyx.c -o build/temp.linux-x86_64-2.6/bzrlib/_bencode_pyx.o
bzrlib/_bencode_pyx.c:2414: warning: ‘__Pyx_GetItemInt’ defined but not used
bzrlib/_bencode_pyx.c:2429: warning: ‘__Pyx_SetItemInt’ defined but not used
creating /home/jml/src/bzr/bzr.dev/bzrlib/bzrlib
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-x86_64-2.6/bzrlib/_bencode_pyx.o -o /home/jml/src/bzr/bzr.dev/bzrlib/bzrlib/_bencode_pyx.so
pyrexc bzrlib/_btree_serializer_pyx.pyx --> bzrlib/_btree_serializer_pyx.c
/usr/lib/python2.6/dist-packages/Pyrex/Compiler/Scanning.py:39: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  import md5
building 'bzrlib._btree_serializer_pyx' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c bzrlib/_btree_serializer_pyx.c -o build/temp.linux-x86_64-2.6/bzrlib/_btree_serializer_pyx.o
bzrlib/_btree_serializer_pyx.c: In function ‘__pyx_f_6bzrlib_21_btree_serializer_pyx_safe_string_from_size’:
bzrlib/_btree_serializer_pyx.c:187: warning: cast from pointer to integer of different size
bzrlib/_btree_serializer_pyx.c: In function ‘__pyx_f_6bzrlib_21_btree_serializer_pyx_safe_interned_string_from_size’:
bzrlib/_btree_serializer_pyx.c:239: warning: cast from pointer to integer of different size
...

On karmic with 1.16,

jml@truth:~/src/bzr/bzr.1.16$ make
building extension modules.
python setup.py build_ext -i
running build_ext
building 'bzrlib._bencode_pyx' extension
creating build
creating build/temp.linux-x86_64-2.6
creating build/temp.linux-x86_64-2.6/bzrlib
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c bzrlib/_bencode_pyx.c -o build/temp.linux-x86_64-2.6/bzrlib/_bencode_pyx.o
bzrlib/_bencode_pyx.c:2414: warning: ‘__Pyx_GetItemInt’ defined but not used
bzrlib/_bencode_pyx.c:2429: warning: ‘__Pyx_SetItemInt’ defined but not used
creating /home/jml/src/bzr/bzr.1.16/bzrlib/bzrlib
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-x86_64-2.6/bzrlib/_bencode_pyx.o -o /home/jml/src/bzr/bzr.1.16/bzrlib/bzrlib/_bencode_pyx.so
building 'bzrlib._btree_serializer_c' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c bzrlib/_btree_serializer_c.c -o build/temp.linux-x86_64-2.6/bzrlib/_btree_serializer_c.o
bzrlib/_btree_serializer_c.c: In function ‘__pyx_f_19_btree_serializer_c_safe_string_from_size’:
bzrlib/_btree_serializer_c.c:144: warning: cast from pointer to integer of different size
bzrlib/_btree_serializer_c.c: In function ‘__pyx_f_19_btree_serializer_c_safe_interned_string_from_size’:
bzrlib/_btree_serializer_c.c:200: warning: cast from pointer to integer of different s...

Read more...

Robert Collins (lifeless) wrote :

It's affecting 1.16 and trunk then:
(from jml's paste: bzrlib/bzrlib/_bencode_pyx.so)

I'm suspecting karmic.

Robert Collins (lifeless) wrote :

John, I've subscribed you because the karmic PPA builds may need some love to Do The Right Thing.

cortesi (aldo) wrote :

It's a shame this has gone unfixed for a few months - it seems to affect all Python C extensions, and makes development a royal pain.

2009/8/26 cortesi <email address hidden>:
> It's a shame this has gone unfixed for a few months - it seems to affect
> all Python C extensions, and makes development a royal pain.

Do you mean all of them for bzr, or all across all python projects in Karmic?
--
Martin <http://launchpad.net/~mbp/>

I'm seeing this problem with C extensions in my own projects. An extension declaration like this:

       Extension(
            "module.extension",
            sources = ["module/extension.c"],
        )

Results in an object file at "module/module/extension.so" rather than "module/extension.so". The fact that this happens with all my projects and only on Karmic seems to show that there's been some modification to the Python build mechanism that's causing this. Also, note that Python 2.5 doesn't seem to have this problem - only Python 2.6.

Regards,

Aldo

James Westby (james-w) wrote :

Martin,

Why did you add a "bzr (Ubuntu)" task to this bug, I don't see this
happening with the bzr package in Ubuntu.

Thanks,

James

2009/8/26 James Westby <email address hidden>:
> Martin,
>
> Why did you add a "bzr (Ubuntu)" task to this bug, I don't see this
> happening with the bzr package in Ubuntu.

Because I thought it might be a problem there. If not, I'll mark it
invalid and apologize for the noise.

--
Martin <http://launchpad.net/~mbp/>

Changed in bzr (Ubuntu):
status: New → Invalid
James Westby (james-w) wrote :

On Wed Aug 26 22:44:40 UTC 2009 Martin Pool wrote:
> 2009/8/26 James Westby <email address hidden>:
> > Martin,
> >
> > Why did you add a "bzr (Ubuntu)" task to this bug, I don't see this
> > happening with the bzr package in Ubuntu.
>
> Because I thought it might be a problem there. If not, I'll mark it
> invalid and apologize for the noise.

No problem, I just wondered if you knew something I didn't.

I made sure to check the contents of the .deb before uploading, and
they were definitely in bzrlib/*.so as they should be.

Thanks,

James

cortesi (aldo) wrote :

This problem only affects in-place builds (i.e. "python setup.py build_ext
-i"), which is something that developers will stumble over while doing
development. Distribution builds don't seem to be affected.

On Thu, Aug 27, 2009 at 12:31 PM, James Westby
<<email address hidden><jw%<email address hidden>>
> wrote:

> On Wed Aug 26 22:44:40 UTC 2009 Martin Pool wrote:
> > 2009/8/26 James Westby <<email address hidden><jw%<email address hidden>>
> >:
> > > Martin,
> > >
> > > Why did you add a "bzr (Ubuntu)" task to this bug, I don't see this
> > > happening with the bzr package in Ubuntu.
> >
> > Because I thought it might be a problem there. If not, I'll mark it
> > invalid and apologize for the noise.
>
> No problem, I just wondered if you knew something I didn't.
>
> I made sure to check the contents of the .deb before uploading, and
> they were definitely in bzrlib/*.so as they should be.
>
> Thanks,
>
> James
>
> --
> C extensions placed in wrong directory
> https://bugs.launchpad.net/bugs/392355
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Bazaar Version Control System: Triaged
> Status in “bzr” package in Ubuntu: Invalid
> Status in “python-defaults” package in Ubuntu: New
>
> Bug description:
> C extensions are placed in bzrlib/bzrlib and thus are not used.
>
> $make
> setup.py build_ext -i
> running build_ext
> building 'bzrlib._bencode_pyx' extension
> creating .../iter-changes-partial-parents/bzrlib/bzrlib
> gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
> build/temp.linux-x86_64-2.6/bzrlib/_bencode_pyx.o -o
> .../iter-changes-partial-parents/bzrlib/bzrlib/_bencode_pyx.so
> building 'bzrlib._btree_serializer_pyx' extension
> gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
> build/temp.linux-x86_64-2.6/bzrlib/_btree_serializer_pyx.o -o
> .../iter-changes-partial-parents/bzrlib/bzrlib/_btree_serializer_pyx.so
> ...
>

--
Aldo Cortesi
www.nullcube.com
+64 210 718 900

Changed in bzr:
assignee: nobody → Ian Clatworthy (ian-clatworthy)

So to summarise, this bug appears to be impacting all python developers on karmic building extensions via setup.py. Bazaar is affected but not the problem per se. Furthermore, it isn't affecting developers on jaunty, nor the packaging of Bazaar on karmic. Is the critical priority justified?

I can dig more into this but I'll need to upgrade one of my boxes to karmic first. That's not a problem but I wasn't planning to do that prior to 2.0 shipping unless I had to.

Martin Pool (mbp) wrote :

I have a Karmic machine, and I'll try to reproduce it there.

Arguably this is not critical, though because we don't warn on missing extensions people might get a very unfoundedly bad impression if they're not built. otoh most people should run from packages. This might be shallow so if it is, we'll fix it, otherwise downgrade it.

Changed in bzr:
assignee: Ian Clatworthy (ian-clatworthy) → Martin Pool (mbp)

On Mon, 2009-08-31 at 05:02 +0000, Ian Clatworthy wrote:

> I can dig more into this but I'll need to upgrade one of my boxes to
> karmic first. That's not a problem but I wasn't planning to do that
> prior to 2.0 shipping unless I had to.

I don't think this should delay 2.0 - AFAICT its not a bzr bug, just
affects bzr.

-Rob

Martin Pool (mbp) on 2009-09-08
summary: - C extensions placed in wrong directory
+ C extensions placed in wrong directory on karmic
Martin Pool (mbp) wrote :

@ian see http://blog.sourcefrog.net/2009/09/setting-up-ubuntu-chroot-for.html for what I used to reproduce this.

I can still confirm it's happening in current Karmic. However, since it can be reproduced in other packages I'm going to downgrade it in bzr. I do think the Karmic python maintainers should take it more seriously...

Changed in bzr:
importance: Critical → High
status: Triaged → Confirmed
description: updated
Colin Watson (cjwatson) on 2009-09-08
affects: python-defaults (Ubuntu) → python2.6 (Ubuntu)
Changed in python2.6 (Ubuntu):
importance: Undecided → High
James Westby (james-w) wrote :

Hi,

It looks to me like http://bugs.python.org/issue6365 needs to be fixed on
the 2.6 branch as well.

Thanks,

James

James Westby (james-w) wrote :

Note http://bugs.python.org/issue6403 as well.

Thanks,

James

Matthias Klose (doko) wrote :

looking ... the patch was explicitely blocked on the 2.6 branch

Changed in python2.6 (Ubuntu):
assignee: nobody → Matthias Klose (doko)
Matthias Klose (doko) wrote :

python2.6 is now updated in karmic. please could somebody recheck?

2009/9/11 Matthias Klose <email address hidden>:
> python2.6 is now updated in karmic. please could somebody recheck?

Looks good to me with 2.6.2-0ubuntu5. Thanks!

--
Martin <http://launchpad.net/~mbp/>

Changed in bzr:
status: Confirmed → Invalid
Martin Pool (mbp) wrote :

Not a bzr upstream bug.

Matthias Klose (doko) wrote :

> Looks good to me with 2.6.2-0ubuntu5. Thanks!

closing

Changed in python2.6 (Ubuntu):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.