Crashes when C calls Python callback with Python3 due to local ctypes module build configuration

Bug #909292 reported by Iven Hsu on 2011-12-28
36
This bug affects 5 people
Affects Status Importance Assigned to Milestone
pygobject
Unknown
Critical
python3.2 (Debian)
Fix Released
Unknown
python3.2 (Ubuntu)
High
Unassigned
Precise
Undecided
Unassigned

Bug Description

The sample code is attached, both happens in new installed Ubuntu 11.10 and Ubuntu 12.04.

Iven Hsu (ivenvd) wrote :
Martin Pitt (pitti) on 2012-02-16
Changed in pygobject (Ubuntu):
status: New → Triaged
importance: Undecided → High
Changed in pygobject:
importance: Unknown → Medium
status: Unknown → Confirmed
Changed in pygobject:
importance: Medium → Critical
Martin Pitt (pitti) on 2012-04-25
summary: - Segment fault when using set_cell_data_func in python3
+ Crashes when C calls Python callback with Python3
Changed in pygobject (Ubuntu):
assignee: nobody → Martin Pitt (pitti)
status: Triaged → In Progress
Changed in pygobject (Debian):
status: Unknown → Confirmed
Martin Pitt (pitti) on 2012-05-03
affects: pygobject (Ubuntu) → python3.2 (Ubuntu)
Changed in python3.2 (Ubuntu):
assignee: Martin Pitt (pitti) → nobody
status: In Progress → Triaged
Download full text (3.7 KiB)

After a multiple-days debugging orgy I finally found what's triggering this. Please see https://bugzilla.gnome.org/show_bug.cgi?id=669157 for the whole story, but copying the last comment here for convenience:

The Debian package runs this snippet between configure and make, to configure static (builtin) vs. dynamic (extensions) modules:

        egrep \
          "^#($$(awk -v ORS='|' '$$2 ~ /^extension$$/ {print $$1}'
debian/PVER-minimal.README.Debian.in)XX)" \
          Modules/Setup.dist \
            | sed -e 's/^#//' -e 's/-Wl,-Bdynamic//;s/-Wl,-Bstatic//' \
            >> $(1)/Modules/Setup.local

When I disable this part, the local build works.

That code results in this Modules/Setup.local:

----------------- 8< ---------------------
# Edit this file for local setup changes
array arraymodule.c # array objects
math mathmodule.c _math.c # -lm # math library functions, e.g. sin()
_struct _struct.c # binary structure packing/unpacking
time timemodule.c _time.c # -lm # time operations and variables
_random _randommodule.c # Random number generator
atexit atexitmodule.c # Register functions to be run at
interpreter-shutdown
_elementtree -DUSE_PYEXPAT_CAPI _elementtree.c # elementtree accelerator
_pickle _pickle.c # pickle accelerator
_datetime _datetimemodule.c # datetime accelerator
_bisect _bisectmodule.c # Bisection algorithms
_heapq _heapqmodule.c # Heap queue algorithm
unicodedata unicodedata.c # static Unicode character database
fcntl fcntlmodule.c # fcntl(2) and ioctl(2)
spwd spwdmodule.c # spwd(3)
grp grpmodule.c # grp(3)
select selectmodule.c # select(2); not on ancient System V
_socket socketmodule.c
_ssl _ssl.c -lssl -lcrypto
_posixsubprocess _posixsubprocess.c # POSIX subprocess module helper
_hashlib _hashopenssl.c -lssl -lcrypto
syslog syslogmodule.c # syslog daemon interface
binascii binascii.c
_ctypes _ctypes/_ctypes.c _ctypes/callbacks.c _ctypes/callproc.c
_ctypes/stgdict.c _ctypes/cfield.c _ctypes/malloc_closure.c -lffi
zlib zlibmodule.c -I$(prefix)/include -L$(exec_prefix)/lib -lz
pyexpat pyexpat.c -lexpat
----------------- 8< ---------------------

Bisecting this quickly leads to the _ctypes module configuration.

So a small reproducer based on the upstream Python 3.2 build is:

 * Configure and build pygobject with PYTHON=python3.2 (or specify the full
path for a local python build)

 * Build python 3.2 with the broken _ctypes module configuration:

$ cd /tmp/
$ tar xf python3.2_3.2.3.orig.tar.gz
$ cd python3.2-3.2.3
$ ./configure --with-wide-unicode # to be compatible with the Debian/Ubuntu
python 3.2
$ echo '_ctypes _ctypes/_ctypes.c _ctypes/callbacks.c _ctypes/callproc.c
_ctypes/stgdict.c _ctypes/cfield.c _ctypes/malloc_closure.c -lffi' >>
Modules/Setup.local
$ make -j4

Then in the pygobject built tree, run

$ (cd tests; TEST_NAMES=test_everything.TestCallbacks PYTHONPATH=..:.
LD_LIBRARY_PATH=./.libs GI_TYPELIB_PATH=. /tmp/python3.2-3.2.3/python
./runtests.py)

which reproduces the failure.

Conversely, if I change the ctypes line in debian/PVER-minimal.README.Debian.in from

  _ctypes extension

to

  _ctypes ...

Read more...

summary: - Crashes when C calls Python callback with Python3
+ Crashes when C calls Python callback with Python3 due to local ctypes
+ module build configuration
affects: pygobject (Debian) → python3.2 (Debian)
Matthias Klose (doko) on 2012-05-03
Changed in python3.2 (Ubuntu Precise):
status: New → In Progress
Martin Pitt (pitti) wrote :

The current theory is that things go wrong because libffi is loaded from both python itself (due to building in the ctypes extension) as well as dynamically through libgirepository.

I extended the test program to use ctypes and gi.repository at the same time, and both work with my local packages which build ctypes as a separate extension again.

Hello Iven, or anyone else affected,

Accepted python3.2 into precise-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in python3.2 (Ubuntu Precise):
status: In Progress → Fix Committed
tags: added: verification-needed
Dmitry Shachnev (mitya57) wrote :
tags: added: verification-failed
removed: verification-needed
Martin Pitt (pitti) wrote :

-0ubuntu3 built now.

tags: added: verification-needed
removed: verification-failed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python3.2 - 3.2.3-0ubuntu3

---------------
python3.2 (3.2.3-0ubuntu3) precise-proposed; urgency=low

  * Build _ctypes as an extension, not a builtin. LP: #909292.
  * Mark symbols defined in the _ctypes extension as optional.
 -- Matthias Klose <email address hidden> Thu, 03 May 2012 13:20:08 +0200

Changed in python3.2 (Ubuntu):
status: Triaged → Fix Released
Iven Hsu (ivenvd) wrote :

After updated, the bug seems fixed, thanks.

Martin Pitt (pitti) wrote :

I confirm that the pygobject test suite now works fine with this python3.2 version. As libpython3.2 changed ABI, I'd also like to verify that its reverse dependencies still work before calling this verified:

Reverse Depends:
  python3-uno
  znc-python
  yafaray
  uwsgi-plugin-python3
  python3-uno
  python3-pyside.qtxml
  python3-pyside.qtwebkit
  python3-pyside.qtuitools
  python3-pyside.qttest
  python3-pyside.qtsvg
  python3-pyside.qtsql
  python3-pyside.qtscript
  python3-pyside.qtopengl
  python3-pyside.qtnetwork
  python3-pyside.qthelp
  python3-pyside.qtgui
  python3-pyside.qtdeclarative
  python3-pyside.qtcore
  python3-pyside.phonon
  libshiboken-py3-1.1
  libpyside-py3-1.1
  libapache2-mod-wsgi-py3
  blender
  python3.2-dev
  postgresql-plpython3-9.1

Martin Pitt (pitti) wrote :

I verified most of the reverse dependencies. The ones that are still left are:

yafaray
libapache2-mod-wsgi-py3

For those I had a look at which symbols they need, and grepped for the removed symbols:

$ dpkg -L yafaray | grep .so | xargs nm -D | egrep 'PyC|stgdict|PyDict_SetItemProxy|PyDict_GetItemProxy'
                 U PyCapsule_GetPointer
                 U PyCapsule_Import
                 U PyCapsule_New

$ nm -D /usr/lib/apache2/modules/mod_wsgi.so-3.2 | egrep 'PyC|stgdict|PyDict_SetItemProxy|PyDict_GetItemProxy'
                 U PyCFunction_NewEx
                 U PyCapsule_New

These both look fine, these symbols were not removed.

Martin Pitt (pitti) wrote :

The test script which uses both ctypes and gi also works fine.

tags: added: verification-done
removed: verification-needed
Changed in pygobject:
status: Confirmed → Unknown
Colin Watson (cjwatson) wrote :

I ran into this while trying to port ubiquity and usb-creator. The python3.2 package in -proposed lets me get past the segfaults and onto the next error. Thanks!

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python3.2 - 3.2.3-0ubuntu3

---------------
python3.2 (3.2.3-0ubuntu3) precise-proposed; urgency=low

  * Build _ctypes as an extension, not a builtin. LP: #909292.
  * Mark symbols defined in the _ctypes extension as optional.
 -- Matthias Klose <email address hidden> Thu, 03 May 2012 13:20:08 +0200

Changed in python3.2 (Ubuntu Precise):
status: Fix Committed → Fix Released
Changed in python3.2 (Debian):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.