Support extension module reloading

Bug #820377 reported by shen139
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
lxml
Triaged
Wishlist
Unassigned

Bug Description

A C program crashes after the second call to Py_Finalize() if lxml is used.

Tested with Python 2.5, 2.7 and 3.2. (etree.LXML_VERSION: (2, 3, 0, 0)) under Ubuntu 11.04.

The C code:
-[
#include <Python.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[])
{
    int x;

    for(x=0;x<10;x++)
    {
  printf(" + Py_Initialize (%d)...\n", x);
  Py_Initialize();

  printf(" + PyRun_SimpleString...\n");
  PyRun_SimpleString("from lxml import etree\n" \
   "print ' - etree.LXML_VERSION: ', etree.LXML_VERSION\n" \
   "print ' - etree.parse()'\n" \
     "tree = etree.parse('test.xml')\n" \
     "print ' - ok'\n");

  printf(" + Py_Finalize...\n\n");
  Py_Finalize();
    }

    return 1;
}
]-

Compiled with:

shen139@vega:~/test$ gcc test-lxml.c -I/usr/include/python2.7 -I/usr/include/python2.7 -o test-lxml.o -L/usr/lib/python2.7/config -lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions

Using:
Python 2.7.1+ (r271:86832, Apr 11 2011, 18:05:24)
[GCC 4.5.2] on linux2

The output:
-[
shen139@vega:~/test$ ./test-lxml.o
 + Py_Initialize (0)...
 + PyRun_SimpleString...
   - etree.LXML_VERSION: (2, 3, 0, 0)
   - etree.parse()
     - ok
 + Py_Finalize...

 + Py_Initialize (1)...
 + PyRun_SimpleString...
   - etree.LXML_VERSION: (2, 3, 0, 0)
   - etree.parse()
Segmentation fault
]-

Using:
Version: 3.2 (r32:88445, Mar 25 2011, 19:26:27)
[GCC 4.5.2]
Platform: linux2

Backtrace:
-[
#0 0x001ec991 in PyObject_Malloc () from /usr/lib/libpython3.2mu.so.1.0
#1 0x001ea045 in _PyObject_New () from /usr/lib/libpython3.2mu.so.1.0
#2 0x00205cda in ?? () from /usr/lib/libpython3.2mu.so.1.0
#3 0x0020f3b5 in PyUnicodeUCS4_DecodeUTF8Stateful () from /usr/lib/libpython3.2mu.so.1.0
#4 0x0020f963 in PyUnicodeUCS4_DecodeUTF8 () from /usr/lib/libpython3.2mu.so.1.0
#5 0x00213f29 in PyUnicodeUCS4_Decode () from /usr/lib/libpython3.2mu.so.1.0
#6 0x007eeca8 in ?? () from /usr/lib/python3/dist-packages/lxml/etree.cpython-32mu.so
#7 0x00820b6c in ?? () from /usr/lib/python3/dist-packages/lxml/etree.cpython-32mu.so
#8 0x00821210 in ?? () from /usr/lib/python3/dist-packages/lxml/etree.cpython-32mu.so
...
...
#526 0x00821210 in ?? () from /usr/lib/python3/dist-packages/lxml/etree.cpython-32mu.so
#527 0x00821210 in ?? () from /usr/lib/python3/dist-packages/lxml/etree.cpython-32mu.so
#528 0x00821210 in ?? () from /usr/lib/python3/dist-packages/lxml/etree.cpython-32mu.so
#529 0x00821210 in ?? () from /usr/lib/python3/dist-packages/lxml/etree.cpython-32mu.so
#530 0x00821210 in ?? () from /usr/lib/python3/dist-packages/lxml/etree.cpython-32mu.so
#531 0x00821210 in ?? () from /usr/lib/python3/dist-packages/lxml/etree.cpython-32mu.so

...
...
.... many many many more......
]-

shen139 (shen139)
description: updated
Revision history for this message
scoder (scoder) wrote :

My guess is that this is simply because extension module unloading is not supported in CPython, especially not in 2.x.

Changed in lxml:
importance: Undecided → Wishlist
status: New → Triaged
Revision history for this message
scoder (scoder) wrote :

This can theoretically be made to work in Py3.5+ now, but needs support in Cython as well. Probably still a lot of work to do on lxml side then, though.

summary: - Segmentation fault after Py_Finalize
+ Support extension module reloading
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.