active etree method in thread crashes python process on shutdown

[Bjørnar Snoksrud]

Encountered on:
Python : sys.version_info(major=2, minor=7, micro=6, releaselevel='final', serial=0)
lxml.etree : (3, 3, 3, 0)
libxml used : (2, 9, 1)
libxml compiled : (2, 9, 1)
libxslt used : (1, 1, 28)
libxslt compiled : (1, 1, 28)

And reproduced on:
Python : sys.version_info(major=2, minor=7, micro=5, releaselevel='final', serial=0)
lxml.etree : (3, 3, 5, 0)
libxml used : (2, 9, 1)
libxml compiled : (2, 9, 1)
libxslt used : (1, 1, 28)
libxslt compiled : (1, 1, 28)

When a thread is running etree.fromstring while the main process is trying to finalize the interpreter on shutdown, the fromstring call might cause a segfault.

Following is a minimal reproduction case. It segfaults every ~15 runs on my machine:

from lxml import etree
import threading

class Thread(threading.Thread):
    def __init__(self):
        super(Thread, self).__init__()
        self.daemon = True

    def run(self):
        while True:
            self.foo = etree.fromstring('<Test/>')

Thread().start()

>

Sample stacktrace:

Core was generated by `/usr/bin/python -B -m <program>'.
Program terminated with signal 11, Segmentation fault.
#0 0x000001fff6cfc488 in ?? () from /lib/python2.7/site-packages/lxml/etree.so
$1 = -99
XXX Signaled thread
#0 0x000001fff6cfc488 in ?? () from /lib/python2.7/site-packages/lxml/etree.so
#1 0x000001fff6cfca40 in ?? () from /lib/python2.7/site-packages/lxml/etree.so
#2 0x000001fff6e86d90 in ?? () from /lib/python2.7/site-packages/lxml/etree.so
#3 0x000001fff7cb40b8 in PyCFunction_Call () from /lib/libpython2.7.so.1.0
#4 0x000001fff7d65220 in PyEval_EvalFrameEx () from /lib/libpython2.7.so.1.0
#5 0x000001fff7d67128 in PyEval_EvalFrameEx () from /lib/libpython2.7.so.1.0
#6 0x000001fff7d67128 in PyEval_EvalFrameEx () from /lib/libpython2.7.so.1.0
#7 0x000001fff7d68368 in PyEval_EvalCodeEx () from /lib/libpython2.7.so.1.0
#8 0x000001fff7c89f70 in ?? () from /lib/libpython2.7.so.1.0
#9 0x000001fff7c38d18 in PyObject_Call () from /lib/libpython2.7.so.1.0
#10 0x000001fff7c5ec00 in ?? () from /lib/libpython2.7.so.1.0
#11 0x000001fff7c38d18 in PyObject_Call () from /lib/libpython2.7.so.1.0
#12 0x000001fff7d5b068 in PyEval_CallObjectWithKeywords () from /lib/libpython2.7.so.1.0
#13 0x000001fff7db6d28 in ?? () from /lib/libpython2.7.so.1.0
#14 0x000001fff7ba8f48 in ?? () from /lib/libpthread.so.0
#15 0x000001fff7922740 in clone () from /lib/libc.so.6
$2 = -99
XXX All Threads

Thread 2 (LWP 942):
#0 0x000001fff7bb3548 in sem_wait () from /lib/libpthread.so.0
#1 0x000001fff7dae618 in PyThread_acquire_lock () from /lib/libpython2.7.so.1.0
#2 0x000001fff7d68698 in PyEval_RestoreThread () from /lib/libpython2.7.so.1.0
#3 0x000001fff7c764f0 in ?? () from /lib/libpython2.7.so.1.0
#4 0x000001fff7c79ea8 in ?? () from /lib/libpython2.7.so.1.0
#5 0x000001fff7cae800 in ?? () from /lib/libpython2.7.so.1.0
#6 0x000001fff7cae800 in ?? () from /lib/libpython2.7.so.1.0
#7 0x000001fff7d81388 in _PyImport_Fini () from /lib/libpython2.7.so.1.0
#8 0x000001fff7d98e48 in Py_Finalize () from /lib/libpython2.7.so.1.0
#9 0x000001fff7d99118 in Py_Exit () from /lib/libpython2.7.so.1.0
#10 0x000001fff7d99408 in ?? () from /lib/libpython2.7.so.1.0
#11 0x000001fff7d99858 in PyErr_PrintEx () from /lib/libpython2.7.so.1.0
#12 0x000001fff7db0e78 in ?? () from /lib/libpython2.7.so.1.0
#13 0x000001fff7db1f48 in Py_Main () from /lib/libpython2.7.so.1.0
#14 0x000001fff77ee4c8 in __libc_start_main () from /lib/libc.so.6
#15 0x00000000010007b0 in _start ()

[scoder (scoder)]

I'd consider patches that fix this, but I don't really consider it important. Doing actual processing in daemon threads sounds like a bad design in the first place.