Invalid use of xmlIO: crash on xmlCharEncCloseFunc()

[Victor Stinner]

lxml 1.3.3 crashed on my Ubuntu Hardy box (i386). Valgrind told me that it's a crash on xmlCharEncCloseFunc() call, called after xmlOutputBufferClose(). I read libxml2 source code: xmlOutputBufferClose() already closes the encoder handler, so the xmlCharEncCloseFunc() call is useless and INVALID!

I wrote a patch for serializer.pxi to disable the invalid call.

Valgrind output:
==14052== Invalid read of size 4
==14052== at 0x59798A5: xmlCharEncCloseFunc (encoding.c:2114)
==14052== by 0x58C98C9: __pyx_f_5etree_12_ElementTree_write (etree.c:25239)
==14052== by 0x80C8EEB: PyEval_EvalFrameEx (ceval.c:3564)
==14052== by 0x80C9314: PyEval_EvalFrameEx (ceval.c:3650)
==14052== by 0x80CA114: PyEval_EvalCodeEx (ceval.c:2831)
==14052== by 0x81123E0: function_call (funcobject.c:517)
==14052== by 0x805C9E6: PyObject_Call (abstract.c:1860)
==14052== by 0x80C709E: PyEval_EvalFrameEx (ceval.c:3844)
==14052== by 0x80CA114: PyEval_EvalCodeEx (ceval.c:2831)
==14052== by 0x81123E0: function_call (funcobject.c:517)
==14052== by 0x805C9E6: PyObject_Call (abstract.c:1860)
==14052== by 0x8062B43: instancemethod_call (classobject.c:2509)
==14052== Address 0x4EF2308 is 0 bytes inside a block of size 20 free'd
==14052== at 0x402237F: free (vg_replace_malloc.c:233)
==14052== by 0x597990B: xmlCharEncCloseFunc (encoding.c:2134)
==14052== by 0x59A6444: xmlOutputBufferClose (xmlIO.c:2353)
==14052== by 0x58C98BE: __pyx_f_5etree_12_ElementTree_write (etree.c:25236)
==14052== by 0x80C8EEB: PyEval_EvalFrameEx (ceval.c:3564)
==14052== by 0x80C9314: PyEval_EvalFrameEx (ceval.c:3650)
==14052== by 0x80CA114: PyEval_EvalCodeEx (ceval.c:2831)
==14052== by 0x81123E0: function_call (funcobject.c:517)
==14052== by 0x805C9E6: PyObject_Call (abstract.c:1860)
==14052== by 0x80C709E: PyEval_EvalFrameEx (ceval.c:3844)
==14052== by 0x80CA114: PyEval_EvalCodeEx (ceval.c:2831)
==14052== by 0x81123E0: function_call (funcobject.c:517)

My code is too huge to extract simple Python example to reproduce the bug, sorry. But I tested my patch: patched lxml doesn't write the error anymore (in Valgrind).

[scoder (scoder)]

Funny how there are still bugs in old code that just do not show up for ages. :)

Thanks for tracking this down.

[Victor Stinner (vstinner)]

"Funny how there are still bugs in old code that just do not show up for ages. :)"

As written in Valgrind output, the code uses "free'd" memory, and so the behaviour changes with the version of libc, with the OS (and OS version), with the memory usage, etc. In my case, the bug only occurs 33% of the time.

Thanks for quickly apply it ;-)

[scoder (scoder)]

fixed in 2.0.6