Zope 3

Segfaults under Load during Garbage Collection

Bug #181833 reported by Kevin McDermott on 2008-01-10

Affects		Status	Importance	Assigned to	Milestone
	Zope 3	Fix Released	High	Gary Poster

Bug Description

This is hitting a standard Zope3 svn://svn.zope.org/repos/main/Zope3/tags/3.4.0b2 (r82643) server under load, with varying concurrency.

Core was generated by `python2.4 /srv/landscape.canonical.com/landscape/landscape'.
Program terminated with signal 11, Segmentation fault.
#0 visit_decref (op=0x0, data=0x0) at ../Modules/gcmodule.c:269
269 ../Modules/gcmodule.c: No such file or directory.
        in ../Modules/gcmodule.c
(gdb) bt
#0 visit_decref (op=0x0, data=0x0) at ../Modules/gcmodule.c:269
#1 0x00002abfa3dc7896 in proxy_traverse (self=0x2aaaad54c190, visit=0x4a0b50 <visit_decref>, arg=0x0)
    at src/zope/security/_proxy.c:305
#2 0x00000000004a13a5 in collect (generation=0) at ../Modules/gcmodule.c:294
#3 0x00000000004a1f45 in _PyObject_GC_New (tp=0x5fed40) at ../Modules/gcmodule.c:872
#4 0x000000000044530e in tuple_iter (seq=0x2abfa68e9bd0) at ../Objects/tupleobject.c:805
#5 0x0000000000414e7f in PyObject_GetIter (o=0x2abfa68e9bd0) at ../Objects/abstract.c:2228
#6 0x00000000004715a0 in PyEval_EvalFrame (f=0x34328e0) at ../Python/ceval.c:2107
#7 0x00000000004767d6 in PyEval_EvalCodeEx (co=0x2abfa49b6b20, globals=<value optimized out>,
    locals=<value optimized out>, args=0x2, argcount=2, kws=0x2abfa49c9e54, kwcount=0, defs=0x0, defcount=0,
    closure=0x0) at ../Python/ceval.c:2741
#8 0x00000000004bf233 in function_call (func=0x2abfa4a4f5f0, arg=0x2aaaabd9cf80, kw=0x0)
    at ../Objects/funcobject.c:548
#9 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#10 0x0000000000419930 in instancemethod_call (func=<value optimized out>, arg=0x2aaaabd9cf80, kw=0x0)
    at ../Objects/classobject.c:2532
#11 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#12 0x000000000044ebf8 in slot_tp_init (self=<value optimized out>, args=0x2e58cd0, kwds=0x0)
    at ../Objects/typeobject.c:4774
#13 0x000000000044b121 in type_call (type=0x1291810, args=0x2e58cd0, kwds=0x0) at ../Objects/typeobject.c:435
#14 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#15 0x0000000000472619 in PyEval_EvalFrame (f=0x4572100) at ../Python/ceval.c:3776
#16 0x0000000000475546 in PyEval_EvalFrame (f=0x343fde0) at ../Python/ceval.c:3651
#17 0x0000000000475546 in PyEval_EvalFrame (f=0x297dd90) at ../Python/ceval.c:3651
#18 0x0000000000475546 in PyEval_EvalFrame (f=0x3006c60) at ../Python/ceval.c:3651
#19 0x0000000000475546 in PyEval_EvalFrame (f=0x3a27a00) at ../Python/ceval.c:3651
#20 0x0000000000475546 in PyEval_EvalFrame (f=0x3175f50) at ../Python/ceval.c:3651
#21 0x00000000004767d6 in PyEval_EvalCodeEx (co=0x2abfa4bf05e0, globals=<value optimized out>,
    locals=<value optimized out>, args=0x2aaaad907298, argcount=3, kws=0x0, kwcount=0, defs=0x0, defcount=0,
    closure=0x0) at ../Python/ceval.c:2741
#22 0x00000000004bf233 in function_call (func=0x2abfa4c5fed8, arg=0x2aaaad907280, kw=0x0)
    at ../Objects/funcobject.c:548
#23 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#24 0x0000000000419930 in instancemethod_call (func=<value optimized out>, arg=0x2aaaad907280, kw=0x0)
    at ../Objects/classobject.c:2532
#25 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#26 0x00002abfa3dc8a75 in proxy_call (self=0x3af9dd0, args=0x31ae248, kwds=0x0) at src/zope/security/_proxy.c:461
#27 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#28 0x0000000000472619 in PyEval_EvalFrame (f=0x38bbb40) at ../Python/ceval.c:3776
#29 0x00000000004767d6 in PyEval_EvalCodeEx (co=0x2abfa65b12d0, globals=<value optimized out>,
    locals=<value optimized out>, args=0x2aaaad907c98, argcount=3, kws=0x2b19970, kwcount=0, defs=0x0, defcount=0,
    closure=0x0) at ../Python/ceval.c:2741
#30 0x00000000004bf1cc in function_call (func=0x2abfa65b5f50, arg=0x2aaaad907c80, kw=0x36c97c0)
    at ../Objects/funcobject.c:548
#31 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#32 0x0000000000473fd0 in PyEval_EvalFrame (f=0x46a0ab0) at ../Python/ceval.c:3845
#33 0x00000000004767d6 in PyEval_EvalCodeEx (co=0x2abfa6584ce0, globals=<value optimized out>,
    locals=<value optimized out>, args=0x29b7460, argcount=3, kws=0x29b7478, kwcount=0, defs=0x0, defcount=0,
    closure=0x2abfa65b6098) at ../Python/ceval.c:2741
#34 0x0000000000474a5a in PyEval_EvalFrame (f=0x29b72b0) at ../Python/ceval.c:3661
#35 0x00000000004767d6 in PyEval_EvalCodeEx (co=0x2abfa659b570, globals=<value optimized out>,
    locals=<value optimized out>, args=0x291ca40, argcount=2, kws=0x291ca50, kwcount=0, defs=0x0, defcount=0,
    closure=0x2aaaac719cb0) at ../Python/ceval.c:2741
#36 0x0000000000474a5a in PyEval_EvalFrame (f=0x291c890) at ../Python/ceval.c:3661
#37 0x0000000000475546 in PyEval_EvalFrame (f=0x69f7d0) at ../Python/ceval.c:3651
#38 0x0000000000475546 in PyEval_EvalFrame (f=0x323e250) at ../Python/ceval.c:3651
#39 0x00000000004767d6 in PyEval_EvalCodeEx (co=0x2abfa65be880, globals=<value optimized out>,
    locals=<value optimized out>, args=0x1, argcount=1, kws=0x4771f78, kwcount=0, defs=0x2abfa65c27a8, defcount=1,
    closure=0x0) at ../Python/ceval.c:2741
#40 0x0000000000474a5a in PyEval_EvalFrame (f=0x4771dd0) at ../Python/ceval.c:3661
#41 0x0000000000475546 in PyEval_EvalFrame (f=0x35f0cd0) at ../Python/ceval.c:3651
#42 0x00000000004767d6 in PyEval_EvalCodeEx (co=0x2abfa65c4f10, globals=<value optimized out>,
    locals=<value optimized out>, args=0x2aaaad2fc1a8, argcount=1, kws=0x0, kwcount=0, defs=0x0, defcount=0,
    closure=0x0) at ../Python/ceval.c:2741
#43 0x00000000004bf233 in function_call (func=0x2abfa65e3668, arg=0x2aaaad2fc190, kw=0x0)
    at ../Objects/funcobject.c:548
#44 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#45 0x0000000000419930 in instancemethod_call (func=<value optimized out>, arg=0x2aaaad2fc190, kw=0x0)
    at ../Objects/classobject.c:2532
#46 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#47 0x000000000044efb6 in slot_tp_call (self=<value optimized out>, args=0x2abfa0a7a050, kwds=0x0)
    at ../Objects/typeobject.c:4546
#48 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#49 0x00002abfa3dc8a75 in proxy_call (self=0x2e6e310, args=0x2abfa0a7a050, kwds=0x0)
    at src/zope/security/_proxy.c:461
#50 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#51 0x0000000000473fd0 in PyEval_EvalFrame (f=0x38ac7d0) at ../Python/ceval.c:3845
#52 0x0000000000475546 in PyEval_EvalFrame (f=0x2d2e220) at ../Python/ceval.c:3651
#53 0x00000000004767d6 in PyEval_EvalCodeEx (co=0x2abfa3c9fc00, globals=<value optimized out>,
    locals=<value optimized out>, args=0xf737d8, argcount=3, kws=0xf737f0, kwcount=0, defs=0x2abfa3ca7d58,
    defcount=2, closure=0x0) at ../Python/ceval.c:2741
#54 0x0000000000474a5a in PyEval_EvalFrame (f=0xf73630) at ../Python/ceval.c:3661
#55 0x0000000000475546 in PyEval_EvalFrame (f=0x3a46880) at ../Python/ceval.c:3651
#56 0x00000000004767d6 in PyEval_EvalCodeEx (co=0x2abfa3cab3b0, globals=<value optimized out>,
    locals=<value optimized out>, args=0x1, argcount=1, kws=0x334a988, kwcount=1, defs=0x2abfa3ca0ce8, defcount=1,
    closure=0x0) at ../Python/ceval.c:2741
#57 0x0000000000474a5a in PyEval_EvalFrame (f=0x334a7c0) at ../Python/ceval.c:3661
#58 0x00000000004767d6 in PyEval_EvalCodeEx (co=0x2abfa33c0f80, globals=<value optimized out>,
    locals=<value optimized out>, args=0x4140068, argcount=3, kws=0x0, kwcount=0, defs=0x0, defcount=0,
    closure=0x0) at ../Python/ceval.c:2741
#59 0x00000000004bf233 in function_call (func=0x2abfa4165aa0, arg=0x4140050, kw=0x0) at ../Objects/funcobject.c:548
#60 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#61 0x0000000000419930 in instancemethod_call (func=<value optimized out>, arg=0x4140050, kw=0x0)
    at ../Objects/classobject.c:2532
#62 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#63 0x000000000044efb6 in slot_tp_call (self=<value optimized out>, args=0x2aaaaf4216c8, kwds=0x0)
    at ../Objects/typeobject.c:4546
#64 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#65 0x0000000000472619 in PyEval_EvalFrame (f=0x3be3f10) at ../Python/ceval.c:3776
#66 0x00000000004767d6 in PyEval_EvalCodeEx (co=0x2abfa0b3a9d0, globals=<value optimized out>,
    locals=<value optimized out>, args=0x2aaaaf054528, argcount=1, kws=0x34f37c0, kwcount=0, defs=0x0, defcount=0,
    closure=0x0) at ../Python/ceval.c:2741
#67 0x00000000004bf1cc in function_call (func=0x2abfa33ef758, arg=0x2aaaaf054510, kw=0x3e1ff00)
    at ../Objects/funcobject.c:548
#68 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#69 0x0000000000473fd0 in PyEval_EvalFrame (f=0x32f3a70) at ../Python/ceval.c:3845
#70 0x00000000004767d6 in PyEval_EvalCodeEx (co=0x2abfa2ac6ea0, globals=<value optimized out>,
    locals=<value optimized out>, args=0x43ead88, argcount=3, kws=0x49d2e10, kwcount=0, defs=0x0, defcount=0,
    closure=0x0) at ../Python/ceval.c:2741
#71 0x00000000004bf1cc in function_call (func=0x2abfa2bea8c0, arg=0x43ead70, kw=0x3e06d40)
    at ../Objects/funcobject.c:548
#72 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#73 0x0000000000473fd0 in PyEval_EvalFrame (f=0x14e0820) at ../Python/ceval.c:3845
#74 0x00000000004767d6 in PyEval_EvalCodeEx (co=0x2abfa2ad0340, globals=<value optimized out>, locals=<value optimized out>, args=0x2aaaac59c158, argcount=3, kws=0x2b08340, kwcount=0, defs=0x0, defcount=0,
    closure=0x0) at ../Python/ceval.c:2741
#75 0x00000000004bf1cc in function_call (func=0x2abfa2beaaa0, arg=0x2aaaac59c140, kw=0x287d500)
    at ../Objects/funcobject.c:548
#76 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#77 0x0000000000473fd0 in PyEval_EvalFrame (f=0x242f4b0) at ../Python/ceval.c:3845
#78 0x00000000004767d6 in PyEval_EvalCodeEx (co=0x2abfa5ab6260, globals=<value optimized out>,
    locals=<value optimized out>, args=0x2abfa650cb60, argcount=2, kws=0x2f25810, kwcount=0, defs=0x0, defcount=0,
    closure=0x0) at ../Python/ceval.c:2741
#79 0x00000000004bf1cc in function_call (func=0x2abfa487cc80, arg=0x2abfa650cb48, kw=0x6a6610)
    at ../Objects/funcobject.c:548
#80 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#81 0x0000000000473fd0 in PyEval_EvalFrame (f=0x1762390) at ../Python/ceval.c:3845
#82 0x0000000000475546 in PyEval_EvalFrame (f=0xdca830) at ../Python/ceval.c:3651
#83 0x00000000004767d6 in PyEval_EvalCodeEx (co=0x2abfa13833b0, globals=<value optimized out>,
    locals=<value optimized out>, args=0x2abfa68e99e8, argcount=1, kws=0x0, kwcount=0, defs=0x0, defcount=0,
    closure=0x0) at ../Python/ceval.c:2741
#84 0x00000000004bf233 in function_call (func=0x2abfa138e230, arg=0x2abfa68e99d0, kw=0x0)
    at ../Objects/funcobject.c:548
#85 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#86 0x0000000000419930 in instancemethod_call (func=<value optimized out>, arg=0x2abfa68e99d0, kw=0x0)
    at ../Objects/classobject.c:2532
#87 0x0000000000413bf0 in PyObject_Call (func=0x0, arg=0x0, kw=0x0) at ../Objects/abstract.c:1795
#88 0x000000000046faf1 in PyEval_CallObjectWithKeywords (func=0x2abfa5607280, arg=0x2abfa0a7a050, kw=0x0)
    at ../Python/ceval.c:3435
#89 0x00000000004a25ed in t_bootstrap (boot_raw=0x2d34850) at ../Modules/threadmodule.c:434
#90 0x00002abfa0b7d3ca in start_thread () from /lib/libpthread.so.0
#91 0x00002abfa10e355d in clone () from /lib/libc.so.6
#92 0x0000000000000000 in ?? ()

See original description

Kevin McDermott (bigkevmcd) on 2008-01-10

description:

updated

Revision history for this message

Kevin McDermott (bigkevmcd) wrote on 2008-01-10:

Patch contributed by Gustavo Niemeyer :-) Edit (513 bytes, text/plain)

By using the Py_VISIT macro to check for NULL objects we guard against visiting already deallocated objects.

Revision history for this message

Michael Hudson-Doyle (mwhudson) wrote on 2008-08-28:

So this is affecting Launchpad too. However, we think it's a pyscopg2 refcounting bug: it looks like the only way that self->proxy.proxy_object can be NULL is if the object has been deallocated (and so the traverse is accessing deallocated memory), and certainly if self->proxy.proxy_object can be NULL on a live proxy object then there are problems all over the module.

Revision history for this message

James Henstridge (jamesh) wrote on 2008-08-28:

Looking at the code, self->proxy.proxy_object can be NULL if tp_clear() has been called on the object (which doesn't necessarily imply that it has been deallocated.

Gustavo's patch fixes tp_traverse() to handle the case where tp_clear() has been called. That said, much of the code in the _proxy.c and _zope_proxy_proxy.c seem to assume that the proxy_object pointer will be valid. This indicates to me that the object should not be clearing the pointer on tp_clear(). Perhaps the object shouldn't even be implementing tp_clear().

Revision history for this message

Gary Poster (gary) wrote on 2009-08-11:

This is in what will become zope.security 3.7.1. I will probably make a release tomorrow.