meliae hits type_traverse assertion when dumping ctypes objects under Py_DEBUG

[Robert Xiao]

This is probably a fairly obscure bug, but it's worth posting about in case it can be fixed (or the information is useful to someone else).

In several versions of Python, type_traverse in Objects/typeobject.c contains the assertion

assert(type->tp_flags & Py_TPFLAGS_HEAPTYPE);

meliae developers know about it, so they made a check in _scanner_core.c which tries to avoid tickling this assertion:

    if (Py_TYPE(c_obj)->tp_traverse == NULL
        || (Py_TYPE(c_obj)->tp_traverse == PyType_Type.tp_traverse
            && !PyType_HasFeature((PyTypeObject*)c_obj, Py_TPFLAGS_HEAPTYPE)))
    {
        /* Obviously we don't traverse if there is no traverse function. But
         * also, if this is a 'Type' (class definition), then
         * PyTypeObject.tp_traverse has an assertion about whether this type is
         * a HEAPTYPE. In debug builds, this can trip and cause failures, even
         * though it doesn't seem to hurt anything.
         * See: https://bugs.launchpad.net/bugs/586122
         */
        do_traverse = 0;
    }

This isn't a complete solution because type_traverse can be called indirectly from some custom tp_traverse. In fact, this is precisely what ctypes.PyCPointerType does:

static int
CDataType_traverse(PyTypeObject *self, visitproc visit, void *arg)
{
    StgDictObject *dict = PyType_stgdict((PyObject *)self);
    if (dict)
        Py_VISIT(dict->proto);
    return PyType_Type.tp_traverse((PyObject *)self, visit, arg);
}
/*...*/
    (traverseproc)CDataType_traverse, /* tp_traverse */
    (inquiry)CDataType_clear, /* tp_clear */

To trigger this bug requires the following steps:
1) Build Python 3.2 using Py_DEBUG
2) Build meliae against this Python
3) Run the following script:
import ctypes
import sys
import meliae.scanner
meliae.scanner.dump_all_referenced('/tmp/foo', sys.getobjects(0))

Here's what I get from running that:

Python 3.2.1 (default, Nov 20 2011, 14:49:05)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
[60256 refs]
>>> import sys
[60258 refs]
>>> import meliae.scanner
[61527 refs]
>>> meliae.scanner.dump_all_referenced('/tmp/foo', sys.getobjects(0))
Assertion failed: (type->tp_flags & Py_TPFLAGS_HEAPTYPE), function type_traverse, file Objects/typeobject.c, line 2600.
Abort trap

If you don't import ctypes, the dump succeeds without incident.

Workaround:
I just compiled typeobject.c with
    if(!(type->tp_flags & Py_TPFLAGS_HEAPTYPE))
        return -1;
instead of the assert.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/22/2011 7:22 AM, Robert Xiao wrote:
> Public bug reported:
>
> This is probably a fairly obscure bug, but it's worth posting about
> in case it can be fixed (or the information is useful to someone
> else).
>

I agree this affects meliae, but what can we actually do about it?

John
=:->

 status: incomplete

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7WBwQACgkQJdeBCYSNAAMVrgCeLWTAwMoEyrMiao0Fb4PRAryu
ZL4AoKH7v4i8rbaYCUree6iMW61GejR+
=Xpfr
-----END PGP SIGNATURE-----

[Ryan Kelly (rfkelly)]

I just hit this bug myself, and have worked up a solution for it. It's not pretty but it seems to work for me.

It's possible to work around this issue inside meliae by extending the Py_TPFLAGS_HEAPTYPE check, so that it covers both the PyType_Type.tp_traverse base case, and the StructType_Type.tp_traverse that's causing problems in this bug.

Unfortunately, you can't get a static reference to StructType_Type since it's hidden away inside the _ctypes.so shared library. So instead I added some code to look it up at runtime.

This is clearly a hack and will add extra function call overhead at runtime, so I'm not really expecting it to be merged. But maybe it will be helpful for others who get stuck on this issue.