pylint spews Exception RuntimeError: maximum recursion depth exceeded while calling a Python object

[Forest]

When operating on a file that calls email.message.Message.get(), pylint 0.15.2-1 spews repeated copies of this error message:

Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in <type 'exceptions.AttributeError'> ignored

Here is a test program that triggers pylint's complaints:

import email.parser
parser = email.parser.HeaderParser()
message = parser.parsestr( '')
message.get('From')

[Forest (foresto)]

I tried to submit a bug report on logilab.org, but I don't see a way to do so.
http://www.logilab.org/857

[afayolle (alexandre-fayolle)]

Hi,

I cannot reproduce this bug with an up-to-date version of pylint and astng. Simplest way to fix it is to upgrade.

Btw, easiest way of communicating with pylint maintainers is to subscribe to the mailing list (http://lists.logilab.org/mailman/listinfo/python-projects) and send a mail to <email address hidden>.

Thanks for your interest in Pylint

--
Alexandre

[Edward K. Ream (edreamleo)]

I am getting this problem with all files of my Leo sources on Ubuntu 9.10 Karmic

pylint 0.18.0, (latest available package)
astng 0.19.0, common 0.39.0
Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
[GCC 4.4.1]

For example, from Leo's bzr repository: (the top-level directory):

pylint leo/core/leoFileCommands.py > pylint.out

Messages to console, but not to pylint.out.

I'd call this a show-stopper. Please do something about this.

Edward K. Ream

[Dane Elwell (dane-elwell)]

If the simplest solution is to upgrade, can we upgrade the version in the repositories?

[Eric Pierce (wapcaplet88)]

I'm getting this error in Gentoo with these versions:

pylint-0.19.0
astng-0.19.3
python-2.6.4

Pylint still seems to work fine, but this error is a little annoying.

[Edward K. Ream (edreamleo)]

I get this on all my files on Windows as well:

pylint 0.19.0,
astng 0.19.3, common 0.46.0
Python 2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 32 bit (Intel)]

I do not believe pylint is working as well as it should. It is missing serious errors, for example, undefined methods.

[Forest (foresto)]

I installed pylint 0.19 (from ubuntu lucid packages) and it is still spewing repeated copies of that error message. It no longer does it with the test program I included in my first comment, but it does with plenty of others. I'm attaching a different program that triggers the misbehavior.

[Forest (foresto)]

More information here:

http://lists.logilab.org/pipermail/python-projects/2009-December/002124.html

[Forest (foresto)]

I tried to add my findings to that bug report, but despite having logged in at logilab, their bug report page keeps presenting me with the "register or log in message". How frustrating. If anyone from the pylint/astng projects is reading, could you please add to that bug report the slightly different error message I reported here, as well as the test code I just attached?

Also, if a logilab.org site owner is reading, please consider improving usability on your site. It's rather difficult to figure out how to file a bug report, and apparently impossible to comment on one.

[Forest (foresto)]

Here's the bug report i was referring to:

http://www.logilab.org/ticket/19641

[Edward K. Ream (edreamleo)]

I can now demonstrate exactly where the unbound recursion happens.

The following is a debugger that warns me when the stack gets too big. It is a subclass of pdb, but it overrides two methods in bdb.py. Here it is::

class debugger(pdb.Pdb):

    def __init__(self,completekey='tab',stdin=None,stdout=None):
        pdb.Pdb.__init__(self,completekey,stdin,stdout)
        self.stackLimit = 200 # Found by trial and error.

    # dispatch_call (override bdb)
    def dispatch_call(self, frame, arg):
        # XXX 'arg' is no longer used

        # EKR: new code.
        stack,junk = self.get_stack(frame,None)
        if len(stack) > self.stackLimit:
            print('(debugger.stop_here) stackCount %s' % len(stack))
            self.interaction(frame,None)

        # Existing code.
        if self.botframe is None:
            # First call of dispatch since reset()
            self.botframe = frame.f_back # (CT) Note that this may also be None!
            return self.trace_dispatch
        if not (self.stop_here(frame) or self.break_anywhere(frame)):
            # No need to trace this function
            return # None
        self.user_call(frame, arg)
        if self.quitting: raise BdbQuit
        return self.trace_dispatch

    # set_continue (override bdb)
    def set_continue(self):

        # Don't stop except at breakpoints or when finished
        self._set_stopinfo(self.botframe, None)

        # EKR: Always run with overhead
        if False:
            if not self.breaks:
                # no breakpoints; run without debugger overhead
                sys.settrace(None)
                frame = sys._getframe().f_back
                while frame and frame is not self.botframe:
                    del frame.f_trace
                    frame = frame.f_back

And here is the stack trace when the stack limit tripped. It includes some print statements I inserted while stepping through the code.

Breakpoint 1 at c:\python26\lib\site-packages\logilab_astng-0.19.3-py2.6.egg\logilab\astng\node_classes.py:256
> c:\python26\lib\site-packages\pylint-0.19.0-py2.6.egg\pylint\lint.py(782)__init__()
-> self._rcfile = None
(Pdb) disable 1
(Pdb) c
manager.py: astng_from_file: not in cache: c:\leo.repo\until\pylintTest.py
manager.py: astng_from_file: calling file_build(pylintTest)
manager.py: astng_from_file: not in cache: c:\leo.repo\until\leo\__init__.py
manager.py: astng_from_file: calling file_build(leo)
manager.py: astng_from_file: not in cache: c:\leo.repo\until\leo\core\__init__.py
manager.py: astng_from_file: calling file_build(leo.core)
manager.py: astng_from_file: not in cache: c:\leo.repo\until\leo\core\leoGlobals.py
manager.py: astng_from_file: calling file_build(leo.core.leoGlobals)
manager.py: astng_from_file: not in cache: c:\leo.repo\until\leo\core\leoApp.py
manager.py: astng_from_file: calling file_build(leo.core.leoApp)
(debugger.stop_here) stackCount 201
> c:\python26\lib\site-packages\logilab_astng-0.19.3-py2.6.egg\logilab\astng\inference.py(52)_set_proxied()
-> def _set_proxied(const):
(Pdb) w
('pdb.py:print_stack_trace', 201)
  c:\python26\lib\bdb.py(404)runcall()
-> res = func(*args, **kwds)
  ...

[Edward K. Ream (edreamleo)]

The failing code appears to be _set_proxied in inference.py.

def _set_proxied(const):

    if not hasattr(const, '__proxied'):
        const.__proxied = _CONST_PROXY[const.value.__class__]
    return const.__proxied
nodes.Const._proxied = property(_set_proxied)

The call to hasattr can call Proxy.__getattr__ in infutils.py:

def __getattr__(self, name):
        if name == '_proxied':
            return getattr(self.__class__, '_proxied')
        if name in self.__dict__:
            return self.__dict__[name]
        return getattr(self._proxied, name)

As the trace shows, the last line of __getattr__ can invoke _setProxied, apparently directly.

My head is about to explode :-) In particular, the const argument to _set_proxied apparently represents None without being None. The debugger reports const to be:

<Instance of __builtin__.NoneType at (an address)>

but "const is None" evaluates to False. Ditto for various other tries, such as isinstance(const,something-or-other)

As if that were not enough, did you notice _proxied vs. __proxied?

I do hope somebody who actually understand this code will fix it. To repeat, pylint has been failing to catch serious bugs in my app as a result of this problem. I've pretty much lost confidence in pylint for now.

Edward

[Edward K. Ream (edreamleo)]

The following hack seems to repair the damage. Change:

def _set_proxied(const):
    if not hasattr(const, '__proxied'):
        const.__proxied = _CONST_PROXY[const.value.__class__]
    return const.__proxied

To:

def _set_proxied(const):
    const._proxied = _CONST_PROXY[const.value.__class__]
    return const._proxied

As you can see, there are two major changes to the code:

1. The code uses _proxied instead of __proxied.

I don't see any other uses of __proxied anywhere,
so this would seem like what was originally intended,
but there are absolutely no guarantees.

2. The code always sets const._proxied.

In my test, _set_proxied wasn't called often enough for this to be a concern.
Presumably the GC can handle the "wasted" space.

*Warning*: I've just begun to test this. Please use caution. It will be good to hear reports from others about how this works for them.

[Tom Fleck (tjf-ocsystems)]

Thanks EKR. We've been getting the same thing with 2.6.4 on linux on all our code.
I tried your change, but got an exception elsewhere whose traceback ended at

    for infered in callee.infer_call_result(self, callcontext):
  File "/usr/local/lib/python2.6/site-packages/logilab/astng/infutils.py", line
298, in infer_call_result
    for node in self._proxied.igetattr('__call__', context):
AttributeError: 'NoneType' object has no attribute 'igetattr'

Myself, I think the code really does intend to use a different variable __proxied but the call to hasattr() causes recursion. I just changed it as follows:

def _set_proxied(const):
    return _CONST_PROXY[const.value.__class__]

and recursion and above problem went away. Not to say this is the right fix, just that it's quiet now.
.

[Edward K. Ream (edreamleo)]

> Thanks EKR.

You're welcome. Tracking down this bug was fun.

> Myself, I think the code really does intend to use a different variable __proxied but the call to hasattr() causes recursion. I just changed it as follows:

def _set_proxied(const):
    return _CONST_PROXY[const.value.__class__]

This works for me, and it seems like a more elegant solution.

Edward

[sthenault (sylvain-thenault)]

I've checked in that patch to upstream repository, thank you guys

[darkpotpot (julien-gamr7)]

Unfortunately it's not solving the issue with Panda3D :/ I still have the maximum recursion depth
For more info, see the second comment made by sthenault in this topic : http://www.logilab.org/ticket/19641

[sthenault (sylvain-thenault)]

huum, I've no time to check this. I'll reopen the ticket if this is confirmed.

does panda3d use some C-compiled extensions?

[darkpotpot (julien-gamr7)]

Yes. But I just checked the error message and it's not exactly the same :

  File "/usr/local/lib/python2.6/dist-packages/logilab_astng-0.19.3-py2.6.egg/logilab/astng/_nodes.py", line 117, in root
    return self.parent.root()
  File "/usr/local/lib/python2.6/dist-packages/logilab_astng-0.19.3-py2.6.egg/logilab/astng/_nodes.py", line 117, in root
    return self.parent.root()
RuntimeError: maximum recursion depth exceeded

So it must be another bug.

[Pander (pander)]

Is this still a bug that needs fixing?