pylint spews Exception RuntimeError: maximum recursion depth exceeded while calling a Python object

Bug #456870 reported by Forest
60
This bug affects 12 people
Affects Status Importance Assigned to Milestone
pylint
Invalid
Undecided
Unassigned
pylint (Ubuntu)
Undecided
Unassigned

Bug Description

When operating on a file that calls email.message.Message.get(), pylint 0.15.2-1 spews repeated copies of this error message:

Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in <type 'exceptions.AttributeError'> ignored

Here is a test program that triggers pylint's complaints:

import email.parser
parser = email.parser.HeaderParser()
message = parser.parsestr( '')
message.get('From')

Revision history for this message
Forest (foresto) wrote :

I tried to submit a bug report on logilab.org, but I don't see a way to do so.
http://www.logilab.org/857

affects: launchpad-foundations → pylint
Changed in pylint:
status: New → Invalid
status: Invalid → New
Revision history for this message
afayolle (alexandre-fayolle) wrote :

Hi,

I cannot reproduce this bug with an up-to-date version of pylint and astng. Simplest way to fix it is to upgrade.

Btw, easiest way of communicating with pylint maintainers is to subscribe to the mailing list (http://lists.logilab.org/mailman/listinfo/python-projects) and send a mail to <email address hidden>.

Thanks for your interest in Pylint

--
Alexandre

Revision history for this message
Edward K. Ream (edreamleo) wrote :

I am getting this problem with all files of my Leo sources on Ubuntu 9.10 Karmic

pylint 0.18.0, (latest available package)
astng 0.19.0, common 0.39.0
Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
[GCC 4.4.1]

For example, from Leo's bzr repository: (the top-level directory):

pylint leo/core/leoFileCommands.py > pylint.out

Messages to console, but not to pylint.out.

I'd call this a show-stopper. Please do something about this.

Edward K. Ream

Revision history for this message
Dane Elwell (dane-elwell) wrote :

If the simplest solution is to upgrade, can we upgrade the version in the repositories?

Revision history for this message
Eric Pierce (wapcaplet88) wrote :

I'm getting this error in Gentoo with these versions:

pylint-0.19.0
astng-0.19.3
python-2.6.4

Pylint still seems to work fine, but this error is a little annoying.

Revision history for this message
Edward K. Ream (edreamleo) wrote :

I get this on all my files on Windows as well:

pylint 0.19.0,
astng 0.19.3, common 0.46.0
Python 2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 32 bit (Intel)]

I do not believe pylint is working as well as it should. It is missing serious errors, for example, undefined methods.

Revision history for this message
Forest (foresto) wrote :

I installed pylint 0.19 (from ubuntu lucid packages) and it is still spewing repeated copies of that error message. It no longer does it with the test program I included in my first comment, but it does with plenty of others. I'm attaching a different program that triggers the misbehavior.

Revision history for this message
Forest (foresto) wrote :
Revision history for this message
Forest (foresto) wrote :

I tried to add my findings to that bug report, but despite having logged in at logilab, their bug report page keeps presenting me with the "register or log in message". How frustrating. If anyone from the pylint/astng projects is reading, could you please add to that bug report the slightly different error message I reported here, as well as the test code I just attached?

Also, if a logilab.org site owner is reading, please consider improving usability on your site. It's rather difficult to figure out how to file a bug report, and apparently impossible to comment on one.

Revision history for this message
Forest (foresto) wrote :

Here's the bug report i was referring to:

http://www.logilab.org/ticket/19641

Forest (foresto)
summary: pylint spews Exception RuntimeError: maximum recursion depth exceeded
- while calling a Python object, when parsing email.message.Message.get()
+ while calling a Python object
Revision history for this message
Edward K. Ream (edreamleo) wrote :
Download full text (9.4 KiB)

I can now demonstrate exactly where the unbound recursion happens.

The following is a debugger that warns me when the stack gets too big. It is a subclass of pdb, but it overrides two methods in bdb.py. Here it is::

class debugger(pdb.Pdb):

    def __init__(self,completekey='tab',stdin=None,stdout=None):
        pdb.Pdb.__init__(self,completekey,stdin,stdout)
        self.stackLimit = 200 # Found by trial and error.

    # dispatch_call (override bdb)
    def dispatch_call(self, frame, arg):
        # XXX 'arg' is no longer used

        # EKR: new code.
        stack,junk = self.get_stack(frame,None)
        if len(stack) > self.stackLimit:
            print('(debugger.stop_here) stackCount %s' % len(stack))
            self.interaction(frame,None)

        # Existing code.
        if self.botframe is None:
            # First call of dispatch since reset()
            self.botframe = frame.f_back # (CT) Note that this may also be None!
            return self.trace_dispatch
        if not (self.stop_here(frame) or self.break_anywhere(frame)):
            # No need to trace this function
            return # None
        self.user_call(frame, arg)
        if self.quitting: raise BdbQuit
        return self.trace_dispatch

    # set_continue (override bdb)
    def set_continue(self):

        # Don't stop except at breakpoints or when finished
        self._set_stopinfo(self.botframe, None)

        # EKR: Always run with overhead
        if False:
            if not self.breaks:
                # no breakpoints; run without debugger overhead
                sys.settrace(None)
                frame = sys._getframe().f_back
                while frame and frame is not self.botframe:
                    del frame.f_trace
                    frame = frame.f_back

And here is the stack trace when the stack limit tripped. It includes some print statements I inserted while stepping through the code.

Breakpoint 1 at c:\python26\lib\site-packages\logilab_astng-0.19.3-py2.6.egg\logilab\astng\node_classes.py:256
> c:\python26\lib\site-packages\pylint-0.19.0-py2.6.egg\pylint\lint.py(782)__init__()
-> self._rcfile = None
(Pdb) disable 1
(Pdb) c
manager.py: astng_from_file: not in cache: c:\leo.repo\until\pylintTest.py
manager.py: astng_from_file: calling file_build(pylintTest)
manager.py: astng_from_file: not in cache: c:\leo.repo\until\leo\__init__.py
manager.py: astng_from_file: calling file_build(leo)
manager.py: astng_from_file: not in cache: c:\leo.repo\until\leo\core\__init__.py
manager.py: astng_from_file: calling file_build(leo.core)
manager.py: astng_from_file: not in cache: c:\leo.repo\until\leo\core\leoGlobals.py
manager.py: astng_from_file: calling file_build(leo.core.leoGlobals)
manager.py: astng_from_file: not in cache: c:\leo.repo\until\leo\core\leoApp.py
manager.py: astng_from_file: calling file_build(leo.core.leoApp)
(debugger.stop_here) stackCount 201
> c:\python26\lib\site-packages\logilab_astng-0.19.3-py2.6.egg\logilab\astng\inference.py(52)_set_proxied()
-> def _set_proxied(const):
(Pdb) w
('pdb.py:print_stack_trace', 201)
  c:\python26\lib\bdb.py(404)runcall()
-> res = func(*args, **kwds)
  ...

Read more...

Revision history for this message
Edward K. Ream (edreamleo) wrote :

The failing code appears to be _set_proxied in inference.py.

def _set_proxied(const):

    if not hasattr(const, '__proxied'):
        const.__proxied = _CONST_PROXY[const.value.__class__]
    return const.__proxied
nodes.Const._proxied = property(_set_proxied)

The call to hasattr can call Proxy.__getattr__ in infutils.py:

def __getattr__(self, name):
        if name == '_proxied':
            return getattr(self.__class__, '_proxied')
        if name in self.__dict__:
            return self.__dict__[name]
        return getattr(self._proxied, name)

As the trace shows, the last line of __getattr__ can invoke _setProxied, apparently directly.

My head is about to explode :-) In particular, the const argument to _set_proxied apparently represents None without being None. The debugger reports const to be:

<Instance of __builtin__.NoneType at (an address)>

but "const is None" evaluates to False. Ditto for various other tries, such as isinstance(const,something-or-other)

As if that were not enough, did you notice _proxied vs. __proxied?

I do hope somebody who actually understand this code will fix it. To repeat, pylint has been failing to catch serious bugs in my app as a result of this problem. I've pretty much lost confidence in pylint for now.

Edward

Revision history for this message
Edward K. Ream (edreamleo) wrote :

The following hack seems to repair the damage. Change:

def _set_proxied(const):
    if not hasattr(const, '__proxied'):
        const.__proxied = _CONST_PROXY[const.value.__class__]
    return const.__proxied

To:

def _set_proxied(const):
    const._proxied = _CONST_PROXY[const.value.__class__]
    return const._proxied

As you can see, there are two major changes to the code:

1. The code uses _proxied instead of __proxied.

I don't see any other uses of __proxied anywhere,
so this would seem like what was originally intended,
but there are absolutely no guarantees.

2. The code always sets const._proxied.

In my test, _set_proxied wasn't called often enough for this to be a concern.
Presumably the GC can handle the "wasted" space.

*Warning*: I've just begun to test this. Please use caution. It will be good to hear reports from others about how this works for them.

Revision history for this message
Tom Fleck (tjf-ocsystems) wrote :

Thanks EKR. We've been getting the same thing with 2.6.4 on linux on all our code.
I tried your change, but got an exception elsewhere whose traceback ended at

    for infered in callee.infer_call_result(self, callcontext):
  File "/usr/local/lib/python2.6/site-packages/logilab/astng/infutils.py", line
298, in infer_call_result
    for node in self._proxied.igetattr('__call__', context):
AttributeError: 'NoneType' object has no attribute 'igetattr'

Myself, I think the code really does intend to use a different variable __proxied but the call to hasattr() causes recursion. I just changed it as follows:

def _set_proxied(const):
    return _CONST_PROXY[const.value.__class__]

and recursion and above problem went away. Not to say this is the right fix, just that it's quiet now.
.

Revision history for this message
Edward K. Ream (edreamleo) wrote :

> Thanks EKR.

You're welcome. Tracking down this bug was fun.

> Myself, I think the code really does intend to use a different variable __proxied but the call to hasattr() causes recursion. I just changed it as follows:

def _set_proxied(const):
    return _CONST_PROXY[const.value.__class__]

This works for me, and it seems like a more elegant solution.

Edward

Revision history for this message
sthenault (sylvain-thenault) wrote :

I've checked in that patch to upstream repository, thank you guys

Revision history for this message
darkpotpot (julien-gamr7) wrote :

Unfortunately it's not solving the issue with Panda3D :/ I still have the maximum recursion depth
For more info, see the second comment made by sthenault in this topic : http://www.logilab.org/ticket/19641

Revision history for this message
sthenault (sylvain-thenault) wrote :

huum, I've no time to check this. I'll reopen the ticket if this is confirmed.

does panda3d use some C-compiled extensions?

Revision history for this message
darkpotpot (julien-gamr7) wrote :

Yes. But I just checked the error message and it's not exactly the same :

  File "/usr/local/lib/python2.6/dist-packages/logilab_astng-0.19.3-py2.6.egg/logilab/astng/_nodes.py", line 117, in root
    return self.parent.root()
  File "/usr/local/lib/python2.6/dist-packages/logilab_astng-0.19.3-py2.6.egg/logilab/astng/_nodes.py", line 117, in root
    return self.parent.root()
RuntimeError: maximum recursion depth exceeded

So it must be another bug.

Nikolaus Rath (nikratio)
Changed in pylint (Ubuntu):
status: New → Confirmed
Changed in pylint:
importance: Undecided → Unknown
status: New → Unknown
Revision history for this message
Pander (pander) wrote :

Is this still a bug that needs fixing?

Pander (pander)
tags: added: 15.10 python
Pander (pander)
Changed in pylint (Ubuntu):
status: Confirmed → Invalid
Changed in pylint:
importance: Unknown → Undecided
status: Unknown → New
importance: Undecided → Unknown
status: New → Unknown
importance: Unknown → Undecided
status: Unknown → New
status: New → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.