AST parsing hits recursion limit

[David Wyde]

The parsing of certain ASTs will hit Python's recursion limit. This happens if AST nodes are nested to a depth greater than "sys.getrecursionlimit()", which defaults to 1000 for me.

A simple example:

    python -c 'print ("+" * 1000) + "1"' > unary.py
    bandit unary.py

When this occurs, Bandit will report an exception:

    Exception RuntimeError: RuntimeError('maximum recursion depth exceeded',) in <generator object iter_fields at 0x108ab2eb0> ignored
    [manager] ERROR Exception occurred when executing tests against /Users/dwyde/test/unary.py. Run "bandit --debug /Users/dwyde/test/unary.py" to see the full traceback.

The start of the traceback with --debug:

    [manager] DEBUG Exception string: maximum recursion depth exceeded while calling a Python object
    [manager] DEBUG Exception traceback: Traceback (most recent call last):
      File "/Users/dwyde/code/bandit/bandit/core/manager.py", line 259, in _parse_file
        score = self._execute_ast_visitor(fname, data, nosec_lines)
      File "/Users/dwyde/code/bandit/bandit/core/manager.py", line 290, in _execute_ast_visitor
        score = res.process(data)
      File "/Users/dwyde/code/bandit/bandit/core/node_visitor.py", line 278, in process
        self.generic_visit(f_ast)
      File "/Users/dwyde/code/bandit/bandit/core/node_visitor.py", line 242, in generic_visit
        if self.pre_visit(item):
      File "/Users/dwyde/code/bandit/bandit/core/node_visitor.py", line 187, in pre_visit
        LOG.debug(ast.dump(node))
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 110, in dump
        return _format(node)
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 94, in _format
        fields = [(a, _format(b)) for a, b in iter_fields(node)]
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 94, in _format
        fields = [(a, _format(b)) for a, b in iter_fields(node)]

Found when running Bandit against
https://bitbucket.org/logilab/astroid/src/1.4.0/astroid/tests/testdata/python2/data/joined_strings.py

[Travis McPeak (travis-mcpeak)]

This is a cool bug but I'm not sure what to do about it.

Seems like Bandit is probably doing the best it can in this case?

[David Wyde (david-wyde)]

I'm not sure what to do about this one. My first thought was to catch an exception, but pre-3.5 it's going to be a RuntimeError. Identifying the recursion error by exception string might be a bit goofy.

Note that the same idea also causes an error in an SQL-checking plugin (with strings and fewer repetitions):

$ python -c 'print "+".join(["\"a\""] * 600)' > repeat.py
$ bandit repeat.py
[main] INFO profile include tests: None
[main] INFO profile exclude tests: None
[main] INFO cli include tests: None
[main] INFO cli exclude tests: None
[main] INFO running on Python 2.7.10
[node_visitor] INFO Unable to find qualified name for module: repeat.py
[tester] ERROR Bandit internal error running: hardcoded_sql_expressions on file repeat.py at line 1: maximum recursion depth exceeded in cmpTraceback (most recent call last):
  File "/Users/dwyde/bandit/bandit/core/tester.py", line 64, in run_tests
    result = test(context)
  File "/Users/dwyde/bandit/bandit/plugins/injection_sql.py", line 92, in hardcoded_sql_expressions
    val = _evaluate_ast(context.node)
  File "/Users/dwyde/bandit/bandit/plugins/injection_sql.py", line 81, in _evaluate_ast
    out = utils.concat_string(node, node.parent)
  File "/Users/dwyde/bandit/bandit/core/utils.py", line 270, in concat_string
    _get(node, bits, stop)
  File "/Users/dwyde/bandit/bandit/core/utils.py", line 259, in _get
    if isinstance(node.left, ast.BinOp)
  File "/Users/dwyde/bandit/bandit/core/utils.py", line 259, in _get
    if isinstance(node.left, ast.BinOp)
  File "/Users/dwyde/bandit/bandit/core/utils.py", line 259, in _get
    if isinstance(node.left, ast.BinOp)
  File "/Users/dwyde/bandit/bandit/core/utils.py", line 259, in _get
    if isinstance(node.left, ast.BinOp)
...
  File "/Users/dwyde/bandit/bandit/core/utils.py", line 256, in _get
    if node != stop:
RuntimeError: maximum recursion depth exceeded in cmp