No filename displayed when exception stops testrunner

[Stanislaw Pitucha]

If the exception suddenly stops the test runner, there's no relevant filename displayed. Additionally, exception could be queued for display at the end and bandit could attempt processing the remaining files.

Example output:

50.. 100.. 150.. 200.. 250.. 300.. 350.. Traceback (most recent call last):
  File ".tox/py27/bin/bandit", line 10, in <module>
    sys.exit(main())
  File "/home/viraptor/Projects/bandit/bandit/bandit.py", line 230, in main
    b_mgr.run_tests()
  File "/home/viraptor/Projects/bandit/bandit/core/manager.py", line 223, in run_tests
    fname, fdata, self.b_ma, self.b_ts)
  File "/home/viraptor/Projects/bandit/bandit/core/manager.py", line 256, in _execute_ast_visitor
    score = res.process(fdata)
.....
  File "/home/viraptor/Projects/bandit/bandit/core/node_visitor.py", line 307, in generic_visit
    self.visit(node=value)
  File "/home/viraptor/Projects/bandit/bandit/core/node_visitor.py", line 275, in visit
    self.context['linerange'] = b_utils.linerange_fix(node)
  File "/home/viraptor/Projects/bandit/bandit/core/utils.py", line 312, in linerange_fix
    lines = linerange(node)
  File "/home/viraptor/Projects/bandit/bandit/core/utils.py", line 295, in linerange
    for n in ast.walk(node):
  File "/usr/lib/python2.7/ast.py", line 213, in walk
    todo.extend(iter_child_nodes(node))
  File "/usr/lib/python2.7/ast.py", line 178, in iter_child_nodes
    for name, field in iter_fields(node):
RuntimeError: maximum recursion depth exceeded

This doesn't indicate in which file did the exception happen.

[Ian Cordasco (icordasc)]

This is because we don't catch a generic error (or RuntimeError specifically) when analyzing files. We need to find the right level to catch a generic exception such that we can find the file we were analyzing.

[Jamie Finnigan (jamiefinnigan)]

Another example of this happening. Exception occurs while scanning a file, but not indicator of which file.

  $ bandit -r .
  [main] INFO profile include tests: None
  [main] INFO profile exclude tests: None
  [main] INFO cli include tests: None
  [main] INFO cli exclude tests: None
  [main] INFO running on Python 2.7.10
  28993 [0.. Traceback (most recent call last):
    File "/home/foo/source/hpe/bandit/bin/bandit", line 11, in <module>
      sys.exit(main())
    File "/home/foo/source/hpe/bandit/local/lib/python2.7/site-packages/bandit/cli/main.py", line 335, in main
      b_mgr.run_tests()
    File "/home/foo/source/hpe/bandit/local/lib/python2.7/site-packages/bandit/core/manager.py", line 238, in run_tests
      nosec_lines)
    File "/home/foo/source/hpe/bandit/local/lib/python2.7/site-packages/bandit/core/manager.py", line 274, in _execute_ast_visitor
      score = res.process(data)
    File "/home/foo/source/hpe/bandit/local/lib/python2.7/site-packages/bandit/core/node_visitor.py", line 278, in process
    f_ast = ast.parse(data)
    File "/usr/lib/python2.7/ast.py", line 37, in parse
      return compile(source, filename, mode, PyCF_ONLY_AST)
  TypeError: compile() expected string without null bytes

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/304279

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/304279
Committed: https://git.openstack.org/cgit/openstack/bandit/commit/?id=b6c8b9f01b4b0ac56ac6d8474852ee1bc5908543
Submitter: Jenkins
Branch: master

commit b6c8b9f01b4b0ac56ac6d8474852ee1bc5908543
Author: Jamie Finnigan <email address hidden>
Date: Mon Apr 11 11:26:58 2016 -0700

    Catch general exception on per-file basis

    This modifies the Bandit manager to catch a general Exception on a
    per-file basis. When an exception does occur, the name of the file is
    emitted and the file is logged as a 'skipped file' for inclusion in
    the end-of-run output. When run in debug mode, a traceback will also
    be printed.

    The change also adds a new test targeting this case, along with a new
    example file (nonsense2.py is gzipped nonsense.py) to trigger the
    test.

    Change-Id: I86e648890dddcc5c2fff7dd9844678e990b0cd63
    Closes-Bug: #1498258