ValueError: Length too long _write_packet

[Attila Fazekas]

https://bazaar.launchpad.net/~subunit/subunit/master/view/head:/python/subunit/v2.py#L202 Handling the longer than 4M packets is not implemented causing exceptions like:

Traceback (most recent call last):
  File "/bin/ostestr", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/os_testr/ostestr.py", line 263, in main
    exit(ostestr(sys.argv[1:]))
  File "/usr/lib/python2.7/site-packages/os_testr/ostestr.py", line 259, in ostestr
    return _select_and_call_runner(opts, regex, others)
  File "/usr/lib/python2.7/site-packages/os_testr/ostestr.py", line 212, in _select_and_call_runner
    black_regex=opts.black_regex)
  File "/usr/lib/python2.7/site-packages/os_testr/ostestr.py", line 167, in call_testr
    black_regex=black_regex)
  File "/usr/lib/python2.7/site-packages/stestr/commands/run.py", line 319, in run_command
    abbreviate=abbreviate)
  File "/usr/lib/python2.7/site-packages/stestr/commands/run.py", line 389, in _run_tests
    return run_tests()
  File "/usr/lib/python2.7/site-packages/stestr/commands/run.py", line 386, in run_tests
    abbreviate=abbreviate)
  File "/usr/lib/python2.7/site-packages/stestr/commands/load.py", line 173, in load
    result.stopTestRun()
  File "/usr/lib/python2.7/site-packages/testtools/testresult/real.py", line 411, in stopTestRun
    domap(methodcaller('stopTestRun'), self.targets)
  File "/usr/lib/python2.7/site-packages/testtools/testresult/real.py", line 390, in domap
    return list(map(*args, **kwargs))
  File "/usr/lib/python2.7/site-packages/stestr/repository/file.py", line 325, in stopTestRun
    run.get_test().run(inserter)
  File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 1006, in run
    self._run(result, self.decorated.run)
  File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 1000, in _run
    return run_method(result)
  File "/usr/lib/python2.7/site-packages/subunit/__init__.py", line 1197, in run
    protocol.lineReceived(line)
  File "/usr/lib/python2.7/site-packages/subunit/__init__.py", line 567, in lineReceived
    self._state.lineReceived(line)
  File "/usr/lib/python2.7/site-packages/subunit/__init__.py", line 404, in lineReceived
    self.details_parser.lineReceived(line)
  File "/usr/lib/python2.7/site-packages/subunit/details.py", line 119, in lineReceived
    self._parse_state(line)
  File "/usr/lib/python2.7/site-packages/subunit/details.py", line 84, in _look_for_content
    self._state.endDetails()
  File "/usr/lib/python2.7/site-packages/subunit/__init__.py", line 399, in endDetails
    self._report_outcome()
  File "/usr/lib/python2.7/site-packages/subunit/__init__.py", line 430, in _report_outcome
    details=self.details_parser.get_details())
  File "/usr/lib/python2.7/site-packages/testtools/testresult/real.py", line 1157, in addFailure
    return self.decorated.addFailure(test, details=details)
  File "/usr/lib/python2.7/site-packages/testtools/testresult/real.py", line 1334, in addError
    self._convert(test, err, details, 'fail')
  File "/usr/lib/python2.7/site-packages/testtools/testresult/real.py", line 1358, in _convert
    mime_type=mime_type, test_id=test_id, timestamp=now)
  File "/usr/lib/python2.7/site-packages/testtools/testresult/real.py", line 415, in status
    domap(methodcaller('status', *args, **kwargs), self.targets)
  File "/usr/lib/python2.7/site-packages/testtools/testresult/real.py", line 390, in domap
    return list(map(*args, **kwargs))
  File "/usr/lib/python2.7/site-packages/testtools/testresult/real.py", line 516, in status
    target.status(**kwargs)
  File "/usr/lib/python2.7/site-packages/testtools/testresult/real.py", line 415, in status
    domap(methodcaller('status', *args, **kwargs), self.targets)
  File "/usr/lib/python2.7/site-packages/testtools/testresult/real.py", line 390, in domap
    return list(map(*args, **kwargs))
  File "/usr/lib/python2.7/site-packages/stestr/repository/memory.py", line 175, in status
    self._hook.status(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/testtools/testresult/real.py", line 415, in status
    domap(methodcaller('status', *args, **kwargs), self.targets)
  File "/usr/lib/python2.7/site-packages/testtools/testresult/real.py", line 390, in domap
    return list(map(*args, **kwargs))
  File "/usr/lib/python2.7/site-packages/subunit/v2.py", line 120, in status
    route_code=route_code, timestamp=timestamp)
  File "/usr/lib/python2.7/site-packages/subunit/v2.py", line 208, in _write_packet
    raise ValueError("Length too long: %r" % base_length)
ValueError: Length too long: 5426744

[XiaojueGuan (xiaojuegaun)]

I was facing the same error:

py37 run-test-pre: PYTHONHASHSEED='1697893785'
py37 run-test: commands[0] | stestr run
Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: oslo_context.context, urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
Exception ignored in: <function _after_fork at 0x7f96afe16dd0>
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 1373, in _after_fork
    assert len(_active) == 1
AssertionError:
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/steveguan/code/Openstack/nova/.tox/py37/lib/python3.7/site-packages/stestr/subunit_runner/run.py", line 99, in <module>
    main()
  File "/home/steveguan/code/Openstack/nova/.tox/py37/lib/python3.7/site-packages/stestr/subunit_runner/run.py", line 90, in main
    testRunner=partial(runner, stdout=sys.stdout))
  File "/home/steveguan/code/Openstack/nova/.tox/py37/lib/python3.7/site-packages/stestr/subunit_runner/program.py", line 190, in __init__
    runner.list(self.test, loader=self.testLoader)
  File "/home/steveguan/code/Openstack/nova/.tox/py37/lib/python3.7/site-packages/stestr/subunit_runner/run.py", line 66, in list
    mime_type="text/plain;charset=utf8")
  File "/home/steveguan/code/Openstack/nova/.tox/py37/lib/python3.7/site-packages/subunit/v2.py", line 120, in status
    route_code=route_code, timestamp=timestamp)
  File "/home/steveguan/code/Openstack/nova/.tox/py37/lib/python3.7/site-packages/subunit/v2.py", line 208, in _write_packet
    raise ValueError("Length too long: %r" % base_length)
ValueError: Length too long: 19626708

=========================
Failures during discovery
=========================

================================================================================
The above traceback was encountered during test discovery which imports all the found test modules in the specified test_path.
ERROR: InvocationError for command /home/steveguan/code/Openstack/nova/.tox/py37/bin/stestr run (exited with code 100)
____________________________________________________________________________________ summary _____________________________________________________________________________________
ERROR: py37: commands failed

[Jean-Paul Calderone (exarkun)]

Since subunit is supposed to represent my test results, it's a bit annoying that this arbitrary length limitation is imposed. If my tests produce output larger than this limit, I think that should be my business and subunit shouldn't get in the way.

That said, if subunit is going to impose this limit, it would be great if there were more visibility into the issue. Perhaps rather than discarding the whole value, some context could at least be included in the error. Right now, I can't tell what test provoked this behavior (it crashes my test runner and the test method isn't on the call stack). Nor can I tell which piece of output this is. If possible, it would be great if there were at least a truncated version of the output available.