zope.testing --subunit allows bad output on stdout, which can break subunit processing and tests
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Launchpad itself |
Fix Released
|
High
|
Graham Binns |
Bug Description
The parallel testing had a test run that only reported about 3000 tests even though the full 16000+ ran. This was because the stream it processed included the following test: http://
Obviously, """No handlers could be found for logger "lazr.smtptest"
This bug's title partially directs a solution by attributing responsibility. This comes from a discussion with Robert in which he said that subunit shouldn't be expected to handle malformed output of this sort. Similarly, we could look for all sources of these complaints in the Launchpad tests (probably from a __del__ method when something is being garbage collected) but that will be error-prone, difficult, and not future-proof.
Instead, the test runner itself ought to isolate the subunit output. We could do that either by making the subunit stream not go to stdout, or make everything else not go to stdout (e.g., "sys.stdout = sys.stderr"). Robert notes that the former solution has the disadvantage that it does not allow piping of subunit output, which is intended to be one of the primary user interfaces for the stream. To pursue the latter solution, we may have test breakage, now and in the future, but it preserves the subunit interface and he prefers it. He points out that diagnosis of future test breakage might be helped if we can turn the sys.stdout redirection to stderr off and on.
I'd prefer that the --subunit flag to the testrunner imply that the stdout isolation should occur; that is, by default when you ask for a subunit stream, you should get a clean subunit stream, with the stdout->stderr redirection for everything else writing to stdout from the tests. For debugging, you can simply not run with --subunit, which will not perform the redirection of stdout. Further, if you really want to use --subunit but disable the stdout/stderr switch, maybe we could have another switch (--disable-
This change should be made in our zope.testing fork.
Related branches
- Benji York (community): Approve (code)
- Diff: 0 lines
- Graham Binns (community): Approve (code)
-
Diff: 14 lines (+3/-1)1 file modifiedversions.cfg (+3/-1)
- Gary Poster (community): Approve
-
Diff: 93 lines (+17/-9)2 files modifiedlib/lp/services/tests/test_stacktrace.py (+8/-4)
versions.cfg (+9/-5)
Changed in launchpad: | |
assignee: | nobody → Graham Binns (gmb) |
status: | Triaged → In Progress |
summary: |
zope.testing --subunit allows bad output on stdout, which can break - subunit processing + subunit processing and tests |
Changed in launchpad: | |
status: | In Progress → Fix Committed |
status: | Fix Committed → In Progress |
tags: | added: qa-untestable |
Changed in launchpad: | |
status: | In Progress → Fix Committed |
Changed in launchpad: | |
status: | Fix Committed → In Progress |
tags: |
added: qa-untestable removed: qa-needstesting |
Changed in launchpad: | |
status: | Fix Committed → Fix Released |
Another idea from Robert, which would be cool: "make the replacement sys.stdout grab a traceback and report the object triggering the write--e.g. a wrapper around sys.stderr with a custom write() + writelines() methods"