Zope layer setup and teardown 'tests' cannot be filtered by testr

[Graham Binns]

What happens:
If you run the Launchpad tests with --subunit, it will report the layer setup and teardown notices as tests, which they aren't. This leads to test count inflation, which gets very confusing for developers (especially when you're running parallel tests, which means that you add about 20 such notices per lxc instance).

Proposed solution (from lifeless):
 - Update the zope.testing subunit support so that layer setup and teardown messages are tagged with "layer".
    - already the case
 - Subunit needs to know about tags.
    - already the case
 - Testr needs to not count or record layer-tagged things as tests.
    - needs to be configurable, and is pending
 - Testr should still dump the layer-tagged messages to the raw stream on disk.
    - same as above

Sub-bugs:
 bug 914281 - noop - case of 518016
 bug 914279 - fixed
 bug 914278 - fixed
 bug 518016 - fixed

[Jonathan Lange (jml)]

On Tue, Jan 10, 2012 at 9:06 AM, Graham Binns <email address hidden> wrote:
> Public bug reported:
>
> What happens:
> If you run the Launchpad tests with --subunit, it will report the layer setup and teardown notices as tests, which they aren't. This leads to test count inflation, which gets very confusing for developers (especially when you're running parallel tests, which means that you add about 20 such notices per lxc instance).
>
> Proposed solution (from lifeless):
>  - Update the zope.testing subunit support so that layer setup and teardown messages are tagged with "layer".
>  - Subunit needs to know about tags.
>  - Testr needs to not count or record layer-tagged things as tests.
>  - Testr should still dump the layer-tagged messages to the raw stream on disk.

Some thoughts:

 * zope.testing already outputs a 'zope:layer' tag after emitting the
start of the test that represents the layer
 * It treats layers as tests to provide a place to put layer
setUp/tearDown errors
 * It would be nicer for testr to not know about layers, but rather to
achieve the same result with (not yet existing) subunit filtering
tools

jml

[Graham Binns (gmb)]

On 10 January 2012 10:29, Jonathan Lange <email address hidden> wrote:
> Some thoughts:
>
>  * zope.testing already outputs a 'zope:layer' tag after emitting the
> start of the test that represents the layer
>  * It treats layers as tests to provide a place to put layer
> setUp/tearDown errors
>  * It would be nicer for testr to not know about layers, but rather to
> achieve the same result with (not yet existing) subunit filtering
> tools

Yeah, we just found that :). Thanks.