testr can have confusing behaviour when a test id is repeated during a stream given to testr load

[Brad Crittenden]

The attached log file is a value subunit stream which 'testr load' identifies as having one failure, but 'testr failing' does not show the failure afterwards.

The cause of this is this:

subunit-ls < .testrepository/0 | grep distribution_txt
distribution_txt
distribution_txt

The test id is being reused and test ids are treated as primary keys.

There are a few options that testr can do better at:
 - it could warn on duplicate test ids (they are generally bad)
 - it could add a unique prefix/suffix for its own handling but still use the original when running test commands.
 - ???

Of the three warning is easiest and possibly sufficient.

[Robert Collins (lifeless)]

Works for me - the load line shows:

....

ZopelessLayer.testSetUp 2445 calls taking 1.9s.
ZopelessLayer.testTearDown 2445 calls taking negligible time.

Total duration of profiled methods 2284.9 seconds.

Total duration of test run 12725.0 seconds.

Tests failed (exit code 1)
make: *** [check] Error 1
======================================================================
FAIL: distribution_txt
----------------------------------------------------------------------
Text attachment: traceback
------------
Traceback (most recent call last):
  File "/usr/lib/python2.6/unittest.py", line 279, in run
    testMethod()
  File "/usr/lib/python2.6/doctest.py", line 2152, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for distribution.txt
  File "lib/lp/soyuz/tests/../doc/distribution.txt", line 0

----------------------------------------------------------------------
File "lib/lp/soyuz/tests/../doc/distribution.txt", line 86, in distribution.txt
Failed example:
    gentoo.newSeries('gentoo-two', 'Gentoo Two',
                     'Gentoo Two Dot Oh', 'Gentoo 2', 'G2',
                     '2.0', None, gentoo.owner)
Differences (ndiff with -expected +actual):
      Traceback (most recent call last):
    - ...
    + File "/usr/lib/python2.6/doctest.py", line 1248, in __run
    + compileflags, 1) in test.globs
    + File "<doctest distribution.txt[19]>", line 1, in <module>
    + gentoo.newSeries('gentoo-two', 'Gentoo Two',
    - Unauthorized: (<Distribution...>, 'newSeries', 'launchpad.Append')
    ? ^^^ ^^^ ^^
    + Unauthorized: (<Distribution 'Gentoo' (gentoo)>, 'newSeries', 'launchpad.Moderate')
    ? ^^^^^^^^^^^^^^^^^^ ^^^ ^^^^

------------
id: 0 tests: 9540 failures: 1 skips: 23

[Robert Collins (lifeless)]

What version of testr are you using?

[Brad Crittenden (bac)]

Hi Robert,

I'm using testrepository 0.0.3-1ubuntu1

The 'testr load' and 'testr last' commands show the failure and summarize with:

id: 0 tests: 9540 failures: 1 skips: 23

So it recognizes one failure.

I would expect 'testr failing' to show that same failure. Instead it produces no output.

Even with the caveat given in 'testr help failing' I would expect 'testr failing' to show the same as 'testr last' in this situation.