testr can have confusing behaviour when a test id is repeated during a stream given to testr load

Bug #620388 reported by Brad Crittenden on 2010-08-19
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Testrepository
Wishlist
Unassigned

Bug Description

The attached log file is a value subunit stream which 'testr load' identifies as having one failure, but 'testr failing' does not show the failure afterwards.

The cause of this is this:

subunit-ls < .testrepository/0 | grep distribution_txt
distribution_txt
distribution_txt

The test id is being reused and test ids are treated as primary keys.

There are a few options that testr can do better at:
 - it could warn on duplicate test ids (they are generally bad)
 - it could add a unique prefix/suffix for its own handling but still use the original when running test commands.
 - ???

Of the three warning is easiest and possibly sufficient.

Brad Crittenden (bac) wrote :
Robert Collins (lifeless) wrote :

Works for me - the load line shows:

....

ZopelessLayer.testSetUp 2445 calls taking 1.9s.
ZopelessLayer.testTearDown 2445 calls taking negligible time.

Total duration of profiled methods 2284.9 seconds.

Total duration of test run 12725.0 seconds.

Tests failed (exit code 1)
make: *** [check] Error 1
======================================================================
FAIL: distribution_txt
----------------------------------------------------------------------
Text attachment: traceback
------------
Traceback (most recent call last):
  File "/usr/lib/python2.6/unittest.py", line 279, in run
    testMethod()
  File "/usr/lib/python2.6/doctest.py", line 2152, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for distribution.txt
  File "lib/lp/soyuz/tests/../doc/distribution.txt", line 0

----------------------------------------------------------------------
File "lib/lp/soyuz/tests/../doc/distribution.txt", line 86, in distribution.txt
Failed example:
    gentoo.newSeries('gentoo-two', 'Gentoo Two',
                     'Gentoo Two Dot Oh', 'Gentoo 2', 'G2',
                     '2.0', None, gentoo.owner)
Differences (ndiff with -expected +actual):
      Traceback (most recent call last):
    - ...
    + File "/usr/lib/python2.6/doctest.py", line 1248, in __run
    + compileflags, 1) in test.globs
    + File "<doctest distribution.txt[19]>", line 1, in <module>
    + gentoo.newSeries('gentoo-two', 'Gentoo Two',
    - Unauthorized: (<Distribution...>, 'newSeries', 'launchpad.Append')
    ? ^^^ ^^^ ^^
    + Unauthorized: (<Distribution 'Gentoo' (gentoo)>, 'newSeries', 'launchpad.Moderate')
    ? ^^^^^^^^^^^^^^^^^^ ^^^ ^^^^

------------
id: 0 tests: 9540 failures: 1 skips: 23

Changed in testrepository:
status: New → Incomplete
Robert Collins (lifeless) wrote :

What version of testr are you using?

Brad Crittenden (bac) wrote :

Hi Robert,

I'm using testrepository 0.0.3-1ubuntu1

The 'testr load' and 'testr last' commands show the failure and summarize with:

id: 0 tests: 9540 failures: 1 skips: 23

So it recognizes one failure.

I would expect 'testr failing' to show that same failure. Instead it produces no output.

Even with the caveat given in 'testr help failing' I would expect 'testr failing' to show the same as 'testr last' in this situation.

summary: - testr lies with some ec2 log files
+ sometimes testr does not record failure in .testrepository/failing
summary: - sometimes testr does not record failure in .testrepository/failing
+ sometimes testr does not record failure in .testrepository/failing when
+ test id is reused
description: updated
summary: - sometimes testr does not record failure in .testrepository/failing when
- test id is reused
+ testr should warn when a test id is repeated in a load operation
summary: - testr should warn when a test id is repeated in a load operation
+ testr can have confusing behaviour when a test id is repeated during a
+ stream given to testr load
description: updated
description: updated
Brad Crittenden (bac) on 2010-11-29
Changed in testrepository:
status: Incomplete → Confirmed
Changed in testrepository:
status: Confirmed → Triaged
importance: Undecided → Wishlist
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments