testr reports wrong work # on tests some times

[Sean Dague]

Using the new subunit-trace.py filter in tempest we've discovered that some times the worker # for a subunit stream is incorrect. On a 4 worker environment we can often get 1, 2, 3, 3 (noting no 0) as reported workers for the tests. There is an attached tempest partial run with subunit test lists, and subunit streams demonstrating this.

[Matt Riedemann (mriedem)]

This shows up a lot: http://goo.gl/IZi6pO

[Robert Collins (lifeless)]

Hi, thanks for filing this. The trace you got is great, but its missing what testrepository saved to disk, so I can't see whats wrong :(

[Robert Collins (lifeless)]

To reproduce, in the testrepository tree itself run

./testr run --parallel --subunit | ~/work/openstack/tempest/tools/subunit-trace.py

 - WARNING: missing Worker 0! Race in testr accounting.
 - Worker 1 (170 tests) => 0:00:01.895823s
 - WARNING: missing Worker 2! Race in testr accounting.
 - Worker 3 (174 tests) => 0:00:01.917335s

[Robert Collins (lifeless)]

I am thinking this is a testtools issue -

greedy evaluating the tests iterator in ConcurrentStreamTestSuite triggers

 - WARNING: missing Worker 0! Race in testr accounting.
 - WARNING: missing Worker 1! Race in testr accounting.
 - WARNING: missing Worker 2! Race in testr accounting.
 - Worker 3 (344 tests) => 0:00:01.939864s

reliably - 100% of the time

[Robert Collins (lifeless)]

Ok, StreamTagger is being constructed with bad data by the make tests lambda:

        super(StreamTagger, self).__init__(targets)
        self.add = frozenset(add or ())
        self.discard = frozenset(discard or ())
        print self.add

frozenset(['worker-3'])
frozenset(['worker-3'])
frozenset(['worker-3'])
frozenset(['worker-3'])