bin/test --load-list does not preserve test order
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Launchpad itself |
Fix Released
|
High
|
Brad Crittenden |
Bug Description
When trying to duplicate intermittent test failures to see if they are related to the tests that ran before them, we have branches to let us get the tests run on a particular worker (see bug 664171) from the subunit output. Once you have the list, you want to run the tests in that order.
The supported way to run the tests from a list of them is using `bin/test --load-list LIST_NAME` where LIST_NAME is a file of test names separated by newlines. Unfortunately, after the tests are aggregated into layers (expected, and should mirror what happened in the previous test run), they are sorted alphabetically. The only way to disable the alphabetization is to use --shuffle, which randomizes the list. There is no way to maintain the original order specified in the list of test names.
Maintaining the original order is the only way to duplicate the test run exactly. We should be able to do this as one of our diagnostic tools.
This would be a change to our zope.testing fork.
Related branches
- Gary Poster (community): Approve
-
Diff: 61 lines (+24/-9)1 file modifiedlib/lp/services/testing/customresult.py (+24/-9)
Changed in launchpad: | |
assignee: | nobody → Brad Crittenden (bac) |
status: | Triaged → In Progress |
Changed in launchpad: | |
status: | Fix Committed → Fix Released |
Fixed in stable r15186 (http:// bazaar. launchpad. net/~launchpad- pqm/launchpad/ stable/ revision/ 15186) by a commit, but not testable.