bin/test --load-list does not preserve test order

Reported by Gary Poster on 2012-04-26
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
High
Brad Crittenden

Bug Description

When trying to duplicate intermittent test failures to see if they are related to the tests that ran before them, we have branches to let us get the tests run on a particular worker (see bug 664171) from the subunit output. Once you have the list, you want to run the tests in that order.

The supported way to run the tests from a list of them is using `bin/test --load-list LIST_NAME` where LIST_NAME is a file of test names separated by newlines. Unfortunately, after the tests are aggregated into layers (expected, and should mirror what happened in the previous test run), they are sorted alphabetically. The only way to disable the alphabetization is to use --shuffle, which randomizes the list. There is no way to maintain the original order specified in the list of test names.

Maintaining the original order is the only way to duplicate the test run exactly. We should be able to do this as one of our diagnostic tools.

This would be a change to our zope.testing fork.

Related branches

lp:~bac/launchpad/ordered-load-list
Merged into lp:launchpad at revision 15186
Gary Poster (community): Approve on 2012-05-01
Brad Crittenden (bac) on 2012-05-01
Changed in launchpad:
assignee: nobody → Brad Crittenden (bac)
status: Triaged → In Progress
Launchpad QA Bot (lpqabot) wrote :

Fixed in stable r15186 (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15186) by a commit, but not testable.

tags: added: qa-untestable
Changed in launchpad:
status: In Progress → Fix Committed
William Grant (wgrant) on 2012-05-03
Changed in launchpad:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers