test regressions should be calculated per-test, not per-package

Bug #1767237 reported by Steve Langasek on 2018-04-27
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Auto Package Testing
Wishlist
Unassigned
britney
Wishlist
Unassigned

Bug Description

currently the britney autopkgtest policy calculates whether a test failure is a regression on a per-package basis: if there has previously been a successful test run of the tests for this package on this architecture where all the tests pass, any test failure is a regression. If there has previously a run of the tests for this package where *some* of the tests pass, no test failure is a regression even if there is an increase in the set of tests that are failing.

It would be nice to be able to surface individual test results to britney so that we could gate more cleanly - and even to be able to force-badtest individual tests.

Steve Langasek (vorlon) on 2018-04-27
Changed in auto-package-testing:
importance: Undecided → Wishlist
Changed in britney:
importance: Undecided → Wishlist
Iain Lane (laney) wrote :

The individual test results are in the artifacts "summary" file, so this can be done.

From my end "wishlist" is about right, not least since it is quite common for packages to bundle many tests into one autopkgtest test so this is probably not going to end up being as useful as you might hope.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers