Don't run tests that never pass (or don't run tests with force-badtest)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Auto Package Testing |
New
|
Undecided
|
Unassigned |
Bug Description
Running tests for results that's ignored anyway is wasting resources and adding delay to proposed migration.
I propose not running the tests for which the results will be surely ignored. Instead skip those tests and possibly perform whole archive testing of the release pocket in an additional queue.
I've collected some statistics to back this proposal in https:/
$ make
sqlite3 autopkgtest.db < arch-speed.sql 2>&1 | tee arch-speed.report
Average test run time in seconds per architecture (in Groovy):
amd64|346.0
arm64|927.0
armhf|694.0
i386|371.0
ppc64el|409.0
s390x|315.0
Total time needed to run each test once on the latest version (in hours, per architecture, in Groovy):
amd64|1284.0
arm64|3331.0
armhf|2506.0
i386|849.0
ppc64el|1482.0
s390x|1122.0
sqlite3 autopkgtest.db < pass-analysis.sql 2>&1 | tee pass-analysis.
Number of packages with tests in Focal and Groovy
amd64|14271
arm64|14099
armhf|14145
i386|13693
ppc64el|14173
s390x|14108
Number of packages in Focal and Groovy not passing a single test
amd64|989
arm64|1118
armhf|1326
i386|2318
ppc64el|1180
s390x|1311
Around 5-10% of tests never passed and roughly this is the percentage of load we would not put on the CI infrastructure by skipping them. I can give a better estimate if needed.
What is the proposal? I don't find 'surely ignored' to be descriptive enough.
Is it: if the result is going to be hinted away anyway, do not queue the tests?
I could get on board with that probably. The history *can* be useful, but if we eventually couple this with 'baseline' retesting then we will get to keep some of the benefits of that.