Extra triggers invalidate passing test results and also may cause test failures

[Balint Reczey]

Since a recent change in the autopkgtest infrastructure many additional packages are added as triggers automatically when a package is being tested in -proposed.
This can be observed at https://autopkgtest.ubuntu.com/packages/g/glibc/groovy/amd64 , for example when the new glibc version are tested for the first time.
When retrying the test from update_excuses.html the extra triggers are not added, just the package trying to enter the release pocket. (Also seen on the link above.)

If the first try fails the package is stuck in -proposed until someone retries the test (or an other test is run also using that package/version).

As I understand the extra triggers are added to increase the likelihood of passing the test, but they can also be the reason for the test failing. In case the test fails requeuing the test without the extra triggers would probably be the best option since that would let regression-free packages through without manual intervention.

Letting regression-free packages without manual intervention is most likely a design goal of the Auto Package Testing project, but I can't find a good reference for that.

[Iain Lane (laney)]

I think this is right. We should make groups out of packages that have to go in together but adding too much from -proposed can cause false results either way, if a package in proposed either breaks or fixes another one and is used in the test unnecessarily.

i.e. if A in proposed Depends B (>> release-pocket-version), and B is in proposed too, we should add a trigger on B when running A's tests. There are real world cases where apt will not resolve the right dependencies without this. But if A also Depends C, and C is in proposed too, but the release pocket version satisfies the dependency, then we should NOT add the trigger. C could be totally broken and then we'd falsely/unfairly hold A in proposed.

I guess some testcases demonstrating the right/wrong behaviour would be a good first step, here.

[Iain Lane (laney)]

btw, since I was just working around this spot (not on this bug, I'm afraid), I got reminded of this issue. Just thought I'd note that the code for this is here:

https://git.launchpad.net/~ubuntu-release/britney/+git/britney2-ubuntu/tree/britney2/policies/autopkgtest.py#n541

[Balint Reczey (rbalint)]

Thinking a bit more about adding extra triggering packages by default the bigger problem is not failures, but the test that pass only with packages from -proposed, because passing test results don't connect packages in migration: LP: #1908508.

The autopkgtests should answer if migrating a particular package to the release pocket breaks anything. By adding extra triggers and not migrating those with the package the tests can't answer that question anymore, resulting more likely regressions in the release pocket.

It is not clear how many test runs are saved by adding the extra triggers nor the negative impact on the release pocket's quality, but I'd like to not have extra triggers until LP: #1908508 is fixed.

[Iain Lane (laney)]

I don't want to get into a which-is-worse argument about the old behaviour versus the current.

The intention there is to trigger tests together which already have to go in together because of relationships that already exist. No extra requirements are required from proposed-migration to achieve that because those relationships should already enforce this grouping.

If you would like to work on this bug, Balint, with a view to making that grouping correct as I outlined in comment #1, I would welcome that.

Viewed in that way, the other bug is a separate issue which we can discuss.

[Balint Reczey (rbalint)]

@laney OK, I may have misunderstood the intent about the triggers together, I try to formalize my understanding how it could be safe.

Let's take systemd triggering open-iscsi:

https://autopkgtest.ubuntu.com/packages/o/open-iscsi/hirsute/amd64

2.1.2-1ubuntu1 systemd/247.1-4ubuntu1 2020-12-18 17:45:57 UTC 0h 15m 53s rbalint pass log   artifacts

vs.

2.1.2-1ubuntu1 systemd/247.1-4ubuntu1 conntrack-tools/1:1.4.6-1 debootstrap/1.0.123ubuntu2 dpdk/19.11.5-1 dq/20181021-1 gpsd/3.20-12build2 hddemux/0.4-7ubuntu2 libreswan/3.32-3ubuntu2 libvirt-dbus/1.4.0-2 mdadm/4.1-8ubuntu1 netplan.io/0.101-0ubuntu2 nftables/0.9.7-1 nix/2.3.7+dfsg1-1build1 pystemd/0.7.0-4build2 python-dbusmock/0.19-1 samba/2:4.13.2+dfsg-3ubuntu1 suricata/1:6.0.1-2 tinyssh/20190101-1build1 tpm2-abrmd/2.3.3-1build1 tpm2-pkcs11/1.2.0-1ubuntu2 webhook/2.6.9-1build2 2020-12-18 04:00:50 UTC 0h 11m 45s - fail log   artifacts   ♻

If the added triggers are the minimal set of packages systemd can't migrate without, then yes, it is correct to add them from systemd's migration's POV, because if systemd migrates the release pocket will include them.

OTOH, if an extra trigger is added because it can't migrate without systemd, then regarding systemd's migration this is not correct because if only systemd migrates then the test was not valid.

As a result, to not create falsely passing tests all the packages which are in the trigger set must require all the rest of the packages in the trigger set, i.e. none of them can migrate if any of the rest is not migrating.

The trivial solution is using only single triggers, but it is possible to calculate the minimal set.

(There is also a hidden assumption, that package relationships don't change, and for set sizes > 1 sometimes this does not stand due to the inflow and outflow of other packages.)