autopkgtest doesn't isolate testbeds (sometimes!)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
autopkgtest (Ubuntu) |
New
|
Undecided
|
Unassigned |
Bug Description
autopkgtest is "clever" about when it resets the testbed betweens tests, and this is a Bad Thing (tm) when it breaks assumptions about isolation. Specifically, in lib/adt_testbed.py under Testbed.reset is found the following code:
def reset(self, deps_new, with_recommends):
'''Reset the testbed, if possible and necessary'''
if 'revert' in self.caps and (
[d for d in self.deps_installed if d not in deps_new]):
pl = self.command(
Paraphrasing, if the test-bed is capable of reversion ('revert' in self.caps), and the current test's dependencies (self.deps_
If one makes the (fairly reasonable?) assumption that tests operate in isolation to one another this can lead to some ... surprising results. Consider the following d/t/control:
Test-Command: touch /etc/foo
Depends: build-essential
Restrictions: isolation-machine, needs-root
Test-Command: test -e /etc/foo
Depends: build-essential
Restrictions: isolation-machine, needs-root
If the tests are truly isolated, this *should* fail ... but it doesn't. Reverse the order of the tests, and of course it does fail. Or add another package to the Depends of the first test (so it's a superset of the second test) and it fails.
Having spent quite some time failing to debug something, because I'd made the faulty assumption that tests were actually isolated, this level of "clever" is annoying and should either be removed or at least documented, preferably in the man-page so it jumps out at me next time I've forgotten this!
Hi,
On 15-03-2024 3:49 p.m., Dave Jones wrote:
> autopkgtest is "clever" about when it resets the testbed betweens tests,
> and this is a Bad Thing (tm) when it breaks assumptions about isolation.
Ironically https:/ /bugs.debian. org/1063533 has a request in the opposite
direction...
> If the tests are truly isolated, this *should* fail ... but it doesn't.
> Reverse the order of the tests, and of course it does fail. Or add
> another package to the Depends of the first test (so it's a superset of
> the second test) and it fails.
Ack.
> Having spent quite some time failing to debug something, because I'd
> made the faulty assumption that tests were actually isolated, this level
> of "clever" is annoying and should either be removed or at least
> documented, preferably in the man-page so it jumps out at me next time
> I've forgotten this!
Currently I'm in favor of documenting, also because I know that there
are packages out there that rely on this. It does means that yes, the
order of the stanza matter.
Paul