autopkgtest doesn't isolate testbeds (sometimes!)

[Dave Jones]

autopkgtest is "clever" about when it resets the testbed betweens tests, and this is a Bad Thing (tm) when it breaks assumptions about isolation. Specifically, in lib/adt_testbed.py under Testbed.reset is found the following code:

    def reset(self, deps_new, with_recommends):
        '''Reset the testbed, if possible and necessary'''

        adtlog.debug('testbed reset: modified=%s, deps_installed=%s(r: %s), deps_new=%s(r: %s)' %
                     (self.modified, self.deps_installed, self.recommends_installed,
                      deps_new, with_recommends))
        if 'revert' in self.caps and (
                self.modified or self.recommends_installed != with_recommends or
                [d for d in self.deps_installed if d not in deps_new]):
            adtlog.debug('testbed reset')
            pl = self.command('revert', (), 1)
            self._opened(pl)
        self.modified = False

Paraphrasing, if the test-bed is capable of reversion ('revert' in self.caps), and the current test's dependencies (self.deps_installed) include packages that are *not* present in the next test's dependencies (deps_new), or if the recommends list has changed, then *and only then* is the test bed sent the "revert" command which will reset it completely (in the case of virt-qemu this restarts the VM from scratch). Otherwise it doesn't bother.

If one makes the (fairly reasonable?) assumption that tests operate in isolation to one another this can lead to some ... surprising results. Consider the following d/t/control:

  Test-Command: touch /etc/foo
  Depends: build-essential
  Restrictions: isolation-machine, needs-root

  Test-Command: test -e /etc/foo
  Depends: build-essential
  Restrictions: isolation-machine, needs-root

If the tests are truly isolated, this *should* fail ... but it doesn't. Reverse the order of the tests, and of course it does fail. Or add another package to the Depends of the first test (so it's a superset of the second test) and it fails.

Having spent quite some time failing to debug something, because I'd made the faulty assumption that tests were actually isolated, this level of "clever" is annoying and should either be removed or at least documented, preferably in the man-page so it jumps out at me next time I've forgotten this!

[Paul Gevers (paul-climbing)]

Hi,

On 15-03-2024 3:49 p.m., Dave Jones wrote:
> autopkgtest is "clever" about when it resets the testbed betweens tests,
> and this is a Bad Thing (tm) when it breaks assumptions about isolation.

Ironically https://bugs.debian.org/1063533 has a request in the opposite
direction...

> If the tests are truly isolated, this *should* fail ... but it doesn't.
> Reverse the order of the tests, and of course it does fail. Or add
> another package to the Depends of the first test (so it's a superset of
> the second test) and it fails.

Ack.

> Having spent quite some time failing to debug something, because I'd
> made the faulty assumption that tests were actually isolated, this level
> of "clever" is annoying and should either be removed or at least
> documented, preferably in the man-page so it jumps out at me next time
> I've forgotten this!

Currently I'm in favor of documenting, also because I know that there
are packages out there that rely on this. It does means that yes, the
order of the stanza matter.

Paul

[Dave Jones (waveform)]

> On 15-03-2024 3:49 p.m., Dave Jones wrote:
> > autopkgtest is "clever" about when it resets the testbed betweens
> > tests, and this is a Bad Thing (tm) when it breaks assumptions
> > about isolation.
>
> Ironically https://bugs.debian.org/1063533 has a request in the
> opposite direction...

Heh, well perhaps I can suggest a compromise ...

> > If the tests are truly isolated, this *should* fail ... but it
> > doesn't. Reverse the order of the tests, and of course it does
> > fail. Or add another package to the Depends of the first test (so
> > it's a superset of the second test) and it fails.
>
> Ack.
>
> > Having spent quite some time failing to debug something, because
> > I'd made the faulty assumption that tests were actually isolated,
> > this level of "clever" is annoying and should either be removed or
> > at least documented, preferably in the man-page so it jumps out at
> > me next time I've forgotten this!
>
> Currently I'm in favor of documenting, also because I know that
> there are packages out there that rely on this. It does means that
> yes, the order of the stanza matter.

I'm also in favour of documenting *currently*, on the basis that it'd
be a pretty fundamental change in behaviour and thus likely to cause
breakage. However, I'm pretty strongly of the opinion that a key
assumption that a test should be able to make, is that it is running
on a "clean" test-bed. If a test *cannot* make that assumption, then
there's a whole lotta hoops it needs to jump through before it can do
anything useful, and that doesn't sound like a good system to me. I'll
try and demonstrate below that this is how things currently stand, but
first I had a quick dive through the docs to see what one can expect
from the specification...

The current autopkgtest spec. [1] doesn't say much (or anything?)
about the state of the test-bed at the start of each test. That's
presumably because having a clean test-bed is such a fundamental
assumption that it wasn't considered worth specifying, explicitly, but
perhaps it should be.

The autopkgtest man-page [2] doesn't say anything on the subject
either (that I can find?). The autopkgtest README.running-tests [3]
*does*, but only to warn about the "null" virtualization leaving
clutter on your system, and "chroot" providing no cleanup or revert.

Interestingly, nothing seems to draw a distinction between tests
specified in the same stanza, and tests in separate stanzas. I could
see a reasonable case being made that tests in the same stanza (hence
sharing Depends and Restrictions), could assume a specific running
order, and that the test-bed will *not* be reset between them. In
other words that the following would mean "foo" runs strictly before
"bar", and "bar" will see the changes that "foo" made to the test-bed:

    Tests: foo, bar
    Depends: pkg1, pkg2
    Restrictions: needs-root, isolation-machine

Whilst the following would imply nothing about the running order, and
both "foo" and "bar" could assume clean test-beds:

    Tests: foo
    Depends: pkg1, pkg2
    Restrictions: needs-root, isolation-machine

    Tests: bar
    Depends: pkg1, pkg2
    Restrictions: needs-root, isolation-machine

I menti...