> On 15-03-2024 3:49 p.m., Dave Jones wrote:
> > autopkgtest is "clever" about when it resets the testbed betweens
> > tests, and this is a Bad Thing (tm) when it breaks assumptions
> > about isolation.
> 
> Ironically https://bugs.debian.org/1063533 has a request in the
> opposite direction...

Heh, well perhaps I can suggest a compromise ...

> > If the tests are truly isolated, this *should* fail ... but it
> > doesn't. Reverse the order of the tests, and of course it does
> > fail. Or add another package to the Depends of the first test (so
> > it's a superset of the second test) and it fails.
> 
> Ack.
> 
> > Having spent quite some time failing to debug something, because
> > I'd made the faulty assumption that tests were actually isolated,
> > this level of "clever" is annoying and should either be removed or
> > at least documented, preferably in the man-page so it jumps out at
> > me next time I've forgotten this!
> 
> Currently I'm in favor of documenting, also because I know that
> there are packages out there that rely on this. It does means that
> yes, the order of the stanza matter.

I'm also in favour of documenting *currently*, on the basis that it'd
be a pretty fundamental change in behaviour and thus likely to cause
breakage. However, I'm pretty strongly of the opinion that a key
assumption that a test should be able to make, is that it is running
on a "clean" test-bed. If a test *cannot* make that assumption, then
there's a whole lotta hoops it needs to jump through before it can do
anything useful, and that doesn't sound like a good system to me. I'll
try and demonstrate below that this is how things currently stand, but
first I had a quick dive through the docs to see what one can expect
from the specification...

The current autopkgtest spec. [1] doesn't say much (or anything?)
about the state of the test-bed at the start of each test. That's
presumably because having a clean test-bed is such a fundamental
assumption that it wasn't considered worth specifying, explicitly, but
perhaps it should be.

The autopkgtest man-page [2] doesn't say anything on the subject
either (that I can find?). The autopkgtest README.running-tests [3]
*does*, but only to warn about the "null" virtualization leaving
clutter on your system, and "chroot" providing no cleanup or revert.

Interestingly, nothing seems to draw a distinction between tests
specified in the same stanza, and tests in separate stanzas. I could
see a reasonable case being made that tests in the same stanza (hence
sharing Depends and Restrictions), could assume a specific running
order, and that the test-bed will *not* be reset between them. In
other words that the following would mean "foo" runs strictly before
"bar", and "bar" will see the changes that "foo" made to the test-bed:

    Tests: foo, bar
    Depends: pkg1, pkg2
    Restrictions: needs-root, isolation-machine

Whilst the following would imply nothing about the running order, and
both "foo" and "bar" could assume clean test-beds:

    Tests: foo
    Depends: pkg1, pkg2
    Restrictions: needs-root, isolation-machine

    Tests: bar
    Depends: pkg1, pkg2
    Restrictions: needs-root, isolation-machine

I mention "no assumptions about the running order" because
README.running-tests [3] talks about parallel test runs (under the
QEMU section), in which case there would be no deterministic means of
understanding which test would run on the test-bed prior to another
(assuming any reuse of test-beds in such a scenario).

Still, as things stand at the moment, why can no test be certain of
having a clean test-bed? Consider the separate stanzas case of "foo"
and "bar" above. If a developer wishes to add a new test, "baz", and
they want to be sure of having a clean test-bed they have to insert
the test at the *start* of the d/t/control file:

    Tests: baz
    Depends: pkg1, pkg2
    Restrictions: needs-root, isolation-machine

    Tests: foo
    Depends: pkg1, pkg2
    Restrictions: needs-root, isolation-machine

    Tests: bar
    Depends: pkg1, pkg2
    Restrictions: needs-root, isolation-machine

But, having done so, they may now break the "foo" test if its
dependencies are a superset of the new test's, and "baz" affects the
test-bed in a way that interacts with the "foo" test. In order to
guarantee that "baz" doesn't affect "foo", we could add a redundant
dependency (making baz's dependencies a superset of foo's), but then
we have to consider if that pointless dependency will affect the "baz"
test. Or alternatively we put the "baz" test at the bottom, but now we
have to consider whether any prior test (with a subset of
dependencies) can affect the test-bed in a way that breaks our new
test. Argh!

Ideally, a developer should be able to define a new test in
d/t/control without having to trawl through existing tests to
determine if they're going to break anything.

[1]: https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst

[2]: https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/runner/autopkgtest.1

[3]: https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.running-tests.rst