autopkgtest failures with breezy 3.3.2-1

[Paride Legovini]

There are several autopkgtest failures affecting breezy 3.3.2-1 and preventing it from migrating, e.g.

https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/amd64/b/breezy/20230124_123950_b8982@/log.gz

According to upstream some tests may be flaky. IRC discussion excerpt:

<paride> these errors are suspicious: ModuleNotFoundError: No module named 'breezy.testingplugins'
<jelmer> so there's some tests that populate breezy.testingplugins in sys.modules
<jelmer> they're also meant to clean up the modifications they do
<jelmer> but for some reason that doesn't happen, and then it bites us later on when we iterate over all the modules

The plan is to disable the flaky modules in Ubuntu. Given that the failure is with the test harness and not with the tests themselves this isn't a blind "let's ignore test failures" move.

This also happens in Debian, but an autopkgtest failure is not blocking there. Again according to upstream:

<jelmer> this particular regression does appear to be Debian-system-specific
<jelmer> I've never seen it happen in Breezy's CI

[Paride Legovini (paride)]

I disabled test_plugins by passing -x test_plugins to the command line in debian/tests/testsuite3 per upstream suggestion (thanks!) and it passed locally:

OK (known_failures=46)
4824 tests skipped
autopkgtest [14:16:21]: test testsuite3: -----------------------]
autopkgtest [14:16:21]: test testsuite3: - - - - - - - - - - results - - - - - - - - - -
testsuite3 PASS
autopkgtest [14:16:21]: @@@@@@@@@@@@@@@@@@@@ summary
testsuite3 PASS

[Paride Legovini (paride)]

Uploading breezy_3.3.2-1ubuntu1.dsc
Uploading breezy_3.3.2-1ubuntu1.debian.tar.xz
Uploading breezy_3.3.2-1ubuntu1_source.buildinfo
Uploading breezy_3.3.2-1ubuntu1_source.changes

[Launchpad Janitor (janitor)]

This bug was fixed in the package breezy - 3.3.2-1ubuntu1

---------------
breezy (3.3.2-1ubuntu1) lunar; urgency=medium

  * Autopkgtest: disable flaky test_plugins (LP: #2003868).
    Thanks: Jelmer Vernooij.

 -- Paride Legovini <email address hidden> Wed, 25 Jan 2023 15:12:39 +0100