python-flaky/3.8.1-2 and pytest-rerunfailures/12.0-1 vs pytest/8.2.2-1

[Chris Peterson]

python-flaky's [1] and pytest-rerunfailures' [2] autopkgtests fail with python3.12 and pytest 8.2.2. This is an issue with pytest itself and unittest test cases not being torn down correctly [3].

[1] https://autopkgtest.ubuntu.com/results/autopkgtest-oracular/oracular/amd64/p/python-flaky/20240606_104752_a0cfa@/log.gz
[2] https://objectstorage.prodstack5.canonical.com/swift/v1/AUTH_0f9aae918d5b4744bf7b827671c86842/autopkgtest-oracular/oracular/amd64/p/pytest-rerunfailures/20240606_104615_632a4@/log.gz
[3] https://github.com/pytest-dev/pytest/issues/12424

[Chris Peterson (cpete)]

Bug in upstream about this issue: https://github.com/box/flaky/issues/212

[Chris Peterson (cpete)]

The culprit seems to be pytest setting the test._instance attribute to None instead of deleting the reference. It looks like upstream has the same conclusion [1].

This fixes the issue for python-flaky, but uncovered a separate issue for pytest-rerunfailures. Although there appears to be some related work upstream for this issue as well [2].

Please see the attached debdiff for the proposed fixes in [1] for pytest, which will resolve python-flaky's autopkgtest regression. I will separately look into whether [2] will fix the issue for pytest-rerunfailures.

PPA: https://launchpad.net/~cpete/+archive/ubuntu/pytest

pytest autopkgtest results: https://autopkgtest.ubuntu.com/results/autopkgtest-oracular-cpete-pytest/oracular/amd64/p/pytest/20240607_173550_99668@/log.gz

python-flaky autopkgtest results: https://autopkgtest.ubuntu.com/results/autopkgtest-oracular-cpete-pytest/oracular/amd64/p/python-flaky/20240607_173242_a6c20@/log.gz

[1] https://github.com/pytest-dev/pytest/pull/12436
[2] https://github.com/pytest-dev/pytest-rerunfailures/commit/0ab54f00d68bb4feadae58d2f4e30f6663495571

[Chris Peterson (cpete)]

New debidff with revised patch.

[Chris Peterson (cpete)]

So the patch won't fully fix the issue for pytest-rerunfailures, but since the pytest upstream bug [1] that this patch fixes was originally posted due to a regression in pytest-rerunfailures I think this is a net improvement.

After applying the fix for [1], the error now in pytest-rerunfailures is that non-function teardown fixtures (e.g. class, module, etc.) do not get called correctly. This was previously a problem that upstream tried to solve in [2] but I think pytest 8.2 introduced changes that cause it to regress again. There is an upstream (rerunfailures) bug [3] to track the issue there.

[1] https://github.com/pytest-dev/pytest/issues/12424
[2] https://github.com/pytest-dev/pytest-rerunfailures/commit/0ab54f00d68bb4feadae58d2f4e30f6663495571
[3] https://github.com/pytest-dev/pytest-rerunfailures/issues/267

[Chris Peterson (cpete)]

We could probably be more clear in the changelog on what the patch I provided does. Can we use:

pytest (8.2.2-1ubuntu1) oracular; urgency=medium

  * 0003-pytest8-testcase-assertion.patch: Fixes unittest regression in
    flaky test plugins (LP: #2068642).

 -- Chris Peterson <email address hidden> Fri, 07 Jun 2024 15:48:04 -0700

instead please?

[Dan Bungert (dbungert)]

feedback provided - drop the author field for a patch with origin, tweak the changelog for clarity. Uploading.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "pytest_8.2.2-1_8.2.2-1ubuntu1.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Dan Bungert (dbungert)]

Already sponsored, unsubscribing.