python-flaky/3.8.1-2 and pytest-rerunfailures/12.0-1 vs pytest/8.2.2-1

Bug #2068642 reported by Chris Peterson
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
pytest (Debian)
Fix Released
Unknown
pytest (Ubuntu)
Fix Committed
Undecided
Unassigned
pytest-rerunfailures (Ubuntu)
Invalid
Undecided
Unassigned
python-flaky (Ubuntu)
Invalid
Undecided
Unassigned
Chris Peterson (cpete)
description: updated
Chris Peterson (cpete)
Changed in python-flaky (Ubuntu):
assignee: Chris Peterson (cpete) → nobody
description: updated
Revision history for this message
Chris Peterson (cpete) wrote :

Bug in upstream about this issue: https://github.com/box/flaky/issues/212

Chris Peterson (cpete)
description: updated
description: updated
Changed in python-flaky (Ubuntu):
status: In Progress → New
Changed in pytest (Ubuntu):
status: New → In Progress
assignee: nobody → Chris Peterson (cpete)
summary: - python-flaky/3.8.1-2 vs pytest/8.2.2-1
+ python-flaky/3.8.1-2 and pytest-rerunfailures/12.0-1 vs pytest/8.2.2-1
Revision history for this message
Chris Peterson (cpete) wrote :

The culprit seems to be pytest setting the test._instance attribute to None instead of deleting the reference. It looks like upstream has the same conclusion [1].

This fixes the issue for python-flaky, but uncovered a separate issue for pytest-rerunfailures. Although there appears to be some related work upstream for this issue as well [2].

Please see the attached debdiff for the proposed fixes in [1] for pytest, which will resolve python-flaky's autopkgtest regression. I will separately look into whether [2] will fix the issue for pytest-rerunfailures.

PPA: https://launchpad.net/~cpete/+archive/ubuntu/pytest

pytest autopkgtest results: https://autopkgtest.ubuntu.com/results/autopkgtest-oracular-cpete-pytest/oracular/amd64/p/pytest/20240607_173550_99668@/log.gz

python-flaky autopkgtest results: https://autopkgtest.ubuntu.com/results/autopkgtest-oracular-cpete-pytest/oracular/amd64/p/python-flaky/20240607_173242_a6c20@/log.gz

[1] https://github.com/pytest-dev/pytest/pull/12436
[2] https://github.com/pytest-dev/pytest-rerunfailures/commit/0ab54f00d68bb4feadae58d2f4e30f6663495571

Revision history for this message
Chris Peterson (cpete) wrote :

New debidff with revised patch.

Revision history for this message
Chris Peterson (cpete) wrote :

So the patch won't fully fix the issue for pytest-rerunfailures, but since the pytest upstream bug [1] that this patch fixes was originally posted due to a regression in pytest-rerunfailures I think this is a net improvement.

After applying the fix for [1], the error now in pytest-rerunfailures is that non-function teardown fixtures (e.g. class, module, etc.) do not get called correctly. This was previously a problem that upstream tried to solve in [2] but I think pytest 8.2 introduced changes that cause it to regress again. There is an upstream (rerunfailures) bug [3] to track the issue there.

[1] https://github.com/pytest-dev/pytest/issues/12424
[2] https://github.com/pytest-dev/pytest-rerunfailures/commit/0ab54f00d68bb4feadae58d2f4e30f6663495571
[3] https://github.com/pytest-dev/pytest-rerunfailures/issues/267

Revision history for this message
Chris Peterson (cpete) wrote :

We could probably be more clear in the changelog on what the patch I provided does. Can we use:

pytest (8.2.2-1ubuntu1) oracular; urgency=medium

  * 0003-pytest8-testcase-assertion.patch: Fixes unittest regression in
    flaky test plugins (LP: #2068642).

 -- Chris Peterson <email address hidden> Fri, 07 Jun 2024 15:48:04 -0700

instead please?

Revision history for this message
Dan Bungert (dbungert) wrote :

feedback provided - drop the author field for a patch with origin, tweak the changelog for clarity. Uploading.

Chris Peterson (cpete)
Changed in pytest (Ubuntu):
status: In Progress → New
assignee: Chris Peterson (cpete) → nobody
Dan Bungert (dbungert)
Changed in pytest (Ubuntu):
status: New → Fix Committed
Chris Peterson (cpete)
Changed in python-flaky (Ubuntu):
status: New → Invalid
Changed in pytest-rerunfailures (Ubuntu):
status: New → Invalid
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "pytest_8.2.2-1_8.2.2-1ubuntu1.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Changed in pytest (Debian):
status: Unknown → New
Revision history for this message
Dan Bungert (dbungert) wrote :

Already sponsored, unsubscribing.

Changed in pytest (Debian):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.