Feature request: hooks to catch tests without proper cleanup

Bug #98431 reported by Marius Gedminas
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Zope 3
Won't Fix
Medium
Unassigned
zope.testing
Won't Fix
Medium
Unassigned
zope.testrunner
Invalid
Undecided
Unassigned

Bug Description

When a unit test does not clean up properly after itself (e.g. changes the component registry and fails to call placelessTearDown, or leaves uncommited objects lying around in a transaction) it can cause a completely unrelated test to fail later. When you have many tests, finding the broken one may be difficult.

It would help if the testrunner let us register a number of hooks to be run before each setUp and after each tearDown. Each hook could be an object with two methods (say, beforeSetUp and afterTearDown) taking the current test as an argument. A hook could take a snapshot of the global state (say, call getSite()) in beforeSetUp and compare it in afterTearDown. If there's a mismatch, you've identified a misbehaving test.

I have implemented this for the SchoolTool test runner a while ago. Unfortunatelly the code is GPLed and requests for a licencing change so far were unsuccessful. I am willing to reimplement the feature for the Zope 3 test runner if there's interest.

Tags: core feature
Revision history for this message
Gary Poster (gary) wrote :

+1 on the feature: sounds very useful. The situation with library-level schooltool code is quite a shame. Given the licensing issues, unfortunately, -1 on someone with potential license problems implementing it. Marius, maybe you could write a clear set of requirements on how to do this (i.e., without any code in it) that someone else could implement? I hear that approach around license problems has precedent (are there any IP lawyers in our midst to confirm or deny? ;-) )

Revision history for this message
Steve Alexander (stevea) wrote :

I think the licencing issues around the schooltool testing code can be resolved. I'll talk with Marius privately about this.

Tres Seaver (tseaver)
Changed in zope.testing:
importance: Undecided → Medium
Changed in zope3:
status: New → Won't Fix
Revision history for this message
Marius Gedminas (mgedmin) wrote :

This needs to be fixed in zope.testrunner, not zope.testing.

Changed in zope.testing:
status: New → Won't Fix
Revision history for this message
Marius Gedminas (mgedmin) wrote :

Test layer testSetUp/testTearDown methods could be repurposed for this.

Unfortunately not for unit tests, which are sort of layer-less. (You could introduce your own unit test layer, but then bin/test -u wouldn't run it, unless bug 669070 gets fixed.)

Revision history for this message
Marius Gedminas (mgedmin) wrote :

I did some work on this 6 years ago, in the test-checkers branch of zope.testing in SVN:
http://zope3.pov.lt/trac/log/zope.testing/branches/test-checkers

The end result was functional, but I gave up while trying to write a functional doctest for it, IIRC. Therefore it was never merged, and then lost during the zope.testing.testrunner -> zope.testrunner renaming and subsequent git migration.

Revision history for this message
Colin Watson (cjwatson) wrote :

The zope.testrunner project on Launchpad has been archived at the request of the Zope developers (see https://answers.launchpad.net/launchpad/+question/683589 and https://answers.launchpad.net/launchpad/+question/685285). If this bug is still relevant, please refile it at https://github.com/zopefoundation/zope.testrunner.

Changed in zope.testrunner:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.