Figure out what to do about zope.sendmail incompatibility with Python >= 2.5.1

[Max Bowsher]

After changes to the design of Python threading in 2.5.1, zope.sendmail.delivery's background thread no longer exits at Python exit time, and causes the Python process to hang and require kill -9. There is no upstream solution AFAICS.

Workaround: http://bazaar.launchpad.net/~maxb/zope3/launchpad-3.4-py2.5/revision/9

Need to liase with upstream and figure out what on earth to do here.

[Gary Poster (gary)]

Max, have you asked upstream about this, or shall I?

[Max Bowsher (maxb)]

I have not yet.

I should probably write up a fuller description of the problem, and some test programs.

[Max Bowsher (maxb)]

Description of the situation:

Prior to Python 2.5.1, the atexit handlers were executed when the *main* thread exits. However, http://bugs.python.org/issue1566280 was then filed, the essence of which is that the atexit handlers could tear down cross-thread resources that were still in use by other threads.

For this reason, the shutdown procedure was modified in Python 2.5.1 - now, when the main thread exits, it first makes a private call to threading._shutdown(), which waits until all non-daemon threads have exited, and *only then* proceeds to run the atexit handlers.

Herein lies the problem. zope.sendmail.delivery.QueueProcessorThread attempts to use atexit to notify itself when it should shut down. However, the Python runtime >= 2.5.1 will wait for the QueueProcessorThread to exit before it calls the atexit handlers!

Potential solutions:

(1) To gain behaviour most similar to earlier Python versions, an evil monkeypatch into Python internals like this: http://bazaar.launchpad.net/~maxb/zope3/launchpad-3.4-py2.5/revision/9 works.

(2) Make the QueueProcessorThread a daemon thread. It will be terminated without notification when the interpreter exits, but the current QueueProcessorThread does not attempt to ensure its queue is completely flushed before shutdown anyway, so that should not matter.

I will attach a small Python demo program useful for illustrating and exploring the issue.

[Gary Poster (gary)]

This patch works from Max's experiments to use the setDaemon approach. This seems like the right basic idea--really, the current zope.sendmail code is essentially acting like a daemon thread, with the combination of atexit and the _stopped flag simply trying to make the loop's block run as a unit before the thread can exit.

Therefore, I made the thread a daemon. Reviewing the code, it seemed the only point at which the code would not fail cleanly was after the mail was sent and before the msg was removed--a single block of actions. Therefore, as I described in the comment, I tried to tie those two actions together with a lock.

When the atexit code has to wait for the lock, the process no longer dies for some reason: the daemon flag is no longer honored in that case. Therefore, I continue to use the _stopped flag for communication in that case. The _stopped flag is set before the lock is acquired.

I also changed __stopped to _stopped because I generally don't like the Python mangled name game.

Marius Gedminas and Max have both reviewed this patch. I may try to get Francis Lacoste to also review, I'll verify it works in Launchpad, and then I plan to push it upstream and make a release today or tomorrow. I intend to make it a bugfix release of zope.sendmail: 3.5.2.