py35: Threads mysteriously exit when warnings logged from exception handler for unhashable exception

[Zane Bitter]

There's a bug in the traceback module in Python3: https://bugs.python.org/issue28603

The traceback module will raise a TypeError if any exceptions in the context/cause chain are unhashable. In Python 3, objects of any class that defines an __eq__() method but does not also define a __hash__() method (inheriting one doesn't count) are unhashable. (Explicitly setting __hash__ = None also makes objects of the class unhashable.)

The effect of this bug is Kafkaesque in its monstrosity. Any attempt to log an exception value - whether by using LOG.exception() (or passing exc_info to any log), or just by logging anything at WARNING level or higher from within an exception handler, can trigger the bug. Functions called from within an exception handler count, so the same code that may work when called from outside an exception handler, or from a different exception handler, can mysteriously fail. What's more, the TypeError that is raised will also trigger the bug when logged, since it's __context__ is a pointer to the chain that created the original problem, and is therefore still affected. Even an arbitrary number of levels of catch-all exception handling and logging is insufficient for anything to appear in the log. The effect is as if execution of the thread simply stops for no apparent reason. Good luck debugging that.[1]

I have submitted a pull request to fix this in Python 3.7,[2] and it has been tagged for backporting to Python 3.6. However Python 3.5 is already in security-updates-only mode and is unlikely to ever be patched. We'll be living with this issue for a while (EOL for Python 3.5 is 3 years away), so we need to work around it.

[1] *cough* https://review.openstack.org/510993
[2] https://github.com/python/cpython/pull/4014

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/512853

[Zane Bitter (zaneb)]

The pull request and its backport merged upstream in CPython, so this will be fixed in Python 3.6.4 and 3.7 when they're released.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/512853
Committed: https://git.openstack.org/cgit/openstack/oslo.log/commit/?id=6b0b386146e48470d889e0f1b22c7a5d5798149c
Submitter: Zuul
Branch: master

commit 6b0b386146e48470d889e0f1b22c7a5d5798149c
Author: Zane Bitter <email address hidden>
Date: Mon Oct 16 16:41:44 2017 -0400

    Allow logging of unhashable exceptions in Python 3

    A bug in the Python 3 standard library,
    https://bugs.python.org/issue28603 means that calling the traceback
    module with an exception chain that includes an unhashable exception
    results in a TypeError being raised. (In Python 3, any classes that
    define __eq__() without also explicitly defining - not inheriting -
    __hash__() will be unhashable.) The TypeError itself is also part of
    such a chain, since its __context__ will be the unhashable exception. If
    we don't catch it then the thread will simply exit mysteriously, and no
    amount of logging will be able to reveal to the user what happened.

    Change-Id: I6768574f38b075c15e829df5593a45c6be66b1bc
    Closes-Bug: #1724366

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.log 3.32.0 release.