Intermittent TypeError NoneType in atexit handler of unit tests

[Ewan Mellor]

There is an intermittent problem with the Swift unit tests that results in them raising TypeError in the logging module's atexit handler.

This can be seen on the Jenkins runs for the following merges (noting that these code changes themselves are not the cause of the problem):

https://code.launchpad.net/~notmyname/swift/update_authors/+merge/66385
https://code.launchpad.net/~notmyname/swift/fix_query_parsing/+merge/53917
https://code.launchpad.net/~ttx/swift/version-1.4.3/+merge/69076
https://code.launchpad.net/~cthier/swift/swift3objpaths/+merge/60119

My Jenkins instance now triggers this all the time, so I was forced to dig into it. It looks like a bug in Python's logging module (in my case, Python 2.6.6 (r266:84292, Sep 15 2010, 15:52:39)). logging installs an atexit handler, but this indirectly calls threading.RLock.acquire. The problem is that there seems to be no guarantee from the Python runtime that the threading module won't already be torn down by the time that atexit callback runs. This means that you get the NoneType exception as shown below.

This is related to http://bugs.python.org/issue9501 reported against Python 2.7. It's different though, because the teardown code in the logging module in Python 2.6 is different to that in 2.7.

I have no idea why this is hitting me now, when it wasn't before. I have no idea why this affects Swift and nothing else that I'm running. I have no idea why this doesn't hit other people all the time.

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.6/logging/__init__.py", line 1517, in shutdown
    h.close()
  File "/usr/lib/python2.6/logging/handlers.py", line 757, in close
    logging.Handler.close(self)
  File "/usr/lib/python2.6/logging/__init__.py", line 709, in close
    _acquireLock()
  File "/usr/lib/python2.6/logging/__init__.py", line 201, in _acquireLock
    _lock.acquire()
  File "/usr/lib/python2.6/threading.py", line 117, in acquire
    me = _get_ident()
TypeError: 'NoneType' object is not callable
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.6/logging/__init__.py", line 1517, in shutdown
    h.close()
  File "/usr/lib/python2.6/logging/handlers.py", line 757, in close
    logging.Handler.close(self)
  File "/usr/lib/python2.6/logging/__init__.py", line 709, in close
    _acquireLock()
  File "/usr/lib/python2.6/logging/__init__.py", line 201, in _acquireLock
    _lock.acquire()
  File "/usr/lib/python2.6/threading.py", line 117, in acquire
    me = _get_ident()
TypeError: 'NoneType' object is not callable