The traces both are stuck in logging - looks like another variation of the log rotation hang.

Thread 2
#0 0x00002b4ed1308b50 in ?? () from None
#1 0x0000000003ff2410 in ?? () from None
#2 0x00000000004d4498 in PyThread_acquire_lock (lock=0x3ff2410, waitflag=128) from ../Python/thread_pthread.h
#3 0x00000000004d7932 in lock_PyThread_acquire_lock (self=0x420d108, args=<value optimised out>) from ../Modules/threadmodule.c
#4 0x00000000004a7c5e in call_function () from ../Python/ceval.c
/usr/lib/python2.6/threading.py (117): acquire
/usr/lib/python2.6/logging/__init__.py (621): acquire
/usr/lib/python2.6/logging/__init__.py (669): handle
/usr/lib/python2.6/logging/__init__.py (1206): callHandlers
/usr/lib/python2.6/logging/__init__.py (1174): handle
/usr/lib/python2.6/logging/__init__.py (1152): _log
/usr/lib/python2.6/logging/__init__.py (1047): info
/srv/launchpad.net/production/launchpad-rev-14560/eggs/zc.zservertracelog-1.1.5-py2.6.egg/zc/zservertracelog/tracelog.py (33): _log
/srv/launchpad.net/production/launchpad-rev-14560/eggs/zc.zservertracelog-1.1.5-py2.6.egg/zc/zservertracelog/tracelog.py (95): executeRequest

is acquiring a lock and

Thread 4
#0 0x00002b4ed1308b50 in ?? () from None
#1 0x0000000003ff2410 in ?? () from None
#2 0x00000000004d4498 in PyThread_acquire_lock (lock=0x3ff2410, waitflag=128) from ../Python/thread_pthread.h
#3 0x00000000004d7932 in lock_PyThread_acquire_lock (self=0x420d108, args=<value optimised out>) from ../Modules/threadmodule.c
#4 0x00000000004a7c5e in call_function () from ../Python/ceval.c
/usr/lib/python2.6/threading.py (117): acquire
/usr/lib/python2.6/logging/__init__.py (621): acquire
/srv/launchpad.net/production/launchpad-rev-14560/eggs/ZConfig-2.9.1dev_20110728-py2.6.egg/ZConfig/components/logger/loghandler.py (87): reopen
/srv/launchpad.net/production/launchpad-rev-14560/eggs/ZConfig-2.9.1dev_20110728-py2.6.egg/ZConfig/components/logger/loghandler.py (42): reopenFiles
/srv/launchpad.net/production/launchpad-rev-14560/lib/canonical/launchpad/webapp/sigusr2.py (20): sigusr2_handler
/usr/lib/python2.6/threading.py (117): acquire
/usr/lib/python2.6/logging/__init__.py (621): acquire
/usr/lib/python2.6/logging/__init__.py (669): handle
/usr/lib/python2.6/logging/__init__.py (1206): callHandlers
/usr/lib/python2.6/logging/__init__.py (1174): handle
/usr/lib/python2.6/logging/__init__.py (1152): _log
/usr/lib/python2.6/logging/__init__.py (1047): info
/srv/launchpad.net/production/launchpad-rev-14560/eggs/zc.zservertracelog-1.1.5-py2.6.egg/zc/zservertracelog/tracelog.py (33): _log
/srv/launchpad.net/production/launchpad-rev-14560/eggs/zc.zservertracelog-1.1.5-py2.6.egg/zc/zservertracelog/tracelog.py (71): handle_request
/srv/launchpad.net/production/launchpad-rev-14560/eggs/zope.server-3.6.1-py2.6.egg/zope/server/serverchannelbase.py (119): received
/srv/launchpad.net/production/launchpad-rev-14560/eggs/zope.server-3.6.1-py2.6.egg/zope/server/dualmodechannel.py (85): handle_read

is aquiring the *same* lock, which it looks like it was trying to acquire before the siguse2 handler was called.

So I suspect the signal handler has found a bug with lock reeentrancy.

can we get a more fine grained trace (down to C level line numbers please) - e.g. get a core that we can retrace please.