object server deadlocks when a worker thread logs something

Bug #1710328 reported by Samuel Merritt on 2017-08-12
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
High
Samuel Merritt

Bug Description

When code run in the object server's worker thread tries to log a message using Swift's logger, it can end up deadlocking the whole process.

Here's how the sad story unfolds:

The object server calls some code using tpool_reraise(), e.g. ECDiskFileWriter._finalize_durable(). Something goes wrong in here for some reason, so the code tries to log the error by calling self.manager.logger.exception(). self.manager.logger is a wrapper around a logging.Logger object with one log handler: a logging.handlers.SysLogHandler.

When the message gets to the SysLogHandler, it goes to emit the record by sending it to a syslog server. The SysLogHandler has a mutex that it uses to serialize message emissions, so before it writes the record to a socket, it acquires the mutex.

The SysLogHandler's mutex is a threading.RLock object, but its internals have been monkeypatched by Eventlet. Instead of having a normal mutex that blocks the running pthread, it has a green mutex that only blocks the running greenthread. When some caller tries to acquire the green mutex but it is already held, the mutex object puts the current greenthread on a queue, then sleeps. When the lock holder releases the lock, it will wake up the head of the queue.

Finally, we get to the problem. Code in the worker pthread starts to log, but the SysLogHandler's mutex is held by another greenthread. The current mutex holder is a greenthread in the main pthread, i.e. the one doing WSGI stuff. So, the (one and only) greenthread in the worker pthread enqueues itself for wakeup later, then goes to sleep.

Now, the mutex-holder greenthread finishes its work, releases the lock, and goes to wake the worker pthread's greenthread... and KABOOM.

    Traceback (most recent call last):
      File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 458, in fire_timers
        timer()
      File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
        cb(*args, **kw)
      File "/usr/local/lib/python2.7/dist-packages/eventlet/semaphore.py", line 147, in _do_acquire
        waiter.switch()
    error: cannot switch to a different thread

A greenthread can only run in the pthread in which it was created. When the eventlet hub tries to run the worker pthread's greenthread, it fails. Worse, it leaves the mutex in a still-held state.

Now, for every request that comes in to this object server, a new greenthread is created (like normal), and when that greenthread goes to log, it blocks waiting for the busted mutex. This quickly exhausts available file descriptors as unclosed sockets pile up.

Meanwhile, the worker pthread's greenthread is still sleeping, waiting for someone to wake it up. If your tpool size happens to be 1, then this means any greenthread that tries to call tpool_execute will never receive a response.

One possible fix would be to use a lock that works across greenthreads in different pthreads.

Reviewed: https://review.openstack.org/493636
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=6d160797fc3257942618a7914d526911ebbda328
Submitter: Jenkins
Branch: master

commit 6d160797fc3257942618a7914d526911ebbda328
Author: Samuel Merritt <email address hidden>
Date: Mon Aug 14 10:41:31 2017 -0700

    Fix deadlock when logging from a tpool thread.

    The object server runs certain IO-intensive methods outside the main
    pthread for performance. If one of those methods tries to log, this can
    cause a crash that eventually leads to an object server with hundreds
    or thousands of greenthreads, all deadlocked.

    The short version of the story is that logging.SysLogHandler has a
    mutex which Eventlet monkey-patches. However, the monkey-patched mutex
    sometimes breaks if used across different pthreads, and it breaks in
    such a way that it is still considered held. After that happens, any
    attempt to emit a log message blocks the calling greenthread forever.

    The fix is to use a mutex that works across different greenlets and
    across different pthreads. This patch introduces such a lock based on
    an anonymous pipe.

    Change-Id: I57decefaf5bbed57b97a62d0df8518b112917480
    Closes-Bug: 1710328

Changed in swift:
status: In Progress → Fix Released

This issue was fixed in the openstack/swift 2.15.1 release.

Reviewed: https://review.openstack.org/503453
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=6b4e6aeffaf1c6f7bc69998870796b7f9230ee91
Submitter: Jenkins
Branch: stable/newton

commit 6b4e6aeffaf1c6f7bc69998870796b7f9230ee91
Author: Samuel Merritt <email address hidden>
Date: Mon Aug 14 10:41:31 2017 -0700

    Fix deadlock when logging from a tpool thread.

    The object server runs certain IO-intensive methods outside the main
    pthread for performance. If one of those methods tries to log, this can
    cause a crash that eventually leads to an object server with hundreds
    or thousands of greenthreads, all deadlocked.

    The short version of the story is that logging.SysLogHandler has a
    mutex which Eventlet monkey-patches. However, the monkey-patched mutex
    sometimes breaks if used across different pthreads, and it breaks in
    such a way that it is still considered held. After that happens, any
    attempt to emit a log message blocks the calling greenthread forever.

    The fix is to use a mutex that works across different greenlets and
    across different pthreads. This patch introduces such a lock based on
    an anonymous pipe.

    Change-Id: I57decefaf5bbed57b97a62d0df8518b112917480
    Closes-Bug: 1710328
    (cherry picked from commit 6d160797fc3257942618a7914d526911ebbda328)

tags: added: in-stable-newton
tags: added: in-stable-ocata

Reviewed: https://review.openstack.org/503459
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=69c715c505cf9e5df29dc1dff2fa1a4847471cb6
Submitter: Jenkins
Branch: stable/ocata

commit 69c715c505cf9e5df29dc1dff2fa1a4847471cb6
Author: Samuel Merritt <email address hidden>
Date: Mon Aug 14 10:41:31 2017 -0700

    Fix deadlock when logging from a tpool thread.

    The object server runs certain IO-intensive methods outside the main
    pthread for performance. If one of those methods tries to log, this can
    cause a crash that eventually leads to an object server with hundreds
    or thousands of greenthreads, all deadlocked.

    The short version of the story is that logging.SysLogHandler has a
    mutex which Eventlet monkey-patches. However, the monkey-patched mutex
    sometimes breaks if used across different pthreads, and it breaks in
    such a way that it is still considered held. After that happens, any
    attempt to emit a log message blocks the calling greenthread forever.

    The fix is to use a mutex that works across different greenlets and
    across different pthreads. This patch introduces such a lock based on
    an anonymous pipe.

    Change-Id: I57decefaf5bbed57b97a62d0df8518b112917480
    Closes-Bug: 1710328
    (cherry picked from commit 6d160797fc3257942618a7914d526911ebbda328)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers