commit 69c715c505cf9e5df29dc1dff2fa1a4847471cb6
Author: Samuel Merritt <email address hidden>
Date: Mon Aug 14 10:41:31 2017 -0700
Fix deadlock when logging from a tpool thread.
The object server runs certain IO-intensive methods outside the main
pthread for performance. If one of those methods tries to log, this can
cause a crash that eventually leads to an object server with hundreds
or thousands of greenthreads, all deadlocked.
The short version of the story is that logging.SysLogHandler has a
mutex which Eventlet monkey-patches. However, the monkey-patched mutex
sometimes breaks if used across different pthreads, and it breaks in
such a way that it is still considered held. After that happens, any
attempt to emit a log message blocks the calling greenthread forever.
The fix is to use a mutex that works across different greenlets and
across different pthreads. This patch introduces such a lock based on
an anonymous pipe.
Change-Id: I57decefaf5bbed57b97a62d0df8518b112917480
Closes-Bug: 1710328
(cherry picked from commit 6d160797fc3257942618a7914d526911ebbda328)
Reviewed: https:/ /review. openstack. org/503459 /git.openstack. org/cgit/ openstack/ swift/commit/ ?id=69c715c505c f9e5df29dc1dff2 fa1a4847471cb6
Committed: https:/
Submitter: Jenkins
Branch: stable/ocata
commit 69c715c505cf9e5 df29dc1dff2fa1a 4847471cb6
Author: Samuel Merritt <email address hidden>
Date: Mon Aug 14 10:41:31 2017 -0700
Fix deadlock when logging from a tpool thread.
The object server runs certain IO-intensive methods outside the main
pthread for performance. If one of those methods tries to log, this can
cause a crash that eventually leads to an object server with hundreds
or thousands of greenthreads, all deadlocked.
The short version of the story is that logging. SysLogHandler has a
mutex which Eventlet monkey-patches. However, the monkey-patched mutex
sometimes breaks if used across different pthreads, and it breaks in
such a way that it is still considered held. After that happens, any
attempt to emit a log message blocks the calling greenthread forever.
The fix is to use a mutex that works across different greenlets and
across different pthreads. This patch introduces such a lock based on
an anonymous pipe.
Change-Id: I57decefaf5bbed 57b97a62d0df851 8b112917480 42618a7914d5269 11ebbda328)
Closes-Bug: 1710328
(cherry picked from commit 6d160797fc32579