Race condition in eventletutils Event

Bug #1805706 reported by Zane Bitter on 2018-11-28
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.utils
Undecided
Zane Bitter

Bug Description

Victor Stinner reported[1] a race condition in the class in the eventletutils module that is designed to provide an eventlet Event with an interface compatible with the threading module's Event class:

Is there a race condition on this line in a greenthread A is blocked on the wait, but another greenthread B calls clear() and then set()? B calls self._event.send(), but A is waiting on a different eventlet Event which is no longer used by the oslo.service Event...

eventlet.event.Event implements a reset() method but it's documented as broken and suggest to create a new Event object:

        # this is kind of a misfeature and doesn't work perfectly well,
        # it's better to create a new event rather than reset an old one
# removing documentation so that we don't get new use cases for it

https://github.com/eventlet/eventlet/blob/master/eventlet/event.py#L53

If there is a race condition, maybe we need something like:

def clear(self):
    old_event = self._event
    self._set = False
    self._event = event.Event()
    # Unblock any greenthreads blocked in wait()
    old_event.send(True)

def wait():
    event = self._event
    while True:
        event.wait()
        if event is not self._event:
            # clear has been called, retry
        break
    ...

[1] https://review.openstack.org/#/c/611807/5/oslo_service/loopingcall.py@101

Zane Bitter (zaneb) on 2018-11-28
Changed in oslo.utils:
status: New → Confirmed
assignee: nobody → Zane Bitter (zaneb)
Changed in oslo.utils:
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/618482
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=cc8b51e1e16f6bdc7d6c0e571e2002e70cde098d
Submitter: Zuul
Branch: master

commit cc8b51e1e16f6bdc7d6c0e571e2002e70cde098d
Author: Zane Bitter <email address hidden>
Date: Wed Nov 28 15:50:12 2018 -0500

    Fix race condition in eventletutils Event

    The threading-compatible eventlet Event class has a race condition on
    the wait method. If greenthread A is blocked on the wait, but another
    greenthread B calls clear() and then set(), B calls self._event.send(),
    but A is waiting on a different eventlet Event which is no longer used
    by the oslo.service Event...

    To resolve this, when clearing an Event trigger the underlying eventlet
    Event immediately, then have the wait() method resume waiting on the new
    eventlet Event.

    Change-Id: I81579e2977bb965a5398a2cb4e3e24f5671e856a
    Co-Authored-By: Victor Stinner <email address hidden>
    Co-Authored-By: Hervé Beraud <email address hidden>
    Closes-Bug: #1805706

Changed in oslo.utils:
status: In Progress → Fix Released

This issue was fixed in the openstack/oslo.utils 3.39.1 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers