API workers might not work when sync thread is enabled

Bug #1420278 reported by Salvatore Orlando on 2015-02-10
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Undecided
Unassigned
Icehouse
High
Salvatore Orlando
Juno
High
Salvatore Orlando
vmware-nsx
High
Salvatore Orlando

Bug Description

API workers are started with a fork().
It is well known that this operation uses CoW on the child process - and this should not constitute a problem.
However, the status synch thread is started at plugin initialization and might be already running when the API workers are forked.
The NSX API client, extensively used by this thread, uses an eventlet semaphore to grab backend connections from a pool.
It is therefore possible that when a worker process is forked it receives semaphore which are in "busy" state. Once forked these semaphores are new objects, and they will never be unblocked. The API worker therefore simply hangs.

This behaviour has been confirmed by observation on the field

Changed in neutron:
status: New → Invalid

Fix proposed to branch: master
Review: https://review.openstack.org/154466

Changed in vmware-nsx:
status: New → In Progress

Reviewed: https://review.openstack.org/180145
Committed: https://git.openstack.org/cgit/stackforge/vmware-nsx/commit/?id=a1a773ba64e189d1db04fc600bbe68b204b0a85b
Submitter: Jenkins
Branch: stable/kilo

commit a1a773ba64e189d1db04fc600bbe68b204b0a85b
Author: Salvatore Orlando <email address hidden>
Date: Tue Feb 10 05:10:25 2015 -0800

    Add delay to sync thread startup

    A 5 second initial delay is added to the looping call running
    the status synchronization thread.
    This will prevent API worker processes from hanging because of
    busy semaphore value deriving from the fork procedure.

    Cherry-picked from: 72e02989707ce6b1e162e0f1c2b38ff9f1dcba3c
    Change-Id: Ieeeb2696f46ec9ee4047d6685988215f619134fd
    Closes-Bug: #1420278

tags: added: in-stable-kilo
Changed in vmware-nsx:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers