API workers might not work when sync thread is enabled

[Salvatore Orlando]

API workers are started with a fork().
It is well known that this operation uses CoW on the child process - and this should not constitute a problem.
However, the status synch thread is started at plugin initialization and might be already running when the API workers are forked.
The NSX API client, extensively used by this thread, uses an eventlet semaphore to grab backend connections from a pool.
It is therefore possible that when a worker process is forked it receives semaphore which are in "busy" state. Once forked these semaphores are new objects, and they will never be unblocked. The API worker therefore simply hangs.

This behaviour has been confirmed by observation on the field

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/154466

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/180145

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/180145
Committed: https://git.openstack.org/cgit/stackforge/vmware-nsx/commit/?id=a1a773ba64e189d1db04fc600bbe68b204b0a85b
Submitter: Jenkins
Branch: stable/kilo

commit a1a773ba64e189d1db04fc600bbe68b204b0a85b
Author: Salvatore Orlando <email address hidden>
Date: Tue Feb 10 05:10:25 2015 -0800

    Add delay to sync thread startup

    A 5 second initial delay is added to the looping call running
    the status synchronization thread.
    This will prevent API worker processes from hanging because of
    busy semaphore value deriving from the fork procedure.

    Cherry-picked from: 72e02989707ce6b1e162e0f1c2b38ff9f1dcba3c
    Change-Id: Ieeeb2696f46ec9ee4047d6685988215f619134fd
    Closes-Bug: #1420278