Nova notifier thread does not run in rpc_worker sub-processes

[Carl Baldwin]

This reported to me today by Maru. When an rpc worker is spawned as a sub-process, that happens after the nova notifier thread has already started.

eventlet.hubs.use_hub() is the call in neutron/openstack/common/service.py that causes all thread execution to stop.

From the event let documentation: "Make sure to do this before the application starts doing any I/O! Calling use_hub completely eliminates the old hub, and any file descriptors or timers that it had been managing will be forgotten."

Maru's observation is that this means that thread should not spawn before forking the process if they need to run in the child process. I agree.

The reason that threads spawn is that the plugin gets loaded prior to forking and the thread for the nova notifier is started in the __init__ method of a sub-class of the plugin.

[Carl Baldwin (carl-baldwin)]

I think what we need is a start method on the plugin and base classes that starts any threads so that this method can be called after the fork has been done.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/84578

[Maru Newby (maru)]

FYI mkolesni was the original reporter, I was following up in his absence.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/84798

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/84798
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3be1d7a75c5ec754825e99e0a8d95b4e1521ae4b
Submitter: Jenkins
Branch: master

commit 3be1d7a75c5ec754825e99e0a8d95b4e1521ae4b
Author: Carl Baldwin <email address hidden>
Date: Wed Apr 2 16:53:33 2014 +0000

    Replace loopingcall in notifier with a delayed send

    The loopingcall thread here was started before processes fork and so
    the thread stops working after the fork call. This is a problem that
    will probably need to be worked out in the long run.

    To ensure that this notifier works correctly in all processes, this
    change replaces the persistent loopingcall thread with a thread
    created on demand to delay and batch up notifications. The first
    notification will trigger spawning the thread to wait to send it. Any
    notifications that come in the meantime will notice that there is
    already a thread waiting to send and will return without spawning.

    Change-Id: I519d4e89b8cee341c0e1cfffbce3e77151e8202a
    Closes-Bug: #1301035

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/95509

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/95509
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=48bc7db994dea4c296777c6ea9d65873c03567cd
Submitter: Jenkins
Branch: stable/icehouse

commit 48bc7db994dea4c296777c6ea9d65873c03567cd
Author: Carl Baldwin <email address hidden>
Date: Wed Apr 2 16:53:33 2014 +0000

    Replace loopingcall in notifier with a delayed send

    The loopingcall thread here was started before processes fork and so
    the thread stops working after the fork call. This is a problem that
    will probably need to be worked out in the long run.

    To ensure that this notifier works correctly in all processes, this
    change replaces the persistent loopingcall thread with a thread
    created on demand to delay and batch up notifications. The first
    notification will trigger spawning the thread to wait to send it. Any
    notifications that come in the meantime will notice that there is
    already a thread waiting to send and will return without spawning.

    Conflicts:
            neutron/notifiers/nova.py

    Change-Id: I519d4e89b8cee341c0e1cfffbce3e77151e8202a
    Closes-Bug: #1301035
    (cherry picked from commit 3be1d7a75c5ec754825e99e0a8d95b4e1521ae4b)