Remove threading before process forking

[Dmitriy Ukhlov]

 Forking processes when a few threads are already running is potentially unsafe operation and could cause a lot of problems because only current thread will continue working in child thread. Any locked by other thread resource will remain locked forever.

We faced with this problem during oslo.messaging development and added workaround to hide this problem: https://review.openstack.org/#/c/274255/ I tried to fix this problem in oslo.service: https://review.openstack.org/#/c/270832/ but oslo folks said that this fix is ugly and it is wrong way to add workarounds to common libraries because projects use them incorrectly. I think that is fare.

[Ihar Hrachyshka (ihar-hrachyshka)]

Marked as High since it blocks oslo.messaging team to proceed on Pika driver.

[Ihar Hrachyshka (ihar-hrachyshka)]

Fix is at: https://review.openstack.org/276842

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/276842
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1cafff087194711399ad2a85a9f394f7204d7bdd
Submitter: Jenkins
Branch: master

commit 1cafff087194711399ad2a85a9f394f7204d7bdd
Author: dukhlov <email address hidden>
Date: Fri Feb 5 19:25:42 2016 +0200

    Remove threading before process forking

    Forking a process when multiple threads are running is an unsafe
    operation and could cause a lot of problems because only current
    thread will continue working in child thread. Any locked by other
    thread resource will remain locked forever.

    We faced with this problem during oslo.messaging development and
    added workaround to hide this problem:
    https://review.openstack.org/#/c/274255/

    I tried to fix this problem in oslo.service:
    https://review.openstack.org/#/c/270832/

    but oslo folks said that this fix is ugly and it is wrong way to add
    workarounds to common libraries because projects use them incorrectly.
    I think that is fair.

    So this patch fixes incorrect usage of oslo libraries. In this patch
    I extended functionality of NeutronWorker and add there
    `worker_process_count` parameter which determines how many processes
    should be spawned for this worker. If `worker_process_count` = 0 - don't
    create process and spawn thread in scope of current process for worker

    Then I moved all background tasks to workers and return them by
    `get_workers` method. start_plugin_workers collects plugin's workers
    using `get_workers` method and starts in ProcessLauncher first workers
    with `worker_process_count` > 0 and only after this starts threaded
    workers by simple Launcher

    Closes-bug: #1569404

    Change-Id: I0544f1d47ae53d572adda872847a56fa0b202d2e

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/313088

[Armando Migliaccio (armando-migliaccio)]

This is being reverted.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Richard Theis (<email address hidden>) on branch: master
Review: https://review.openstack.org/313088

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 9.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/313277
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=483c5982c020ff21ceecf1d575c2d8fad2937d6e
Submitter: Jenkins
Branch: master

commit 483c5982c020ff21ceecf1d575c2d8fad2937d6e
Author: Dmitriy Ukhlov <email address hidden>
Date: Fri May 6 08:41:07 2016 +0000

    Revert "Revert "Remove threading before process forking""

    This reverts commit b1cdba1696f5d4ec71d37a773501bd4f9e0cddb9

    Original patch was reverted because it broke neutron plugin's
    backward compatibility and needed more work.

    This patch fixes that problems:
    1) original behaviour of add_agent_status_check,
       start_periodic_l3_agent_status_check and
       start_periodic_dhcp_agent_status_check methods is deprecated but kept
       for using in third part plugins for backward compatibility
    2) new add_agent_status_check_worker, add_periodic_l3_agent_status_check
       and add_periodic_dhcp_agent_status_check method are implemented
       instead and are used for implementing plugins in neutron codebase

    Closes-Bug: #1569404

    Change-Id: I3a32a95489831f0d862930384309eefdc881d8f6

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 9.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/379780

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379780
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=05cd620389aad58d9d1bdb7064dc0ec3034a4e46
Submitter: Jenkins
Branch: master

commit 05cd620389aad58d9d1bdb7064dc0ec3034a4e46
Author: Dariusz Smigiel <email address hidden>
Date: Thu Sep 29 19:22:24 2016 +0000

    Removed deprecated methods for AgentSchedulers

    Removed methods deprecated by
    change I3a32a95489831f0d862930384309eefdc881d8f6

    Change-Id: I00c8c323135e3510896a9b66877fe887285d9281
    Related-Bug: #1569404