Race condition in Process Monitor / disable proccess

[Miguel Angel Ajo]

When we disable a monitored process via ProcessMonitor, and the
monitoring loop is running a

RuntimeError: dictionary changed size during iteration

will happen here:

https://github.com/openstack/neutron/blob/master/neutron/agent/linux/external_process.py#L223

This can be prevented by either, getting the full list of .keys() for the loop, or adding extra locking.

[Miguel Angel Ajo (mangelajo)]

A simple change:

- for service_id in self._process_managers:
+for service_id in self._process_managers.keys():

would fix the issue, as the later:

pm = self._process_managers.get(service_id) // if pm...

will just skip a process manager that has been disabled.

[Miguel Angel Ajo (mangelajo)]

eventually during functional tests run you can find this:

Captured stderr:
~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 455, in fire_timers
        timer()
      File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
        cb(*args, **kw)
      File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 212, in main
        result = function(*args, **kwargs)
      File "neutron/openstack/common/lockutils.py", line 272, in inner
        return f(*args, **kwargs)
      File "neutron/agent/linux/external_process.py", line 273, in _check_child_processes
        for service_id in self._process_managers:
    RuntimeError: dictionary changed size during iteration

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140647
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b25915f1d94e19756c932ae18119bdf872cc6316
Submitter: Jenkins
Branch: master

commit b25915f1d94e19756c932ae18119bdf872cc6316
Author: Miguel Angel Ajo <email address hidden>
Date: Wed Dec 10 12:21:52 2014 +0100

    Fix race condition in ProcessMonitor

    If any new external process was enabled/disabled during
    the process monitoring loop time, a RuntimeError: dictionary
    changed size during iteration was thrown. This is fixed
    by pre-building the service list from the dictionary
    for the iteration.

    Closes-Bug: #1401042

    Change-Id: Id0a3af22fb85992bde35d8c691bee3ddf435639d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/142430

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/142430
Reason: Sorry, not needed since ProcessMonitor is not used in Juno. The actual changes to use it will come in Kilo only.