Caught Exception in dhcp agent sync_state may block or delay configuration of new networks

[Stephen Ma]

In the dhcp-agent.log, sometimes this error is seen.
Dhcp_agent.ini is configured with no router_id defined. There is one dhcp agent managing all dhcp servers in one node.

This is the Traceback:
2013-07-10 16:16:15 ERROR [quantum.agent.dhcp_agent] Unable to sync network state.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/quantum/agent/dhcp_agent.py", line 152, in sync_state
self.disable_dhcp_helper(deleted_id)
File "/usr/lib/python2.7/dist-packages/quantum/agent/dhcp_agent.py", line 197, in disable_dhcp_helper
self.disable_isolated_metadata_proxy(network)
File "/usr/lib/python2.7/dist-packages/quantum/agent/dhcp_agent.py", line 340, in disable_isolated_metadata_proxy
pm.disable()
File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/external_process.py", line 67, in disable
ip_wrapper.netns.execute(cmd)
File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/ip_lib.py", line 407, in execute
check_exit_code=check_exit_code)
File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/utils.py", line 61, in execute
raise RuntimeError(m)
RuntimeError:...

The dhcp_agent.py in commit 1bd456371f9909d5cb33536e84a3fdd7aac40f8c shows:

    def sync_state(self):
        """Sync the local DHCP state with Neutron."""
        LOG.info(_('Synchronizing state'))
        pool = eventlet.GreenPool(cfg.CONF.num_sync_threads)
        known_network_ids = set(self.cache.get_network_ids())

        try:
            active_networks = self.plugin_rpc.get_active_networks_info()
            active_network_ids = set(network.id for network in active_networks)
            for deleted_id in known_network_ids - active_network_ids:
                self.disable_dhcp_helper(deleted_id)

            for network in active_networks:
                pool.spawn_n(self.configure_dhcp_for_network, network)

        except Exception:
            self.needs_resync = True
            LOG.exception(_('Unable to sync network state.'))

When an error happens in the loop of the self.disable_dhcp_helper routine, all networks in the "for network in active_networks" loop will be skipped. So either the configuration of dhcp of network will either be delayed or will never happen.

This routine touches many networks in a system. So any error exceptions should be caught inside disable_dhcp_helper() so that processing of other networks are not blocked -- for any exceptions, bugs or not.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/38655

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/38655
Committed: http://github.com/openstack/neutron/commit/3d5bec962c1e5bb923107f54585c94a9b24f2d30
Submitter: Jenkins
Branch: master

commit 3d5bec962c1e5bb923107f54585c94a9b24f2d30
Author: Stephen Ma <email address hidden>
Date: Thu Jul 25 07:25:48 2013 -0700

    Dhcp agent sync_state may block or delay
    configuration of new networks.

    Fixes Bug 1202722

    Change-Id: I368cb064057d48be1491df6825cc67c265706b50