Tunnel creation race in ovs-agent, if tunnel_update is called from ml2 plugin

[Jakub Libosvar]

There is a race condition in ovs-agent if l2pop is not used.

If ovs agent receives tunnel_update rpc message and starts handling it, it searches in local vlan map and calls ovs-ofctl leading to yield in eventlet. If in another coroutine ovs agent creates a new LocalVLANMapping due to creation of new port for new tenant, local vlan map changes and causes exception, after eventlet starts handling coroutine of tunnel_update.

    2015-04-09 17:41:47.771 6067 ERROR oslo.messaging.rpc.dispatcher [req-1c5353f1-38bd-49f6-8ad1-31ea14484156 ] Exception during message handling: dictionary changed size during iteration
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher incoming.message))
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 347, in tunnel_update
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher tunnel_type)
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1055, in _setup_tunnel_port
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher for network_id, vlan_mapping in self.local_vlan_map.iteritems():
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher RuntimeError: dictionary changed size during iteration
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/178625

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/178625
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e1fd7a8c5d1dd683603a75244f5baf273d018fc7
Submitter: Jenkins
Branch: master

commit e1fd7a8c5d1dd683603a75244f5baf273d018fc7
Author: Jakub Libosvar <email address hidden>
Date: Wed Apr 29 12:04:03 2015 +0200

    Don't use iterator in search for tunnel type

    Changing dictionary size while using iterator causes RuntimeError. This
    can happen in local vlan mapping under certain program flows. This patch
    changes iteritems() to values() that returns list and thus preventing
    from failure if local vlan mapping changes during creating tunnels.

    Change-Id: I8a858d5c53e85f83a582f34205f9afa214cb4d58
    Closes-Bug: 1449944

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: neutron-pecan
Review: https://review.openstack.org/185072