ESX scale: VMs failed to get ip in vxlan setup due to keyerror

[Arun Kumar]

When spawning around 4k VMs in 100 network scenario , around 500 Vms failed to get ip by throwing below keyerror exception.
 And it is getting ip by restarting dhcp (S40network) in the VM.

build used: HOS 4.0 ((build 01-547)
 Attached logs

2016-06-22 12:01:36.112 31012 ERROR networking_vsphere.agent.ovsvapp_agent [-] Failed to handle VM_UPDATED event for VM: 1e2b8347-6303-4417-aab8-e48d238f7902.
 2016-06-22 12:01:36.112 31012 ERROR networking_vsphere.agent.ovsvapp_agent Traceback (most recent call last):
 2016-06-22 12:01:36.112 31012 ERROR networking_vsphere.agent.ovsvapp_agent File "/opt/stack/venv/neutron-20160621T031745Z/lib/python2.7/site-packages/networking_vsphere/agent/ovsvapp_agent.py", line 1087, in _notify_device_updated
 2016-06-22 12:01:36.112 31012 ERROR networking_vsphere.agent.ovsvapp_agent updated_port = self.ports_dict[vnic.port_uuid]
 2016-06-22 12:01:36.112 31012 ERROR networking_vsphere.agent.ovsvapp_agent KeyError: 4db21599-5542-46a3-a994-937986eb230b
 2016-06-22 12:01:36.112 31012 ERROR networking_vsphere.agent.ovsvapp_agent
 2016-06-22 12:01:36.112 31012 ERROR networking_vsphere.agent.ovsvapp_agent [-] This may result in failure of network
provisioning for VirtualMachine 1e2b8347-6303-4417-aab8-e48d238f7902.
 2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent [-] Cause of failure: 4db21599-5542-46a3-a994-937986eb230b.
 2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent Traceback (most recent call last):
2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent File "/opt/stack/venv/neutron-20160621T031745Z/lib/python2.7/site-packages/networking_vsphere/agent/ovsvapp_agent.py", line 964, in process_event
 2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent self._notify_device_updated(vm, host, event.host_changed)
 2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent File "/opt/stack/venv/neutron-20160621T031745Z/lib/python2.7/site-packages/networking_vsphere/common/utils.py", line 91, in inner
 2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent return f(obj, *args, **kw)
 2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent File "/opt/stack/venv/neutron-20160621T031745Z/lib/python2.7/site-packages/networking_vsphere/agent/ovsvapp_agent.py", line 1108, in _notify_device_updated
 2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent raise error.OVSvAppNeutronAgentError(e)
 2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent OVSvAppNeutronAgentError: 4db21599-5542-46a3-a994-937986eb230b
 2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent