VMs failed to get ip in vxlan setup due to keyerror

[Arun Kumar]

When spawning around 4k VMs in 100 network scenario , around 500 Vms failed to get ip by throwing below keyerror exception.
 And it is getting ip by restarting dhcp (S40network) in the VM.

build used: HOS 4.0 ((build 01-547)
 Attached logs

2016-06-22 12:01:36.112 31012 ERROR networking_vsphere.agent.ovsvapp_agent [-] Failed to handle VM_UPDATED event for VM: 1e2b8347-6303-4417-aab8-e48d238f7902.
 2016-06-22 12:01:36.112 31012 ERROR networking_vsphere.agent.ovsvapp_agent Traceback (most recent call last):
 2016-06-22 12:01:36.112 31012 ERROR networking_vsphere.agent.ovsvapp_agent File "/opt/stack/venv/neutron-20160621T031745Z/lib/python2.7/site-packages/networking_vsphere/agent/ovsvapp_agent.py", line 1087, in _notify_device_updated
 2016-06-22 12:01:36.112 31012 ERROR networking_vsphere.agent.ovsvapp_agent updated_port = self.ports_dict[vnic.port_uuid]
 2016-06-22 12:01:36.112 31012 ERROR networking_vsphere.agent.ovsvapp_agent KeyError: 4db21599-5542-46a3-a994-937986eb230b
 2016-06-22 12:01:36.112 31012 ERROR networking_vsphere.agent.ovsvapp_agent
 2016-06-22 12:01:36.112 31012 ERROR networking_vsphere.agent.ovsvapp_agent [-] This may result in failure of network provisioning for VirtualMachine 1e2b8347-6303-4417-aab8-e48d238f7902.
 2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent [-] Cause of failure: 4db21599-5542-46a3-a994-937986eb230b.
 2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent Traceback (most recent call last):
2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent File "/opt/stack/venv/neutron-20160621T031745Z/lib/python2.7/site-packages/networking_vsphere/agent/ovsvapp_agent.py", line 964, in process_event
 2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent self._notify_device_updated(vm, host, event.host_changed)
 2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent File "/opt/stack/venv/neutron-20160621T031745Z/lib/python2.7/site-packages/networking_vsphere/common/utils.py", line 91, in inner
 2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent return f(obj, *args, **kw)
 2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent File "/opt/stack/venv/neutron-20160621T031745Z/lib/python2.7/site-packages/networking_vsphere/agent/ovsvapp_agent.py", line 1108, in _notify_device_updated
 2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent raise error.OVSvAppNeutronAgentError(e)
 2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent OVSvAppNeutronAgentError: 4db21599-5542-46a3-a994-937986eb230b
 2016-06-22 12:01:36.113 31012 ERROR networking_vsphere.agent.ovsvapp_agent

[Armando Migliaccio (armando-migliaccio)]

I see networking_vsphere and HOS in this bug report, are you even sure this is a Neutron bug?

[Arun Kumar (arooncoomar)]

@Armando, Sorry for the confusion. I have marked this as a neutron bug by mistake

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/358324
Committed: https://git.openstack.org/cgit/openstack/networking-vsphere/commit/?id=7f844b009f4cfd14f3910a6a0436eacecf4dc5ec
Submitter: Jenkins
Branch: master

commit 7f844b009f4cfd14f3910a6a0436eacecf4dc5ec
Author: M Arun Kumar <email address hidden>
Date: Fri Aug 5 14:01:05 2016 +0530

    VMs failed to get ip in vxlan setup due to keyerror

    When spawning around 4k VMs in 100 network scenario , around 500 Vms
    failed to get ip by throwing below keyerror exception

    Closes-Bug: #1615357

    Change-Id: I732cbdd208d4fb81712c48b60e9ec995fcb00066

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/365852

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/365852
Committed: https://git.openstack.org/cgit/openstack/networking-vsphere/commit/?id=fa83081609e96618b07a5e20e7bc31c640cf9e22
Submitter: Jenkins
Branch: stable/mitaka

commit fa83081609e96618b07a5e20e7bc31c640cf9e22
Author: M Arun Kumar <email address hidden>
Date: Fri Aug 5 14:01:05 2016 +0530

    VMs failed to get ip in vxlan setup due to keyerror

    When spawning around 4k VMs in 100 network scenario , around 500 Vms
    failed to get ip by throwing below keyerror exception

    Closes-Bug: #1615357

    Conflicts:
     networking_vsphere/agent/ovsvapp_agent.py

    Change-Id: I732cbdd208d4fb81712c48b60e9ec995fcb00066
    (cherry picked from commit 7f844b009f4cfd14f3910a6a0436eacecf4dc5ec)

[Aman Kumar (amank)]

Got the same error because of new OpenvSwitch.

Launchpad bug can be found here:
https://bugs.launchpad.net/networking-vsphere/+bug/1630964

Review link: https://review.openstack.org/#/c/382869/