[R4.1 Build 7]: VM object deletion takes 3 minutes after k8s POD deletion

[Pulkit Tandon]

R4.1 - Build 7

HA K8s setup with control data interfaces provisioned.
3 controllers and 2 computes
1 Kube master and 2 Slaves

Description:
Many sanity test cases failed on a check where we verify that after deletion of a Pod, corresponding VM object gets deleted from the agent.

Observed that in random cases, the VM object does not get deleted.
After running manually, found that agent removes the VM info after 3 minutes.
Thus, test case failed.

Logs can be found in following sanity run:
http://10.204.216.50/Docs/logs/4.1.0.0-7_2017_12_01_00_05_21/logs/

CNI logs attached

[Hari Prasad Killi (haripk)]

This scenario has ECMP. When one VM is deleted, the flows continue as there are other VMs but the flow entry is still holding a reference to the deleted VM. Because of this the VM object is not freed till the flow gets deleted.

[Pulkit Tandon (pulkitt)]

Randomly, any sanity test case failed due to this issue:
Few of the consistent failures are:

TestPodScale.test_pod_with_kube_manager_restart
TestService.test_service_with_kube_manager_restart

[Pulkit Tandon (pulkitt)]

As Hari debugged, the issue of flow getting stuck is not observed anymore in any of the recent builds.
Hence closing this bug for now.
Last run : R5.0-ubuntu16-ocata-103.

Also note that the same issue still exist for few sanity test cases but the reason is different.
Hence logging a different bug for that.

If both these bugs are found to be related, or in case this issue observed again, I will reopen this bug.