Esxi host after removal from inventory and when re-added back some of guestVM broken from Contrail

[Sarath]

Part of removal from inventory, disconnected esxi from inventory. When adding back to inventory
most of VM's came up fine and issue seen couple of VM's
Please find below VM-205-32 got into issue and VM-205-31 works good as seen vcenter-plugin logs

Topology
##########

3 controllers
8 Esx nodes

>>> As seen below Vcenter-plugin logs, VM's assigned IP inline with Contrail.

root@oblocknode04:/var/log/contrail#
root@oblocknode04:/var/log/contrail#
root@oblocknode04:/var/log/contrail# cat contrail-vcenter-plugin.log| grep 205-32
2015-12-10 19:25:29,526 - INFO [pool-1-thread-1:VncDB@545] - Create Virtual Machine : VM:VM-205-32 (uuid=5004d418-e2e0-7e75-e440-b3d72e177bf3), VN:d48ca60e-7c67-355c-b416-86cddb391481, vrouterIp: 172.16.80.110, EsxiHost:10.87.26.205, vlan:1980/1981
2015-12-10 19:25:29,551 - INFO [pool-1-thread-1:ApiConnectorImpl@162] - >> Request: POST, /virtual-machine-interfaces, {"virtual-machine-interface":{"virtual_machine_interface_mac_addresses":{"mac_address":["00:50:56:84:21:11"]},"id_perms":{"enable":true,"user_visible":false,"creator":"vcenter-plugin"},"display_name":"vmi-testvn1-11-VM-205-32","security_group_refs":[{"to":["default-domain","vCenter","default"],"attr":null,"href":null,"uuid":null}],"virtual_machine_refs":[{"to":["5004d418-e2e0-7e75-e440-b3d72e177bf3"],"attr":null,"href":null,"uuid":null}],"virtual_network_refs":[{"to":["default-domain","vCenter","testvn1-11"],"attr":null,"href":null,"uuid":null}],"name":"744f15e8-c5a5-4188-a36c-c79a5739db0e","uuid":"744f15e8-c5a5-4188-a36c-c79a5739db0e","fq_name":["default-domain","vCenter","744f15e8-c5a5-4188-a36c-c79a5739db0e"],"parent_type":"project"}}
2015-12-10 19:25:29,568 - INFO [pool-1-thread-1:ApiConnectorImpl@162] - >> Request: POST, /instance-ips, {"instance-ip":{"instance_ip_secondary":false,"id_perms":{"enable":true,"user_visible":false,"creator":"vcenter-plugin"},"display_name":"ip-testvn1-11-VM-205-32","virtual_network_refs":[{"to":["default-domain","vCenter","testvn1-11"],"attr":null,"href":null,"uuid":null}],"virtual_machine_interface_refs":[{"to":["default-domain","vCenter","744f15e8-c5a5-4188-a36c-c79a5739db0e"],"attr":null,"href":null,"uuid":null}],"name":"847b20a1-dc20-4761-9e7c-26d2e8b48088","uuid":"847b20a1-dc20-4761-9e7c-26d2e8b48088","fq_name":["847b20a1-dc20-4761-9e7c-26d2e8b48088"]}}
2015-12-10 19:25:29,597 - INFO [pool-1-thread-1:VncDB@661] - VRouterAPi Add Port success - interface name:vmi-testvn1-11-VM-205-32(744f15e8-c5a5-4188-a36c-c79a5739db0e), VM=VM-205-32, VN=testvn1-11, vmIpAddress=2.0.1.81, vlan=1980/1981
2015-12-10 19:25:29,597 - INFO [pool-1-thread-1:VncDB@686] - Create Virtual Machine : VM:VM-205-32 (uuid=5004d418-e2e0-7e75-e440-b3d72e177bf3) Done
2015-12-14 00:23:02,801 - INFO [pool-1-thread-1:VncDB@545] - Create Virtual Machine : VM:VM-205-32 (uuid=5004d418-e2e0-7e75-e440-b3d72e177bf3), VN:d48ca60e-7c67-355c-b416-86cddb391481, vrouterIp: 172.16.80.110, EsxiHost:10.87.26.205, vlan:1980/1981
2015-12-14 00:23:10,064 - INFO [pool-1-thread-1:VncDB@545] - Create Virtual Machine : VM:VM-205-32 (uuid=5004d418-e2e0-7e75-e440-b3d72e177bf3), VN:d48ca60e-7c67-355c-b416-86cddb391481, vrouterIp: 172.16.80.110, EsxiHost:10.87.26.205, vlan:1980/1981
2015-12-14 00:23:10,093 - INFO [pool-1-thread-1:ApiConnectorImpl@162] - >> Request: POST, /virtual-machine-interfaces, {"virtual-machine-interface":{"virtual_machine_interface_mac_addresses":{"mac_address":["00:50:56:84:21:11"]},"id_perms":{"enable":true,"user_visible":false,"creator":"vcenter-plugin"},"display_name":"vmi-testvn1-11-VM-205-32","security_group_refs":[{"to":["default-domain","vCenter","default"],"attr":null,"href":null,"uuid":null}],"virtual_machine_refs":[{"to":["5004d418-e2e0-7e75-e440-b3d72e177bf3"],"attr":null,"href":null,"uuid":null}],"virtual_network_refs":[{"to":["default-domain","vCenter","testvn1-11"],"attr":null,"href":null,"uuid":null}],"name":"81d8555d-e601-42e8-8724-cc6fa63a1b04","uuid":"81d8555d-e601-42e8-8724-cc6fa63a1b04","fq_name":["default-domain","vCenter","81d8555d-e601-42e8-8724-cc6fa63a1b04"],"parent_type":"project"}}
2015-12-14 00:23:10,111 - INFO [pool-1-thread-1:ApiConnectorImpl@162] - >> Request: POST, /instance-ips, {"instance-ip":{"instance_ip_secondary":false,"id_perms":{"enable":true,"user_visible":false,"creator":"vcenter-plugin"},"display_name":"ip-testvn1-11-VM-205-32","virtual_network_refs":[{"to":["default-domain","vCenter","testvn1-11"],"attr":null,"href":null,"uuid":null}],"virtual_machine_interface_refs":[{"to":["default-domain","vCenter","81d8555d-e601-42e8-8724-cc6fa63a1b04"],"attr":null,"href":null,"uuid":null}],"name":"f5fbb9e2-e34c-4781-8fab-419aacccb948","uuid":"f5fbb9e2-e34c-4781-8fab-419aacccb948","fq_name":["f5fbb9e2-e34c-4781-8fab-419aacccb948"]}}
2015-12-14 00:23:10,142 - INFO [pool-1-thread-1:VncDB@661] - VRouterAPi Add Port success - interface name:vmi-testvn1-11-VM-205-32(81d8555d-e601-42e8-8724-cc6fa63a1b04), VM=VM-205-32, VN=testvn1-11, vmIpAddress=2.0.1.74, vlan=1980/1981
2015-12-14 00:23:10,142 - INFO [pool-1-thread-1:VncDB@686] - Create Virtual Machine : VM:VM-205-32 (uuid=5004d418-e2e0-7e75-e440-b3d72e177bf3) Done
root@oblocknode04:/var/log/contrail#

root@oblocknode04:/var/log/contrail#
root@oblocknode04:/var/log/contrail# cat contrail-vcenter-plugin.log| grep 205-31
2015-12-10 19:25:29,376 - INFO [pool-1-thread-1:VncDB@545] - Create Virtual Machine : VM:VM-205-31 (uuid=50045f9e-aba9-3032-312a-087ab2cbbc21), VN:d48ca60e-7c67-355c-b416-86cddb391481, vrouterIp: 172.16.80.110, EsxiHost:10.87.26.205, vlan:1980/1981
2015-12-10 19:25:29,401 - INFO [pool-1-thread-1:ApiConnectorImpl@162] - >> Request: POST, /virtual-machine-interfaces, {"virtual-machine-interface":{"virtual_machine_interface_mac_addresses":{"mac_address":["00:50:56:84:e0:4c"]},"id_perms":{"enable":true,"user_visible":false,"creator":"vcenter-plugin"},"display_name":"vmi-testvn1-11-VM-205-31","security_group_refs":[{"to":["default-domain","vCenter","default"],"attr":null,"href":null,"uuid":null}],"virtual_machine_refs":[{"to":["50045f9e-aba9-3032-312a-087ab2cbbc21"],"attr":null,"href":null,"uuid":null}],"virtual_network_refs":[{"to":["default-domain","vCenter","testvn1-11"],"attr":null,"href":null,"uuid":null}],"name":"9475a9df-acbb-477c-9d63-b9c846d5bb68","uuid":"9475a9df-acbb-477c-9d63-b9c846d5bb68","fq_name":["default-domain","vCenter","9475a9df-acbb-477c-9d63-b9c846d5bb68"],"parent_type":"project"}}
2015-12-10 19:25:29,417 - INFO [pool-1-thread-1:ApiConnectorImpl@162] - >> Request: POST, /instance-ips, {"instance-ip":{"instance_ip_secondary":false,"id_perms":{"enable":true,"user_visible":false,"creator":"vcenter-plugin"},"display_name":"ip-testvn1-11-VM-205-31","virtual_network_refs":[{"to":["default-domain","vCenter","testvn1-11"],"attr":null,"href":null,"uuid":null}],"virtual_machine_interface_refs":[{"to":["default-domain","vCenter","9475a9df-acbb-477c-9d63-b9c846d5bb68"],"attr":null,"href":null,"uuid":null}],"name":"9691567e-6955-485e-aca8-f20fbdb9eaff","uuid":"9691567e-6955-485e-aca8-f20fbdb9eaff","fq_name":["9691567e-6955-485e-aca8-f20fbdb9eaff"]}}
2015-12-10 19:25:29,449 - INFO [pool-1-thread-1:VncDB@661] - VRouterAPi Add Port success - interface name:vmi-testvn1-11-VM-205-31(9475a9df-acbb-477c-9d63-b9c846d5bb68), VM=VM-205-31, VN=testvn1-11, vmIpAddress=2.0.1.73, vlan=1980/1981
2015-12-10 19:25:29,449 - INFO [pool-1-thread-1:VncDB@686] - Create Virtual Machine : VM:VM-205-31 (uuid=50045f9e-aba9-3032-312a-087ab2cbbc21) Done
2015-12-14 00:10:45,062 - INFO [pool-1-thread-1:VncDB@545] - Create Virtual Machine : VM:VM-205-31 (uuid=50045f9e-aba9-3032-312a-087ab2cbbc21), VN:d48ca60e-7c67-355c-b416-86cddb391481, vrouterIp: 172.16.80.110, EsxiHost:10.87.26.205, vlan:1980/1981
2015-12-14 00:10:52,083 - INFO [pool-1-thread-1:VncDB@545] - Create Virtual Machine : VM:VM-205-31 (uuid=50045f9e-aba9-3032-312a-087ab2cbbc21), VN:d48ca60e-7c67-355c-b416-86cddb391481, vrouterIp: 172.16.80.110, EsxiHost:10.87.26.205, vlan:1980/1981
2015-12-14 00:10:52,107 - INFO [pool-1-thread-1:ApiConnectorImpl@162] - >> Request: POST, /virtual-machine-interfaces, {"virtual-machine-interface":{"virtual_machine_interface_mac_addresses":{"mac_address":["00:50:56:84:e0:4c"]},"id_perms":{"enable":true,"user_visible":false,"creator":"vcenter-plugin"},"display_name":"vmi-testvn1-11-VM-205-31","security_group_refs":[{"to":["default-domain","vCenter","default"],"attr":null,"href":null,"uuid":null}],"virtual_machine_refs":[{"to":["50045f9e-aba9-3032-312a-087ab2cbbc21"],"attr":null,"href":null,"uuid":null}],"virtual_network_refs":[{"to":["default-domain","vCenter","testvn1-11"],"attr":null,"href":null,"uuid":null}],"name":"12878e63-21dd-43a9-a722-8c6a051e1bac","uuid":"12878e63-21dd-43a9-a722-8c6a051e1bac","fq_name":["default-domain","vCenter","12878e63-21dd-43a9-a722-8c6a051e1bac"],"parent_type":"project"}}
2015-12-14 00:10:52,125 - INFO [pool-1-thread-1:ApiConnectorImpl@162] - >> Request: POST, /instance-ips, {"instance-ip":{"instance_ip_secondary":false,"id_perms":{"enable":true,"user_visible":false,"creator":"vcenter-plugin"},"display_name":"ip-testvn1-11-VM-205-31","virtual_network_refs":[{"to":["default-domain","vCenter","testvn1-11"],"attr":null,"href":null,"uuid":null}],"virtual_machine_interface_refs":[{"to":["default-domain","vCenter","12878e63-21dd-43a9-a722-8c6a051e1bac"],"attr":null,"href":null,"uuid":null}],"name":"990f8d44-f6d9-4da6-9daf-fa0cf09b78e2","uuid":"990f8d44-f6d9-4da6-9daf-fa0cf09b78e2","fq_name":["990f8d44-f6d9-4da6-9daf-fa0cf09b78e2"]}}
2015-12-14 00:10:52,156 - INFO [pool-1-thread-1:VncDB@661] - VRouterAPi Add Port success - interface name:vmi-testvn1-11-VM-205-31(12878e63-21dd-43a9-a722-8c6a051e1bac), VM=VM-205-31, VN=testvn1-11, vmIpAddress=2.0.1.73, vlan=1980/1981
2015-12-14 00:10:52,156 - INFO [pool-1-thread-1:VncDB@686] - Create Virtual Machine : VM:VM-205-31 (uuid=50045f9e-aba9-3032-312a-087ab2cbbc21) Done
root@oblocknode04:/var/log/contrail#
root@oblocknode04:/var/log/contrail#
root@oblocknode04:/var/log/contrail#

## version

root@oblocknode02:~#
root@oblocknode02:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 2.22-112 14
contrail-config 2.22-112 14
contrail-control 2.22-112 14
contrail-dns 2.22-112 14
contrail-f5 2.22-112 14
contrail-fabric-utils 2.22-112 14
contrail-install-packages 2.21.1-14~vcenter 14
contrail-install-vcenter-plugin 2.21.1-11282015 14
contrail-lib 2.22-112 14
contrail-nodemgr 2.22-112 14
contrail-openstack-analytics 2.22-112 14
contrail-openstack-control 2.22-112 14
contrail-openstack-database 2.22-112 14
contrail-openstack-webui 2.22-112 14
contrail-setup 2.22-112 14
contrail-utils 2.22-112 14
contrail-vmware-config 2.22-112 14
contrail-vmware-utils 2.22-112 14
contrail-web-controller 2.22-112 14
contrail-web-core 2.22-112 14
ifmap-python-client 0.1-2 14
ifmap-server 0.3.2-1contrail1 14
python-contrail 2.22-112 14
root@oblocknode02:~#

[Sarath (nsarath)]

Please find below the logs of 3 Controllers and 8 Vrouters

-bash-4.1$
-bash-4.1$ pwd
/users/nsarath/PR/Bug-1526120
-bash-4.1$
-bash-4.1$ ls -l
total 3705400
-rwxrwxrwx 1 nsarath test 1245532160 Dec 14 10:13 Ctrl-A-log.tar*
-rwxrwxrwx 1 nsarath test 1125345280 Dec 14 10:14 Ctrl-B-log.tar*
-rwxrwxrwx 1 nsarath test 1150218240 Dec 14 10:13 Ctrl-C-log.tar*
-rwxrwxrwx 1 nsarath test 32440320 Dec 14 10:13 Vrtr-0-log.tar*
-rwxrwxrwx 1 nsarath test 32614400 Dec 14 10:13 Vrtr-1-log.tar*
-rwxrwxrwx 1 nsarath test 30351360 Dec 14 10:13 Vrtr-2-log.tar*
-rwxrwxrwx 1 nsarath test 32235520 Dec 14 10:13 Vrtr-3-log.tar*
-rwxrwxrwx 1 nsarath test 32634880 Dec 14 10:13 Vrtr-4-log.tar*
-rwxrwxrwx 1 nsarath test 32307200 Dec 14 10:13 Vrtr-5-log.tar*
-rwxrwxrwx 1 nsarath test 32522240 Dec 14 10:13 Vrtr-7-log.tar*
-rwxrwxrwx 1 nsarath test 33177600 Dec 14 10:13 Vrtr-8-log.tar*
-bash-4.1$

[Sachchidanand Vaidya (vaidyasd)]

This issue is seen due to timing issue. vcenter-plugin does AddPort for new VMI discovered in pool. If VM boots up and requests DHCP before, the port is added to vRouter by PLugin.
This issue shouldn't be present on 3.0 as the plugin is modified to be event-driven.
Pls check with mainline code.