On agent restart VMs list are not getting synced between agent and nova-compute

[VishnuVardhan V]

Build No:2272, ubuntu

There are 2 VMs

root@nodea8:~# nova list
+--------------------------------------+----------+--------+------------+-------------+----------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+----------+--------+------------+-------------+----------------+
| 155748f2-07db-4cc0-bacd-152d0f5fe0e4 | vn1-inst | ACTIVE | None | Running | vn1=10.10.10.2 |
| 925c0732-2335-4f77-860e-7f8779878b4f | vn2-inst | ACTIVE | None | Running | vn2=10.10.11.2 |
+--------------------------------------+----------+--------+------------+-------------+----------------+

After restarting agent only one VM (925c0732-2335-4f77-860e-7f8779878b4f) was present in agent. Confirmed via agent introspect (config and oper)

Analysis of agent sandesh traces shows that agent's connection with nova-compute happened twice (version 1 and 2), but all the VMs info was not sent when it was connected with version 2. Detailed logs are shown below.

2014-07-25 15:09:30.653 CfgOpenstackConnect: op = Connect version = 1 file = controller/src/vnsw/agent/openstack/instance_service_server.cc line = 112

2014-07-25 15:09:30.655 CfgOpenstackAddPort: op = Add port_uuid = 4cd4cc81-5562-471a-94f3-7ed259984803 instance_uuid = 155748f2-07db-4cc0-bacd-152d0f5fe0e4 vn_uuid = 02366ad2-3d33-4941-b4b1-7d529cbba0a8 ip_address = 10.10.10.2 tap_name = tap4cd4cc81-55 mac_address = 02:4c:d4:cc:81:55 display_name = vn1-inst hostname = vn1-inst host = nodea8 version = 1 vlan_id = -1 vm_project_uuid = 7a8af205-d2cc-48fe-88d0-bfef01465e13 file = controller/src/vnsw/agent/openstack/instance_service_server.cc line = 98

2014-07-25 15:09:30.655 CfgIntfTrace: intf = tap4cd4cc81-55 vm_name = vn1-inst vm = 155748f2-07db-4cc0-bacd-152d0f5fe0e4 vn = 02366ad2-3d33-4941-b4b1-7d529cbba0a8 ip = 10.10.10.2 op = ADD version = 1 vlan_id = 65535 vm_project = 7a8af205-d2cc-48fe-88d0-bfef01465e13 file = controller/src/vnsw/agent/cfg/cfg_interface.cc line = 106

2014-07-25 15:09:31.826 CfgOpenstackConnect: op = Connect version = 2 file = controller/src/vnsw/agent/openstack/instance_service_server.cc line = 112

2014-07-25 15:09:31.827 CfgOpenstackAddPort: op = Add port_uuid = b4720609-c419-44a9-88e9-769a280685f7 instance_uuid = 925c0732-2335-4f77-860e-7f8779878b4f vn_uuid = 3b394a44-7ca5-4e55-b242-e5e57c02c04d ip_address = 10.10.11.2 tap_name = tapb4720609-c4 mac_address = 02:b4:72:06:09:c4 display_name = vn2-inst hostname = vn2-inst host = nodea8 version = 2 vlan_id = -1 vm_project_uuid = 7a8af205-d2cc-48fe-88d0-bfef01465e13 file = controller/src/vnsw/agent/openstack/instance_service_server.cc line = 98

2014-07-25 15:09:31.827 CfgIntfTrace: intf = tapb4720609-c4 vm_name = vn2-inst vm = 925c0732-2335-4f77-860e-7f8779878b4f vn = 3b394a44-7ca5-4e55-b242-e5e57c02c04d ip = 10.10.11.2 op = ADD version = 2 vlan_id = 65535 vm_project = 7a8af205-d2cc-48fe-88d0-bfef01465e13 file = controller/src/vnsw/agent/cfg/cfg_interface.cc line = 106

2014-07-25 15:10:30.654 CfgIntfInfo: InterfaceConfigStaleCleaner Walk invoked. controller/src/vnsw/agent/openstack/instance_service_server.cc 637

2014-07-25 15:10:30.654 CfgIntfWalkDone: 1 controller/src/vnsw/agent/openstack/instance_service_server.cc 657
2014-07-25 15:10:31.827 CfgIntfInfo: InterfaceConfigStaleCleaner Walk invoked. controller/src/vnsw/agent/openstack/instance_service_server.cc 637

2014-07-25 15:10:31.827 CfgIntfWalkDone: 2 controller/src/vnsw/agent/openstack/instance_service_server.cc 657
2014-07-25 15:10:31.827 CfgIntfTrace: intf = tap4cd4cc81-55 vm_name = vn1-inst vm = 155748f2-07db-4cc0-bacd-152d0f5fe0e4 vn = 02366ad2-3d33-4941-b4b1-7d529cbba0a8 ip = 10.10.10.2 op = DELETE version = 1 vlan_id = 65535 vm_project = 7a8af205-d2cc-48fe-88d0-bfef01465e13 file = controller/src/vnsw/agent/cfg/cfg_interface.cc line = 118

[Hari Prasad Killi (haripk)]

Problem not seen later. Need to reproduce.