control-node ifmap continued to have snat si even after router is deleted

[Vedamurthy Joshi]

R1.10 44 (Ubuntu icehouse multinode setup)

In the test test_basic_snat_behavior_with_subnet_attach_detach in contrail-test, it creates snat setup, removes the VN interface from the router, adds the VN back again and finally cleans up everything.

At the end of it, the script failed because the VN object in control-node config db continued to exist even after the VN and router was deleted.
It was seen that the snat VM's instance IP continued to refer to the VN in control-node config.

The VN is virtual-network:default-domain:TestRouters-82931130:ext_vn-51078609

In control-node introspect : ifmap table entry :
virtual-network:default-domain:TestRouters-82931130:ext_vn-51078609
- - -
obj_info
neighbors
instance-ip:si_808bbb6b-dc7b-4020-8cce-4816611b85ce-right
00:38:46.051751

Logs will be in http://10.204.216.50/Docs/bugs/#

root@nodec22:/var/log/contrail# neutron net-list
+--------------------------------------+-------------------------+---------------------------------------------------+
| id | name | subnets |
+--------------------------------------+-------------------------+---------------------------------------------------+
| dd32ac7e-527c-486f-addb-4118a65ed43a | default-virtual-network | |
| d9efaf6b-d60c-4b48-b5e7-89ce4b29647c | ip-fabric | |
| eae8c123-451b-43cb-948d-d37ec2c827d5 | vn1-31619381 | 0bc7104c-1a59-47ec-ad19-6fbc6dbab0b5 51.6.68.0/24 |
| 2993df96-842a-4f5f-b7f8-93fa36e37a04 | __link_local__ | |
+--------------------------------------+-------------------------+---------------------------------------------------+
root@nodec22:/var/log/contrail#

test logs :
=========
2014-10-04 05:42:38,453 - INFO - Deleting interface with subnet_id aca8ed2d-e966-4304-a6a3-c66d7aaf410f, port_id None from router 808bb
b6b-dc7b-4020-8cce-4816611b85ce
2014-10-04 05:42:39,062 - INFO - Deleting the VM vm_public-53323424
2014-10-04 05:42:44,221 - INFO - Deleting the VM vm_private-26268979
2014-10-04 05:42:49,326 - INFO - Deleting the VN vn_private-41715389
2014-10-04 05:42:49,488 - INFO - VN vn_private-41715389 is not found in API Server
None
2014-10-04 05:42:49,494 - INFO - VN vn_private-41715389 is not present in Agent 10.204.217.69
None
2014-10-04 05:42:49,501 - INFO - VN vn_private-41715389 is not present in Agent 10.204.217.108
None
2014-10-04 05:42:49,507 - INFO - VN vn_private-41715389 is not present in Agent 10.204.217.109
2014-10-04 05:42:49,510 - WARNING - Routing instance for VN vn_private-41715389 is still found in Control-node 10.204.217.7
2014-10-04 05:42:49,513 - WARNING - Routing instance for VN vn_private-41715389 is still found in Control-node 10.204.217.108
2014-10-04 05:42:49,550 - WARNING - Control-node config DB still has VN vn_private-41715389
2014-10-04 05:42:54,601 - INFO - Routing instances and Config db in Control-nodes does not have VN vn_private-41715389 info
2014-10-04 05:42:54,601 - INFO - Deleting RT for VN ext_vn-51078609
2014-10-04 05:42:54,624 - INFO - Deleting the VN ext_vn-51078609
2014-10-04 05:42:55,016 - INFO - VN ext_vn-51078609 is not found in API Server
None
2014-10-04 05:42:55,022 - INFO - VN ext_vn-51078609 is not present in Agent 10.204.217.69
None
2014-10-04 05:42:55,030 - INFO - VN ext_vn-51078609 is not present in Agent 10.204.217.109
2014-10-04 05:42:55,032 - WARNING - Routing instance for VN ext_vn-51078609 is still found in Control-node 10.204.217.7
2014-10-04 05:42:55,036 - WARNING - Routing instance for VN ext_vn-51078609 is still found in Control-node 10.204.217.108
2014-10-04 05:42:55,066 - WARNING - Control-node config DB still has VN ext_vn-51078609
2014-10-04 05:43:00,110 - WARNING - Control-node config DB still has VN ext_vn-51078609
2014-10-04 05:43:05,159 - WARNING - Control-node config DB still has VN ext_vn-51078609
2014-10-04 05:43:10,197 - WARNING - Control-node config DB still has VN ext_vn-51078609

[Vedamurthy Joshi (vedujoshi)]

Prakash debugged on a similar issue :
Have these errors in contrail-api-0.log :

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1168, in _dbe_oper_subscribe
    self._dbe_delete_notification(oper_info)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1270, in _dbe_delete_notification
    (ok, ifmap_result) = method(obj_info)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_ifmap_client_gen.py", line 7949, in _ifmap_virtual_machine_delete
    self._delete_id_pair_meta(r_elem[0].attrib['name'], to_imid, meta_name)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 343, in _delete_id_pair_meta
    _id_to_metas_delete(id_x, id_y, meta_name)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 324, in _id_to_metas_delete
    if meta_name not in self._id_to_metas[id1]:
KeyError: 'contrail:service-instance:default-domain:TestRouters-01229666:si_21072655-bd1c-433b-b818-fb3c838bd734'

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1168, in _dbe_oper_subscribe
    self._dbe_delete_notification(oper_info)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1270, in _dbe_delete_notification
    (ok, ifmap_result) = method(obj_info)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_ifmap_client_gen.py", line 1216, in _ifmap_instance_ip_delete
    self._delete_id_pair_meta(r_elem[0].attrib['name'], to_imid, meta_name)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 343, in _delete_id_pair_meta
    _id_to_metas_delete(id_x, id_y, meta_name)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 324, in _id_to_metas_delete
    if meta_name not in self._id_to_metas[id1]:
KeyError: 'contrail:virtual-machine-interface:default-domain:TestRouters-01229666:default-domain__TestRouters-01229666__c2a0c10f-f6f2-4b51-bdb7-9f3d062f760e__1-right'

[Sachin Bansal (sbansal)]

Already fixed in mainline with commit 4196290833e41fc9488d2ec119ddb77597cb5a94

[Rudra Rugge (rrugge)]

Sachins fixes are for https://bugs.launchpad.net/juniperopenstack/+bug/1365322. Please ignore his comments or commit id above this message.

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/3619
Committed: http://github.org/Juniper/contrail-controller/commit/f2dc24032032b4a6dbe952200464704bee77787d
Submitter: Zuul
Branch: R1.10

commit f2dc24032032b4a6dbe952200464704bee77787d
Author: Sachin Bansal <email address hidden>
Date: Thu Oct 9 12:28:45 2014 -0700

set pending_field_update after updating static routes so that it is reflected in api server

Change-Id: If42a04863d598286ddb18727340897d0d8302ed0
Closes-Bug: 1377467

[Rudra Rugge (rrugge)]

ifmap still has residual instance-ip entries. Ajay believes that it is related to rabbitmq restart and ha mode not being set.

[Ashish Ranjan (aranjan-n)]

Review pending

[Hampapur Ajay (hajay)]

Can this be retried after https://review.opencontrail.org/#/c/5234/ is in?