After upgrading to 3.0.3-B3, npot able to delete or add vip ..Get the below trace
WARNING:contrail-api:__default__ [SYS_NOTICE]: VncApiError: RabbitMQ connection ESTABLISHED <Connection: amqp://guest:**@13.1.0.12:5672// at 0x7feca2c1d950>
13.1.0.12 - - [2016-10-22 16:29:07] "GET /virtual-ip/6157dce3-98b7-40c9-975b-7862db468143?exclude_back_refs=True&exclude_children=True HTTP/1.1" 200 1827 0.067809
13.1.0.12 - - [2016-10-22 16:29:07] "GET /virtual-ip/6157dce3-98b7-40c9-975b-7862db468143?exclude_back_refs=True&exclude_children=True HTTP/1.1" 200 1827 0.029701
13.1.0.12 - - [2016-10-22 16:29:07] "POST /ref-update HTTP/1.1" 200 122 0.074072
DEBUG:api-0:Sending request(xid=184): Delete(path=u'/fq-name-to-uuid/instance_ip:6157dce3-98b7-40c9-975b-7862db468143', version=-1)
DEBUG:api-0:Received response(xid=184): True
DEBUG:api-0:Sending request(xid=185): Delete(path=u'/api-server/subnets/default-domain:PROJ1:VIP1-NET1:145.14.0.0/24/2433613827', version=-1)
DEBUG:api-0:Received response(xid=185): True
13.1.0.12 - - [2016-10-22 16:29:08] "DELETE /instance-ip/4095a655-74a0-4368-8ec7-cc3ac2dfd0e0 HTTP/1.1" 200 115 0.208828
13.1.0.12 - - [2016-10-22 16:29:11] "GET /virtual-machine-interface/b86e08a3-c45f-4146-be76-1d0f236336d2?fields=instance_ip_back_refs%2Cinterface_route_table_refs HTTP/1.1" 200 2680 0.478803
DEBUG:api-0:Sending request(xid=186): Delete(path=u'/fq-name-to-uuid/virtual_machine_interface:default-domain:PROJ1:default-domain__PROJ1__9c4d69ce-a044-46cf-a97f-46d999f0b448__1__right__1', version=-1)
DEBUG:api-0:Received response(xid=186): True
13.1.0.12 - - [2016-10-22 16:29:11] "DELETE /virtual-machine-interface/b86e08a3-c45f-4146-be76-1d0f236336d2 HTTP/1.1" 200 115 0.125459
13.1.0.12 - - [2016-10-22 16:29:11] "POST /ref-update HTTP/1.1" 200 122 0.027159
13.1.0.12 - - [2016-10-22 16:29:11] "GET /instance-ip/b88f0480-6946-40db-b704-2e2e6c24e932?exclude_back_refs=True&exclude_children=True HTTP/1.1" 200 1733 0.030372
DEBUG:api-0:Sending request(xid=187): Delete(path=u'/fq-name-to-uuid/virtual_machine_interface:default-domain:PROJ1:default-domain__PROJ1__9c4d69ce-a044-46cf-a97f-46d999f0b448__1__left__2', version=-1)
ERROR:contrail-api:__default__ [SYS_ERR]: VncApiError: <type 'exceptions.KeyError'>
Python 2.7.6: /usr/bin/python
Sat Oct 22 16:29:11 2016
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py in _dbe_subscribe_callback(self=<vnc_cfg_api_server.vnc_cfg_ifmap.VncServerKombuClient object>, oper_info={u'imid': u'contrail:virtual-machine-interface:default-dom...9c4d69ce-a044-46cf-a97f-46d999f0b448__1__left__2', u'obj_dict': {u'display_name': u'default-domain__PROJ1__9c4d69ce-a044-46cf-a97f-46d999f0b448__1__left__2', u'fq_name': [u'default-domain', u'PROJ1', u'default-domain__PROJ1__9c4d69ce-a044-46cf-a97f-46d999f0b448__1__left__2'], u'id_perms': {u'created': u'2016-10-22T23:23:26.510008', u'creator': None, u'description': None, u'enable': True, u'last_modified': u'2016-10-22T23:28:01.393166', u'permissions': {u'group': u'KeystoneServiceAdmin', u'group_access': 7, u'other_access': 7, u'owner': u'admin', u'owner_access': 7}, u'user_visible': True, u'uuid': {u'uuid_lslong': 12626384721742063782L, u'uuid_mslong': 7604013375239702565}}, u'parent_href': u'http://13.1.0.10:8082/project/38f6091b-0ee1-4154-93d5-5122c54bfc40', u'parent_type': u'project', u'parent_uuid': u'38f6091b-0ee1-4154-93d5-5122c54bfc40', u'perms2': {u'global_access': 0, u'owner': u'38f6091b-0ee1-4154-93d5-5122c54bfc40', u'owner_access': 7, u'share': []}, u'routing_instance_refs': [{u'attr': {u'direction': u'both', u'dst_mac': None, u'ipv6_service_chain_address': None, u'mpls_label': None, u'protocol': None, u'service_chain_address': None, u'src_mac': None, u'vlan_tag': None}, u'href': u'http://13.1.0.10:8082/routing-instance/056f167d-d0c6-4083-8086-16b986e23d4b', u'to': [u'default-domain', u'PROJ1', u'POOL1-NET1', u'POOL1-NET1'], u'uuid': u'056f167d-d0c6-4083-8086-16b986e23d4b'}], u'security_group_refs': [{u'attr': None, u'href': u'http://13.1.0.10:8082/security-group/e74b5c76-2730-46a2-817d-882131220788', u'to': [u'default-domain', u'PROJ1', u'default'], u'uuid': u'e74b5c76-2730-46a2-817d-882131220788'}], u'uuid': u'6986e1e0-827b-4825-af39-ee01c8dae8a6', ...}, u'oper': u'DELETE', u'parent_imid': u'contrail:project:default-domain:PROJ1', u'type': u'virtual-machine-interface', u'uuid': u'6986e1e0-827b-4825-af39-ee01c8dae8a6'})
965 self._dbe_update_notification(oper_info)
966 elif oper_info['oper'] == 'DELETE':
967 self._dbe_delete_notification(oper_info)
968
969 trace_msg(trace, 'MessageBusNotifyTraceBuf', self._sandesh)
self = <vnc_cfg_api_server.vnc_cfg_ifmap.VncServerKombuClient object>
self._dbe_delete_notification = <bound method VncServerKombuClient._dbe_delete_n...erver.vnc_cfg_ifmap.VncServerKombuClient object>>
oper_info = {u'imid': u'contrail:virtual-machine-interface:default-dom...9c4d69ce-a044-46cf-a97f-46d999f0b448__1__left__2', u'obj_dict': {u'display_name': u'default-domain__PROJ1__9c4d69ce-a044-46cf-a97f-46d999f0b448__1__left__2', u'fq_name': [u'default-domain', u'PROJ1', u'default-domain__PROJ1__9c4d69ce-a044-46cf-a97f-46d999f0b448__1__left__2'], u'id_perms': {u'created': u'2016-10-22T23:23:26.510008', u'creator': None, u'description': None, u'enable': True, u'last_modified': u'2016-10-22T23:28:01.393166', u'permissions': {u'group': u'KeystoneServiceAdmin', u'group_access': 7, u'other_access': 7, u'owner': u'admin', u'owner_access': 7}, u'user_visible': True, u'uuid': {u'uuid_lslong': 12626384721742063782L, u'uuid_mslong': 7604013375239702565}}, u'parent_href': u'http://13.1.0.10:8082/project/38f6091b-0ee1-4154-93d5-5122c54bfc40', u'parent_type': u'project', u'parent_uuid': u'38f6091b-0ee1-4154-93d5-5122c54bfc40', u'perms2': {u'global_access': 0, u'owner': u'38f6091b-0ee1-4154-93d5-5122c54bfc40', u'owner_access': 7, u'share': []}, u'routing_instance_refs': [{u'attr': {u'direction': u'both', u'dst_mac': None, u'ipv6_service_chain_address': None, u'mpls_label': None, u'protocol': None, u'service_chain_address': None, u'src_mac': None, u'vlan_tag': None}, u'href': u'http://13.1.0.10:8082/routing-instance/056f167d-d0c6-4083-8086-16b986e23d4b', u'to': [u'default-domain', u'PROJ1', u'POOL1-NET1', u'POOL1-NET1'], u'uuid': u'056f167d-d0c6-4083-8086-16b986e23d4b'}], u'security_group_refs': [{u'attr': None, u'href': u'http://13.1.0.10:8082/security-group/e74b5c76-2730-46a2-817d-882131220788', u'to': [u'default-domain', u'PROJ1', u'default'], u'uuid': u'e74b5c76-2730-46a2-817d-882131220788'}], u'uuid': u'6986e1e0-827b-4825-af39-ee01c8dae8a6', ...}, u'oper': u'DELETE', u'parent_imid': u'contrail:project:default-domain:PROJ1', u'type': u'virtual-machine-interface', u'uuid': u'6986e1e0-827b-4825-af39-ee01c8dae8a6'}
/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py in _dbe_delete_notification(self=<vnc_cfg_api_server.vnc_cfg_ifmap.VncServerKombuClient object>, obj_info={u'imid': u'contrail:virtual-machine-interface:default-dom...9c4d69ce-a044-46cf-a97f-46d999f0b448__1__left__2', u'obj_dict': {u'display_name': u'default-domain__PROJ1__9c4d69ce-a044-46cf-a97f-46d999f0b448__1__left__2', u'fq_name': [u'default-domain', u'PROJ1', u'default-domain__PROJ1__9c4d69ce-a044-46cf-a97f-46d999f0b448__1__left__2'], u'id_perms': {u'created': u'2016-10-22T23:23:26.510008', u'creator': None, u'description': None, u'enable': True, u'last_modified': u'2016-10-22T23:28:01.393166', u'permissions': {u'group': u'KeystoneServiceAdmin', u'group_access': 7, u'other_access': 7, u'owner': u'admin', u'owner_access': 7}, u'user_visible': True, u'uuid': {u'uuid_lslong': 12626384721742063782L, u'uuid_mslong': 7604013375239702565}}, u'parent_href': u'http://13.1.0.10:8082/project/38f6091b-0ee1-4154-93d5-5122c54bfc40', u'parent_type': u'project', u'parent_uuid': u'38f6091b-0ee1-4154-93d5-5122c54bfc40', u'perms2': {u'global_access': 0, u'owner': u'38f6091b-0ee1-4154-93d5-5122c54bfc40', u'owner_access': 7, u'share': []}, u'routing_instance_refs': [{u'attr': {u'direction': u'both', u'dst_mac': None, u'ipv6_service_chain_address': None, u'mpls_label': None, u'protocol': None, u'service_chain_address': None, u'src_mac': None, u'vlan_tag': None}, u'href': u'http://13.1.0.10:8082/routing-instance/056f167d-d0c6-4083-8086-16b986e23d4b', u'to': [u'default-domain', u'PROJ1', u'POOL1-NET1', u'POOL1-NET1'], u'uuid': u'056f167d-d0c6-4083-8086-16b986e23d4b'}], u'security_group_refs': [{u'attr': None, u'href': u'http://13.1.0.10:8082/security-group/e74b5c76-2730-46a2-817d-882131220788', u'to': [u'default-domain', u'PROJ1', u'default'], u'uuid': u'e74b5c76-2730-46a2-817d-882131220788'}], u'uuid': u'6986e1e0-827b-4825-af39-ee01c8dae8a6', ...}, u'oper': u'DELETE', u'parent_imid': u'contrail:project:default-domain:PROJ1', u'type': u'virtual-machine-interface', u'uuid': u'6986e1e0-827b-4825-af39-ee01c8dae8a6'})
1075 finally:
1076 (ok, ifmap_result) = self._ifmap_db.object_delete(obj_info['type'],
1077 obj_info)
1078 if not ok:
1079 self.config_log(ifmap_result, level=SandeshLevel.SYS_ERR)
I am looking into it.