[ubuntu-havana-R1.10-#16] SI cleanup is failing

[Prashant Shetty]

SI cleanup is failing in this regression[R1.10 #16] with below trace. Can you analyze the below logs collected.
All logs/cores will be at /cs-shared/test_runs/nodea35/2014_08_17_21_32_58 on nodeb10.englab.juniper.net

Logs:

 u'oper': u'DELETE',
 u'parent_imid': u'contrail:project:default-domain:admin',
 u'type': u'virtual-machine-interface',
 u'uuid': u'31873f5f-4826-4224-8799-ERROR:vnc_cfg_api_server.vnc_cfg_ifmap:Exception in _dbe_oper_subscribe :
<type 'exceptions.KeyError'>
Python 2.7.3: /usr/bin/python
Sun Aug 17 22:29:21 2014

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py in _dbe_oper_subscribe(self=<vnc_cfg_api_server.vnc_cfg_ifmap.VncKombuClient object>)
 1085 self._dbe_update_notification(oper_info)
 1086 elif oper_info['oper'] == 'DELETE':
 1087 self._dbe_delete_notification(oper_info)
 1088 except Exception as e:
 1089 string_buf = cStringIO.StringIO()
self = <vnc_cfg_api_server.vnc_cfg_ifmap.VncKombuClient object>
self._dbe_delete_notification = <bound method VncKombuClient._dbe_delete_notific..._api_server.vnc_cfg_ifmap.VncKombuClient object>>
oper_info = {u'imid': u'contrail:virtual-machine:112053a1-9310-4126-950e-375d1915cf60', u'obj_dict': {u'display_name': u'112053a1-9310-4126-950e-375d1915cf60', u'fq_name': [u'112
053a1-9310-4126-950e-375d1915cf60'], u'id_perms': {u'created': u'2014-08-17T16:52:40.230431', u'description': None, u'enable': True, u'last_modified': u'2014-08-17T16:52:40.40069
6', u'permissions': {u'group': u'cloud-admin-group', u'group_access': 7, u'other_access': 7, u'owner': u'cloud-admin', u'owner_access': 7}, u'uuid': {u'uuid_lslong': 107405830343
17557600L, u'uuid_mslong': 1234078251321671974}}, u'service_instance_refs': [{u'attr': None, u'href': u'http://0.0.0.0:9100/service-instance/d4420238-3fa8-44b6-bb37-1e0dcfb8defd'
, u'to': [u'ERROR'], u'uuid': u'd4420238-3fa8-44b6-bb37-1e0dcfb8defd'}], u'uuid': u'112053a1-9310-4126-950e-375d1915cf60'}, u'oper': u'DELETE', u'type': u'virtual-machine', u'uui
d': u'112053a1-9310-4126-950e-375d1915cf60'}

 /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py in _dbe_delete_notification(self=<vnc_cfg_api_server.vnc_cfg_ifmap.VncKombuClient object>, obj_info={u'imid'
: u'contrail:virtual-machine:112053a1-9310-4126-950e-375d1915cf60', u'obj_dict': {u'display_name': u'112053a1-9310-4126-950e-375d1915cf60', u'fq_name': [u'112053a1-9310-4126-950e
-375d1915cf60'], u'id_perms': {u'created': u'2014-08-17T16:52:40.230431', u'description': None, u'enable': True, u'last_modified': u'2014-08-17T16:52:40.400696', u'permissions':
{u'group': u'cloud-admin-group', u'group_access': 7, u'other_access': 7, u'owner': u'cloud-admin', u'owner_access': 7}, u'uuid': {u'uuid_lslong': 10740583034317557600L, u'uuid_ms
long': 1234078251321671974}}, u'service_instance_refs': [{u'attr': None, u'href': u'http://0.0.0.0:9100/service-instance/d4420238-3fa8-44b6-bb37-1e0dcfb8defd', u'to': [u'ERROR'],
 u'uuid': u'd4420238-3fa8-44b6-bb37-1e0dcfb8defd'}], u'uuid': u'112053a1-9310-4126-950e-375d1915cf60'}, u'oper': u'DELETE', u'type': u'virtual-machine', u'uuid': u'112053a1-9310-
4126-950e-375d1915cf60'})
 1177 method_name = obj_info['type'].replace('-', '_')
 1178 method = getattr(self._ifmap_db, "_ifmap_%s_delete" % (method_name))
 1179 (ok, ifmap_result) = method(obj_info)
 1180 if not ok:
 1181 raise Exception(ifmap_result)
ok undefined
ifmap_result undefined
method = <bound method VncIfmapClient._ifmap_virtual_mach..._api_server.vnc_cfg_ifmap.VncIfmapClient object>>
.
.
.
The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1087, in _dbe_oper_subscribe
    self._dbe_delete_notification(oper_info)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1179, in _dbe_delete_notification
    (ok, ifmap_result) = method(obj_info)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_ifmap_client_gen.py", line 7949, in _ifmap_virtual_machine_delete
    self._delete_id_pair_meta(r_elem[0].attrib['name'], to_imid, meta_name)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 287, in _delete_id_pair_meta
    _id_to_metas_delete(id_x, id_y, meta_name)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 268, in _id_to_metas_delete
    if meta_name not in self._id_to_metas[id1]:
KeyError: 'contrail:service-instance:default-domain:admin:mirror_si_1'

[Rudra Rugge (rrugge)]

Fix committed for R1.10
https://github.com/Juniper/contrail-controller/commit/56b0fa54b893afd024452fde79395936e9867b11

[Rudra Rugge (rrugge)]

Prashant - is this relevant for mainline?