build 2709: [contrail-api]: __default__ [SYS_ERR]: VncApiError: <type 'exceptions.KeyError'>

[Ankit Jain]

"/var/log/contrail/contrail-api.log" 1299L, 234132C
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of exceptions.KeyError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.KeyError object>
    __repr__ = <method-wrapper '__repr__' of exceptions.KeyError object>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.KeyError object>
    __setstate__ = <built-in method __setstate__ of exceptions.KeyError object>
    __sizeof__ = <built-in method __sizeof__ of exceptions.KeyError object>
    __str__ = <method-wrapper '__str__' of exceptions.KeyError object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of exceptions.KeyError object>
    args = (u'contrail:routing-instance:default-domain:admin:right_vn:right_vn',)
    message = u'contrail:routing-instance:default-domain:admin:right_vn:right_vn'

The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1117, in _dbe_subscribe_callback
    self._dbe_delete_notification(oper_info)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1220, in _dbe_delete_notification
    obj_info)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 318, in object_delete
    existing_metas = self._object_read_to_meta_index(ifmap_id)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 273, in _object_read_to_meta_index
    return self._id_to_metas[ifmap_id].copy()
KeyError: u'contrail:routing-instance:default-domain:admin:right_vn:right_vn'

02/03/2016 01:41:00 AM [contrail-api]: __default__ [SYS_ERR]: VncApiError: <type 'exceptions.KeyError'>
Python 2.7.6: /usr/bin/python
Wed Feb 3 01:41:00 2016

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py in _dbe_subscribe_callback(self=<vnc_cfg_api_server.vnc_cfg_ifmap.VncServerKombuClient object>, oper_info={u'imid': u'contrail:routing-instance:default-domain:admin:right_vn:right_vn', u'obj_dict': {u'display_name': u'right_vn', u'fq_name': [u'default-domain', u'admin', u'right_vn', u'right_vn'], u'id_perms': {u'created': u'2016-02-02T20:06:53.803173', u'creator': None, u'description': None, u'enable': True, u'last_modified': u'2016-02-02T20:07:04.021884', u'permissions': {u'group': u'admin', u'group_access': 7, u'other_access': 7, u'owner': u'contrail-api', u'owner_access': 7}, u'user_visible': True, u'uuid': {u'uuid_lslong': 11883975026415117185L, u'uuid_mslong': 13411258877860597606L}}, u'parent_href': u'http://10.204.217.53:8082/virtual-network/3cf10189-a861-4914-a3ec-5af27bb2da89', u'parent_type': u'virtual-network', u'parent_uuid': u'3cf10189-a861-4914-a3ec-5af27bb2da89', u'perms2': {u'global_access': 0, u'owner': None, u'owner_access': 7, u'share': []}, u'route_target_refs': [{u'attr': {u'import_export': None}, u'href': u'http://10.204.217.53:8082/route-target/bcd4709f-54c1-454d-b455-f3ac7fd5bc86', u'to': [u'target:64512:8000005'], u'uuid': u'bcd4709f-54c1-454d-b455-f3ac7fd5bc86'}], u'routing_instance_has_pnf': False, u'routing_instance_is_default': True, ...}, u'oper': u'DELETE', u'parent_imid': u'contrail:virtual-network:default-domain:admin:right_vn', u'type': u'routing-instance', u'uuid': u'ba1e5ce4-bdaf-4f66-a4ec-5c5291162381'})
 1115 self._dbe_update_notification(oper_info)
 1116 elif oper_info['oper'] == 'DELETE':
 1117 self._dbe_delete_notification(oper_info)
 1118
 1119 trace_msg(trace, 'MessageBusNotifyTraceBuf', self._sandesh)
self = <vnc_cfg_api_server.vnc_cfg_ifmap.VncServerKombuClient object>
self._dbe_delete_notification = <bound method VncServerKombuClient._dbe_delete_n...erver.vnc_cfg_ifmap.VncServerKombuClient object>>

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/16934
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/16934
Committed: http://github.org/Juniper/contrail-controller/commit/b305b6604deefcce92dcc5d256db4f8909ca92da
Submitter: Zuul
Branch: master

commit b305b6604deefcce92dcc5d256db4f8909ca92da
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri Feb 5 14:43:41 2016 -0800

Checking for the presence of the meta key before trying to access it.

Change-Id: I81446749ada2221a837a361efdcbbbb82b8cbc59
Closes-Bug: 1541434

[Martin Millnert (r-martin-5)]

Running nightly builds from master (i.e. having incorporated the change above) I was struck by something eeringly similar, in attached error log.

It appears the API call to delete routing-instance is issued twice, where the first succeeds and the second fails due to the routing-instance not existing anymore.