api: ifmap update fails when messages are recevied out of order

[Senthilnathan Murugappan]

ERROR:contrail-api:__default__ [SYS_ERR]: VncApiError: <type 'exceptions.KeyError'>
Python 2.7.6: /usr/bin/python
Fri May 13 10:45:59 2016

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py in _dbe_subscribe_callback(self=<vnc_cfg_api_server.vnc_cfg_ifmap.VncServerKombuClient object>, oper_info={u'oper': u'UPDATE', u'type': u'virtual-machine-interface', u'uuid': u'f0b8ced6-fd72-4cca-97a5-ae16ad9f492b'})
  812 self._dbe_create_notification(oper_info)
  813 if oper_info['oper'] == 'UPDATE':
  814 self._dbe_update_notification(oper_info)
  815 elif oper_info['oper'] == 'DELETE':
  816 self._dbe_delete_notification(oper_info)
self = <vnc_cfg_api_server.vnc_cfg_ifmap.VncServerKombuClient object>
self._dbe_update_notification = <bound method VncServerKombuClient._dbe_update_n...erver.vnc_cfg_ifmap.VncServerKombuClient object>>
oper_info = {u'oper': u'UPDATE', u'type': u'virtual-machine-interface', u'uuid': u'f0b8ced6-fd72-4cca-97a5-ae16ad9f492b'}

 /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py in _dbe_update_notification(self=<vnc_cfg_api_server.vnc_cfg_ifmap.VncServerKombuClient object>, obj_info={u'oper': u'UPDATE', u'type': u'virtual-machine-interface', u'uuid': u'f0b8ced6-fd72-4cca-97a5-ae16ad9f492b'}) 884 method_name = obj_info['type'].replace('-', '_') 885 method = getattr(self._ifmap_db, "_ifmap_%s_update" % (method_name)) 886 (ok, ifmap_result) = method(ifmap_id, new_obj_dict)
  887 if not ok: 888 raise Exception(ifmap_result)ok = Trueifmap_result undefinedmethod = <bound method VncIfmapClient._ifmap_virtual_mach..._api_server.vnc_cfg_ifmap.VncIfmapClient object>>ifmap_id = u'contrail:virtual-machine-interface:default-dom...b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2'new_obj_dict = {'display_name': u'default-domain__TestPublicTenant__si_b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2', 'fq_name': [u'default-domain', u'TestPublicTenant', u'default-domain__TestPublicTenant__si_b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2'], 'id_perms': {u'created': u'2016-05-13T10:45:59.417750', u'creator': None, u'description': None, u'enable': True, u'last_modified': u'2016-05-13T10:45:59.606126', u'permissions': {u'group': u'KeystoneAdmin', u'group_access': 7, u'other_access': 7, u'owner': u'admin', u'owner_access': 7}, u'user_visible': False, u'uuid': {u'uuid_lslong': 10927331483309787435L, u'uuid_mslong': 17345841387587914954L}}, 'parent_href': u'http://0.0.0.0:9100/project/b9b26e6d-75c0-4738-a9be-c2c673879328', 'parent_type': u'project', 'parent_uuid': 'b9b26e6d-75c0-4738-a9be-c2c673879328', 'security_group_refs': [{'attr': None, 'href': 'http://0.0.0.0:9100/security-group/0aaeff11-2945-4cc9-9dec-99362d592ed9', 'to': [u'default-domain', u'TestPublicTenant', u'default'], 'uuid': '0aaeff11-2945-4cc9-9dec-99362d592ed9'}], 'uuid': u'f0b8ced6-fd72-4cca-97a5-ae16ad9f492b', 'virtual_machine_interface_mac_addresses': {u'mac_address': [u'02:b9:17:2d:f0:70']}, 'virtual_machine_interface_properties': {u'interface_mirror': None, u'local_preference': 200, u'service_interface_type': u'left', u'sub_interface_vlan_tag': None}, ...}
 /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_ifmap_client_gen.py in _ifmap_virtual_machine_interface_update(self=<vnc_cfg_api_server.vnc_cfg_ifmap.VncIfmapClient object>, ifmap_id=u'contrail:virtual-machine-interface:default-dom...b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2', new_obj_dict={'display_name': u'default-domain__TestPublicTenant__si_b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2', 'fq_name': [u'default-domain', u'TestPublicTenant', u'default-domain__TestPublicTenant__si_b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2'], 'id_perms': {u'created': u'2016-05-13T10:45:59.417750', u'creator': None, u'description': None, u'enable': True, u'last_modified': u'2016-05-13T10:45:59.606126', u'permissions': {u'group': u'KeystoneAdmin', u'group_access': 7, u'other_access': 7, u'owner': u'admin', u'owner_access': 7}, u'user_visible': False, u'uuid': {u'uuid_lslong': 10927331483309787435L, u'uuid_mslong': 17345841387587914954L}}, 'parent_href': u'http://0.0.0.0:9100/project/b9b26e6d-75c0-4738-a9be-c2c673879328', 'parent_type': u'project', 'parent_uuid': 'b9b26e6d-75c0-4738-a9be-c2c673879328', 'security_group_refs': [{'attr': None, 'href': 'http://0.0.0.0:9100/security-group/0aaeff11-2945-4cc9-9dec-99362d592ed9', 'to': [u'default-domain', u'TestPublicTenant', u'default'], 'uuid': '0aaeff11-2945-4cc9-9dec-99362d592ed9'}], 'uuid': u'f0b8ced6-fd72-4cca-97a5-ae16ad9f492b', 'virtual_machine_interface_mac_addresses': {u'mac_address': [u'02:b9:17:2d:f0:70']}, 'virtual_machine_interface_properties': {u'interface_mirror': None, u'local_preference': 200, u'service_interface_type': u'left', u'sub_interface_vlan_tag': None}, ...}) 7653 def _ifmap_virtual_machine_interface_update(self, ifmap_id, new_obj_dict):
 7654 # read in refs from ifmap to determine which ones become inactive after update 7655 existing_metas = self._ifmap_virtual_machine_interface_read_to_meta_index(ifmap_id)
 7656
 7657 # remove properties that are no longer active
existing_metas undefinedself = <vnc_cfg_api_server.vnc_cfg_ifmap.VncIfmapClient object>self._ifmap_virtual_machine_interface_read_to_meta_index = <bound method VncIfmapClient._ifmap_virtual_mach..._api_server.vnc_cfg_ifmap.VncIfmapClient object>>ifmap_id = u'contrail:virtual-machine-interface:default-dom...b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2'
 /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_ifmap_client_gen.py in _ifmap_virtual_machine_interface_read_to_meta_index(self=<vnc_cfg_api_server.vnc_cfg_ifmap.VncIfmapClient object>, ifmap_id=u'contrail:virtual-machine-interface:default-dom...b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2', field_names=None) 7646 metas = {} 7647 for meta_name in metas_to_read:
 7648 if meta_name in self._id_to_metas[ifmap_id]: 7649 metas[meta_name] = self._id_to_metas[ifmap_id][meta_name] 7650 return metasmeta_name = u'virtual-machine-interface-qos-forwarding-class'self = <vnc_cfg_api_server.vnc_cfg_ifmap.VncIfmapClient object>
ifmap_id = u'contrail:virtual-machine-interface:default-dom...b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2'
<type 'exceptions.KeyError'>: u'contrail:virtual-machine-interface:default-domain:TestPublicTenant:default-domain__TestPublicTenant__si_b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2'
    __class__ = <type 'exceptions.KeyError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.KeyError object>
    __dict__ = {}
    __doc__ = 'Mapping key not found.'
    __format__ = <built-in method __format__ of exceptions.KeyError object>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.KeyError object>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.KeyError object>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.KeyError object>
    __hash__ = <method-wrapper '__hash__' of exceptions.KeyError object>
    __init__ = <method-wrapper '__init__' of exceptions.KeyError object>
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of exceptions.KeyError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.KeyError object>
    __repr__ = <method-wrapper '__repr__' of exceptions.KeyError object>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.KeyError object>
    __setstate__ = <built-in method __setstate__ of exceptions.KeyError object>
    __sizeof__ = <built-in method __sizeof__ of exceptions.KeyError object>
    __str__ = <method-wrapper '__str__' of exceptions.KeyError object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of exceptions.KeyError object>
    args = (u'contrail:virtual-machine-interface:default-dom...b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2',)
    message = u'contrail:virtual-machine-interface:default-dom...b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2'

The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 814, in _dbe_subscribe_callback
    self._dbe_update_notification(oper_info)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 886, in _dbe_update_notification
    (ok, ifmap_result) = method(ifmap_id, new_obj_dict)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_ifmap_client_gen.py", line 7655, in _ifmap_virtual_machine_interface_update
    existing_metas = self._ifmap_virtual_machine_interface_read_to_meta_index(ifmap_id)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_ifmap_client_gen.py", line 7648, in _ifmap_virtual_machine_interface_read_to_meta_index
    if meta_name in self._id_to_metas[ifmap_id]:
KeyError: u'contrail:virtual-machine-interface:default-domain:TestPublicTenant:default-domain__TestPublicTenant__si_b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2'

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20796
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20796
Committed: http://github.org/Juniper/contrail-controller/commit/b3467d8ad5ebaff04f065c4240efe101343dcbeb
Submitter: Zuul
Branch: master

commit b3467d8ad5ebaff04f065c4240efe101343dcbeb
Author: Ignatious Johnson Christopher <email address hidden>
Date: Tue May 31 22:54:51 2016 +0000

Preserving the state of the object even if the CREATE msg
is queued after UPDATE in rabbit. Handled by reading the
database and publishing to IFMAP.

Change-Id: Ic556ceafde02bb927385025a97a93553fcc1d1da
Closes-Bug: 1583332

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20820
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20824
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20824
Committed: http://github.org/Juniper/contrail-generateDS/commit/0fb677e3a0518eafee1f065efd7d57775775f53a
Submitter: Zuul
Branch: R2.22.x

commit 0fb677e3a0518eafee1f065efd7d57775775f53a
Author: Ignatious Johnson Christopher <email address hidden>
Date: Wed Jun 1 05:37:17 2016 +0000

Skip publishing to IFMAP when updating the object,
if udated msg is queued before create msg.

Subsequent create msg will read from the database
and publish to IFMAP

Change-Id: I8bc18a1de34bc599572b35d045aea88769bf82aa
Partial-Bug: 1583332

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20856
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20820
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20856
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20856
Committed: http://github.org/Juniper/contrail-controller/commit/69e430fa44d01458aa65fc80a101ce9911ae8ed5
Submitter: Zuul
Branch: R2.22.x

commit 69e430fa44d01458aa65fc80a101ce9911ae8ed5
Author: Ignatious Johnson Christopher <email address hidden>
Date: Thu Jun 2 05:13:47 2016 +0000

Preserving the state of the object even if the CREATE msg
is queued after UPDATE in rabbit. Handled by reading the
database and publishing to IFMAP.

Change-Id: I67f1f2d435d9d587787db73bdd6db08b713f5be1
Closes-Bug: 1583332

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20820
Committed: http://github.org/Juniper/contrail-controller/commit/6c4c59614bb8b86a9c54dd35ec0676e04ada2bda
Submitter: Zuul
Branch: R3.0

commit 6c4c59614bb8b86a9c54dd35ec0676e04ada2bda
Author: Ignatious Johnson Christopher <email address hidden>
Date: Tue May 31 22:54:51 2016 +0000

Preserving the state of the object even if the CREATE msg
is queued after UPDATE in rabbit. Handled by reading the
database and publishing to IFMAP.

Change-Id: Ic556ceafde02bb927385025a97a93553fcc1d1da
Closes-Bug: 1583332

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21085
Submitter: Hampapur Ajay (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/21085
Committed: http://github.org/Juniper/contrail-controller/commit/528dea92daabd3806995079cea65922ee4246077
Submitter: Zuul
Branch: master

commit 528dea92daabd3806995079cea65922ee4246077
Author: Hampapur Ajay <email address hidden>
Date: Fri Jun 10 10:27:59 2016 -0700

Read object from db on create notification. Reading it in
ifmap:object_create will lead to double reads from db on startup.

Change-Id: I043d007ef4ff02f3744fd44d2f962b9af67e9f37
Closes-Bug: 1583332

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21192
Submitter: Sachin Bansal (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/21192
Committed: http://github.org/Juniper/contrail-controller/commit/077906c5e2f1c050eb7a76b3c128effc5ca21e97
Submitter: Zuul
Branch: R3.0

commit 077906c5e2f1c050eb7a76b3c128effc5ca21e97
Author: Hampapur Ajay <email address hidden>
Date: Fri Jun 10 10:27:59 2016 -0700

Read object from db on create notification. Reading it in
ifmap:object_create will lead to double reads from db on startup.

Change-Id: I043d007ef4ff02f3744fd44d2f962b9af67e9f37
Closes-Bug: 1583332
(cherry picked from commit 528dea92daabd3806995079cea65922ee4246077)