api: ifmap update fails when messages are recevied out of order

Bug #1583332 reported by Senthilnathan Murugappan
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
Won't Fix
Undecided
Ignatious Johnson Christopher
R2.21.x
Won't Fix
Undecided
Ignatious Johnson Christopher
R2.22.x
Fix Committed
Medium
Ignatious Johnson Christopher
R3.0
Fix Committed
Medium
Ignatious Johnson Christopher
Trunk
Fix Committed
Medium
Ignatious Johnson Christopher

Bug Description

ERROR:contrail-api:__default__ [SYS_ERR]: VncApiError: <type 'exceptions.KeyError'>
Python 2.7.6: /usr/bin/python
Fri May 13 10:45:59 2016

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py in _dbe_subscribe_callback(self=<vnc_cfg_api_server.vnc_cfg_ifmap.VncServerKombuClient object>, oper_info={u'oper': u'UPDATE', u'type': u'virtual-machine-interface', u'uuid': u'f0b8ced6-fd72-4cca-97a5-ae16ad9f492b'})
  812 self._dbe_create_notification(oper_info)
  813 if oper_info['oper'] == 'UPDATE':
  814 self._dbe_update_notification(oper_info)
  815 elif oper_info['oper'] == 'DELETE':
  816 self._dbe_delete_notification(oper_info)
self = <vnc_cfg_api_server.vnc_cfg_ifmap.VncServerKombuClient object>
self._dbe_update_notification = <bound method VncServerKombuClient._dbe_update_n...erver.vnc_cfg_ifmap.VncServerKombuClient object>>
oper_info = {u'oper': u'UPDATE', u'type': u'virtual-machine-interface', u'uuid': u'f0b8ced6-fd72-4cca-97a5-ae16ad9f492b'}

 /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py in _dbe_update_notification(self=<vnc_cfg_api_server.vnc_cfg_ifmap.VncServerKombuClient object>, obj_info={u'oper': u'UPDATE', u'type': u'virtual-machine-interface', u'uuid': u'f0b8ced6-fd72-4cca-97a5-ae16ad9f492b'}) 884 method_name = obj_info['type'].replace('-', '_') 885 method = getattr(self._ifmap_db, "_ifmap_%s_update" % (method_name)) 886 (ok, ifmap_result) = method(ifmap_id, new_obj_dict)
  887 if not ok: 888 raise Exception(ifmap_result)ok = Trueifmap_result undefinedmethod = <bound method VncIfmapClient._ifmap_virtual_mach..._api_server.vnc_cfg_ifmap.VncIfmapClient object>>ifmap_id = u'contrail:virtual-machine-interface:default-dom...b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2'new_obj_dict = {'display_name': u'default-domain__TestPublicTenant__si_b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2', 'fq_name': [u'default-domain', u'TestPublicTenant', u'default-domain__TestPublicTenant__si_b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2'], 'id_perms': {u'created': u'2016-05-13T10:45:59.417750', u'creator': None, u'description': None, u'enable': True, u'last_modified': u'2016-05-13T10:45:59.606126', u'permissions': {u'group': u'KeystoneAdmin', u'group_access': 7, u'other_access': 7, u'owner': u'admin', u'owner_access': 7}, u'user_visible': False, u'uuid': {u'uuid_lslong': 10927331483309787435L, u'uuid_mslong': 17345841387587914954L}}, 'parent_href': u'http://0.0.0.0:9100/project/b9b26e6d-75c0-4738-a9be-c2c673879328', 'parent_type': u'project', 'parent_uuid': 'b9b26e6d-75c0-4738-a9be-c2c673879328', 'security_group_refs': [{'attr': None, 'href': 'http://0.0.0.0:9100/security-group/0aaeff11-2945-4cc9-9dec-99362d592ed9', 'to': [u'default-domain', u'TestPublicTenant', u'default'], 'uuid': '0aaeff11-2945-4cc9-9dec-99362d592ed9'}], 'uuid': u'f0b8ced6-fd72-4cca-97a5-ae16ad9f492b', 'virtual_machine_interface_mac_addresses': {u'mac_address': [u'02:b9:17:2d:f0:70']}, 'virtual_machine_interface_properties': {u'interface_mirror': None, u'local_preference': 200, u'service_interface_type': u'left', u'sub_interface_vlan_tag': None}, ...}
 /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_ifmap_client_gen.py in _ifmap_virtual_machine_interface_update(self=<vnc_cfg_api_server.vnc_cfg_ifmap.VncIfmapClient object>, ifmap_id=u'contrail:virtual-machine-interface:default-dom...b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2', new_obj_dict={'display_name': u'default-domain__TestPublicTenant__si_b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2', 'fq_name': [u'default-domain', u'TestPublicTenant', u'default-domain__TestPublicTenant__si_b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2'], 'id_perms': {u'created': u'2016-05-13T10:45:59.417750', u'creator': None, u'description': None, u'enable': True, u'last_modified': u'2016-05-13T10:45:59.606126', u'permissions': {u'group': u'KeystoneAdmin', u'group_access': 7, u'other_access': 7, u'owner': u'admin', u'owner_access': 7}, u'user_visible': False, u'uuid': {u'uuid_lslong': 10927331483309787435L, u'uuid_mslong': 17345841387587914954L}}, 'parent_href': u'http://0.0.0.0:9100/project/b9b26e6d-75c0-4738-a9be-c2c673879328', 'parent_type': u'project', 'parent_uuid': 'b9b26e6d-75c0-4738-a9be-c2c673879328', 'security_group_refs': [{'attr': None, 'href': 'http://0.0.0.0:9100/security-group/0aaeff11-2945-4cc9-9dec-99362d592ed9', 'to': [u'default-domain', u'TestPublicTenant', u'default'], 'uuid': '0aaeff11-2945-4cc9-9dec-99362d592ed9'}], 'uuid': u'f0b8ced6-fd72-4cca-97a5-ae16ad9f492b', 'virtual_machine_interface_mac_addresses': {u'mac_address': [u'02:b9:17:2d:f0:70']}, 'virtual_machine_interface_properties': {u'interface_mirror': None, u'local_preference': 200, u'service_interface_type': u'left', u'sub_interface_vlan_tag': None}, ...}) 7653 def _ifmap_virtual_machine_interface_update(self, ifmap_id, new_obj_dict):
 7654 # read in refs from ifmap to determine which ones become inactive after update 7655 existing_metas = self._ifmap_virtual_machine_interface_read_to_meta_index(ifmap_id)
 7656
 7657 # remove properties that are no longer active
existing_metas undefinedself = <vnc_cfg_api_server.vnc_cfg_ifmap.VncIfmapClient object>self._ifmap_virtual_machine_interface_read_to_meta_index = <bound method VncIfmapClient._ifmap_virtual_mach..._api_server.vnc_cfg_ifmap.VncIfmapClient object>>ifmap_id = u'contrail:virtual-machine-interface:default-dom...b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2'
 /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_ifmap_client_gen.py in _ifmap_virtual_machine_interface_read_to_meta_index(self=<vnc_cfg_api_server.vnc_cfg_ifmap.VncIfmapClient object>, ifmap_id=u'contrail:virtual-machine-interface:default-dom...b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2', field_names=None) 7646 metas = {} 7647 for meta_name in metas_to_read:
 7648 if meta_name in self._id_to_metas[ifmap_id]: 7649 metas[meta_name] = self._id_to_metas[ifmap_id][meta_name] 7650 return metasmeta_name = u'virtual-machine-interface-qos-forwarding-class'self = <vnc_cfg_api_server.vnc_cfg_ifmap.VncIfmapClient object>
ifmap_id = u'contrail:virtual-machine-interface:default-dom...b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2'
<type 'exceptions.KeyError'>: u'contrail:virtual-machine-interface:default-domain:TestPublicTenant:default-domain__TestPublicTenant__si_b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2'
    __class__ = <type 'exceptions.KeyError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.KeyError object>
    __dict__ = {}
    __doc__ = 'Mapping key not found.'
    __format__ = <built-in method __format__ of exceptions.KeyError object>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.KeyError object>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.KeyError object>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.KeyError object>
    __hash__ = <method-wrapper '__hash__' of exceptions.KeyError object>
    __init__ = <method-wrapper '__init__' of exceptions.KeyError object>
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of exceptions.KeyError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.KeyError object>
    __repr__ = <method-wrapper '__repr__' of exceptions.KeyError object>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.KeyError object>
    __setstate__ = <built-in method __setstate__ of exceptions.KeyError object>
    __sizeof__ = <built-in method __sizeof__ of exceptions.KeyError object>
    __str__ = <method-wrapper '__str__' of exceptions.KeyError object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of exceptions.KeyError object>
    args = (u'contrail:virtual-machine-interface:default-dom...b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2',)
    message = u'contrail:virtual-machine-interface:default-dom...b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2'

The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 814, in _dbe_subscribe_callback
    self._dbe_update_notification(oper_info)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 886, in _dbe_update_notification
    (ok, ifmap_result) = method(ifmap_id, new_obj_dict)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_ifmap_client_gen.py", line 7655, in _ifmap_virtual_machine_interface_update
    existing_metas = self._ifmap_virtual_machine_interface_read_to_meta_index(ifmap_id)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_ifmap_client_gen.py", line 7648, in _ifmap_virtual_machine_interface_read_to_meta_index
    if meta_name in self._id_to_metas[ifmap_id]:
KeyError: u'contrail:virtual-machine-interface:default-domain:TestPublicTenant:default-domain__TestPublicTenant__si_b6fcf742-9b6f-4cf6-b3d0-16fa46fc7378__1__left__2'

Tags: config
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/20796
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/20796
Committed: http://github.org/Juniper/contrail-controller/commit/b3467d8ad5ebaff04f065c4240efe101343dcbeb
Submitter: Zuul
Branch: master

commit b3467d8ad5ebaff04f065c4240efe101343dcbeb
Author: Ignatious Johnson Christopher <email address hidden>
Date: Tue May 31 22:54:51 2016 +0000

Preserving the state of the object even if the CREATE msg
is queued after UPDATE in rabbit. Handled by reading the
database and publishing to IFMAP.

Change-Id: Ic556ceafde02bb927385025a97a93553fcc1d1da
Closes-Bug: 1583332

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/20820
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/20824
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/20824
Committed: http://github.org/Juniper/contrail-generateDS/commit/0fb677e3a0518eafee1f065efd7d57775775f53a
Submitter: Zuul
Branch: R2.22.x

commit 0fb677e3a0518eafee1f065efd7d57775775f53a
Author: Ignatious Johnson Christopher <email address hidden>
Date: Wed Jun 1 05:37:17 2016 +0000

Skip publishing to IFMAP when updating the object,
if udated msg is queued before create msg.

Subsequent create msg will read from the database
and publish to IFMAP

Change-Id: I8bc18a1de34bc599572b35d045aea88769bf82aa
Partial-Bug: 1583332

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/20856
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/20820
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/20856
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/20856
Committed: http://github.org/Juniper/contrail-controller/commit/69e430fa44d01458aa65fc80a101ce9911ae8ed5
Submitter: Zuul
Branch: R2.22.x

commit 69e430fa44d01458aa65fc80a101ce9911ae8ed5
Author: Ignatious Johnson Christopher <email address hidden>
Date: Thu Jun 2 05:13:47 2016 +0000

Preserving the state of the object even if the CREATE msg
is queued after UPDATE in rabbit. Handled by reading the
database and publishing to IFMAP.

Change-Id: I67f1f2d435d9d587787db73bdd6db08b713f5be1
Closes-Bug: 1583332

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/20820
Committed: http://github.org/Juniper/contrail-controller/commit/6c4c59614bb8b86a9c54dd35ec0676e04ada2bda
Submitter: Zuul
Branch: R3.0

commit 6c4c59614bb8b86a9c54dd35ec0676e04ada2bda
Author: Ignatious Johnson Christopher <email address hidden>
Date: Tue May 31 22:54:51 2016 +0000

Preserving the state of the object even if the CREATE msg
is queued after UPDATE in rabbit. Handled by reading the
database and publishing to IFMAP.

Change-Id: Ic556ceafde02bb927385025a97a93553fcc1d1da
Closes-Bug: 1583332

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/21085
Submitter: Hampapur Ajay (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/21085
Committed: http://github.org/Juniper/contrail-controller/commit/528dea92daabd3806995079cea65922ee4246077
Submitter: Zuul
Branch: master

commit 528dea92daabd3806995079cea65922ee4246077
Author: Hampapur Ajay <email address hidden>
Date: Fri Jun 10 10:27:59 2016 -0700

Read object from db on create notification. Reading it in
ifmap:object_create will lead to double reads from db on startup.

Change-Id: I043d007ef4ff02f3744fd44d2f962b9af67e9f37
Closes-Bug: 1583332

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/21192
Submitter: Sachin Bansal (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/21192
Committed: http://github.org/Juniper/contrail-controller/commit/077906c5e2f1c050eb7a76b3c128effc5ca21e97
Submitter: Zuul
Branch: R3.0

commit 077906c5e2f1c050eb7a76b3c128effc5ca21e97
Author: Hampapur Ajay <email address hidden>
Date: Fri Jun 10 10:27:59 2016 -0700

Read object from db on create notification. Reading it in
ifmap:object_create will lead to double reads from db on startup.

Change-Id: I043d007ef4ff02f3744fd44d2f962b9af67e9f37
Closes-Bug: 1583332
(cherry picked from commit 528dea92daabd3806995079cea65922ee4246077)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.