schema-transformer crashes when deleting a RT

[Sylvain Afchain]

On a production platform we good severals schema-transformer crash with this traceback

<class 'cfgm_common.exceptions.NoIdError'>
Python 2.7.6: /usr/bin/python
Fri May 22 10:16:15 2015

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in launch_arc(transformer=<schema_transformer.to_bgp.SchemaTransformer object>, ssrc_mapc=<cfgm_common.ifmap.client.client instance>)
 3656 result = arc_mapc.call('poll', pollreq)
 3657 try:
 3658 transformer.process_poll_result(result)
 3659 except Exception as e:
 3660 try:
transformer = <schema_transformer.to_bgp.SchemaTransformer object>
transformer.process_poll_result = <bound method SchemaTransformer.process_poll_res...ema_transformer.to_bgp.SchemaTransformer object>>
result = '<?xml version="1.0" encoding="UTF-8" standalone=...lResult></ns2:response></ns3:Body></ns3:Envelope>'

 /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in process_poll_result(self=<schema_transformer.to_bgp.SchemaTransformer object>, poll_result_str='<?xml version="1.0" encoding="UTF-8" standalone=...lResult></ns2:response></ns3:Body></ns3:Envelope>')
 3487 continue
 3488 if service_chain in (remote_service_chain_list or []):
 3489 service_chain.destroy()
 3490 else:
 3491 service_chain.delete()
service_chain = <schema_transformer.to_bgp.ServiceChain object>
service_chain.destroy = <bound method ServiceChain.destroy of <schema_transformer.to_bgp.ServiceChain object>>

 /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in destroy(self=<schema_transformer.to_bgp.ServiceChain object>)
 1768 service_ri1 = vn1_obj.rinst.get(service_name1)
 1769 if service_ri1 is not None:
 1770 vn1_obj.delete_routing_instance(service_ri1)
 1771 del vn1_obj.rinst[service_name1]
 1772 if vn2_obj:
vn1_obj = <schema_transformer.to_bgp.VirtualNetworkST object>
vn1_obj.delete_routing_instance = <bound method VirtualNetworkST.delete_routing_in...hema_transformer.to_bgp.VirtualNetworkST object>>
service_ri1 = <schema_transformer.to_bgp.RoutingInstanceST object>

 /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in delete_routing_instance(self=<schema_transformer.to_bgp.VirtualNetworkST object>, ri=<schema_transformer.to_bgp.RoutingInstanceST object>, old_ri_list=None)
  632 for ri2 in vn.rinst.values():
  633 if ri.get_fq_name_str() in ri2.connections:
  634 ri2.delete_connection(ri)
  635 rtgt_list = ri.obj.get_route_target_refs()
  636 ri_fq_name_str = ri.obj.get_fq_name_str()
ri2 = <schema_transformer.to_bgp.RoutingInstanceST object>
ri2.delete_connection = <bound method RoutingInstanceST.delete_connectio...ema_transformer.to_bgp.RoutingInstanceST object>>
ri = <schema_transformer.to_bgp.RoutingInstanceST object>

 /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in delete_connection(self=<schema_transformer.to_bgp.RoutingInstanceST object>, ri2=<schema_transformer.to_bgp.RoutingInstanceST object>)
 1389 self.connections.discard(ri2.get_fq_name_str())
 1390 ri2.connections.discard(self.get_fq_name_str())
 1391 self.obj = _vnc_lib.routing_instance_read(id=self.obj.uuid)
 1392 self.obj.del_routing_instance(ri2.obj)
 1393 _vnc_lib.routing_instance_update(self.obj)
self = <schema_transformer.to_bgp.RoutingInstanceST object>
self.obj = <vnc_api.gen.resource_client.RoutingInstance object>
global _vnc_lib = <vnc_api.vnc_api.VncApi object>
_vnc_lib.routing_instance_read = <bound method VncApi.routing_instance_read of <vnc_api.vnc_api.VncApi object>>
builtinid = <built-in function id>
self.obj.uuid = u'dd00dbb3-3a04-486d-b765-fad3b9c28a01'

 /usr/lib/python2.7/dist-packages/vnc_api/gen/vnc_api_client_gen.py in routing_instance_read(self=<vnc_api.vnc_api.VncApi object>, fq_name=None, fq_name_str=None, id=u'dd00dbb3-3a04-486d-b765-fad3b9c28a01', ifmap_id=None, fields=None)
 5314 query_params = {'exclude_back_refs':True,
 5315 'exclude_children':True,}
 5316 content = self._request_server(rest.OP_GET, uri, query_params)
 5317
 5318 obj_dict = json.loads(content)['routing-instance']
content undefined
self = <vnc_api.vnc_api.VncApi object>
self._request_server = <bound method VncApi._request_server of <vnc_api.vnc_api.VncApi object>>
global rest = <module 'cfgm_common.rest' from '/usr/lib/python2.7/dist-packages/cfgm_common/rest.pyc'>
rest.OP_GET = 2
uri = u'/routing-instance/dd00dbb3-3a04-486d-b765-fad3b9c28a01'
query_params = {'exclude_back_refs': True, 'exclude_children': True}

 /usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py in _request_server(self=<vnc_api.vnc_api.VncApi object>, op=2, url=u'/routing-instance/dd00dbb3-3a04-486d-b765-fad3b9c28a01', data={'exclude_back_refs': True, 'exclude_children': True}, retry_on_error=True, retry_after_authn=False)
  362 elif status == 404:
  363 raise NoIdError('Error: oper %s url %s body %s response %s'
  364 % (op, url, data, content))
  365 elif status == 403:
  366 raise PermissionDenied(content)
op = 2
url = u'/routing-instance/dd00dbb3-3a04-486d-b765-fad3b9c28a01'
data = {'exclude_back_refs': True, 'exclude_children': True}
content = u'No routing-instance object found for id dd00dbb3-3a04-486d-b765-fad3b9c28a01'
<class 'cfgm_common.exceptions.NoIdError'>: Unknown id: Error: oper 2 url /routing-instance/dd00dbb3-3a04-486d-b765-fad3b9c28a01 body {'exclude_back_refs': True, 'exclude_children': True} response No routing-instance object found for id dd00dbb3-3a04-486d-b765-fad3b9c28a01
    __class__ = <class 'cfgm_common.exceptions.NoIdError'>
    __delattr__ = <method-wrapper '__delattr__' of NoIdError object>
    __dict__ = {'_unknown_id': u"Error: oper 2 url /routing-instance/dd00dbb3-3...ound for id dd00dbb3-3a04-486d-b765-fad3b9c28a01"}
    __doc__ = None
    __format__ = <built-in method __format__ of NoIdError object>
    __getattribute__ = <method-wrapper '__getattribute__' of NoIdError object>
    __getitem__ = <method-wrapper '__getitem__' of NoIdError object>
    __getslice__ = <method-wrapper '__getslice__' of NoIdError object>
    __hash__ = <method-wrapper '__hash__' of NoIdError object>
    __init__ = <bound method NoIdError.__init__ of NoIdError()>
    __module__ = 'cfgm_common.exceptions'
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of NoIdError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of NoIdError object>
    __repr__ = <method-wrapper '__repr__' of NoIdError object>
    __setattr__ = <method-wrapper '__setattr__' of NoIdError object>
    __setstate__ = <built-in method __setstate__ of NoIdError object>
    __sizeof__ = <built-in method __sizeof__ of NoIdError object>
    __str__ = <bound method NoIdError.__str__ of NoIdError()>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of NoIdError object>
    __weakref__ = None
    _unknown_id = u"Error: oper 2 url /routing-instance/dd00dbb3-3...ound for id dd00dbb3-3a04-486d-b765-fad3b9c28a01"
    args = ()
    message = ''

The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 3658, in launch_arc
    transformer.process_poll_result(result)
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 3489, in process_poll_result
    service_chain.destroy()
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 1770, in destroy
    vn1_obj.delete_routing_instance(service_ri1)
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 634, in delete_routing_instance
    ri2.delete_connection(ri)
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 1391, in delete_connection
    self.obj = _vnc_lib.routing_instance_read(id=self.obj.uuid)
  File "/usr/lib/python2.7/dist-packages/vnc_api/gen/vnc_api_client_gen.py", line 5316, in routing_instance_read
    content = self._request_server(rest.OP_GET, uri, query_params)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 364, in _request_server
    % (op, url, data, content))
NoIdError: Unknown id: Error: oper 2 url /routing-instance/dd00dbb3-3a04-486d-b765-fad3b9c28a01 body {'exclude_back_refs': True, 'exclude_children': True} response No routing-instance object found for id dd00dbb3-3a04-486d-b765-fad3b9c28a01

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11361
Submitter: Sylvain Afchain (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11362
Submitter: Sylvain Afchain (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/11361
Committed: http://github.org/Juniper/contrail-controller/commit/7ee732a689aa32ed914dbe4bf3d15d0b5e6b51c4
Submitter: Zuul
Branch: master

commit 7ee732a689aa32ed914dbe4bf3d15d0b5e6b51c4
Author: Sylvain Afchain <email address hidden>
Date: Mon Jun 8 08:23:18 2015 +0000

Fix schema crash when deleting RT

Change-Id: Ib277781c2b6862fceb9590f05f4021d6c6f83140
Closes-Bug: #1462921

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/12870
Submitter: Sachin Bansal (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/12870
Committed: http://github.org/Juniper/contrail-controller/commit/72bfd5b9dd1ade5632d1f88a7ed03748c3e179f8
Submitter: Zuul
Branch: R2.22-dev

commit 72bfd5b9dd1ade5632d1f88a7ed03748c3e179f8
Author: Sylvain Afchain <email address hidden>
Date: Mon Jun 8 08:23:18 2015 +0000

Fix schema crash when deleting RT

Change-Id: Ib277781c2b6862fceb9590f05f4021d6c6f83140
Closes-Bug: #1462921
(cherry picked from commit 7ee732a689aa32ed914dbe4bf3d15d0b5e6b51c4)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13476
Submitter: Jean-Philippe Braun (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13476
Committed: http://github.org/Juniper/contrail-controller/commit/621b547706983ca13ca545e0267b3dadc6b85ce2
Submitter: Zuul
Branch: R2.20

commit 621b547706983ca13ca545e0267b3dadc6b85ce2
Author: Sylvain Afchain <email address hidden>
Date: Mon Jun 8 08:23:18 2015 +0000

Fix schema crash when deleting RT

Change-Id: Ib277781c2b6862fceb9590f05f4021d6c6f83140
Closes-Bug: #1462921
(cherry picked from commit 7ee732a689aa32ed914dbe4bf3d15d0b5e6b51c4)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11362
Submitter: ?douard Thuleau (<email address hidden>)