schema crash with RT delete

[Senthilnathan Murugappan]

Observed the below errors during sanity run of transit-vn test via heat using ST version-2.

Other issue observed was.
1) ST says the SC got created however the Service-RI's are not found in api-server.
   (ST shouldnt have marked it as created)
2) From ST introspect it was observed that the VN had Service-RI in obj_refs however the object is not found in Snh_StObjectReq?object_type=routing_instance
   (Need to find out why Service-RIs are not created)

The above issues are not reproducible.
Will copy the log files to /cs-shared/bugs/<BugID>/

contrail-version:
contrail-install-packages 3.2.0.0-2746~mitaka 2746

<class 'cfgm_common.exceptions.RefsExistError'>
Python 2.7.6: /usr/bin/python
Tue Oct 18 04:37:32 2016

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in _vnc_subscribe_callback(self=<schema_transformer.to_bgp.SchemaTransformer object>, oper_info={u'imid': u'contrail:route-target:target:64001:8000028', u'obj_dict': {u'display_name': u'target:64001:8000028', u'fq_name': [u'target:64001:8000028'], u'id_perms': {u'created': u'2016-10-17T23:07:31.169847', u'creator': None, u'description': None, u'enable': True, u'last_modified': u'2016-10-17T23:07:31.169847', u'permissions': {u'group': u'cloud-admin-group', u'group_access': 7, u'other_access': 7, u'owner': u'cloud-admin', u'owner_access': 7}, u'user_visible': True, u'uuid': {u'uuid_lslong': 10811834330744670415L, u'uuid_mslong': 18226946591566090056L}}, u'perms2': {u'global_access': 0, u'owner': u'5a7bf6bd1a2f4503b8b60b4bacc2ad41', u'owner_access': 7, u'share': []}, u'uuid': u'fcf31f5a-c250-4f48-960b-5a0a4e68c8cf'}, u'oper': u'DELETE', u'type': u'route_target', u'uuid': u'fcf31f5a-c250-4f48-960b-5a0a4e68c8cf'})
  309 DBBaseST.get_obj_type_map(), self._REACTION_MAP)
  310 dependency_tracker.evaluate(obj_type, obj)
  311 obj_class.delete(obj.name)
  312 else:
  313 # unknown operation
obj_class = <class 'schema_transformer.config_db.RouteTargetST'>
obj_class.delete = <bound method __metaclass__.delete of <class 'schema_transformer.config_db.RouteTargetST'>>
obj = <schema_transformer.config_db.RouteTargetST object>
obj.name = 'target:64001:8000028'

 /usr/lib/python2.7/dist-packages/cfgm_common/vnc_db.py in delete(cls=<class 'schema_transformer.config_db.RouteTargetST'>, key='target:64001:8000028')
   82 if obj is None:
   83 return
   84 obj.delete_obj()
   85 del cls._dict[key]
   86 # end delete
obj = <schema_transformer.config_db.RouteTargetST object>
obj.delete_obj = <bound method RouteTargetST.delete_obj of <schema_transformer.config_db.RouteTargetST object>>

 /usr/lib/python2.7/dist-packages/schema_transformer/config_db.py in delete_obj(self=<schema_transformer.config_db.RouteTargetST object>)
 1327
 1328 def delete_obj(self):
 1329 self._vnc_lib.route_target_delete(fq_name=[self.name])
 1330 # end delete_obj
 1331 # end RoutTargetST
self = <schema_transformer.config_db.RouteTargetST object>
self._vnc_lib = <vnc_api.vnc_api.VncApi object>
self._vnc_lib.route_target_delete = <functools.partial object>
fq_name undefined
self.name = 'target:64001:8000028'

 /usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py in wrapper(self=<vnc_api.vnc_api.VncApi object>, *args=('route-target',), **kwargs={'fq_name': ['target:64001:8000028']})
   40 retry_on_error=False)
   41 self._parse_homepage(homepage)
   42 return func(self, *args, **kwargs)
   43 return wrapper
   44
func = <function _object_delete>
self = <vnc_api.vnc_api.VncApi object>
args = ('route-target',)
kwargs = {'fq_name': ['target:64001:8000028']}

 /usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py in _object_delete(self=<vnc_api.vnc_api.VncApi object>, res_type='route-target', fq_name=['target:64001:8000028'], id=u'08debe02-630c-43b2-a6a0-ae79af40f81a', ifmap_id=None)
  530 uri = obj_cls.resource_uri_base[res_type] + '/' + id
  531
  532 content = self._request_server(rest.OP_DELETE, uri)
  533 # end _object_delete
  534
content undefined
self = <vnc_api.vnc_api.VncApi object>
self._request_server = <bound method VncApi._request_server of <vnc_api.vnc_api.VncApi object>>
global rest = <module 'cfgm_common.rest' from '/usr/lib/python2.7/dist-packages/cfgm_common/rest.pyc'>
rest.OP_DELETE = 4
uri = u'/route-target/08debe02-630c-43b2-a6a0-ae79af40f81a'

 /usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py in _request_server(self=<vnc_api.vnc_api.VncApi object>, op=4, url=u'/route-target/08debe02-630c-43b2-a6a0-ae79af40f81a', data=None, retry_on_error=True, retry_after_authn=False, retry_count=30)
  729 return self._request(op, url, data=data, retry_on_error=retry_on_error,
  730 retry_after_authn=retry_after_authn,
  731 retry_count=retry_count)
  732 #end _request_server
  733
retry_count = 30

 /usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py in _request(self=<vnc_api.vnc_api.VncApi object>, op=4, url=u'/route-target/08debe02-630c-43b2-a6a0-ae79af40f81a', data=None, retry_on_error=True, retry_after_authn=False, retry_count=30)
  807 raise OverQuota(content)
  808 elif status == 409:
  809 raise RefsExistError(content)
  810 elif status == 504:
  811 # Request sent to API server, but no response came within 50s
global RefsExistError = <class 'cfgm_common.exceptions.RefsExistError'>
content = u"Delete when resource still referred: ['http://...-instance/cd2a3672-2225-48da-914e-2f2a6520b9f1']"
<class 'cfgm_common.exceptions.RefsExistError'>: Delete when resource still referred: ['http://10.204.216.33:8082/routing-instance/cd2a3672-2225-48da-914e-2f2a6520b9f1']
    __class__ = <class 'cfgm_common.exceptions.RefsExistError'>
    __delattr__ = <method-wrapper '__delattr__' of RefsExistError object>
    __dict__ = {}
    __doc__ = None
    __format__ = <built-in method __format__ of RefsExistError object>
    __getattribute__ = <method-wrapper '__getattribute__' of RefsExistError object>
    __getitem__ = <method-wrapper '__getitem__' of RefsExistError object>
    __getslice__ = <method-wrapper '__getslice__' of RefsExistError object>
    __hash__ = <method-wrapper '__hash__' of RefsExistError object>
    __init__ = <method-wrapper '__init__' of RefsExistError object>
    __module__ = 'cfgm_common.exceptions'
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of RefsExistError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of RefsExistError object>
    __repr__ = <method-wrapper '__repr__' of RefsExistError object>
    __setattr__ = <method-wrapper '__setattr__' of RefsExistError object>
    __setstate__ = <built-in method __setstate__ of RefsExistError object>
    __sizeof__ = <built-in method __sizeof__ of RefsExistError object>
    __str__ = <method-wrapper '__str__' of RefsExistError object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of RefsExistError object>
    __weakref__ = None
    args = (u"Delete when resource still referred: ['http://...-instance/cd2a3672-2225-48da-914e-2f2a6520b9f1']",)
    message = u"Delete when resource still referred: ['http://...-instance/cd2a3672-2225-48da-914e-2f2a6520b9f1']"

The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 311, in _vnc_subscribe_callback
    obj_class.delete(obj.name)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_db.py", line 84, in delete
    obj.delete_obj()
  File "/usr/lib/python2.7/dist-packages/schema_transformer/config_db.py", line 1329, in delete_obj
    self._vnc_lib.route_target_delete(fq_name=[self.name])
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 42, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 532, in _object_delete
    content = self._request_server(rest.OP_DELETE, uri)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 731, in _request_server
    retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 809, in _request
    raise RefsExistError(content)
RefsExistError: Delete when resource still referred: ['http://10.204.216.33:8082/routing-instance/cd2a3672-2225-48da-914e-2f2a6520b9f1']

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25262
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25263
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25262
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25263
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25262
Committed: http://github.org/Juniper/contrail-controller/commit/7cd871f52e256ef64c0d3d0a5b7f6385e5e4e99f
Submitter: Zuul
Branch: master

commit 7cd871f52e256ef64c0d3d0a5b7f6385e5e4e99f
Author: Ranjeet R <email address hidden>
Date: Mon Oct 24 18:32:30 2016 -0700

Fixes: schema crash with RT delete

When a RT is deleted, we call a RT delete
on the object again. Providing a new helper
function so that the RT can be deleted without
calling the delete method.

Closes-Bug 1635048

Change-Id: I506db27382b7d64d9451a619956f6e9edc785385

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25263
Committed: http://github.org/Juniper/contrail-controller/commit/965276083296e54fafb4c87d18450885a970a812
Submitter: Zuul
Branch: R3.2

commit 965276083296e54fafb4c87d18450885a970a812
Author: Ranjeet R <email address hidden>
Date: Mon Oct 24 18:38:35 2016 -0700

Fixes: schema crash with RT delete

When a RT is deleted, we call a RT delete
on the object again. Providing a new helper
function so that the RT can be deleted without
calling the delete method.

Closes-Bug 1635048

Change-Id: Ie438341f51c350989bcafcc58fb07650480e191c

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/29455
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/29456
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/29457
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/29456
Committed: http://github.org/Juniper/contrail-controller/commit/83033b87499bc8bd822f7c433812c2e96cbf664f
Submitter: Zuul (<email address hidden>)
Branch: R3.0

commit 83033b87499bc8bd822f7c433812c2e96cbf664f
Author: Ranjeet R <email address hidden>
Date: Mon Oct 24 18:38:35 2016 -0700

Fixes: schema crash with RT delete

When a RT is deleted, we call a RT delete
on the object again. Providing a new helper
function so that the RT can be deleted without
calling the delete method.

Closes-Bug 1635048

Change-Id: Ie438341f51c350989bcafcc58fb07650480e191c
(cherry picked from commit 965276083296e54fafb4c87d18450885a970a812)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/29457
Committed: http://github.org/Juniper/contrail-controller/commit/b8e431d2b14bc1ffa77af9e78bc46e1eb9d4ca1c
Submitter: Zuul (<email address hidden>)
Branch: R3.0.3.x

commit b8e431d2b14bc1ffa77af9e78bc46e1eb9d4ca1c
Author: Ranjeet R <email address hidden>
Date: Mon Oct 24 18:38:35 2016 -0700

Fixes: schema crash with RT delete

When a RT is deleted, we call a RT delete
on the object again. Providing a new helper
function so that the RT can be deleted without
calling the delete method.

Closes-Bug 1635048

Change-Id: Ie438341f51c350989bcafcc58fb07650480e191c
(cherry picked from commit 965276083296e54fafb4c87d18450885a970a812)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/29455
Committed: http://github.org/Juniper/contrail-controller/commit/69cbf35bb4bf0640757bdb5a1fc89c0728346a65
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 69cbf35bb4bf0640757bdb5a1fc89c0728346a65
Author: Ranjeet R <email address hidden>
Date: Mon Oct 24 18:38:35 2016 -0700

Fixes: schema crash with RT delete

When a RT is deleted, we call a RT delete
on the object again. Providing a new helper
function so that the RT can be deleted without
calling the delete method.

Closes-Bug 1635048

Change-Id: Ie438341f51c350989bcafcc58fb07650480e191c
(cherry picked from commit 965276083296e54fafb4c87d18450885a970a812)