[Build "Ubuntu 12.04.3 LTS" 2.0-2441~havana] : Same instance ip deleted twice,causing exception in neutron

[Sandip Dey]

Logs saved at : nodeb10.englab.juniper.net:/cs-shared/test_runs/nodeb2/2014_11_06_22_16_03

Please see the logs as below.Same instance ip was tried to be deleted twice, casing neutron error and in turn port does not get deleted and finally virtual network not getting deleted.

Prakash did some initial analysis.

127.0.0.1 - - [2014-11-06 22:58:13] "DELETE /instance-ip/84a96651-ba07-49f1-9ffe-d1009913f153 HTTP/1.1" 200 115 0.086180
INFO:api-0:Received error(xid=221) NoNodeError((), {})
INFO:api-0:Sending request(xid=223): GetData(path=u'/api-server/subnets/default-domain:TestECMPSanity-90268094:in_network_vn1-16714651:35.145.34.0/24/0596713986', watcher=None)
127.0.0.1 - - [2014-11-06 22:58:13] "GET /virtual-machine-interface/bc89d16e-e715-4145-b9d2-6757dad22e98?fields=logical_router_back_refs%2Cinstance_ip_back_refs%2Cfloating_ip_back_refs HTTP/1.1" 200 3467 0.006365
INFO:api-0:Received error(xid=222) NoNodeError((), {})
INFO:api-0:Sending request(xid=224): Create(path=u'/api-server/subnets/default-domain:TestECMPSanity-90268094:in_network_vn1-16714651:35.145.34.0/24/0596713986', data='f0aa0531-cf1d-49f9-b741-de2979647de7', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0)
INFO:api-0:Received error(xid=223) NoNodeError((), {})
INFO:api-0:Sending request(xid=225): Create(path=u'/api-server/subnets/default-domain:TestECMPSanity-90268094:in_network_vn1-16714651:35.145.34.0/24/0596713986', data='6031376e-43ed-4872-81af-e4079a3b379d', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0)
INFO:api-0:Sending request(xid=226): Delete(path=u'/fq-name-to-uuid/virtual_machine_interface:default-domain:TestECMPSanity-90268094:default-domain__TestECMPSanity-90268094__7d5c8930-1c34-499c-b50e-31e22db8dd3d__1-left', version=-1)
INFO:api-0:Received response(xid=224): u'/api-server/subnets/default-domain:TestECMPSanity-90268094:in_network_vn1-16714651:35.145.34.0/24/0596713986'
127.0.0.1 - - [2014-11-06 22:58:13] "DELETE /instance-ip/84a96651-ba07-49f1-9ffe-d1009913f153 HTTP/1.1" 409 154 0.114358

Second delete causing the error as

vnc_api error
===========
The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_interface.py", line 418, in plugin_delete_port
    cfgdb.port_delete(port['id'])
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 3458, in port_delete
    instance_ip_id=iip_back_ref['uuid'])
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 423, in _instance_ip_delete
    self._vnc_lib.instance_ip_delete(id=instance_ip_id)
  File "/usr/lib/python2.7/dist-packages/vnc_api/gen/vnc_api_client_gen.py", line 516, in instance_ip_delete
    content = self._request_server(rest.OP_DELETE, uri)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 342, in _request_server
    retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 387, in _request
    raise RefsExistError(content)
RefsExistError: NotFoundException(_message=None)

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/bottle.py", line 764, in _handle
    return route.call(**args)
  File "/usr/lib/python2.7/dist-packages/bottle.py", line 1625, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python2.7/dist-packages/bottle.py", line 1575, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_interface.py", line 470, in plugin_http_post_port
    return self.plugin_delete_port(context, port)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_interface.py", line 422, in plugin_delete_port
    raise e
RefsExistError: NotFoundException(_message=None)

neutron error
===========

2014-11-06 22:58:14.135 ERROR [neutron.api.v2.resource] delete failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 84, in resource
    result = method(request=request, **args)
  File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 432, in delete
    obj_deleter(request.context, id, **kwargs)
  File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 499, in delete_port
    self._delete_resource('port', context, port_id)
  File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 304, in _delete_resource
    obj_name=res_type)
  File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 235, in _raise_contrail_error
    raise exc.NeutronException(**info)
NeutronException: An unknown exception occurred.
2014-11-06 22:58:14.174 ERROR [neutron.api.v2.resource] delete failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 84, in resource
    result = method(request=request, **args)
  File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 432, in delete
    obj_deleter(request.context, id, **kwargs)
  File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 499, in delete_port
    self._delete_resource('port', context, port_id)
  File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 304, in _delete_resource
    obj_name=res_type)
  File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 235, in _raise_contrail_error
    raise exc.NeutronException(**info)
NeutronException: An unknown exception occurred.

[Sachin Bansal (sbansal)]

nodeb10 is not accessible without a juniper unix account. Could you please copy the logs at a place that is accessible.

[Sachin Bansal (sbansal)]

This is most likely fixed with this commit: http://github.org/Juniper/contrail-controller/commit/75a56bf73e2a24e78299c78e81896c204c47cc58

Could you please try on a build that includes this fix?

[Vedamurthy Joshi (vedujoshi)]

Per comment#2, it should be fixed in master..marking the state accordingly