API Server keeps throwing NoIDError on a deleted VN , causing neutron reqs from nova to fail

[Vedamurthy Joshi]

2.0-2405 Ubuntu Single node Havana

Sanity was stuck for a long long time during this run : http://anamika.englab.juniper.net:8080/job/ubuntu-12-04_havana_Single_Node_Sanity/518/

nova logs used to have neutron exceptions
'nova list --all-tenants' cmd used to fail as well.

Log files and cassandra logs will be in http://10.204.216.50/Docs/bugs/#

contrail-api log :
===============
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_interface.py", line 428, in plugin_get_ports
    ports_info = cfgdb.port_list(context, filters)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 3452, in port_list
    port_iip_objs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 663, in _port_list
    port_info = self._port_vnc_to_neutron(port_obj, memo_req)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 1791, in _port_vnc_to_neutron
    net_obj = self._virtual_network_read(net_id=net_id)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 324, in _virtual_network_read
    fields=fields)
  File "/usr/lib/python2.7/dist-packages/vnc_api/gen/vnc_api_client_gen.py", line 4977, in virtual_network_read
    content = self._request_server(rest.OP_GET, uri, query_params)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 366, in _request_server
    % (op, url, data, content))
NoIdError: Unknown id: Error: oper 2 url /virtual-network/fb595eba-527d-471b-a164-580b74cdc403 body {'exclude_back_refs': True, 'exclude_children': True} response No virtual-network object found for id fb595eba-527d-471b-a164-580b74cdc403

nova-api.log :
============

2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack Traceback (most recent call last):
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 119, in __call__
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack return req.get_response(self.application)2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack application, catch_exc_info=False)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack return resp(environ, start_response)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 539, in __call__2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack return self.app(env, start_response)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack return resp(environ, start_response)2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack return resp(environ, start_response)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack response = self.app(environ, start_response)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack return resp(environ, start_response)2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 917, in __call__
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack content_type, body, accept)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1002, in _process_stack
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack request, action_args)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 890, in post_process_extensions
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack **action_args)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 583, in detail
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack self._extend_servers(req, list(resp_obj.obj['servers']))
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 533, in _extend_servers
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack .get_instances_security_groups_bindings(context))
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 287, in get_instances_security_groups_bindings
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack ports = neutron.list_ports().get('ports')
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 108, in with_params
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack ret = self.function(instance, *args, **kwargs)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 306, in list_ports
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack **_params)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1343, in list
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack for r in self._pagination(collection, path, **params):
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1356, in _pagination
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack res = self.get(path, params=params)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1329, in get
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack headers=headers, params=params)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1314, in retry_request
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack headers=headers, params=params)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1257, in do_request
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack self._handle_fault_response(status_code, replybody)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1227, in _handle_fault_response
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack exception_handler_v20(status_code, des_error_body)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack message=msg)
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack NeutronClientException: 500-{u'NeutronError': {u'message': u'An unknown exception occurred.', u'type': u'NeutronException', u'detail': u''}}
2014-10-23 12:58:44.289 4985 TRACE nova.api.openstack
2014-10-23 12:58:44.292 4985 INFO nova.api.openstack [req-e3a063fc-215a-4775-b5ad-1e384c382850 f016792a994d45c5b00a23d95f738a0c 06c84b9b36a540548ca579a29664060e] http://10.204.216.36:8774/v1.1/06c84b9b36a540548ca579a29664060e/servers/detail returned with HTTP 500
2014-10-23 12:58:44.293 4985 INFO nova.osapi_compute.wsgi.server [req-e3a063fc-215a-4775-b5ad-1e384c382850 f016792a994d45c5b00a23d95f738a0c 06c84b9b36a540548ca579a29664060e] 10.204.216.36 "GET /v1.1/06c84b9b36a540548ca579a29664060e/servers/detail HTTP/1.1" status: 500 len: 335 time: 0.1657028

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/4553
Committed: http://github.org/Juniper/contrail-controller/commit/66a44bdb7304f989420129da4f909b978239bb47
Submitter: Zuul
Branch: master

commit 66a44bdb7304f989420129da4f909b978239bb47
Author: Hampapur Ajay <email address hidden>
Date: Wed Nov 12 23:15:50 2014 -0800

Ignore deleted network during port-list as it may have disappeared.

In case of parallel tests, it is possible that during port list,
we collect a port which along with its network is deleted by the
time it is transformed to neutron representation. Ignore NoIdError
in such cases and proceed to remaining ports in the list.

Change-Id: I204e2bf9c216223ae2a918e20121950396554283
Fixes-Bug: #1384582

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/4876
Committed: http://github.org/Juniper/contrail-controller/commit/beb31bd4aab1ff63dce983a93ccf2ab818539d49
Submitter: Zuul
Branch: R1.10

commit beb31bd4aab1ff63dce983a93ccf2ab818539d49
Author: Hampapur Ajay <email address hidden>
Date: Wed Nov 12 23:15:50 2014 -0800

Ignore deleted network during port-list as it may have disappeared.

In case of parallel tests, it is possible that during port list,
we collect a port which along with its network is deleted by the
time it is transformed to neutron representation. Ignore NoIdError
in such cases and proceed to remaining ports in the list.

Change-Id: I204e2bf9c216223ae2a918e20121950396554283
Fixes-Bug: #1384582
(cherry picked from commit 66a44bdb7304f989420129da4f909b978239bb47)