Juniper Openstack

ServerManager :: Neutron client exceptions seen on HA setup.

  1. Series r3.1
  2. Bug #1635366
Bug #1635366 reported by Ritam Gangopadhyay
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Won't Fix
High
Dheeraj Gautam
R3.0.3.x
Won't Fix
High
Dheeraj Gautam
R3.1
Fix Committed
High
Dheeraj Gautam
R3.2
Fix Committed
High
Dheeraj Gautam
Juniper Openstack r3.2.0.0-fcs "r3.2.0.0"
Trunk
Fix Committed
High
Dheeraj Gautam
Juniper Openstack r4.0.0.0-fcs "r4.0.0.0"

Bug Description

Sanity scripts are failing with nova client exception since R3.0.3. It's seen that nova api calls neutron client internally which doesn't have anything to do with the particular test script and is a general code flow. Here neutron client exceptions are seen in the nova api logs which match in time stamp with the test script failures.

Trace as below:-

2016-10-19 14:21:17.482 31940 ERROR nova.api.openstack [req-df5ee8f1-b78f-4702-969f-7076bceb306d fddbcbfb1d1a4df2bb4ec4cab09b3b2c ec76a56b1ccc4e1e80111ce70a40ad36 - - -] Caught error: Authentication required
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack Traceback (most recent call last):
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 125, in call
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack return req.get_response(self.application)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack application, catch_exc_info=False)2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack app_iter = application(self.environ, start_response)2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in call
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack return resp(environ, start_response)2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py
", line 634, in __call__2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack return self._call_app(env, start_response)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 554, in _call_app
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack return self._app(env, _fake_start_response)
[11:39:22 PM] sudheendra rao: 2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in call
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack return resp(environ, start_response)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in call
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack return resp(environ, start_response)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in call
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack response = self.app(environ, start_response)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in call
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack return resp(environ, start_response)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in call
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack resp = self.call_func(req, args, *self.kwargs)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack return self.func(req, args, *kwargs)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 756, in call
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack content_type, body, accept)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 847, in _process_stack
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack request, action_args)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 710, in post_process_extensions
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack **action_args)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 461, in show
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack return self._show(req, resp_obj)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 457, in _show
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack self._extend_servers(req, [resp_obj.obj['server']])
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 422, in _extend_servers
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack servers))
[11:39:34 PM] sudheendra rao: 2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 355, in get_instances_security_groups_bindings
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack security_groups = self._get_secgroups_from_port_list(ports, neutron)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 339, in _get_secgroups_from_port_list
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack search_results = neutron.list_security_groups(**sg_search_opts)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 99, in with_params
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack ret = self.function(instance, args, *kwargs)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 812, in list_security_groups
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack retrieve_all, **_params)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 304, in list
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack for r in self._pagination(collection, path, **params):
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 317, in _pagination
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack res = self.get(path, params=params)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 290, in get
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack headers=headers, params=params)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 267, in retry_request
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack headers=headers, params=params)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 208, in do_request
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack self._handle_fault_response(status_code, replybody)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 182, in _handle_fault_response
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack exception_handler_v20(status_code, des_error_body)
[11:39:46 PM] sudheendra rao: 2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 80, in exception_handler_v20
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack message=message)
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack NeutronClientException: Authentication required
2016-10-19 14:21:17.482 31940 TRACE nova.api.openstack

Ritam Gangopadhyay (ritam)
Changed in juniperopenstack:
milestone: r3.0.3.1 → none
assignee: nobody → Abhay Joshi (abhayj)
milestone: none → r3.0.3.1
Revision history for this message
Abhay Joshi (abhayj) wrote :

Pre-existing. Won't fix in 3.0.3.1.

Changed in juniperopenstack:
assignee: Abhay Joshi (abhayj) → Dheeraj Gautam (dgautam)
milestone: r3.0.3.1 → r3.2.0.0-fcs
Sudheendra Rao (sudheendra-k)
information type: Proprietary → Public
Revision history for this message
Dheeraj Gautam (dgautam) wrote :

Updates so far:
================
This issue comes with Sanity and test cases ran in parallel. When running individual testcases or running sanity in serial, This issue doesn't appear.

Other possible cause was continuous restart of haproxy by hamon script. Restart of haproxy will cause all mysql and nova connections to reset and cause above reported issues. https://review.opencontrail.org/#/c/25626/ review fixes haproxy restart issue.

Revision history for this message
Dheeraj Gautam (dgautam) wrote :

Issue is not seen on recent build of R3.2-4, so closing for now.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.