[3.2.0.0-8~mitaka] Neutron Traceback while loading scale config

[chhandak]

Observed this Neutron Server Traceback while loading scale config

What Script was doing:

Creating Physical Interface
Create VN
Creating Logical Interface
Creating Server (BMS)
Associating to LIF

Same is repeated in parallel for 4 TOR

2016-12-07 16:23:52.251 6165 INFO neutron.wsgi [-] 172.17.90.1 - - [07/Dec/2016 16:23:52] "POST /v2.0/ports.json HTTP/1.1" 401 281 0.002376
2016-12-07 16:23:57.311 6172 INFO neutron.wsgi [req-27bc4241-244f-4a64-99f8-93314f63c059 c8a8aae457f2464281d4972304532ff7 1c6eb4ffa690412185ec7d7453f57b94 - - -] 172.17.90.1 - - [07/Dec/2016 16:23:57] "POST /v2.0/ports.json HTTP/1.1" 201 838 4.086853
2016-12-07 16:24:03.439 6136 INFO neutron.wsgi [req-0b6a698c-faf5-48e7-9871-5612a4bbc3e4 cebcd14afff94b018bdd7e11b219981b 745b02c70dc941e6887e027a5da3f463 - - -] 172.17.90.1 - - [07/Dec/2016 16:24:03] "POST /v2.0/ports.json HTTP/1.1" 201 839 5.172971
2016-12-07 16:24:10.288 6133 INFO neutron.wsgi [req-0150fa5a-9278-400e-9f83-57f1e396b9a9 cebcd14afff94b018bdd7e11b219981b 745b02c70dc941e6887e027a5da3f463 - - -] 172.17.90.1 - - [07/Dec/2016 16:24:10] "POST /v2.0/ports.json HTTP/1.1" 201 839 5.723876
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource [req-a6ae6049-2921-4605-b645-699d0da58963 d7b3a2b3fc6e48488439272cd4c5fc99 d0b229c2ba1e4e379928201a5f35b027 - - -] create failed
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource Traceback (most recent call last):
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 84, in resource
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource result = method(request=request, **args)
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 410, in create
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource return self._create(request, body, **kwargs)
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource self.force_reraise()
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 521, in _create
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource obj = do_create(body)
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 503, in do_create
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource request.context, reservation.reservation_id)
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource self.force_reraise()
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 496, in do_create
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource return obj_creator(request.context, **kwargs)
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin_base.py", line 472, in create_port
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource port = self._create_resource('port', context, port)
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 260, in _create_resource
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource obj_name=res_type)
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 242, in _transform_response
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource plugin_base._raise_contrail_error(info, obj_name)
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin_base.py", line 111, in _raise_contrail_error
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource raise exc.NeutronException(**info)
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource NeutronException: An unknown exception occurred.
2016-12-07 16:24:13.251 6136 ERROR neutron.api.v2.resource
2016-12-07 16:24:13.256 6136 INFO neutron.wsgi [req-a6ae6049-2921-4605-b645-699d0da58963 d7b3a2b3fc6e48488439272cd4c5fc99 d0b229c2ba1e4e379928201a5f35b027 - - -] 172.17.90.1 - - [07/Dec/2016 16:24:13] "POST /v2.0/ports.json HTTP/1.1" 500 314 1.959660

[Sachin Bansal (sbansal)]

Please provide contrail-api-0-stdout.log after the problem happens.

[chhandak (chhandak)]

Logs copied to /auto/cores/1648274

[Sachin Bansal (sbansal)]

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_interface.py", line 406, in plugin_create_port
    net_info = cfgdb.port_create(context, port['resource'])
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 2494, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 3709, in port_create
    ports = self._virtual_machine_interface_list(back_ref_id=net_id)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 478, in _virtual_machine_interface_list
    fields=n_extra_fields)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 42, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 537, in _objects_list
    filters=filters)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 42, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 1194, in resource_list
    data = query_params)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 755, in _request_server
    retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/context.py", line 37, in wrapper
    return fn(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 83, in _request
    get_context().request, user_token)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1795, in get_auth_headers_from_token
    return self._auth_svc.get_auth_headers_from_token(request, token)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_auth_keystone.py", line 280, in get_auth_headers_from_token
    return auth_middleware(request.headers.environ, self.start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 467, in __call__
    response = req.get_response(self._app)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1292, in call_application
    return (captured[0], captured[1], app_iter)
IndexError: list index out of range

[Sachin Bansal (sbansal)]

I see the above traceback as the reason for neutron traceback. This is identical to what is reported in https://bugs.launchpad.net/juniperopenstack/r3.0/+bug/1608815. That bug supposed to be fixed in this version. Assigning to Deepinder to take a look.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27219
Submitter: Deepinder Setia (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27224
Submitter: Deepinder Setia (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27219
Committed: http://github.org/Juniper/contrail-controller/commit/1a71bb254990e53db4218cbf0e2000ee0d9daa44
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 1a71bb254990e53db4218cbf0e2000ee0d9daa44
Author: Deepinder Setia <email address hidden>
Date: Tue Dec 13 15:55:08 2016 -0800

I think start_response must called inside pseudo-app to keep WSGI
gods happy. Start_response wasn't getting called prior to this fix
if user token was invalid.

Change-Id: I84b0f6790cfd9f2c479f81951974bd5e51ba2fd1
Closes-Bug: #1648274

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27224
Committed: http://github.org/Juniper/contrail-controller/commit/9daafd84f3eba464080a40818ff673944d4e4ca7
Submitter: Zuul (<email address hidden>)
Branch: master

commit 9daafd84f3eba464080a40818ff673944d4e4ca7
Author: Deepinder Setia <email address hidden>
Date: Tue Dec 13 15:55:08 2016 -0800

I think start_response must called inside pseudo-app to keep WSGI
gods happy. Start_response wasn't getting called prior to this fix
if user token was invalid.

Change-Id: I84b0f6790cfd9f2c479f81951974bd5e51ba2fd1
Closes-Bug: #1648274