Juniper Openstack

while continuously booting and deleting 10 vms, boot/delete fails randomly

Bug #1548626 reported by Vedamurthy Joshi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Won't Fix
High
Hampapur Ajay
Trunk
In Progress
High
Hampapur Ajay

Bug Description

R3.0 2714 Ubuntu 14.04 Kilo multi-node setup
===========
while :
do
nova boot --nic net-id=0de455d7-1cc7-43f8-af95-69606e0b78e5 --image 5d69c762-2294-4517-937d-b3628faae05a --flavor 2 --min-count 10 vn1-servers ;
nova list | grep vn1-servers | awk '{print $2}' | xargs -L1 nova delete
done

2016-02-23 10:45:21.821 19882 ERROR nova.api.openstack [req-836c1e85-a4ff-4358-a3d0-d75fc46c7168 ba0a7bd4bede42eaaf3c4f4f47936da3 6412d041dd49437488453a7df596dff3 - - -] Caught error: Port 01d6c6d6-4f29-4414-ab63-9e8610c963e7 could not be found
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack Traceback (most recent call last):
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 125, in __call__
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack return req.get_response(self.application)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack application, catch_exc_info=False)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack return resp(environ, start_response)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 634, in __call__
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack return self._call_app(env, start_response)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 554, in _call_app
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack return self._app(env, _fake_start_response)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack return resp(environ, start_response)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack return resp(environ, start_response)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack response = self.app(environ, start_response)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack return resp(environ, start_response)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 756, in __call__
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack content_type, body, accept)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 821, in _process_stack
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 911, in dispatch
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack return method(req=request, **action_args)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 636, in create
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack self._handle_create_exception(*sys.exc_info())
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 465, in _handle_create_exception
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack six.reraise(*exc_info)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 621, in create
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack check_server_group_quota=check_server_group_quota)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 149, in inner
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack rv = f(*args, **kwargs)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1543, in create
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack check_server_group_quota=check_server_group_quota)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1151, in _create_instance
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack auto_disk_config, reservation_id, max_count)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 873, in _validate_and_build_base_options
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack requested_networks, max_count)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 520, in _check_requested_networks
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack max_count)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1029, in validate_networks
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack ports = neutron.list_ports(tenant_id=context.project_id)['ports']
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 99, in with_params
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack ret = self.function(instance, *args, **kwargs)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 505, in list_ports
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack **_params)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 304, in list
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack for r in self._pagination(collection, path, **params):
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 317, in _pagination
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack res = self.get(path, params=params)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 290, in get
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack headers=headers, params=params)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 267, in retry_request
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack headers=headers, params=params)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 208, in do_request
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack self._handle_fault_response(status_code, replybody)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 182, in _handle_fault_response
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack exception_handler_v20(status_code, des_error_body)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 67, in exception_handler_v20
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack status_code=status_code)
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack PortNotFoundClient: Port 01d6c6d6-4f29-4414-ab63-9e8610c963e7 could not be found
2016-02-23 10:45:21.821 19882 TRACE nova.api.openstack
2016-02-23 10:45:21.964 19882 INFO nova.api.openstack [req-836c1e85-a4ff-4358-a3d0-d75fc46c7168 ba0a7bd4bede42eaaf3c4f4f47936da3 6412d041dd49437488453a7df596dff3 - - -] http://10.204.216.184:8774/v1.1/6412d041dd49437488453a7df596dff3/servers returned with HTTP 500

Tags: config
Revision history for this message
Vedamurthy Joshi (vedujoshi) wrote :
Download full text (3.5 KiB)

Corresponding nova-api log for delete :

2016-02-23 10:50:14.473 2792 ERROR nova.network.neutronv2.api [req-8441dee4-9f0c-40a1-84fa-8ade648cafd8 ba0a7bd4bede42eaaf3c4f4f47936da3 6412d041dd49437488453a7df596dff3 - - -] Unable to clear device ID for port 'None'
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api Traceback (most recent call last):
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 366, in _unbind_ports
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api port_client.update_port(port_id, port_req_body)
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 99, in with_params
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api ret = self.function(instance, *args, **kwargs)
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 520, in update_port
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api return self.put(self.port_path % (port), body=body)
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 299, in put
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api headers=headers, params=params)
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 267, in retry_request
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api headers=headers, params=params)
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 208, in do_request
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api self._handle_fault_response(status_code, replybody)
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 182, in _handle_fault_response
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api exception_handler_v20(status_code, des_error_body)
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 80, in exception_handler_v20
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api message=message)
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api NeutronClientException: 404 Not Found
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api The resource could not be found.
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api
2016-02-23 10:50:14.473 2792 TRACE nova.network.neutronv2.api
2016-02-23 10:50:14.806 2792 INFO nova.compute.manager [req-b4f3e9a6-479a-42b7-b5e8-242fabc959d3 ba0a7bd4bede42eaaf3c4f4f47936da3 6412d041dd49437488453a7df596dff3 - - -] [instance: 3966ce8e-b294-42ba-8b49-e2b0928b9f73] Terminating instance
2016-02-23 10...

Read more...

Revision history for this message
Vedamurthy Joshi (vedujoshi) wrote :
Revision history for this message
Vedamurthy Joshi (vedujoshi) wrote :
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/17710
Submitter: Hampapur Ajay (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/17710
Committed: http://github.org/Juniper/contrail-controller/commit/fbecaeec0b98640030193da9b7fdea35ebbf6553
Submitter: Zuul
Branch: master

commit fbecaeec0b98640030193da9b7fdea35ebbf6553
Author: Hampapur Ajay <email address hidden>
Date: Tue Feb 23 18:04:39 2016 -0800

Don't call port_read() api during port_list. Instead use vmi_read
and ignore NoIdError.

Change-Id: I6ea3595a94cf289539bfa31631c5c9a444d7be24
Partial-Bug: 1548626

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Bug update]

bug update...

Ashish Ranjan (aranjan-n)
no longer affects: juniperopenstack/r3.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.