Juniper Openstack

Bulk delete of docker VM gets stuck

Bug #1454976 reported by Vinod Nair
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
Fix Committed
High
Ignatious Johnson Christopher
Juniper Openstack r2.20-fcs
Trunk
Fix Committed
High
Ignatious Johnson Christopher
Juniper Openstack r3.0-fcs

Bug Description

Ona cluster with Docker as the hyperviosor, on trying to delete a bunch of Instances which are in Error state, the operation doesnot complete , see the below trace

root@cmbu-vse2100-2:/opt/contrail/utils# nova delete 4e930ba8-2ed0-46a3-9261-8da530389374
Request to delete server 4e930ba8-2ed0-46a3-9261-8da530389374 has been accepted.

2015-05-14 00:07:29.620 2645 AUDIT nova.compute.manager [req-a98bc38e-b0cf-4c4f-819b-cb2253517caa None] [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] Terminating instance
2015-05-14 00:07:34.811 2645 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2015-05-14 00:07:35.468 2645 AUDIT nova.compute.resource_tracker [-] Total physical ram (MB): 72230, total allocated virtual ram (MB): 52224
2015-05-14 00:07:35.468 2645 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 653
2015-05-14 00:07:35.469 2645 AUDIT nova.compute.resource_tracker [-] Total usable vcpus: 1, total allocated vcpus: 0
2015-05-14 00:07:35.469 2645 AUDIT nova.compute.resource_tracker [-] PCI stats: []
2015-05-14 00:07:35.495 2645 INFO nova.scheduler.client.report [-] Compute_service record updated for ('cmbu-vxa2010-22', 'cmbu-vxa2010-22')
2015-05-14 00:07:35.496 2645 INFO nova.compute.resource_tracker [-] Compute_service record updated for cmbu-vxa2010-22:cmbu-vxa2010-22
2015-05-14 00:07:39.711 2645 ERROR nova.compute.manager [req-a98bc38e-b0cf-4c4f-819b-cb2253517caa None] [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] Setting instance vm_state to ERROR
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] Traceback (most recent call last):
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2516, in do_terminate_instance
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] self._delete_instance(context, instance, bdms, quotas)
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 131, in inner
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] rv = f(*args, **kwargs)
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2485, in _delete_instance
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] quotas.rollback()
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] six.reraise(self.type_, self.value, self.tb)
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2462, in _delete_instance
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] self._shutdown_instance(context, instance, bdms)
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2392, in _shutdown_instance
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] requested_networks)
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] six.reraise(self.type_, self.value, self.tb)
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2381, in _shutdown_instance
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] block_device_info)
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/driver.py", line 429, in destroy
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] self.soft_delete(instance)
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/driver.py", line 414, in soft_delete
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] container_id = self._get_container_id(instance)
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/driver.py", line 234, in _get_container_id
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] return self._find_container_by_name(instance['name']).get('id')
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/driver.py", line 225, in _find_container_by_name
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] for info in self.list_instances(inspect=True):
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/driver.py", line 175, in list_instances
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] info = self.docker.inspect_container(container['id'])
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/client.py", line 36, in wrapper
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] out = f(*args, **kwds)
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 695, in inspect_container
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] self._get(self._url("/containers/{0}/json".format(container))),
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 82, in _get
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] return self.get(url, **self._set_request_timeout(kwargs))
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/client.py", line 36, in wrapper
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] out = f(*args, **kwds)
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 395, in get
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] return self.request('GET', url, **kwargs)
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/client.py", line 36, in wrapper
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] out = f(*args, **kwds)
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] resp = self.send(prep, **send_kwargs)
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/client.py", line 36, in wrapper
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] out = f(*args, **kwds)
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 486, in send
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] r = adapter.send(request, **kwargs)
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 387, in send
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] raise Timeout(e)
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374] Timeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=10)
2015-05-14 00:07:39.711 2645 TRACE nova.compute.manager [instance: 4e930ba8-2ed0-46a3-9261-8da530389374]
2015-05-14 00:07:39.810 2645 INFO nova.scheduler.client.report [req-a98bc38e-b0cf-4c4f-819b-cb2253517caa None] Compute_service record updated for ('cmbu-vxa2010-22', 'cmbu-vxa2010-22')
2015-05-14 00:07:39.942 2645 INFO nova.scheduler.client.report [req-a98bc38e-b0cf-4c4f-819b-cb2253517caa None] Compute_service record updated for ('cmbu-vxa2010-22', 'cmbu-vxa2010-22')
2015-05-14 00:07:39.944 2645 ERROR oslo.messaging.rpc.dispatcher [req-a98bc38e-b0cf-4c4f-819b-cb2253517caa ] Exception during message handling: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=10)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher payload)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 301, in decorated_function
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher pass
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 287, in decorated_function
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 351, in decorated_function
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 329, in decorated_function
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 317, in decorated_function
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2528, in terminate_instance
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher do_terminate_instance(instance, bdms)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 272, in inner
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2526, in do_terminate_instance
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher self._set_instance_error_state(context, instance)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2516, in do_terminate_instance
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher self._delete_instance(context, instance, bdms, quotas)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 131, in inner
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher rv = f(*args, **kwargs)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2485, in _delete_instance
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher quotas.rollback()
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2462, in _delete_instance
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher self._shutdown_instance(context, instance, bdms)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2392, in _shutdown_instance
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher requested_networks)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2381, in _shutdown_instance
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher block_device_info)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/driver.py", line 429, in destroy
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher self.soft_delete(instance)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/driver.py", line 414, in soft_delete
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher container_id = self._get_container_id(instance)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/driver.py", line 234, in _get_container_id
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher return self._find_container_by_name(instance['name']).get('id')
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/driver.py", line 225, in _find_container_by_name
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher for info in self.list_instances(inspect=True):
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/driver.py", line 175, in list_instances
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher info = self.docker.inspect_container(container['id'])
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/client.py", line 36, in wrapper
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher out = f(*args, **kwds)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 695, in inspect_container
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher self._get(self._url("/containers/{0}/json".format(container))),
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 82, in _get
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher return self.get(url, **self._set_request_timeout(kwargs))
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/client.py", line 36, in wrapper
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher out = f(*args, **kwds)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 395, in get
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher return self.request('GET', url, **kwargs)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/client.py", line 36, in wrapper
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher out = f(*args, **kwds)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher resp = self.send(prep, **send_kwargs)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/client.py", line 36, in wrapper
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher out = f(*args, **kwds)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 486, in send
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher r = adapter.send(request, **kwargs)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 387, in send
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher raise Timeout(e)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher Timeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=10)
2015-05-14 00:07:39.944 2645 TRACE oslo.messaging.rpc.dispatcher

nova show 4e930ba8-2ed0-46a3-9261-8da530389374
+--------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | cmbu-vxa2010-22 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | cmbu-vxa2010-22 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000069 |
| OS-EXT-STS:power_state | 4 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-05-14T03:18:54Z |
| fault | {"message": "UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=10)", "code": 500, "details": " File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 317, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2528, in terminate_instance |
| | do_terminate_instance(instance, bdms) |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py\", line 272, in inner |
| | return f(*args, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2526, in do_terminate_instance |
| | self._set_instance_error_state(context, instance) |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 82, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2516, in do_terminate_instance |
| | self._delete_instance(context, instance, bdms, quotas) |
| | File \"/usr/lib/python2.7/dist-packages/nova/hooks.py\", line 131, in inner |
| | rv = f(*args, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2485, in _delete_instance |
| | quotas.rollback() |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 82, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2462, in _delete_instance |
| | self._shutdown_instance(context, instance, bdms) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2392, in _shutdown_instance |
| | requested_networks) |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 82, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2381, in _shutdown_instance |
| | block_device_info) |
| | File \"/usr/lib/python2.7/dist-packages/novadocker/virt/docker/driver.py\", line 429, in destroy |
| | self.soft_delete(instance) |
| | File \"/usr/lib/python2.7/dist-packages/novadocker/virt/docker/driver.py\", line 414, in soft_delete |
| | container_id = self._get_container_id(instance) |
| | File \"/usr/lib/python2.7/dist-packages/novadocker/virt/docker/driver.py\", line 234, in _get_container_id |
| | return self._find_container_by_name(instance['name']).get('id') |
| | File \"/usr/lib/python2.7/dist-packages/novadocker/virt/docker/driver.py\", line 225, in _find_container_by_name |
| | for info in self.list_instances(inspect=True): |
| | File \"/usr/lib/python2.7/dist-packages/novadocker/virt/docker/driver.py\", line 175, in list_instances |
| | info = self.docker.inspect_container(container['id']) |
| | File \"/usr/lib/python2.7/dist-packages/novadocker/virt/docker/client.py\", line 36, in wrapper |
| | out = f(*args, **kwds) |
| | File \"/usr/local/lib/python2.7/dist-packages/docker/client.py\", line 695, in inspect_container |
| | self._get(self._url(\"/containers/{0}/json\".format(container))), |
| | File \"/usr/local/lib/python2.7/dist-packages/docker/client.py\", line 82, in _get |
| | return self.get(url, **self._set_request_timeout(kwargs)) |
| | File \"/usr/lib/python2.7/dist-packages/novadocker/virt/docker/client.py\", line 36, in wrapper |
| | out = f(*args, **kwds) |
| | File \"/usr/lib/python2.7/dist-packages/requests/sessions.py\", line 395, in get |
| | return self.request('GET', url, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/novadocker/virt/docker/client.py\", line 36, in wrapper |
| | out = f(*args, **kwds) |
| | File \"/usr/lib/python2.7/dist-packages/requests/sessions.py\", line 383, in request |
| | resp = self.send(prep, **send_kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/novadocker/virt/docker/client.py\", line 36, in wrapper |
| | out = f(*args, **kwds) |
| | File \"/usr/lib/python2.7/dist-packages/requests/sessions.py\", line 486, in send |
| | r = adapter.send(request, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/requests/adapters.py\", line 387, in send |
| | raise Timeout(e) |
| | ", "created": "2015-05-14T07:07:39Z"} |
| flavor | m1.tiny (1) |
| hostId | 56fe2e4f0df48955e156ca774d2ef8068ad02971b8af9071211b0913 |
| id | 4e930ba8-2ed0-46a3-9261-8da530389374 |
| image | phusion-baseimage-enablesshd (86526b4d-1c0a-419c-950a-25662416abf5) |
| key_name | - |
| metadata | {} |
| name | VM6 |
| os-extended-volumes:volumes_attached | [] |
| status | ERROR |
| tenant_id | 9a72e9d550f34814af071ea507175746 |
| updated | 2015-05-14T07:07:39Z |
| user_id | fc7848f3307144e3b4a4137e84f0a1d1 |
+--------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Tags: docker
OpenContrail Admin (ci-admin-f)
information type: Proprietary → Public
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/10731
Committed: http://github.org/Juniper/contrail-packaging/commit/22ac9793701d0844dea10bbcffdd3eabe4a2f821
Submitter: Zuul
Branch: R2.20

commit 22ac9793701d0844dea10bbcffdd3eabe4a2f821
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri May 22 16:33:26 2015 -0700

Removing veth interfaces if the contrainer launch failed
Closes-Bug: 1452151
Closes-Bug: 1454976

Using inet6 argument with ip command to configure ipv6 address to container
Closes-Bug: 1455616

Configure network using statically instead of using dhclient to avoid
it writting to the compute hosts resolv.conf
Closes-Bug: 1455168

Periodically check the vrouter agent conncetion and publish the port informaton to
the vrouter agent.
Closes-Bug: 1454655

Change-Id: I3f59055da0748df2a8dddfd82cf2eb062377dfab

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/10730
Committed: http://github.org/Juniper/contrail-packaging/commit/70bbb08f57773280a40c6f1565f853918c0bb120
Submitter: Zuul
Branch: master

commit 70bbb08f57773280a40c6f1565f853918c0bb120
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri May 22 16:09:24 2015 -0700

Removing veth interfaces if the contrainer launch failed
Closes-Bug: 1452151
Closes-Bug: 1454976

Using inet6 argument with ip command to configure ipv6 address to container
Closes-Bug: 1455616

Configure network using statically instead of using dhclient to avoid
it writting to the compute hosts resolv.conf
Closes-Bug: 1455168

Periodically check the vrouter agent conncetion and publish the port informaton to
the vrouter agent.
Closes-Bug: 1454655

Change-Id: I79360edd27ca59743f5ee6afee928f77c6a31435

Revision history for this message
Ignatious Johnson Christopher (ijohnson-x) wrote :

Reviewed: https://review.opencontrail.org/10730
Committed: http://github.org/Juniper/contrail-packaging/commit/70bbb08f57773280a40c6f1565f853918c0bb120
Submitter: Zuul
Branch: master

commit 70bbb08f57773280a40c6f1565f853918c0bb120
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri May 22 16:09:24 2015 -0700

Removing veth interfaces if the contrainer launch failed
Closes-Bug: 1452151
Closes-Bug: 1454976

Using inet6 argument with ip command to configure ipv6 address to container
Closes-Bug: 1455616

Configure network using statically instead of using dhclient to avoid
it writting to the compute hosts resolv.conf
Closes-Bug: 1455168

Periodically check the vrouter agent conncetion and publish the port informaton to
the vrouter agent.
Closes-Bug: 1454655

Change-Id: I79360edd27ca59743f5ee6afee928f77c6a31435

Revision history for this message
Ignatious Johnson Christopher (ijohnson-x) wrote :

Reviewed: https://review.opencontrail.org/10731
Committed: http://github.org/Juniper/contrail-packaging/commit/22ac9793701d0844dea10bbcffdd3eabe4a2f821
Submitter: Zuul
Branch: R2.20

commit 22ac9793701d0844dea10bbcffdd3eabe4a2f821
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri May 22 16:33:26 2015 -0700

Removing veth interfaces if the contrainer launch failed
Closes-Bug: 1452151
Closes-Bug: 1454976

Using inet6 argument with ip command to configure ipv6 address to container
Closes-Bug: 1455616

Configure network using statically instead of using dhclient to avoid
it writting to the compute hosts resolv.conf
Closes-Bug: 1455168

Periodically check the vrouter agent conncetion and publish the port informaton to
the vrouter agent.
Closes-Bug: 1454655

Change-Id: I3f59055da0748df2a8dddfd82cf2eb062377dfab

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.