Zun

Delete a container that not exist in compute node will be error

Bug #1736885 reported by Kevin Zhao
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Zun
Invalid
Medium
Kevin Zhao

Bug Description

Create a zun container.
Delete the container with docker command.
Then use zun to delete the container will fail.

We should handle the error response from docker client and check the error message.
If is "No such container", we should container deletion for DB, not raise an error.
Then the delete process will stay at "Deleting".

12月 07 11:31:05 zun-aio zun-compute[655]: 2017-12-07 11:31:05.655 ERROR zun.compute.manager [req-e7de38e9-70bc-4685-8c64-183361df4ce4 admin admin] Error occurred while calling Docker delete API: Docker internal error: 404 Client Error: Not Found ("No such container: 8eac35913e9d945f10833257468ce51488e4aa07088a41406bd98f9bcf5df9b3").: DockerError: Docker internal error: 404 Client Error: Not Found ("No such container: 8eac35913e9d945f10833257468ce51488e4aa07088a41406bd98f9bcf5df9b3").
12月 07 11:31:05 zun-aio zun-compute[655]: 2017-12-07 11:31:05.672 DEBUG oslo_concurrency.lockutils [req-e7de38e9-70bc-4685-8c64-183361df4ce4 admin admin] Lock "fd12f6ec-efe0-4ff0-94b7-53fd25b8db28" released by "zun.compute.manager.do_container_delete" :: held 0.165s from (pid=655) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:285
12月 07 11:31:05 zun-aio zun-compute[655]: Traceback (most recent call last):
12月 07 11:31:05 zun-aio zun-compute[655]: File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
12月 07 11:31:05 zun-aio zun-compute[655]: timer()
12月 07 11:31:05 zun-aio zun-compute[655]: File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
12月 07 11:31:05 zun-aio zun-compute[655]: cb(*args, **kw)
12月 07 11:31:05 zun-aio zun-compute[655]: File "/opt/stack/zun/zun/common/utils.py", line 151, in context_wrapper
12月 07 11:31:05 zun-aio zun-compute[655]: func(*args, **kwargs)
12月 07 11:31:05 zun-aio zun-compute[655]: File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
12月 07 11:31:05 zun-aio zun-compute[655]: return f(*args, **kwargs)
12月 07 11:31:05 zun-aio zun-compute[655]: File "/opt/stack/zun/zun/compute/manager.py", line 363, in do_container_delete
12月 07 11:31:05 zun-aio zun-compute[655]: self._do_container_delete(context, container, force)
12月 07 11:31:05 zun-aio zun-compute[655]: File "/opt/stack/zun/zun/compute/manager.py", line 379, in _do_container_delete
12月 07 11:31:05 zun-aio zun-compute[655]: self._fail_container(context, container, six.text_type(e))
12月 07 11:31:05 zun-aio zun-compute[655]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
12月 07 11:31:05 zun-aio zun-compute[655]: self.force_reraise()
12月 07 11:31:05 zun-aio zun-compute[655]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
12月 07 11:31:05 zun-aio zun-compute[655]: six.reraise(self.type_, self.value, self.tb)
12月 07 11:31:05 zun-aio zun-compute[655]: File "/opt/stack/zun/zun/compute/manager.py", line 372, in _do_container_delete
12月 07 11:31:05 zun-aio zun-compute[655]: self.driver.delete(context, container, force)
12月 07 11:31:05 zun-aio zun-compute[655]: File "/opt/stack/zun/zun/container/docker/driver.py", line 288, in delete
12月 07 11:31:05 zun-aio zun-compute[655]: raise
12月 07 11:31:05 zun-aio zun-compute[655]: File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
12月 07 11:31:05 zun-aio zun-compute[655]: self.gen.throw(type, value, traceback)
12月 07 11:31:05 zun-aio zun-compute[655]: File "/opt/stack/zun/zun/container/docker/utils.py", line 48, in docker_client
12月 07 11:31:05 zun-aio zun-compute[655]: six.reraise(type(desired_exc), desired_exc, sys.exc_info()[2])
12月 07 11:31:05 zun-aio zun-compute[655]: File "/opt/stack/zun/zun/container/docker/utils.py", line 44, in docker_client
12月 07 11:31:05 zun-aio zun-compute[655]: **client_kwargs
12月 07 11:31:05 zun-aio zun-compute[655]: File "/opt/stack/zun/zun/container/docker/driver.py", line 279, in delete
12月 07 11:31:05 zun-aio zun-compute[655]: self._cleanup_network_for_container(container, network_api)
12月 07 11:31:05 zun-aio zun-compute[655]: File "/opt/stack/zun/zun/container/docker/driver.py", line 296, in _cleanup_network_for_container
12月 07 11:31:05 zun-aio zun-compute[655]: container, docker_net, neutron_network_id=neutron_net)
12月 07 11:31:05 zun-aio zun-compute[655]: File "/opt/stack/zun/zun/network/kuryr_network.py", line 296, in disconnect_container_from_network
12月 07 11:31:05 zun-aio zun-compute[655]: network_name)
12月 07 11:31:05 zun-aio zun-compute[655]: File "/usr/local/lib/python2.7/dist-packages/docker/utils/decorators.py", line 19, in wrapped
12月 07 11:31:05 zun-aio zun-compute[655]: return f(self, resource_id, *args, **kwargs)
12月 07 11:31:05 zun-aio zun-compute[655]: File "/usr/local/lib/python2.7/dist-packages/docker/utils/decorators.py", line 34, in wrapper
12月 07 11:31:05 zun-aio zun-compute[655]: return f(self, *args, **kwargs)
12月 07 11:31:05 zun-aio zun-compute[655]: File "/usr/local/lib/python2.7/dist-packages/docker/api/network.py", line 263, in disconnect_container_from_network
12月 07 11:31:05 zun-aio zun-compute[655]: self._raise_for_status(res)
12月 07 11:31:05 zun-aio zun-compute[655]: File "/usr/local/lib/python2.7/dist-packages/docker/api/client.py", line 222, in _raise_for_status
12月 07 11:31:05 zun-aio zun-compute[655]: raise create_api_error_from_http_exception(e)
12月 07 11:31:05 zun-aio zun-compute[655]: File "/usr/local/lib/python2.7/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
12月 07 11:31:05 zun-aio zun-compute[655]: raise cls(e, response=response, explanation=explanation)
12月 07 11:31:05 zun-aio zun-compute[655]: DockerError: Docker internal error: 404 Client Error: Not Found ("No such container: 8eac35913e9d945f10833257468ce51488e4aa07088a41406bd98f9bcf5df9b3").

Kevin Zhao (kevin-zhao)
Changed in zun:
assignee: nobody → Kevin Zhao (kevin-zhao)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to zun (master)

Fix proposed to branch: master
Review: https://review.openstack.org/526309

Changed in zun:
status: New → In Progress
hongbin (hongbin034)
Changed in zun:
importance: Undecided → Medium
Revision history for this message
Kevin Zhao (kevin-zhao) wrote :

This bug could not reproduce with master branch

Changed in zun:
status: In Progress → Invalid
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on zun (master)

Change abandoned by Kevin Zhao (<email address hidden>) on branch: master
Review: https://review.openstack.org/526309

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.