docker virt driver does not stop containers if spawn fails to setup network

Bug #1242815 reported by Boden R
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Guangya Liu (Jay Lau)

Bug Description

Using docker virt driver (i.e. compute_driver = docker.DockerDriver in /etc/nova/nova.conf) -- if nova.virt.docker.driver.spawn() fails in setup_network(), the started container is not stopped prior to the driver reporting error.

A snippet from docker virt driver's spawn() method:

        self.docker.start_container(container_id)
        try:
            self._setup_network(instance, network_info)
        except Exception as e:
            msg = _('Cannot setup network: {0}')
            raise exception.InstanceDeployFailure(msg.format(e),
                                                  instance_id=instance['name'])

As shown above, the docker container is started and then networking is attempted -- if the _setup_network() call fails, and exception is raised but the docker container is left running... I had this happen to me and after the _setup_network() error I issued a "docker ps" command -- my container was still running until I stopped it manually with "docker stop CID".

The result of the above is container sprawl.

Tags: docker
Revision history for this message
Guangya Liu (Jay Lau) (jay-lau-513) wrote :

I also encounter this, will provide a patch later.

Changed in nova:
assignee: nobody → Jay Lau (jay-lau-513)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/53045

Changed in nova:
status: New → In Progress
Revision history for this message
Sébastien Han (sebastien-han) wrote :

Btw how did you fix your network issue?

tags: added: docker
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/64987
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a32b2069aa80c365756a17e6d4e4788b5009115c
Submitter: Jenkins
Branch: master

commit a32b2069aa80c365756a17e6d4e4788b5009115c
Author: Daniel Kuffner <email address hidden>
Date: Tue Oct 22 10:33:41 2013 +0800

    Destroy docker container if spawn fails to set up network

    If docker driver failed to set up network when spawn an instance,
    then the docker driver should also destroy the container to make
    sure no garbage container left for such case.

    Change-Id: I4496182d9f6a9bfd86b0a32edb19dcd975467e91
    Closes-Bug: #1242815

Changed in nova:
status: In Progress → Fix Committed
Changed in nova:
milestone: none → icehouse-3
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: icehouse-3 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.