Supervisord unable to restart containers after docker daemon restart

[Kamil Szczygiel]

After restarting docker service, supervisord is unable to restart containers due to entries in /proc/mounts containing container mappings.

Relevant Docker issue: https://github.com/docker/docker/issues/5684

Workaround: Remove container entries from /proc/mounts. After a while, supervisord is able to start all containers successfully.

[Matthew Mosesohn (raytrac3r)]

Kamil, restarting docker daemon should restart all previously running containers. Can you share steps you used to reproduce and a copy of /var/log/docker?

[Kamil Szczygiel (kamil-szczygiel)]

Hi Matthew,

You can reproduce that just by restarting docker service. I've attached docker log.

[Dmitry Klenov (dklenov)]

Matthew, do you have enough data for issue reproduction?

[Matthew Mosesohn (raytrac3r)]

Kamil, we haven't seen this issue in Docker 1.4.1 so much, but it was quite prevalent in Docker 0.10, which we used previously. This logic would try to clear any pending mounts for the container:

https://github.com/openstack/fuel-library/blob/master/files/fuel-docker-utils/functions.sh#L253-L255

I'll try to reproduce your issue locally, but there are two short term workarounds you can try:
Option 1: Reboot. It's not so graceful, but it works
Option 2 : service docker stop; grep docker /proc/mounts/ | awk '{print $2}' | sort -r | xargs --no-run-if-empty -n1 umount -l;service docker start; dockerctl start all

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/240826

[Matthew Mosesohn (raytrac3r)]

I actually found the cause. We should use dockerctl instead of plain docker commands because of upstream bug https://github.com/docker/docker/issues/4036 . It is fixed in Docker 1.7, but we will not be updating docker in Fuel versions below 8.0. My patch https://review.openstack.org/#/c/240826/ contains a workaround that should solve this.

[Kamil Szczygiel (kamil-szczygiel)]

Matthew, thank you for workarounds.

[Matthew Mosesohn (raytrac3r)]

Kamil, the workaround is actually much shorter. Just after running "service docker restart", run "dockerctl start all". The umount contained in the start_container function is operating exactly as expected.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/240826
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=a1fbd3723c808d005f8b95d05103490df4ff2d17
Submitter: Jenkins
Branch: master

commit a1fbd3723c808d005f8b95d05103490df4ff2d17
Author: Matthew Mosesohn <email address hidden>
Date: Mon Nov 2 14:20:10 2015 +0300

    Use dockerctl start for supervisorctl container launch

    If docker daemon is restarted, the mount points are not safely
    torn down. dockerctl cleans these up before trying to start
    the container and therefore should be used.

    There was an original change to use direct docker commands
    because you could not start a container from a different
    Fuel release, but now this is possible and works. Therefore,
    this patch depends on the patch that fixes it.

    Depends-On: Ia864b0a9843f927aa85945e5ab0b2da4bec3a440
    Change-Id: Ie3e471d1c394fac9228b7e2bf4b6877775680b3b
    Closes-Bug: #1511294

[Denis Meltsaykin (dmeltsaykin)]

Matt, could you please clarify how this affects already deployed fuel or new deployments from ISO?

[Matthew Mosesohn (raytrac3r)]

Denis, quite often when a container is stopped, docker fails to correctly unmount the container's disk. We thought this was fixed when upgrading to Docker 1.4.2, but we were mistaken. The issue persists. If the containers are already mounted when the container is trying to start, it will fail to start. It's a workaround to solve a docker bug.

[Aleksei Stepanov (penguinolog)]

Verified on fuel-8.0-525-2016-02-05_01-55-59.iso

[Rodion Tikunov (rtikunov)]

Closing this bug as Won't Fix for MOS 6.1 as we have finished active support for those releases and don't merge non-critical and non-security fixes there anymore.

[Alexey Stupnikov (astupnikov)]

We no longer support MOS5.1, MOS6.0, MOS6.1
We deliver only Critical/Security fixes to MOS7.0, MOS8.0.
We deliver only High/Critical/Security fixes to MOS9.2.