upgrades fails while restarting some containers

[Eduardo Gonzalez]

While upgrading Newton to Ocata.
kolla_toolbox and nova_libvirt are not recreated.
I reproduced this behaviour in all upgrade attempts with both centos binary and source.

Instead of recreated are removed and fails.
This is a hard issue because nova_libvirt is not present until the next upgrade attempt.

Errors:

TASK [common : Starting kolla-toolbox container] *******************************
fatal: [localhost]: FAILED! => {"changed": true, "failed": true, "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_Xixy18/ansible_module_kolla_docker.py\", line 774, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/tmp/ansible_Xixy18/ansible_module_kolla_docker.py\", line 592, in start_container\\n self.remove_container()\\n File \"/tmp/ansible_Xixy18/ansible_module_kolla_docker.py\", line 480, in remove_container\\n force=True\\n File \"/usr/lib/python2.7/site-packages/docker/utils/decorators.py\", line 21, in wrapped\\n return f(self, resource_id, *args, **kwargs)\\n File \"/usr/lib/python2.7/site-packages/docker/api/container.py\", line 278, in remove_container\\n self._raise_for_status(res)\\n File \"/usr/lib/python2.7/site-packages/docker/client.py\", line 174, in _raise_for_status\\n raise errors.APIError(e, response, explanation=explanation)\\nAPIError: 500 Server Error: Internal Server Error (\"{\"message\":\"Unable to remove filesystem for 0a45bbf35052607f252fd6d5ec21017086544432d5b78351632b9a42f90d6313: remove /var/lib/docker/containers/0a45bbf35052607f252fd6d5ec21017086544432d5b78351632b9a42f90d6313/shm: device or resource busy\"}\")\\n'"}

RUNNING HANDLER [nova : Restart nova-libvirt container] ************************
fatal: [localhost]: FAILED! => {"changed": true, "failed": true, "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_M3Oedo/ansible_module_kolla_docker.py\", line 774, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/tmp/ansible_M3Oedo/ansible_module_kolla_docker.py\", line 581, in recreate_or_restart_container\\n self.remove_container()\\n File \"/tmp/ansible_M3Oedo/ansible_module_kolla_docker.py\", line 480, in remove_container\\n force=True\\n File \"/usr/lib/python2.7/site-packages/docker/utils/decorators.py\", line 21, in wrapped\\n return f(self, resource_id, *args, **kwargs)\\n File \"/usr/lib/python2.7/site-packages/docker/api/container.py\", line 278, in remove_container\\n self._raise_for_status(res)\\n File \"/usr/lib/python2.7/site-packages/docker/client.py\", line 174, in _raise_for_status\\n raise errors.APIError(e, response, explanation=explanation)\\nAPIError: 500 Server Error: Internal Server Error (\"{\"message\":\"Unable to remove filesystem for cb03dca6fd61086a321b99646f26a31b6d79cfec6a10a4155fb4ef21a35a4014: remove /var/lib/docker/containers/cb03dca6fd61086a321b99646f26a31b6d79cfec6a10a4155fb4ef21a35a4014/shm: device or resource busy\"}\")\\n'"}

[Jeffrey Zhang (jeffrey4l)]

which docker-engine are u using?
i saw this in docker 1.13. but docker 1.12.x is OK.

I guess there are some bug in docker 1.13. shm file is not release by docker or container.

[Eduardo Gonzalez (egonzalez90)]

Faced similar issue with 1.12.6 while upgrading with an instance running.
Almost all containers are restarted, but libvirt is removed and exist with similar error:
RUNNING HANDLER [nova : Restart nova-libvirt container] ************************
fatal: [localhost]: FAILED! => {"changed": true, "failed": true, "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_1FNRnQ/ansible_module_kolla_docker.py\", line 781, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/tmp/ansible_1FNRnQ/ansible_module_kolla_docker.py\", line 588, in recreate_or_restart_container\\n self.remove_container()\\n File \"/tmp/ansible_1FNRnQ/ansible_module_kolla_docker.py\", line 487, in remove_container\\n force=True\\n File \"/usr/lib/python2.7/site-packages/docker/utils/decorators.py\", line 21, in wrapped\\n return f(self, resource_id, *args, **kwargs)\\n File \"/usr/lib/python2.7/site-packages/docker/api/container.py\", line 278, in remove_container\\n self._raise_for_status(res)\\n File \"/usr/lib/python2.7/site-packages/docker/client.py\", line 174, in _raise_for_status\\n raise errors.APIError(e, response, explanation=explanation)\\nAPIError: 500 Server Error: Internal Server Error (\"{\"message\":\"Driver devicemapper failed to remove root filesystem 2e9ca10f7ad0950fe867e1a2a6a67d1874543f6678f40877f51b9b2094adb76b: Device is Busy\"}\")\\n'"}

[Jeffrey Zhang (jeffrey4l)]

i tried upgrade on docker 1.12.6, and it works.

weird docker ;(

[Jeffrey Zhang (jeffrey4l)]

docker upstream bug: https://github.com/moby/moby/issues/17902

When this error happens, the container is removed actually. even though docker doesn't remove it on filesystem. A workaround is assume the container is removed if we can not found it in docker ps -a

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/488346

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/488346
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=ea1ae405ba6255cdde2af88fb000fd3b79ea3af1
Submitter: Jenkins
Branch: master

commit ea1ae405ba6255cdde2af88fb000fd3b79ea3af1
Author: Jeffrey Zhang <email address hidden>
Date: Fri Jul 28 14:30:45 2017 +0800

    Assume the container is removed if it is not show in docker ps

    In some case, docker can not remove container and raise following error
    message:

        Unable to remove filesystem for xxx remove
        /var/lib/docker/containers/xxx/shm: device or resource busy

    But the container is removed. This patch assumes container is
    removed if only container name is not shown in docker ps.

    Closes-Bug: #1662598
    Change-Id: I079d5ec6178018403ec7a49c975f137e27eb9ad4

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/488306
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=8cf51b3fe5b7bfc1fe6776bdeb07c607cd8baf39
Submitter: Jenkins
Branch: stable/ocata

commit 8cf51b3fe5b7bfc1fe6776bdeb07c607cd8baf39
Author: Jeffrey Zhang <email address hidden>
Date: Fri Jul 28 14:30:45 2017 +0800

    Assume the container is removed if it is not show in docker ps

    In some case, docker can not remove container and raise following error
    message:

        Unable to remove filesystem for xxx remove
        /var/lib/docker/containers/xxx/shm: device or resource busy

    But the container is removed. This patch assumes container is
    removed if only container name is not shown in docker ps.

    Closes-Bug: #1662598
    Change-Id: I079d5ec6178018403ec7a49c975f137e27eb9ad4
    (cherry picked from commit ea1ae405ba6255cdde2af88fb000fd3b79ea3af1)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 5.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 4.0.3 release.