Saw this issue everytime if I am deploying the setup second time in both single and multi nodes.
Command used for deployment. Version which I am deploying is pike and kolla-ansible version I am using is 6.0.0
# kolla-ansible deploy -i /root/kolla/multinode
Restart of kolla-toolbox container is getting failed with the following error:
~~~
RUNNING HANDLER [common : Restart kolla-toolbox container] *********************************************************************************************************************************
fatal: [control01]: FAILED! => {"changed": true, "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_0BU43U/ansible_module_kolla_docker.py\", line 823, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/tmp/ansible_0BU43U/ansible_module_kolla_docker.py\", line 606, in recreate_or_restart_container\\n self.start_container()\\n File \"/tmp/ansible_0BU43U/ansible_module_kolla_docker.py\", line 633, in start_container\\n self.dc.start(container=self.params.get(\\'name\\'))\\n File \"/usr/lib/python2.7/site-packages/docker/utils/decorators.py\", line 19, in wrapped\\n return f(self, resource_id, *args, **kwargs)\\n File \"/usr/lib/python2.7/site-packages/docker/api/container.py\", line 1063, in start\\n self._raise_for_status(res)\\n File \"/usr/lib/python2.7/site-packages/docker/api/client.py\", line 227, in _raise_for_status\\n raise create_api_error_from_http_exception(e)\\n File \"/usr/lib/python2.7/site-packages/docker/errors.py\", line 31, in create_api_error_from_http_exception\\n raise cls(e, response=response, explanation=explanation)\\nNotFound: 404 Client Error: Not Found (\"oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:364: container init caused \\\\\"rootfs_linux.go:54: mounting \\\\\\\\\\\\\"/var/lib/docker/containers/a62ab8c26940bb77d14a1bcdcbfa9a7ab02be7edfc8e781390056eb109c79518/secrets\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\"/var/lib/docker/overlay2/7145d35d5634d7929e293a6bc1ffe1ac0e772aa22847111e4f8c477f71c5c22a/merged\\\\\\\\\\\\\" at \\\\\\\\\\\\\"/var/lib/docker/overlay2/7145d35d5634d7929e293a6bc1ffe1ac0e772aa22847111e4f8c477f71c5c22a/merged/run/secrets\\\\\\\\\\\\\" caused \\\\\\\\\\\\\"no such file or directory\\\\\\\\\\\\\"\\\\\"\"\\n\")\\n'"}
~~~
Deleted the kolla-toolbox container using `docker rm <container id>` command but still it always failed with above error.
I don't have anything in merged directory.
~~~
[root@kolla-controller ~]# cd /var/lib/docker/overlay2/7145d35d5634d7929e293a6bc1ffe1ac0e772aa22847111e4f8c477f71c5c22a/
[root@kolla-controller 7145d35d5634d7929e293a6bc1ffe1ac0e772aa22847111e4f8c477f71c5c22a]# ls
diff link lower merged work
[root@kolla-controller 7145d35d5634d7929e293a6bc1ffe1ac0e772aa22847111e4f8c477f71c5c22a]# cd merged/
[root@kolla-controller merged]# ls
[root@kolla-controller merged]#
~~~
overlay2 is used as docker storage backend.
Workaround is to delete the kolla-toolbox, destroy the deployment, restart and then start with deployment again which is not a feasible solution.
Restarting docker seemed to work for me.