kolla-ansible -i ./all-in-one deploy error @ RUNNING HANDLER [common : Restart kolla-toolbox container]

[Jair Bolivar]

When installing openstack `stable/xena` branch, bootstrap-servers [pass], all-in-one prechecks [pass], but when running all-in-one deploy the installer stops at this stage:

RUNNING HANDLER [common : Restart kolla-toolbox container] with the error:

fatal: [localhost]: FAILED! => {"changed": true, "msg": "'Traceback (most recent call last):\\n File \"/usr/local/lib/python3.8/dist-packages/docker/api/client.py\", line 268, in _raise_for_status\\n response.raise_for_status()\\n File \"/usr/lib/python3/dist-packages/requests/models.py\", line 940, in raise_for_status\\n raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/containers/kolla_toolbox/start\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \"/tmp/ansible_kolla_docker_payload_9khtvvr_/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 1241, in main\\n File \"/tmp/ansible_kolla_docker_payload_9khtvvr_/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 892, in recreate_or_restart_container\\n File \"/tmp/ansible_kolla_docker_payload_9khtvvr_/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 924, in start_container\\n File \"/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py\", line 19, in wrapped\\n return f(self, resource_id, *args, **kwargs)\\n File \"/usr/local/lib/python3.8/dist-packages/docker/api/container.py\", line 1111, in start\\n self._raise_for_status(res)\\n File \"/usr/local/lib/python3.8/dist-packages/docker/api/client.py\", line 270, in _raise_for_status\\n raise create_api_error_from_http_exception(e)\\n File \"/usr/local/lib/python3.8/dist-packages/docker/errors.py\", line 31, in create_api_error_from_http_exception\\n raise cls(e, response=response, explanation=explanation)\\ndocker.errors.APIError: 500 Server Error for http+docker://localhost/v1.41/containers/kolla_toolbox/start: Internal Server Error (\"path /run is mounted on /run but it is not a shared mount\")\\n'"}

RUNNING HANDLER [common : Restart cron container] ****************************************************************************************************************************

NO MORE HOSTS LEFT ***********************************************************************************************************************************************************

PLAY RECAP *******************************************************************************************************************************************************************
localhost : ok=24 changed=12 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0

Command failed ansible-playbook -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e kolla_action=deploy /usr/local/share/kolla-ansible/ansible/site.yml --inventory ./all-in-one

[Radosław Piliszek (yoctozepto)]

Please provide distribution name and version (I'm guessing Ubuntu 20.04 in this reported case), the running Docker and containerd daemons versions, used Kolla Ansible version (commit id or version from PyPI - whichever you used).
Please also attach output of `mount`, `dpkg -l` and `ps aux`.

Does reboot help?

[Adam Tomas (bkslash)]

I had similar error but in different situation (re-deploy after changing two nodes to different ones but with disks from original hosts).
After a while I've noticed, that docker overlays weren't mounted

cat /proc/1/mountinfo | grep shared

So I tried to find why. After

docker info | grep Root

it turned out that docker root dir
was pointing to /var/snap/docker/common/var-lib-docker instead of /var/lib/docker, where all previous data was located. I had to modify /lib/systemd/system/docker.service to point to proper dir:

ExecStart=/usr/bin/dockerd -g /var/lib/docker -H (... the rest of original command)

Maybe the problem is similar in your situation.
You can also try:

mount --make-shared /

[Radosław Piliszek (yoctozepto)]

Hmm, then it looks like a conflict with snapd, maybe Ubuntu changed some defaults. I'm pretty sure we test Ubntu upstream without snapd installed.

[Launchpad Janitor (janitor)]

[Expired for kolla-ansible because there has been no activity for 60 days.]