neutron failed to deploy in ubuntu binary

[Lingfeng Xiong]

neutron in master failed to deploy in a multi-node deployment (ubuntu binary)

Log:
TASK: [neutron | Starting neutron-dhcp-agent container] ***********************
skipping: [compute-node]
failed: [controller] => {"changed": true, "failed": true}
msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/neutron_dhcp_agent/start',),)

in controller:
ae2838c514e2 kolla:4000/kollaglue/ubuntu-binary-neutron-dhcp-agent:2.0.0 "kolla_start" 28 seconds ago Created neutron_dhcp_agent

docker start ae2838c514e2:
Error response from daemon: Cannot start container ae2838c514e24836b45a8ff4528ddc6bfb935fca60f3a32691354b7c5a044670: Path /run/netns is mounted on /run but it is not a shared mount.
Error: failed to start containers: ae2838c514e2

[Steven Dake (sdake)]

I believe this is fixed with using the shared flag in systemd. I believe Sam has a patch to modify the docs to resolve this problem.

[Lingfeng Xiong (xionglingfeng)]

Yes, I saw that patchset today
https://review.openstack.org/#/c/281279/

Thank you for your reply.

[Liyi Meng (liyi-meng)]

Hi Steven, how this fix problem in ubuntu where there is no systemd? I cherry-pick Sam's change. it make no difference for me. :(

[Liyi Meng (liyi-meng)]

commit d077fa279c69c9a56d2bae138e5b8cfa62991d78
Merge: 8812271 94a79b1
Author: Jenkins <email address hidden>
Date: Wed Feb 17 15:06:25 2016 +0000

    Merge "Trove ubuntu binary container"

Deploy with ubuntu binary

Here is my error:

<pre>
TASK: [neutron | Starting neutron-dhcp-agent container] ***********************
skipping: [k-blade-4]
skipping: [k-blade-5]
failed: [k-blade-1] => {"changed": true, "failed": true}
msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/neutron_dhcp_agent/start',),)
failed: [k-blade-3] => {"changed": true, "failed": true}
msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/neutron_dhcp_agent/start',),)
failed: [k-blade-2] => {"changed": true, "failed": true}
msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/neutron_dhcp_agent/start',),)
</pre>

On the controller node
<pre>
time="2016-02-18T14:45:46.198232044Z" level=warning msg="Auto-creating non-existent volume host path /run/netns, this is deprecated and will be removed soon"
time="2016-02-18T14:45:46.264529981Z" level=error msg="error locating sandbox id 712989fd759ec8902903cad4b7c562c12805c363c2b4c8d7c284b6cb5bd87ed9: sandbox 712989fd759ec8902903cad4b7c562c12805c363c2b4c8d7c284b6cb5bd87ed9 not found"
time="2016-02-18T14:45:46.265495865Z" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/lib/docker/containers/73f4ec8885377419c1694ab77dea494c1f23a88cf4bc300a87aa0304ff94b135/shm: invalid argument"
time="2016-02-18T14:45:46.265640402Z" level=error msg="Error unmounting container 73f4ec8885377419c1694ab77dea494c1f23a88cf4bc300a87aa0304ff94b135: not mounted"
time="2016-02-18T14:45:46.266043143Z" level=error msg="Handler for POST /v1.22/containers/neutron_dhcp_agent/start returned error: Cannot start container 73f4ec8885377419c1694ab77dea494c1f23a88cf4bc300a87aa0304ff94b135: Path /run/netns is mounted on /run but it is not a shared mount."
</pre>

Try to run the container manually
<pre>
docker start neutron_dhcp_agent
Error response from daemon: Cannot start container 73f4ec8885377419c1694ab77dea494c1f23a88cf4bc300a87aa0304ff94b135: Path /run/netns is mounted on /run but it is not a shared mount.
Error: failed to start containers: neutron_dhcp_agent
</pre>

sudo grep shared /proc/self/mountinfo
342 335 0:17 / /run rw,relatime shared:1 - tmpfs none rw
163 342 0:4 / /run/docker/netns/default rw shared:2 - nsfs nsfs rw

[Liyi Meng (liyi-meng)]

OK, I found a workaround for my problem. Before running deployment. I should run this command first.

 sudo mkdir -p /run/netns && sudo mount -o rw,relatime -t tmpfs tmpfs /run/netns && sudo mount --make-rshared /run

@Sam Yaple, this seems more reliable than simply run "mount --make-shared /run". do you think it is a good idea to incorporate this into the ubuntu playbook? Do you want me to upate the document?

[Liyi Meng (liyi-meng)]

OK, my apology. My way of doing thing is not stable either. Sometimes work, sometimes not.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

One small comment: I'm using
- Ubuntu source
- All-in-one
and I also have this problem.

DOCKER LOGS:
DEBU[0156] copying image data from ba7b8918b37a0274ef74d6e2550bfc76312a8d665844c44aeafe4a25726de348:/var/lib/neutron/kolla, to neutron_metadata_socket
DEBU[0156] Calling GET /v1.22/containers/json
DEBU[0156] GET /v1.22/containers/json?all=1&limit=-1&trunc_cmd=0&size=0
DEBU[0156] Calling POST /v1.22/containers/neutron_dhcp_agent/start
DEBU[0156] POST /v1.22/containers/neutron_dhcp_agent/start
DEBU[0156] form data: {}
DEBU[0156] container mounted via layerStore: /var/lib/docker/aufs/mnt/8ec90df91cdb2a4590f6bba4d31aa4c4158010650c5fe93224b028df0000880a
WARN[0156] Auto-creating non-existent volume host path /run/netns, this is deprecated and will be removed soon
ERRO[0157] error locating sandbox id 7fbffe1a9572be17d6202e6af0dfc6c93cf8c4cfc1cbef3498a71b6595aeccc3: sandbox 7fbffe1a9572be17d6202e6af0dfc6c93cf8c4cfc1cbef3498a71b6595aeccc3 not found
WARN[0157] failed to cleanup ipc mounts:
failed to umount /var/lib/docker/containers/ba7b8918b37a0274ef74d6e2550bfc76312a8d665844c44aeafe4a25726de348/shm: invalid argument
ERRO[0157] Error unmounting container ba7b8918b37a0274ef74d6e2550bfc76312a8d665844c44aeafe4a25726de348: not mounted
ERRO[0157] Handler for POST /v1.22/containers/neutron_dhcp_agent/start returned error: Cannot start container ba7b8918b37a0274ef74d6e2550bfc76312a8d665844c44aeafe4a25726de348: Path /run/netns is mounted on /run but it is not a shared mount.

[Mauricio Lima (mliima) (mauricio-lima)]

Fix merged -> https://review.openstack.org/#/c/281031/