kolla-toolbox container failed and stick on create stage

[Jeffrey Zhang]

[root@localhost kolla]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
eebe54757c12 hub.xcodest.me:5000/lokolla/centos-source-kolla-toolbox:2.0.0 "/bin/sleep infinity" 38 seconds ago Created kolla_toolbox
21eb292ed9e7 hub.xcodest.me:5000/lokolla/centos-source-rsyslog:2.0.0 "kolla_start" 38 seconds ago Up 37 seconds rsyslog

[root@localhost kolla]# docker logs rsyslog
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Copying service configuration files
INFO:__main__:Removing existing destination: /etc/rsyslog.conf
INFO:__main__:Copying /var/lib/kolla/config_files/rsyslog.conf to /etc/rsyslog.conf
INFO:__main__:Setting permissions for /etc/rsyslog.conf
INFO:__main__:Writing out command to execute
Running command: '/usr/sbin/rsyslogd -n'
rsyslogd: cannot create '/run/kolla/log': Address already in use
[root@localhost kolla]# docker logs kolla_toolbox
Timestamp: 2016-02-11 07:31:16.11548407 -0500 EST
Code: System error

Message: not a directory

Frames:
---
0: setupRootfs
Package: github.com/opencontainers/runc/libcontainer
File: rootfs_linux.go@40
---
1: Init
Package: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
File: standard_init_linux.go@57
---
2: StartInitialization
Package: github.com/opencontainers/runc/libcontainer.(*LinuxFactory)
File: factory_linux.go@240
---
3: initializer
Package: github.com/docker/docker/daemon/execdriver/native
File: init.go@35
---
4: Init
Package: github.com/docker/docker/pkg/reexec
File: reexec.go@26
---
5: main
Package: main
File: docker.go@18
---
6: main
Package: runtime
File: proc.go@111
---
7: goexit
Package: runtime
File: asm_amd64.s@1721

I am using SSD + brfs filesystem. So I suppose the root cause is that after the rsyslog container start but before the rsyslog service finished initialization( i.e. /run/kolla/log is not created ), the kolla_toolbox started and create the /run/kolla/log as folder.

The solution is: wait for a moment and ensure the /run/kolla/log is created before starting the kolla_toolbox container.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/279064

[Éric Lemoine (elemoine)]

Have you seen https://bugs.launchpad.net/kolla/+bug/1544156 ? I've seen this problem after a reboot or Docker Daemon restart, as described in my bug report. Do you have other ways to reproduce the bug? If so please indicate how.

[Jeffrey Zhang (jeffrey4l)]

@elemoine, yes, they are almost the same thing, I saw that issue , too. But the solution is not the same. When restarting, nothing can be done to ensure the /run/kolla/log to be created before the kolla_toolbox starting. IIRC, there should be a docker issue for this kind of situation.

[Éric Lemoine (elemoine)]

@jeffrey4l, I'd like to know how I can reproduce the issue you are observing (without reboot or Docker Engine restart).

[Jeffrey Zhang (jeffrey4l)]

@elemonie

rm -rf /run/kolla/log
docker run -it --rm -v /run/kolla/log:/dev/log centos ls -alh /dev/log
OK, but /dev/log will be a folder

rm -rf /run/kolla/log
docker run -it --rm -v /dev:/dev centos ls -alh /dev/log
OK, the /dev/log will be the same with the host /dev/log ( just mount bind)

rm -rf /run/kolla/log
docker run -it --rm -v /dev:/dev -v /run/kolla/log:/dev/log centos ls -alh /dev/log
Error, this is the issue case.

rm -rf /run/kolla/log
touch /run/kolla/log
docker run -it --rm -v /dev:/dev -v /run/kolla/log:/dev/log centos ls -alh /dev/log
OK. the /dev/log is the same with the host /run/kolla/log( a normal file, just mount bind)

conclusion: ( Mostly are guess, I am not familiar with the docker source code)
when mounting, if the inner mount point is a file, and the outer mount point is a a folder(or not exist), the mount will be failed.

It should be like this

mkdir a
touch b
touch c
mount --bind b c # OK
mount --bind a c # Failed

[Jeffrey Zhang (jeffrey4l)]

So the core solution is: make sure the outer mount point is a plain file and created already.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/280918

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/280429
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=690e6853de1215775c672c9cb6be5190887b37da
Submitter: Jenkins
Branch: master

commit 690e6853de1215775c672c9cb6be5190887b37da
Author: SamYaple <email address hidden>
Date: Mon Feb 15 20:49:04 2016 +0000

    Move socket binding to named_volume

    The extend_start.sh script for rsyslog is removed as it is no longer
    needed. Docker no longer binds to /dev/log or /run/kolla/log

    Closes-Bug: #1544545
    Change-Id: Ic0a323a26ee4e9e15baf4598285844a8a4955f23

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/280918
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=7ddad3f3d572182f1f4d42925e17217e7752e182
Submitter: Jenkins
Branch: master

commit 7ddad3f3d572182f1f4d42925e17217e7752e182
Author: SamYaple <email address hidden>
Date: Tue Feb 16 20:28:51 2016 +0000

    Move haproxy socket to volume

    Partial-Bug: #1544545
    Change-Id: I292bcaeacb080ff4c5ab6b42b7d899039d6b19c4

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jeffrey Zhang ([e-mail]) on branch: master
Review: https://review.openstack.org/279064
Reason: Sam's PS fix this issue. https://review.openstack.org/#/c/280429