Rsyslog doesn't work on ubuntu

[Vu Nguyen Duy (CBR09)]

Hi
I'm trying kolla-liberty rc1. rsyslog container doesn't work with kollaglue/ubuntu-source-rsyslog

root@cbr-HP-Compaq-Pro-6300-SFF:~/kolla# docker logs rsyslog
************************************************************
INFO:__main__:Kolla config strategy set to: COPY_ONCE
INFO:__main__:Loading config file at /opt/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Copying service configuration files
INFO:__main__:Removing existing destination: /etc/rsyslog.conf
INFO:__main__:Copying /opt/kolla/config_files/rsyslog.conf to /etc/rsyslog.conf
INFO:__main__:Setting permissions for /etc/rsyslog.conf
INFO:__main__:Writing out command to execute
rsyslogd: warning: ~ action is deprecated, consider using the 'stop' statement instead [try http://www.rsyslog.com/e/2307 ]
rsyslogd: cannot create '/dev/log': Address already in use
rsyslogd: imuxsock does not run because we could not aquire any socket

rsyslogd: activation of module imuxsock failed
***********************************************************
root@cbr-HP-Compaq-Pro-6300-SFF:~/kolla# docker exec -ti rsyslog /bin/bash
root@cbr-HP-Compaq-Pro-6300-SFF:/# cat /var/log/syslog.log
***********************************************************
Sep 30 04:29:17 cbr-HP-Compaq-Pro-6300-SFF rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="1" x-info="http://www.rsyslog.com"] start
Sep 30 04:29:17 cbr-HP-Compaq-Pro-6300-SFF rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [try http://www.rsyslog.com/e/2307 ]
Sep 30 04:29:17 cbr-HP-Compaq-Pro-6300-SFF rsyslogd: cannot create '/dev/log': Address already in use
Sep 30 04:29:17 cbr-HP-Compaq-Pro-6300-SFF rsyslogd: imuxsock does not run because we could not aquire any socket

Sep 30 04:29:17 cbr-HP-Compaq-Pro-6300-SFF rsyslogd-3000: activation of module imuxsock failed
Sep 30 04:29:19 cbr-HP-Compaq-Pro-6300-SFF kernel: [ 8967.331493] aufs au_opts_verify:1612:docker[10480]: dirperm1 breaks the protection by the permission bits on the lower branch
Sep 30 04:29:19 cbr-HP-Compaq-Pro-6300-SFF kernel: [ 8967.409264] aufs au_opts_verify:1612:docker[10480]: dirperm1 breaks the protection by the permission bits on the lower branch
Sep 30 04:29:45 cbr-HP-Compaq-Pro-6300-SFF kernel: [ 8993.094500] aufs au_opts_verify:1612:docker[10490]: dirperm1 breaks the protection by the permission bits on the lower branch
Sep 30 04:29:45 cbr-HP-Compaq-Pro-6300-SFF kernel: [ 8993.173192] aufs au_opts_verify:1612:docker[10495]: dirperm1 breaks the protection by the permission bits on the lower branch
Sep 30 04:29:54 cbr-HP-Compaq-Pro-6300-SFF kernel: [ 9001.910235] aufs au_opts_verify:1612:docker[9643]: dirperm1 breaks the protection by the permission bits on the lower branch
Sep 30 04:29:54 cbr-HP-Compaq-Pro-6300-SFF kernel: [ 9001.987372] aufs au_opts_verify:1612:docker[9643]: dirperm1 breaks the protection by the permission bits on the lower branch
Sep 30 04:29:55 cbr-HP-Compaq-Pro-6300-SFF kernel: [ 9003.221679] aufs au_opts_verify:1612:docker[11644]: dirperm1 breaks the protection by the permission bits on the lower branch
Sep 30 04:35:09 cbr-HP-Compaq-Pro-6300-SFF kernel: [ 9317.314849] aufs au_opts_verify:1612:docker[12137]: dirperm1 breaks the protection by the permission bits on the lower branch
Sep 30 04:36:07 cbr-HP-Compaq-Pro-6300-SFF kernel: [ 9375.268036] aufs au_opts_verify:1612:docker[13099]: dirperm1 breaks the protection by the permission bits on the lower branch
Sep 30 04:36:53 cbr-HP-Compaq-Pro-6300-SFF kernel: [ 9421.411652] aufs au_opts_verify:1612:docker[13087]: dirperm1 breaks the protection by the permission bits on the lower branch
Sep 30 04:36:53 cbr-HP-Compaq-Pro-6300-SFF kernel: [ 9421.487385] aufs au_opts_verify:1612:docker[13133]: dirperm1 breaks the protection by the permission bits on the lower branch
*************************************************************

[Sam Yaple (s8m)]

Please provide the output for the following commands:

# uname -a
# docker info

[Vu Nguyen Duy (CBR09) (nguyenduyvu099)]

Yes.
root@cbr-HP-Compaq-Pro-6300-SFF:~/kolla/ansible# uname -a
Linux cbr-HP-Compaq-Pro-6300-SFF 3.19.0-28-generic #30~14.04.1-Ubuntu SMP Tue Sep 1 09:32:55 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

root@cbr-HP-Compaq-Pro-6300-SFF:~/kolla/ansible# docker info
Containers: 19
Images: 85
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 125
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-28-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 4
Total Memory: 7.675 GiB
Name: cbr-HP-Compaq-Pro-6300-SFF
ID: W5QE:2EGC:UIUZ:BSU4:MM6X:XNTO:SH6I:T7FE:54MI:UUZY:FKHD:UI67
Username: cauberong099
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

root@cbr-HP-Compaq-Pro-6300-SFF:~/kolla/ansible# docker version
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

(Sorry for incorrect my version docker in IRC :D, and I cannot access IRC due to IP banned)

[Sam Yaple (s8m)]

My settings are below. Given we are running the exact same kernel, I would recommend upgrading docker.

root@sy-test01:~/kolla# uname -a
Linux sy-test01 3.19.0-28-generic #30~14.04.1-Ubuntu SMP Tue Sep 1 09:32:55 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@sy-test01:~/kolla# docker info
Containers: 30
Images: 128
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 188
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-28-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 24
Total Memory: 62.75 GiB
Name: sy-test01
ID: M3IP:YZ6T:H4HA:PBT4:HDQ5:FNSJ:33US:AH5G:Y3EF:CI6O:U6VX:DNQX
Username: samyaple
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
root@sy-test01:~/kolla# docker version
Client:
 Version: 1.8.2
 API version: 1.20
 Go version: go1.4.2
 Git commit: 0a8c2e3
 Built: Thu Sep 10 19:19:00 UTC 2015
 OS/Arch: linux/amd64

Server:
 Version: 1.8.2
 API version: 1.20
 Go version: go1.4.2
 Git commit: 0a8c2e3
 Built: Thu Sep 10 19:19:00 UTC 2015
 OS/Arch: linux/amd64

[Vu Nguyen Duy (CBR09) (nguyenduyvu099)]

Try upgrading Docker to 1.8 and disable apparmor, not solve problem. same error for rsyslog container.

[Sam Yaple (s8m)]

Hello,

We worked through this issue out-of-band from this bug. The issue was apparmor. It was not fully removed, it was sticking around in the initramfs after a reboot. A post [1] at the ubuntuforums resolved the issue of removing apparmor with the following steps:

 * apt-get purge apparmor
 * rm -rf /etc/apparmor*
 * update-initramfs -u
 * reboot

I am going to close this out now.

[1] http://ubuntuforums.org/showthread.php?t=1348443

[Vu Nguyen Duy (CBR09) (nguyenduyvu099)]

Hello,

After seeing a bug about rsyslog: https://bugs.launchpad.net/kolla/+bug/1501694. Today, I suddenly thought about my bug (this bug), and it seems it not due apparmor, privileged flag is enough to bypass apparmor. This problem was due /dev/log that was created by another containers before rsyslog container is up and running. The solution is as same as bug1501694, just remove /dev/log.