varlog full on fuel / docker

[planetrobbie]

Environment details:

- MOS 8.0 update 1
- KVM compute nodes
- StackLight deployed

It seems logrotate on Fuel Master Node isn't doing a proper job. Some process seems to keep locks on file which prevent the cleaning of old logs. After a while Fuel master Node can fail due to a completely full /var/log directory.

[root@fuel ~]# lsof +L1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
ruby 12941 root 10u REG 253,11 0 0 302063673
/tmp/0.355920445712242 (deleted)
ruby 12941 root 11u REG 253,11 0 0 302063677
/tmp/0.4962506066101938 (deleted)
ruby 12941 root 12u REG 253,11 0 0 302063678
/tmp/0.9557256875125087 (deleted)
ruby 12941 root 13u REG 253,11 0 0 302063679
/tmp/0.38445258991291265 (deleted)
ruby 12941 root 14u REG 253,11 0 0 302063680
/tmp/0.6314123791710764 (deleted)
ruby 12941 root 15u REG 253,11 0 0 302063681
/tmp/0.8132751736802393 (deleted)
ruby 12941 root 16u REG 253,11 0 0 302063682
/tmp/0.1780995451762869 (deleted)
ruby 12947 root 10u REG 253,11 0 0 302063673
/tmp/0.355920445712242 (deleted)
ruby 12950 root 11u REG 253,11 0 0 302063677
/tmp/0.4962506066101938 (deleted)
ruby 12953 root 12u REG 253,11 0 0 302063678
/tmp/0.9557256875125087 (deleted)
ruby 12959 root 13u REG 253,11 0 0 302063679
/tmp/0.38445258991291265 (deleted)
ruby 12970 root 14u REG 253,11 0 0 302063680
/tmp/0.6314123791710764 (deleted)
ruby 12973 root 15u REG 253,11 0 0 302063681
/tmp/0.8132751736802393 (deleted)
ruby 12977 root 16u REG 253,11 0 0 302063682
/tmp/0.1780995451762869 (deleted)
nginx 18837 root 6w REG 253,2 8667136 0 1441979
/var/log/nginx/access_nailgun.log-20160624 (deleted)
nginx 18837 root 7w REG 253,2 1271296000 0 1442083
/var/log/nginx/error_nailgun.log-20160624 (deleted)
nginx 18837 root 8w REG 253,2 61440 0 1441985
/var/log/nginx/access_repo.log-20160624 (deleted)
nginx 18837 root 9w REG 253,2 5623808 0 1442084
/var/log/nginx/error_repo.log-20160624 (deleted)
nginx 18838 dockerroot 6w REG 253,2 8667136 0 1441979
/var/log/nginx/access_nailgun.log-20160624 (deleted)
nginx 18838 dockerroot 7w REG 253,2 1271296000 0 1442083
/var/log/nginx/error_nailgun.log-20160624 (deleted)
nginx 18838 dockerroot 8w REG 253,2 61440 0 1441985
/var/log/nginx/access_repo.log-20160624 (deleted)
nginx 18838 dockerroot 9w REG 253,2 5623808 0 1442084
/var/log/nginx/error_repo.log-20160624 (deleted)
nginx 18839 dockerroot 6w REG 253,2 8667136 0 1441979
/var/log/nginx/access_nailgun.log-20160624 (deleted)
nginx 18839 dockerroot 7w REG 253,2 1271296000 0 1442083
/var/log/nginx/error_nailgun.log-20160624 (deleted)
nginx 18839 dockerroot 8w REG 253,2 61440 0 1441985
/var/log/nginx/access_repo.log-20160624 (deleted)
nginx 18839 dockerroot 9w REG 253,2 5623808 0 1442084
/var/log/nginx/error_repo.log-20160624 (deleted)
[root@fuel ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9.5G 1.4G 7.7G 16% /
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 0 7.8G 0% /dev/shm
tmpfs 7.8G 42M 7.8G 1% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/mapper/os-var 76G 36G 37G 50% /var
/dev/mapper/os-varlog 54G 54G 0 100% /var/log
/dev/vda3 197M 108M 89M 55% /boot
/dev/vda2 200M 0 200M 0% /boot/efi
[root@fuel ~]# dockerctl restart nginx
[root@fuel ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9.5G 1.4G 7.7G 16% /
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 0 7.8G 0% /dev/shm
tmpfs 7.8G 42M 7.8G 1% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/mapper/os-var 76G 36G 37G 50% /var
/dev/mapper/os-varlog 54G 53G 0 100% /var/log
/dev/vda3 197M 108M 89M 55% /boot
/dev/vda2 200M 0 200M 0% /boot/efi

[planetrobbie (sebbraun)]

Note that a dockerctl restart have resolved the problem with the nginx logs deleted but still open. It's not shown in the message above but it demonstrate there is really an issue with the way the logs are rotated. It shouldn't require a dockerctl restart.

[Max Lvov (usrleon)]

MOS8.0 MU1

nginx has HUGE logs deleted but still opened
http://paste.openstack.org/show/545703/