varlog full on fuel / docker
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Invalid
|
Medium
|
Fuel Sustaining | ||
8.0.x |
Confirmed
|
Medium
|
Kyrylo Galanov | ||
Mitaka |
Invalid
|
Undecided
|
Sravanthi |
Bug Description
Environment details:
- MOS 8.0 update 1
- KVM compute nodes
- StackLight deployed
It seems logrotate on Fuel Master Node isn't doing a proper job. Some process seems to keep locks on file which prevent the cleaning of old logs. After a while Fuel master Node can fail due to a completely full /var/log directory.
[root@fuel ~]# lsof +L1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
ruby 12941 root 10u REG 253,11 0 0 302063673
/tmp/0.
ruby 12941 root 11u REG 253,11 0 0 302063677
/tmp/0.
ruby 12941 root 12u REG 253,11 0 0 302063678
/tmp/0.
ruby 12941 root 13u REG 253,11 0 0 302063679
/tmp/0.
ruby 12941 root 14u REG 253,11 0 0 302063680
/tmp/0.
ruby 12941 root 15u REG 253,11 0 0 302063681
/tmp/0.
ruby 12941 root 16u REG 253,11 0 0 302063682
/tmp/0.
ruby 12947 root 10u REG 253,11 0 0 302063673
/tmp/0.
ruby 12950 root 11u REG 253,11 0 0 302063677
/tmp/0.
ruby 12953 root 12u REG 253,11 0 0 302063678
/tmp/0.
ruby 12959 root 13u REG 253,11 0 0 302063679
/tmp/0.
ruby 12970 root 14u REG 253,11 0 0 302063680
/tmp/0.
ruby 12973 root 15u REG 253,11 0 0 302063681
/tmp/0.
ruby 12977 root 16u REG 253,11 0 0 302063682
/tmp/0.
nginx 18837 root 6w REG 253,2 8667136 0 1441979
/var/log/
nginx 18837 root 7w REG 253,2 1271296000 0 1442083
/var/log/
nginx 18837 root 8w REG 253,2 61440 0 1441985
/var/log/
nginx 18837 root 9w REG 253,2 5623808 0 1442084
/var/log/
nginx 18838 dockerroot 6w REG 253,2 8667136 0 1441979
/var/log/
nginx 18838 dockerroot 7w REG 253,2 1271296000 0 1442083
/var/log/
nginx 18838 dockerroot 8w REG 253,2 61440 0 1441985
/var/log/
nginx 18838 dockerroot 9w REG 253,2 5623808 0 1442084
/var/log/
nginx 18839 dockerroot 6w REG 253,2 8667136 0 1441979
/var/log/
nginx 18839 dockerroot 7w REG 253,2 1271296000 0 1442083
/var/log/
nginx 18839 dockerroot 8w REG 253,2 61440 0 1441985
/var/log/
nginx 18839 dockerroot 9w REG 253,2 5623808 0 1442084
/var/log/
[root@fuel ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9.5G 1.4G 7.7G 16% /
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 0 7.8G 0% /dev/shm
tmpfs 7.8G 42M 7.8G 1% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/mapper/os-var 76G 36G 37G 50% /var
/dev/mapper/
/dev/vda3 197M 108M 89M 55% /boot
/dev/vda2 200M 0 200M 0% /boot/efi
[root@fuel ~]# dockerctl restart nginx
[root@fuel ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9.5G 1.4G 7.7G 16% /
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 0 7.8G 0% /dev/shm
tmpfs 7.8G 42M 7.8G 1% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/mapper/os-var 76G 36G 37G 50% /var
/dev/mapper/
/dev/vda3 197M 108M 89M 55% /boot
/dev/vda2 200M 0 200M 0% /boot/efi
tags: | added: area-library |
Changed in fuel: | |
assignee: | nobody → Fuel Library (Deprecated) (fuel-library) |
milestone: | none → 8.0-updates |
importance: | Undecided → Medium |
status: | New → Confirmed |
Changed in fuel: | |
assignee: | Fuel Library (Deprecated) (fuel-library) → Fuel Sustaining (fuel-sustaining-team) |
Changed in fuel: | |
milestone: | 8.0-updates → 10.0 |
Changed in fuel: | |
status: | Confirmed → Invalid |
tags: | added: customer-found |
Note that a dockerctl restart have resolved the problem with the nginx logs deleted but still open. It's not shown in the message above but it demonstrate there is really an issue with the way the logs are rotated. It shouldn't require a dockerctl restart.