nginx keeps writing to rotated logs
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Released
|
Critical
|
Alexey Stupnikov | ||
7.0.x |
Fix Released
|
Critical
|
Alexey Stupnikov | ||
Mitaka |
Invalid
|
High
|
Alexey Stupnikov |
Bug Description
Fuel 8.0 MU3
Some services inside containers: nginx, postgres, astute - keep open descriptors to files which have been removed. The problem is with nginx, which keeps open rotated log files and keeps writing to them. This causes log rotation to technically fail, and space to be polluted but ever expanding but non-visible files.
How to reproduce:
1. deploy Fuel 8.0
2. wait 2 days
3. run `lsof 2>/dev/null| grep "(deleted)" | grep var/log`
Expected result - no output
Actual result - many open descriptors to logs which have been deleted
Additionelly, as expected in this situation new logs are empty.
Example:
[root@fuel ~]# lsof 2>/dev/null | grep "(deleted)" | grep var/log
nginx 8558 root 2w REG 253,2 832 392504 /var/log/
nginx 8558 root 4w REG 253,2 832 392504 /var/log/
nginx 8558 root 6w REG 253,2 3479159 392513 /var/log/
nginx 8558 root 7w REG 253,2 271475922 392514 /var/log/
nginx 8558 root 8w REG 253,2 332456 392515 /var/log/
nginx 8558 root 9w REG 253,2 178745 392516 /var/log/
nginx 8559 dockerroot 2w REG 253,2 832 392504 /var/log/
nginx 8559 dockerroot 4w REG 253,2 832 392504 /var/log/
nginx 8559 dockerroot 6w REG 253,2 3479159 392513 /var/log/
nginx 8559 dockerroot 7w REG 253,2 271475922 392514 /var/log/
nginx 8559 dockerroot 8w REG 253,2 332456 392515 /var/log/
nginx 8559 dockerroot 9w REG 253,2 178745 392516 /var/log/
[root@fuel ~]# date
Fri Nov 18 02:13:41 UTC 2016
[root@fuel ~]# uptime
02:13:52 up 1 day, 5:58, 1 user, load average: 0.00, 0.01, 0.05
[root@fuel ~]#
Changed in fuel: | |
milestone: | none → 8.0-mu-4 |
milestone: | 8.0-mu-4 → none |
Changed in fuel: | |
milestone: | none → 8.0-updates |
tags: | added: area-library |
Changed in fuel: | |
importance: | High → Critical |
no longer affects: | fuel/8.0.x |
Changed in fuel: | |
assignee: | Fuel Sustaining (fuel-sustaining-team) → MOS Maintenance (mos-maintenance) |
Changed in fuel: | |
status: | In Progress → Fix Committed |
tags: | added: on-verification |
tags: | added: on-verification |
proposed fix for 8.0 and 7.0:
In /etc/logrotate. d/nginx (provided by nginx-1. 6.3-6.el7~ mos1.x86_ 64), change:
/var/log/nginx/*log {
create 0644 nginx nginx
daily
rotate 10
missingok
notifempty
compress
sharedscripts
postrotate
/bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true
endscript
}
to:
/var/log/ docker- logs/nginx/ *log {
create 0644 nginx nginx
daily
rotate 10
missingok
notifempty
compress
sharedscripts
postrotate
/bin/kill -USR1 `pgrep -f /usr/sbin/nginx 2>/dev/null` 2>/dev/null || true
endscript
}
Additionally, remove /var/log/ docker- logs/nginx/ *log from /etc/logrotate. d/fuel. nodaily, otherwise the above change will not work.
For Fuel 9.0, which is also affected due to double logrotate policy - remove nginx from /etc/logrotate. d/fuel. nodaily, the logrotate config installed with nginx is working fine by itself.