[LMA] InfluxDB/Grafana - Log rotation not working

[Mike Nguyen]

The log rotation for the /var/log/influxdb/influxd.log file does not seem to be working.

- There are 2 files, /etc/logrotate.d/influxd and /etc/logrotate.d/influxdb. The influxdb package seems to come with the /etc/logrotate.d/influxd file already (I suppose this file might not have been there in earlier versions?), making the /etc/logrotate.d/influxdb (that seems to be put in place by puppet) not needed anymore.

- The permissions for both of those files was not right, making it so logrotate refused to run the rotation for the influxd.log file. They need to be root/root/0644 for logrotate to accept to run. They were 0664 (and also, I believe, not owned by root, but I've lost the output for that, might need to be checked).

And, of course, the log rotation not working makes it so /var/log filesystem fills up somewhat quickly.

[Simon Pasquier (simon-pasquier)]

Thanks for the report! IIUC this only affects the stable/0.8 branch since we've removed the custom logrotate configuration in the master branch.

[Mike Nguyen (moozoo)]

Oh, my bad, you're right, I forgot to mention the version I'm on, which is indeed 0.8!

id | name | version | package_version
---|-----------------------------|---------|----------------
6 | influxdb_grafana | 0.8.0 | 3.0.0
5 | lma_collector | 0.8.0 | 2.0.0
7 | lma_infrastructure_alerting | 0.8.0 | 3.0.0
8 | elasticsearch_kibana | 0.8.0 | 3.0.0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/0.8
Review: https://review.openstack.org/280659

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/280659
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-influxdb-grafana/commit/?id=b266926e39534662481d61cd864924e827bc595f
Submitter: Jenkins
Branch: stable/0.8

commit b266926e39534662481d61cd864924e827bc595f
Author: Guillaume Thouvenin <email address hidden>
Date: Tue Feb 16 13:40:45 2016 +0100

    Remove logrotate configuration file

    This change removes the logrotate file because it is included in the
    InfluxDB package now.

    Change-Id: I3501d94f8a73297681411f6aa612e36005b83dff
    Closes-Bug: #1535440

[guillaume thouvenin (guillaume-thouvenin)]

Just tested with the fix on my env and today I can see:

root@node-4:/var/log/influxdb# ll
total 67792
drwxr-xr-x 2 influxdb influxdb 4096 Feb 17 07:35 ./
drwxrwxr-x 15 root syslog 4096 Feb 16 08:33 ../
-rw-r--r-- 1 root root 2098758 Feb 17 08:45 influxd.log
-rw-r--r-- 1 root root 67306547 Feb 17 07:35 influxd.log-20160217

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/0.8
Review: https://review.openstack.org/296373

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/296373
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-influxdb-grafana/commit/?id=9b7c01da209427ccd0b521ec28cfe88fd992a4fb
Submitter: Jenkins
Branch: stable/0.8

commit 9b7c01da209427ccd0b521ec28cfe88fd992a4fb
Author: Swann Croiset <email address hidden>
Date: Wed Mar 23 12:02:34 2016 +0100

    Support rolling upgrade from 0.8

    The patch removes explicitly the logrotate configuration installed by the
    0.8 version.

    Change-Id: I069627d663028c541b5fbd64297ed5ddfc1b4396
    Related-Bug: #1535440

[Swann Croiset (swann-w)]

The fix is incomplet because the logrotate configuration of the package still sets a wrong permission.

# logrotate -f -v /etc/logrotate.d/influxd
Ignoring /etc/logrotate.d/influxd because of bad file mode.

Handling 0 logs

This command solves the issue:
# chmod g-w /etc/logrotate.d/influxd

# logrotate -f -v /etc/logrotate.d/influxd
reading config file /etc/logrotate.d/influxd

Handling 1 logs

rotating pattern: /var/log/influxdb/influxd.log forced from command line (7 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/influxdb/influxd.log
  log needs rotating
rotating log /var/log/influxdb/influxd.log, log->rotateCount is 7
dateext suffix '-2016032510'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
destination /var/log/influxdb/influxd.log-2016032510.gz already exists, skipping rotation

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/0.8
Review: https://review.openstack.org/297628

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/297628
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-influxdb-grafana/commit/?id=0ed15b1ba244787f81cd819b4a98aecabbf01338
Submitter: Jenkins
Branch: stable/0.8

commit 0ed15b1ba244787f81cd819b4a98aecabbf01338
Author: Swann Croiset <email address hidden>
Date: Fri Mar 25 12:04:12 2016 +0100

    Fix wrong permission of logrotate configuration file

    Fixes-Bug: #1535440
    Change-Id: Ic0d1755932a6f5380dac94d7d99d805788779759