no logrotate for debug output leads to a disk overflow

[Tigran]

Hi,

Using fuel 6.1 (one of the nightly builds after hard code freeze);
Enabled debug output and today could not log into the dashboard.

Small investigation showed that on one of the controllers the hard drive was 100% used :

root@node-6:/var/log# uptime
 13:29:29 up 14 days, 3:25, 1 user, load average: 2.61, 2.57, 3.76
root@node-6:/var/log# du . -a | sort -n -r | head -n 10
35632432 .
11963944 ./conntrackd-stats.log.2
3900068 ./keystone
3896588 ./keystone/keystone-all.log
3637188 ./neutron-all.log
3003560 ./neutron
2058432 ./keystone-all.log
1946828 ./neutron/server.log
1633108 ./heat-all.log
1511560 ./heat
root@node-6:/var/log# du -sh ./keystone
3.8G ./keystone
root@node-6:/var/log# du . -sh
34G .

[Oleksiy Molchanov (omolchanov)]

I have already a patch for 6.1 https://review.openstack.org/#/c/196609/ for conntrackd.

[Tigran (tigran-avanesov)]

It also might be that some logrotate scripts do not work (not sure if i'm testing correctly):

root@node-6:/etc/logrotate.d# logrotate --force fuel.nodaily
error: error creating output file /var/log/ceilometer/ceilometer-agent-notification.log.1: File exists

root@node-6:/etc/logrotate.d# grep su fuel.nodaily
"/var/log/sudo.log"
  su root syslog
root@node-6:/etc/logrotate.d# ls -ld /var/log/ceilometer
drwxrwxr-x 2 ceilometer ceilometer 4096 Jun 28 00:30 /var/log/ceilometer

and logrotate for the other files in the script are not executed... (e.g. keystone)

[Tigran (tigran-avanesov)]

It also seems that /var/log/upstart/keystone grows so quickly in a debug mode, that it takes all space in default 50Gb root partition before any daily logrotate start working...

[Tomasz 'Zen' Napierala (tzn)]

Changing to medium as any "debug" operations should be very careful and additionally monitored. Fuel does not provide monitoring solution itself.
Note about missing configuration for logrotate is still valid however

[Vladimir Kuklin (vkuklin)]

I do not think that we need to lower priority of this bug to medium - it can affect user production environment whenever he wants to turn on debugging to fix some issues. I suggest we continue working on this bug for 7.0 and set importance to high.

[Sergey Kolekonov (skolekonov)]

Looks like that the main problem in 7.0 is that logs for upstart and atop are not rotated.
upstart logs seems to be useless at all, because services have their own logging. We can disable these logs by adding console=none to upstart scripts, or disable them globally.

[Alex Schultz (alex-schultz)]

which atop logs are not rotated? The atop.log is cleared daily via the restart of atop and we should only be keeping 7 days of atop_YYYYMMDD files. https://review.openstack.org/#/c/179831/ was done to address atop log rotation.

[Sergey Kolekonov (skolekonov)]

Alex, I'll check, maybe it was outdated ISO. So for now I suggest to use use_stderr=false for all components (because upstart is logging stderr from jobs) and add logrotate for upstart logs themselves (we should not disable them because not only openstack components write logs here)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/209513

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/209513
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=8637d96478cb4f44bd051aa675c98cf165970640
Submitter: Jenkins
Branch: master

commit 8637d96478cb4f44bd051aa675c98cf165970640
Author: Sergey Kolekonov <email address hidden>
Date: Wed Aug 5 16:07:01 2015 +0300

    Fix upstart logs management

    - disable use_stderr option for all components because upstart adds to logs
      all output from stderr output of its jobs
    - add upstart logs to common fuel logrotate configuration file
    - add noop tests

    Change-Id: Ifefec5e4c8e3438229018edc7c1aaa53a3a779bd
    Upstream patches: https://review.openstack.org/#/q/topic:bug/1482564,n,z
    Closes-bug: #1469176

[Igor (ipukha)]

Verified on MOS 7.0 ISO #265

[Roman Rufanov (rrufanov)]

customer found on 6.1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/225876

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/225876
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=99c3c1df1a8329d960f6a86cbf230ca603ab75cd
Submitter: Jenkins
Branch: stable/6.1

commit 99c3c1df1a8329d960f6a86cbf230ca603ab75cd
Author: Sergey Kolekonov <email address hidden>
Date: Wed Aug 5 16:07:01 2015 +0300

    Fix upstart logs management

    - disable use_stderr option for all components because upstart adds to logs
      all output from stderr output of its jobs
    - add upstart logs to common fuel logrotate configuration file
    - add noop tests

    Change-Id: Ifefec5e4c8e3438229018edc7c1aaa53a3a779bd
    Upstream patches: https://review.openstack.org/#/q/topic:bug/1482564,n,z
    Closes-bug: #1469176
    (cherry picked from commit 8637d96478cb4f44bd051aa675c98cf165970640)

[Alexey Stupnikov (astupnikov)]

MOS 6.0 is no longer supported, moving to Won't Fix.