[library] Logs from MongoDB server should be written to a separate file instead of syslog

[Artem Panchenko]

Recently I ran long-time Rally performance test of OpenStack environment deployed using Fuel and after 10 hours of testing I observed that MySQL on primary controller is down due to lack of free disk space:

<27>Sep 9 09:58:01 node-1 mysqld: 2014-09-09 09:58:01 8287 [Warning] Disk is full writing './mysql-bin.000008' (Errcode: 28 - No space left on device). Waiting for someone to free space...

It also caused a lot of errors in other services such as nova-api and rabbitmq. Configuration of cluster was the following:

1. CentOS, HA, KVM, NovaFlatDHCP, Sahara, Ceilometer, OpenStack debug logging enabled
2. 1 controller+mongo, 2 controller+cinder+mongo, 1 compute and 2 compute+ceph nodes

And I used default disk partitioning configuration, so for 'root' filesystem was allocated 44 GB of disk space. During performance testing all of free disk space was occupied by huge log file:

[root@node-1 ~]# ls -lah /var/log/messages
-rw-------. 1 root root 30G Sep 9 09:43 /var/log/messages

And most of its content belonged to mongodb server:

[root@node-1 ~]# wc -l /var/log/messages
67681931 /var/log/message

[root@node-1 ~]# awk '{print $5}' /var/log/messages | sort | uniq -c | sort -n | tail -5

    760 kernel:
   1000 nailgun-agent:
  53995 rabbitmq
3930722 haproxy[47952]:
63490772 mongod.27017[16088]:

Here you can find the part of MongoDB logs:

http://paste.openstack.org/show/108840/

As you can see 93% of lines in /var/log/messages file were from MongoDB server and it wasn't rotated by logrotate, because that file is rotated weekly. In my opinion to avoid such issues we should configure rsyslog to write MongoDB logs to a separate file and add it to the logrotate's hourly rotation config (/etc/logrotate.d/20-fuel.conf).

Also, probably it's safer to create a separate LVM volume specially for logs and mount it to the /var/log, so then we could increase it's size in case of 'OpenStack debug logging' option is enabled.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/120588

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/120588
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=f686e9ea00eaa9ae141e283ff2b8b28bacd89f72
Submitter: Jenkins
Branch: master

commit f686e9ea00eaa9ae141e283ff2b8b28bacd89f72
Author: Tomasz 'Zen' Napierala <email address hidden>
Date: Wed Sep 10 23:25:29 2014 +0200

    Adds "/var/log/messages" to hourly log rotation

    Because still many services, including MongoDB, are logging to standard log
    destination we should rotate the logs more agressively. Moreover MongoDB in
    version 2.4 cannot use LogFacility, so it's easier to fix it in logrotate than
    in MonngoDB module itself.

    Change-Id: Ice1bbf4a3bf2d453a41d7660b8742a074ec29216
    Closes-Bug: 1367234

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/120820

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/120821

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/120820
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=395fd9d20a003603cc9ad26e16cb13c1c45e24e6
Submitter: Jenkins
Branch: stable/5.1

commit 395fd9d20a003603cc9ad26e16cb13c1c45e24e6
Author: Tomasz 'Zen' Napierala <email address hidden>
Date: Wed Sep 10 23:25:29 2014 +0200

    Adds "/var/log/messages" to hourly log rotation

    Because still many services, including MongoDB, are logging to standard log
    destination we should rotate the logs more agressively. Moreover MongoDB in
    version 2.4 cannot use LogFacility, so it's easier to fix it in logrotate than
    in MonngoDB module itself.

    Change-Id: Ice1bbf4a3bf2d453a41d7660b8742a074ec29216
    Closes-Bug: 1367234
    (cherry picked from commit f686e9ea00eaa9ae141e283ff2b8b28bacd89f72)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/120821
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=d3ce50bea9ec91ac932238e676decd4d24bc28e8
Submitter: Jenkins
Branch: stable/5.0

commit d3ce50bea9ec91ac932238e676decd4d24bc28e8
Author: Tomasz 'Zen' Napierala <email address hidden>
Date: Wed Sep 10 23:25:29 2014 +0200

    Adds "/var/log/messages" to hourly log rotation

    Because still many services, including MongoDB, are logging to standard log
    destination we should rotate the logs more agressively. Moreover MongoDB in
    version 2.4 cannot use LogFacility, so it's easier to fix it in logrotate than
    in MonngoDB module itself.

    Change-Id: Ice1bbf4a3bf2d453a41d7660b8742a074ec29216
    Closes-Bug: 1367234
    (cherry picked from commit f686e9ea00eaa9ae141e283ff2b8b28bacd89f72)

[Matthew Mosesohn (raytrac3r)]

This just rotates /var/log/messages. I'll move the file to /var/log/mongod.log in rsyslog both on nodes and on Fuel Master and add rotation policy.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/133956

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/133956
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=6d0309ef2fce9b8032185bc01c7122b600844a1b
Submitter: Jenkins
Branch: master

commit 6d0309ef2fce9b8032185bc01c7122b600844a1b
Author: Matthew Mosesohn <email address hidden>
Date: Wed Nov 12 16:49:55 2014 +0400

    Move mongodb logs to mongod.log

    Enables consolidation of MongoDB logs in one location,
    instead of polluting /var/log/messages.

    Added rotation for /var/log/mongod.log on deployed nodes,
    while the current Fuel Master rotation requires no changes.

    Change-Id: I3bbc6b87ed9d125191041c310069136c59d8d1f9
    Closes-Bug: 1367234

[Mykhaylo Slobodyan (mykhaylo-slobodyan)]

Verified in 6.0 ISO 49

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.0"
  api: "1.0"
  build_number: "49"
  build_id: "2014-12-09_22-41-06"
  astute_sha: "16b252d93be6aaa73030b8100cf8c5ca6a970a91"
  fuellib_sha: "2c99931072d951301d395ebd5bf45c8d401301bb"
  ostf_sha: "a9afb68710d809570460c29d6c3293219d3624d4"
  nailgun_sha: "22bd43b89a17843f9199f92d61fc86cb0f8772f1"
  fuelmain_sha: "3aab16667f47dd8384904e27f70f7a87ba15f4ee"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/153546

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/153546
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=ea460229777a356008551c0e15e5fce67eff8623
Submitter: Jenkins
Branch: stable/5.1

commit ea460229777a356008551c0e15e5fce67eff8623
Author: Matthew Mosesohn <email address hidden>
Date: Wed Nov 12 16:49:55 2014 +0400

    Move mongodb logs to mongod.log

    Enables consolidation of MongoDB logs in one location,
    instead of polluting /var/log/messages.

    Added rotation for /var/log/mongod.log on deployed nodes,
    while the current Fuel Master rotation requires no changes.

    Change-Id: I3bbc6b87ed9d125191041c310069136c59d8d1f9
    Closes-Bug: 1367234
    (cherry picked from commit 6d0309ef2fce9b8032185bc01c7122b600844a1b)

[Yury Konov (yukonoff)]

I experience similar behavior on MOS 7.0 + MU1 with LMA plugin.

[root@fuel ~]# df -h /var/log
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-varlog
                       40G 40G 0 100% /var/log

[root@fuel ~]# ls -lah $(find /var/log/ -type f | xargs du -h | sort -h | tail -n 1 | cut -f2)
-rw-r----- 1 root adm 16G Фев 8 15:53 /var/log/docker-logs/remote/node-28.domain.tld/mongod.27017.log