/ fs is full on controller (big /var/log/mongod.log)

[Sergey Galkin]

astute_sha: 3c374c9f7bfbdbcd7ce2f716cd704e3044e6fb41
auth_required: true
build_id: 2014-11-12_22-00-23
build_number: '99'
feature_groups:
- mirantis
fuellib_sha: bc3c6298f203af3c911a6c5aa11760331c9bdd60
fuelmain_sha: 144262798c38c9186ad2d9c71c60a795b1474014
nailgun_sha: 563749b01dd26e1e0978636efa3a0dafa5745fab
ostf_sha: 720cc1308c3a7081736edd167e7928ca61914aaa
production: docker

Steps to reproduce:
1. Install cluster on 100 nodes (3 controllers in HA, 97 computes + ceph) wih ceilometer, ceph, neutron gre
2. Run tempest and rally tests

after ~8-9 hours uptime login to main controller, fs on / is full

[root@node-64 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 15G 15G 0 100% /
tmpfs 16G 37M 16G 1% /dev/shm
/dev/md0 194M 27M 158M 15% /boot
/dev/mapper/mongo-mongodb
                      959G 54G 857G 6% /var/lib/mongo

[root@node-64 ~]# ls -lh /var/log/ | grep mongod.log
-rw------- 1 root adm 3.9G Nov 14 03:01 mongod.log
-rw------- 1 root adm 3.7G Nov 14 03:01 mongod.log.1
-rw------- 1 root adm 13M Nov 14 02:01 mongod.log.4.gz
-rw------- 1 root adm 19M Nov 14 01:01 mongod.log.5.gz

[root@node-64 ~]# uptime
 05:53:05 up 8:50, 0 users, load average: 2.54, 2.46, 2.61

Sorry, can't attache snapshot
[root@fuel rally]# du -sh /var/log/
9.7G /var/log/

[Tomasz 'Zen' Napierala (tzn)]

Matt recently merged this: https://review.openstack.org/#/c/133956/ . MongoDB log should be rotated hourly according to configuration. We will investigate this

[Matthew Mosesohn (raytrac3r)]

They are rotated hourly. Look at the timestamps.

What's interesting is what level of logging is enabled on this system? Normal or debug? Also, rally generates far more traffic on a cluster than normal usage, so it may not affect most users.

[Sergey Galkin (sgalkin)]

Yes, debug enabled.
May be OS deployment script should allocate more space for /dev/mapper/os-root if debug enabled ?

[Sergey Galkin (sgalkin)]

https://bugs.launchpad.net/fuel/+bug/1394864 also