logrotate fails to rotate logs with errors like this (running logrotate -v):
> ... cut ...
> dateext suffix '-20150302'
> glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
> renaming /var/log/docker-logs/nailgun/app.log.4.gz to /var/log/docker-logs/nailgun/app.log.5.gz (rotatecount 4, logstart 1, i 4),
> old log /var/log/docker-logs/nailgun/app.log.4.gz does not exist
> renaming /var/log/docker-logs/nailgun/app.log.3.gz to /var/log/docker-logs/nailgun/app.log.4.gz (rotatecount 4, logstart 1, i 3),
> old log /var/log/docker-logs/nailgun/app.log.3.gz does not exist
> renaming /var/log/docker-logs/nailgun/app.log.2.gz to /var/log/docker-logs/nailgun/app.log.3.gz (rotatecount 4, logstart 1, i 2),
> old log /var/log/docker-logs/nailgun/app.log.2.gz does not exist
> renaming /var/log/docker-logs/nailgun/app.log.1.gz to /var/log/docker-logs/nailgun/app.log.2.gz (rotatecount 4, logstart 1, i 1),
> old log /var/log/docker-logs/nailgun/app.log.1.gz does not exist
> renaming /var/log/docker-logs/nailgun/app.log.0.gz to /var/log/docker-logs/nailgun/app.log.1.gz (rotatecount 4, logstart 1, i 0),
> old log /var/log/docker-logs/nailgun/app.log.0.gz does not exist
> copying /var/log/remote/172.31.2.82/mongod.27017.log to /var/log/remote/172.31.2.82/mongod.27017.log.1
> error: error creating output file /var/log/remote/172.31.2.82/mongod.27017.log.1: File exists
> [root@fuel-juno ~]# ls -l /var/log/remote/[0-9]*/mongod.*.log*
> -rw-r----- 1 root adm 27894054211 Mar 2 08:50 /var/log/remote/172.31.2.82/mongod.27017.log
> -rw-r----- 1 root adm 449987680 Feb 27 19:01 /var/log/remote/172.31.2.82/mongod.27017.log.1
> -rw-r----- 1 root adm 16194075 Feb 27 18:00 /var/log/remote/172.31.2.82/mongod.27017.log.5.gz
This leads to the same problems as described in bug #1383741, where Fuel gets corrupt and unable to, for example, deploy new nodes.
This has not only been a problem with mongod*.log; We've seen it with other files as well. Manually removing the .1 files have lead to logrotate successfully running, but after a while it pops up again.
How logrotate gets to this state we're not sure, and we don't think it's because the mongo log is too big, as we've seen it with smaller files as well. Unfortunately I have not recorded those files as I thought this was a only-happens-once thing.
The main problem for us is currently that the mongo log, as stated earlier, is way too big. The log level of 1 in /etc/mongo.conf on the mongo server sets mongo to log /every write operation/. That's a lot for the 72 nodes we have, and we've barely started using them.
Another thing that I currently can't back up is that I think if a rotation of a file in the middle of the logrotate pattern fails, none of the later files will rotate, again leading to the mongo log never rotating an filling the disk...
Please provide the Fuel version and diagnostic logs snapshot, if possible