rabbitmq logrotate doesn't work properly

[Leontii Istomin]

the logrotate config for rabbitmq removes old logs instead of create a new one. /etc/logrotate.d/rabbitmq-server: http://paste.openstack.org/show/361513/
[root@node-8 ~]# ls -la /var/log/rabbitmq/rabbit\@node-8*
-rw-r----- 1 rabbitmq rabbitmq 0 Jul 10 10:42 /<email address hidden>
-rw-r----- 1 rabbitmq rabbitmq 0 Jul 10 10:42 /<email address hidden>
[root@node-8 ~]# service rabbitmq-server rotate-logs > /dev/null
[root@node-8 ~]# ls -la /var/log/rabbitmq/rabbit\@node-8*
-rw-r----- 1 rabbitmq rabbitmq 0 Jul 10 11:05 /<email address hidden>
-rw-r----- 1 rabbitmq rabbitmq 0 Jul 10 11:05 /<email address hidden>

We need to use the following command:
DATE=-`date +%y_%m_%d_%H_%M_%S`; rabbitmqctl rotate_logs ${DATE}
[root@node-13 ~]# ls -la /var/log/rabbitmq/rabbit\@node-13*
-rw-r----- 1 rabbitmq rabbitmq 0 Jul 10 10:47 /<email address hidden>
-rw-r----- 1 rabbitmq rabbitmq 0 Jul 10 10:47 /<email address hidden>
[root@node-13 ~]# DATE=-`date +%y_%m_%d_%H_%M_%S`; rabbitmqctl rotate_logs ${DATE}
Rotating logs to files with suffix "-15_07_10_11_06_05" ...
...done.
[root@node-13 ~]# ls -la /var/log/rabbitmq/rabbit\@node-13*
-rw-r----- 1 rabbitmq rabbitmq 0 Jul 10 11:06 /<email address hidden>
-rw-r----- 1 rabbitmq rabbitmq 91 Jul 10 11:06 /<email address hidden>15_07_10_11_06_05
-rw-r----- 1 rabbitmq rabbitmq 0 Jul 10 11:06 /<email address hidden>
-rw-r----- 1 rabbitmq rabbitmq 0 Jul 10 11:06 /<email address hidden>15_07_10_11_06_05

api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-19_13-02-31
build_number: '525'
feature_groups:
- mirantis
fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'

[Florek (sflorczak)]

Because ROTATE_SUFFIX in the configuration file is not set the log is cleared.
However "service rabbitmq-server rotate-logs" shouldn't be used in the first place. The proper way to rotate the logs is using: logrotate -f /etc/logrotate.d/rabbitmq-server

[Leontii Istomin (listomin)]

I believe the bug should be in "confirmed" state because we lose rabbitmq logs. If the right way to solve this issue is using logrotate instead of native rabbitmq logrotation please implement that.

[Florek (sflorczak)]

The issue can be reproducing in the following way:
in /var/log/rabbitmq there should be at least two files:
 - <email address hidden> with size less than 100MB (value configured in /etc/logrotate.conf), where X is the number of the node
 - any other file (*.log) with size more than 100MB
whenever logrotate will be run (/usr/sbin/logrotate -v /etc/logrotate.conf) it'll rotate the second file - because its larger than the defined threshold, will not rotate <email address hidden> because it's below threshold and after that the postscript will be run (according to /etc/logrotate.d/rabbitmq-server):
/etc/init.d/rabbitmq-server rotate-logs > /dev/null
Because the "ROTATE_SUFFIX" is not set in the /etc/init.d/rabbitmq-server no logs will be rotated, instead of that content of the /<email address hidden> will be flushed.

The simplest way to fix the issue is to change /etc/logrotate.d/rabbitmq-server in the following way:

delete:

 sharedscripts
        postrotate
            /etc/init.d/rabbitmq-server rotate-logs > /dev/null
        endscript

add:
 copytruncate

This way the "clearing script" will not be run. "copytruncate" will copy current log file to the new file and truncate the file currently used by the rabbitmq. Thanks to that the file handle used by rabbitmq doesn't have to be changed and the process itself doesn't have to be reloaded.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 7.0
Change author: Aleksandr Mogylchenko <email address hidden>
Review: https://review.fuel-infra.org/10372

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 8.0
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/14927

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/14927
Submitter: Pkgs Jenkins <email address hidden>
Branch: 8.0

Commit: c03a5cc1c14c5a0c1426f88f87b628d86d407fb8
Author: Ivan Suzdal <email address hidden>
Date: Tue Dec 22 10:23:29 2015

Remove postscript, add copytruncate to logrotate.

Change-Id: Ide704954dbdac8be479ea4ae9b94479dd523677c
Closes-Bug: #1473405

[Ivan Lozgachev (ilozgachev)]

Verified on Fuel 8.0 build 518, ENV-13

[Alexey Lebedeff (alebedev-a)]

Fixed for centos7 and ubuntu by packaging rabbit 3.6.1, where `rotate_logs` was completely fixed.

[Sergey Kulanov (skulanov)]

Folks, please check for master branch, probably you forgot to fix logrotate script. Please check failed test [1]:

1)
[root@nailgun ~]# rpm -q --changelog rabbitmq-server
* Wed Mar 16 2016 Alexey Lebedeff <email address hidden> - 3.6.1-3~mos1
- d9a7cc1 Fix systemd issues
- 383e912 Tune 3.6.1 for RHEL and backport some patches
- c389137 Import upstream package for 3.6.1
- b0a8190 Bump version for rebuild
- 66e63bf Merge "Remove postscript, add copytruncate to logrotate." into 8.0
- d231c4a Suffix update for rabbitmq-server * Modification made according to https://github.com/openstack/fuel-specs/blob/master/specs/8.0/separate-mos-from-centos.rst#rpm-packages-versioning

2)
[root@nailgun ~]# rpm -ql rabbitmq-server-3.6.1-3.el7~mos1.noarch | grep logrotate
/etc/logrotate.d/rabbitmq-server
[root@nailgun ~]# cat /etc/logrotate.d/rabbitmq-server
/var/log/rabbitmq/*.log {
        weekly
        missingok
        rotate 20
        compress
        delaycompress
        notifempty
        sharedscripts
        postrotate
            /sbin/service rabbitmq-server rotate-logs > /dev/null
        endscript
}

3)
[root@nailgun ~]# service rabbitmq-server rotate-logs
The service command supports only basic LSB actions (start, stop, restart, try-restart, reload, force-reload, status). For other actions, please try to use systemctl.

should be:
        postrotate
            /usr/sbin/rabbitmqctl -q rotate_logs
        endscript

[1]. https://product-ci.infra.mirantis.net/view/9.0_swarm/job/9.0.system_test.ubuntu.logrotate/51/

[Alexey Lebedeff (alebedev-a)]

I've fixed `rabbitmqctl rotate_logs` in upstream version of rabbit - https://github.com/rabbitmq/rabbitmq-server/pull/543 and so patched logrotate script is no longer needed.

[Vitaly Sedelnik (vsedelnik)]

Won't Fix for 7.0-updates because of Medium importance