RabbitMQ log delimiter is incorrect

[Éric Lemoine]

For the parsing of RabbitMQ logs a RegexSplitter is used with '\n(=[^=]+====)' for the delimiter [*]. That delimiter is not correct. It should be '\n\n(=[^=]+====)' (two newline characters).

RabbiMQ logs look like this:

    =INFO REPORT==== 25-Feb-2016::14:44:27 ===
    Starting RabbitMQ 3.5.5 on Erlang R16B03-1
    Copyright (C) 2007-2015 Pivotal Software, Inc.
    Licensed under the MPL. See http://www.rabbitmq.com/

    =INFO REPORT==== 25-Feb-2016::14:44:27 ===
    node : rabbit@operator
    home dir : /var/lib/rabbitmq
    config file(s) : /etc/rabbitmq/rabbitmq.config
    cookie hash : X03MO1qnZdYdgyfeuILPmQ==
    log : /var/log/kolla/rabbitmq/rabbit.log
    sasl log : /var/log/kolla/rabbitmq/rabbit-sasl.log
    database dir : /var/lib/rabbitmq/mnesia/rabbit

    =INFO REPORT==== 25-Feb-2016::14:44:28 ===
    Memory limit set to 1516MB of 3791MB total.

So when using only one newline character in the delimiter there's a problem at the very start of the RabbitMQ log stream. The log stream starts with a new line, so Heka finds a delimiter match on the first two lines. But it does not have content for this log entry so it may stop there.

The solution to this problem is to use '\n\n(=[^=]+====)' as the delimiter. And using that delimiter is more correct anyway.

Also, the deliver_incomplete_final splitter option should be used, for Heka to grab the last line in the RabbitMQ log stream. Without this the last line is not collected until new lines are appended to the stream.

[*] <https://github.com/openstack/fuel-plugin-lma-collector/blob/master/deployment_scripts/puppet/modules/lma_collector/manifests/logs/rabbitmq.pp#L27>
[**] <http://hekad.readthedocs.org/en/v0.10.0/config/splitters/index.html>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/287767

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/287767
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=6fe3df80d5354f93571816039d2ebe3bc71a3b10
Submitter: Jenkins
Branch: master

commit 6fe3df80d5354f93571816039d2ebe3bc71a3b10
Author: Guillaume Thouvenin <email address hidden>
Date: Thu Mar 3 13:39:50 2016 +0100

    Update RabbitMQ log delimiter

    This patch modifies the regex used by the rabbitmq splitter. It adds a
    newline constant at the beginning of the regex to avoid to match the
    first pattern of the log file. We observed that sometimes, the log
    stream starts with a new line, so Heka finds a delimiter match on the
    first two lines. But it does not have content for this log entry so it
    may stop there.

    Known issue: you miss the first log entry.

    Change-Id: Ib2208db842de9f30a95a514f0e51d00d693374e1
    Closes-Bug: #1550222