Heka logstreamer journal missing for OpenStack services

[Éric Lemoine]

Heka logstreamer stores journal files in /var/cache/lma_collector/logstreamer. This is what the directory looks like on my controller node:

    # ls /var/cache/lma_collector/logstreamer/
    keystone-wsgi-admin mysql ovsdb-server pacemaker rabbitmq.node-10-sasl system.cron.log system.debug system.kern.log system.syslog
    keystone-wsgi-main openstack.swift ovs-vswitchd rabbitmq.node-10 system.auth.log system.daemon.log system.haproxy.log system.messages

There are no journal files for openstack.nova, openstack.neutron, etc. I.e. for all the OpenStack logs handled by the "openstack_log.lua" decoder sandbox. (Note that there IS a journal file for openstack.swift, which is handled by another decoder.)

This is a big issue, as it means that the all the logs are reprocessed when the LMA service is restarted!

[Éric Lemoine (elemoine)]

This issue is most probably related to https://review.openstack.org/#/c/255418/.

[Éric Lemoine (elemoine)]

Note: version 0.10.0 of Heka has been released. See https://github.com/mozilla-services/heka/releases/tag/v0.10.0. It is interesting to note that this release fixes a Logstreamer bug: https://github.com/mozilla-services/heka/issues/1199. To be tested.

[Éric Lemoine (elemoine)]

Using Heka version 0.10.0 does not fix the problem. The problem is elsewhere. It's related to the use of the character "/" as the separator in the logstreamer differentiator. The logstreamer uses the Logger name produced by the differentiator as the journal file name, so this name should not include slash characters.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/262808

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/262808
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=e7ee2478545f7fcd989076e5b055b65fb5a8553d
Submitter: Jenkins
Branch: master

commit e7ee2478545f7fcd989076e5b055b65fb5a8553d
Author: Éric Lemoine <email address hidden>
Date: Thu Dec 31 16:34:24 2015 +0100

    Fix OpenStack logstreamer differentiator

    Using "/" as the separator in the Heka logstreamer differentiator
    prevents the logstreamer from creating a journal file. This is
    because the logstreamer uses the Logger name produced by the
    differentiator as the journal file name.

    This change fixes the problem by changing the differentiator separator
    from "/" to "_".

    Change-Id: Iaf838f0e5df370df24f0d9d63a9bf13e37aa4bbe
    Closes-Bug: #1530326

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/263171
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=fcaf530a5f1c1b3f355d5541b73291ad8cf15a46
Submitter: Jenkins
Branch: master

commit fcaf530a5f1c1b3f355d5541b73291ad8cf15a46
Author: Éric Lemoine <email address hidden>
Date: Mon Jan 4 10:07:32 2016 +0100

    Fail if differentiator includes slash

    Make heka::input::logstreamer fail if the differentiator parameter
    includes a slash character.

    Change-Id: Iff87ee8eef2b18a3e84591f7dc05d7c360aef86b
    Related-bug: #1530326