Handle sequential rotating OpenStack logfiles

[Éric Lemoine]

Fuel uses logrotate to rotate logfiles, including OpenStack logfiles. For example, this is the logrotate configuration for nova (in /etc/logrotate.d/nova-common.conf):

    /var/log/nova/*.log {
        daily
        missingok
        compress
        delaycompress
        copytruncate
    }

With this, and the /etc/logrotate.conf configuration file, the following log structure is possible:

    /var/log/nova/nova-api.log
    /var/log/nova/nova-api.log.1
    /var/log/nova/nova-api.log.2

At this point our Heka logstreamer configuration for OpenStack logs (lma_collector::logs::openstack) doesn't handle log rotation, as logfiles ending with .<x> are not read at all.

The Heka logstreamer plugin includes specific support for sequential rotating logfiles. See http://hekad.readthedocs.org/en/latest/pluginconfig/logstreamer.html#single-sequential-rotating-logfile. We should use it.

This is what the heka::input:.logstreamer would look like:

    heka::input::logstreamer { $title:
      config_dir => $lma_collector::params::config_dir,
      log_directory => "/var/log/${title}",
      decoder => 'openstack',
      splitter => 'openstack',
      file_match => '(?P<Service>.+)\.log\.?(?P<Seq>\d*)$',
      differentiator => "['${title}', '/', 'Service']",
      priority => "['^Seq']",
      require => Class['lma_collector::logs::openstack_decoder_splitter'],
      notify => Class['lma_collector::service'],
    }

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/262171

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/262171
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=a97b56dbf6907cbbb5f76cde76cbc5d66bc46eb7
Submitter: Jenkins
Branch: master

commit a97b56dbf6907cbbb5f76cde76cbc5d66bc46eb7
Author: Éric Lemoine <email address hidden>
Date: Tue Dec 29 10:38:10 2015 +0100

    Handle sequential rotating OpenStack logfiles

    Change-Id: I2483f14862ea307d489f27ef36418e37d597ddf1
    Closes-Bug: #1529784