Logrotate configs must be adjusted for containerized services logs

[Bogdan Dobrelya]

Logs collection part is addressed in the spec https://review.openstack.org/#/c/462900

Logrotate configs covering added log paths for containers must come from either openstack packages, or a tripleo package.

[Bogdan Dobrelya (bogdando)]

Perhaps the logrotate scripts could come with the openstack-tripleo-common-containers(-base???) package, but sending SIGHUP to containerized openstack (and not only) services will be a tricky thing . Restarting containers could work as well, but looks too intrusive.

[Jiří Stránský (jistr)]

I wouldn't worry about SIGHUPs -- we should just make it so that the package is present on the node before starting any containers. That's doable both for fresh deployment and upgrade.

As for the package though, that might be more difficult. Tripleo-common is currently on undercloud only, we'd probably have to start installing it on overcloud too, which might be ok but would have to be double checked, as i think we had some issues with installing e.g. python-tripleoclient on overcloud, IIRC.

[Jiří Stránský (jistr)]

Oh you probably meant SIGHUPs from logrotate, not when installing the package, right? :)

[Bogdan Dobrelya (bogdando)]

Right. It must be done to prevent services from writing to open but unlinked files, consuming free space and ignoring rotated log files are now in place.

[Bogdan Dobrelya (bogdando)]

The plan is to ship a logrotate config and a SIGHUPper to clear unlinked & deleted files after it.
It may be done either with a postrotate script for logrotate, or with a cron job. The latter allows to introduce a time splay for sending SIGHUP to services, while the former can only SIGHUP them all at once. Something like an example /etc/logrotate.d/openstack-containers does the former:

/var/log/containers/*/*.log {
 rotate 14
 size 10M
 missingok
 notifempty

 postrotate
  /sbin/lsof -nPs +L1 +D /var/log 2>&1|\
  /bin/perl -ne '/\S+\s+(\d+).*\/var\/log\/.*\(deleted\)/ && print "$1\n"' |\
  /bin/xargs -n1 -r -t kill -HUP
 endscript
}

[Bogdan Dobrelya (bogdando)]

The tripleo-common package is not installed for hosts, but it is installed for all containers, and lsof will be installed as a dependency ( https://review.rdoproject.org/r/#/c/7878/ ). It seems that the best way to launch it is using the kolla cron container. It has both cron and logrotate, and can be configured as a uc/overcloud services to store the logrotate config like in the example above.

Additionally, it can be configured to run the aforementioned cleanup command instead of postrotate script, with a time splay.

[Bogdan Dobrelya (bogdando)]

The dependency is logrotate & cron containerization https://bugs.launchpad.net/tripleo/+bug/1706354

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/490048

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/490050

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/490068

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/490068
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=7776b19a7d90b360bb7078f1349f50e64286b456
Submitter: Jenkins
Branch: master

commit 7776b19a7d90b360bb7078f1349f50e64286b456
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Aug 2 18:07:08 2017 +0200

    Add logrotate-crond container images params

    Related-bug: #1700912

    Change-Id: I3fd918b01edd71e3e9e7c4efe7fb790ae2f37d19
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/490048
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=70987c80c1212e0ad50c4a7600cbf1665f673cc2
Submitter: Jenkins
Branch: master

commit 70987c80c1212e0ad50c4a7600cbf1665f673cc2
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Aug 2 15:42:38 2017 +0200

    Add logrotate-crond configuration

    Generate a cron job and a config for logrotate
    to be run against containerized services logs.

    Related-bug: #1700912

    Change-Id: Ib9d5d8ca236296179182613e1ff625deea168614
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/490050
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8a03456056b22a0abff470dfd81334e369ae4cbc
Submitter: Jenkins
Branch: master

commit 8a03456056b22a0abff470dfd81334e369ae4cbc
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Aug 2 14:48:26 2017 +0200

    Add logrotate with crond service

    Add a docker service template to provide containerized services
    logs rotation with a crond job.
    Add OS::TripleO::Services::LogrotateCrond to CI multinode-containers
    and to all environments among with generic services like Ntp or Kernel.
    Set it to OS::Heat::None for non containerized environments and
    only enable it to the environments/docker.yaml.

    Closes-bug: #1700912

    Change-Id: Ic94373f0a0758e9959e1f896481780674437147d
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0rc1 release candidate.