Logrotate configs must be adjusted for containerized services logs

Bug #1700912 reported by Bogdan Dobrelya on 2017-06-28
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Bogdan Dobrelya

Bug Description

Logs collection part is addressed in the spec https://review.openstack.org/#/c/462900

Logrotate configs covering added log paths for containers must come from either openstack packages, or a tripleo package.

Changed in tripleo:
milestone: none → pike-3
importance: Undecided → High
status: New → Triaged
tags: added: containers
Bogdan Dobrelya (bogdando) wrote :

Perhaps the logrotate scripts could come with the openstack-tripleo-common-containers(-base???) package, but sending SIGHUP to containerized openstack (and not only) services will be a tricky thing . Restarting containers could work as well, but looks too intrusive.

Jiří Stránský (jistr) wrote :

I wouldn't worry about SIGHUPs -- we should just make it so that the package is present on the node before starting any containers. That's doable both for fresh deployment and upgrade.

As for the package though, that might be more difficult. Tripleo-common is currently on undercloud only, we'd probably have to start installing it on overcloud too, which might be ok but would have to be double checked, as i think we had some issues with installing e.g. python-tripleoclient on overcloud, IIRC.

Jiří Stránský (jistr) wrote :

Oh you probably meant SIGHUPs from logrotate, not when installing the package, right? :)

Bogdan Dobrelya (bogdando) wrote :

Right. It must be done to prevent services from writing to open but unlinked files, consuming free space and ignoring rotated log files are now in place.

Bogdan Dobrelya (bogdando) wrote :

The plan is to ship a logrotate config and a SIGHUPper to clear unlinked & deleted files after it.
It may be done either with a postrotate script for logrotate, or with a cron job. The latter allows to introduce a time splay for sending SIGHUP to services, while the former can only SIGHUP them all at once. Something like an example /etc/logrotate.d/openstack-containers does the former:

/var/log/containers/*/*.log {
 rotate 14
 size 10M

  /sbin/lsof -nPs +L1 +D /var/log 2>&1|\
  /bin/perl -ne '/\S+\s+(\d+).*\/var\/log\/.*\(deleted\)/ && print "$1\n"' |\
  /bin/xargs -n1 -r -t kill -HUP

Bogdan Dobrelya (bogdando) wrote :

The tripleo-common package is not installed for hosts, but it is installed for all containers, and lsof will be installed as a dependency ( https://review.rdoproject.org/r/#/c/7878/ ). It seems that the best way to launch it is using the kolla cron container. It has both cron and logrotate, and can be configured as a uc/overcloud services to store the logrotate config like in the example above.

Additionally, it can be configured to run the aforementioned cleanup command instead of postrotate script, with a time splay.

Bogdan Dobrelya (bogdando) wrote :

The dependency is logrotate & cron containerization https://bugs.launchpad.net/tripleo/+bug/1706354

Changed in tripleo:
assignee: nobody → Bogdan Dobrelya (bogdando)
Changed in tripleo:
milestone: pike-3 → pike-rc1

Fix proposed to branch: master
Review: https://review.openstack.org/490048

Changed in tripleo:
status: Triaged → In Progress

Reviewed: https://review.openstack.org/490068
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=7776b19a7d90b360bb7078f1349f50e64286b456
Submitter: Jenkins
Branch: master

commit 7776b19a7d90b360bb7078f1349f50e64286b456
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Aug 2 18:07:08 2017 +0200

    Add logrotate-crond container images params

    Related-bug: #1700912

    Change-Id: I3fd918b01edd71e3e9e7c4efe7fb790ae2f37d19
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Reviewed: https://review.openstack.org/490048
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=70987c80c1212e0ad50c4a7600cbf1665f673cc2
Submitter: Jenkins
Branch: master

commit 70987c80c1212e0ad50c4a7600cbf1665f673cc2
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Aug 2 15:42:38 2017 +0200

    Add logrotate-crond configuration

    Generate a cron job and a config for logrotate
    to be run against containerized services logs.

    Related-bug: #1700912

    Change-Id: Ib9d5d8ca236296179182613e1ff625deea168614
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Reviewed: https://review.openstack.org/490050
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8a03456056b22a0abff470dfd81334e369ae4cbc
Submitter: Jenkins
Branch: master

commit 8a03456056b22a0abff470dfd81334e369ae4cbc
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Aug 2 14:48:26 2017 +0200

    Add logrotate with crond service

    Add a docker service template to provide containerized services
    logs rotation with a crond job.
    Add OS::TripleO::Services::LogrotateCrond to CI multinode-containers
    and to all environments among with generic services like Ntp or Kernel.
    Set it to OS::Heat::None for non containerized environments and
    only enable it to the environments/docker.yaml.

    Closes-bug: #1700912

    Change-Id: Ic94373f0a0758e9959e1f896481780674437147d
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Changed in tripleo:
status: In Progress → Fix Released

This issue was fixed in the openstack/tripleo-heat-templates release candidate.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers