Comment 8 for bug 1776533

Reviewed: https://review.openstack.org/575525
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=823540143f4074a1b7f8adef9626c25aad1cd9b3
Submitter: Zuul
Branch: stable/queens

commit 823540143f4074a1b7f8adef9626c25aad1cd9b3
Author: Michele Baldessari <email address hidden>
Date: Tue Jun 12 19:34:37 2018 +0200

    Fix logrotate_crond issues

    So currently the logrotate_crond container has a few issues issues:
    A) In the postrotate it matches pids multiple times and sends SIGHUPs multiple time to processes:
        ======== /var/log/messages =====
        Jun 3 09:01:15 overcloud-controller-0 logrotate-crond: kill -HUP 1575
        Jun 3 09:01:15 overcloud-controller-0 rsyslogd: [origin software="rsyslogd" swVersion="8.24.0" x-pid="1575" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
        Jun 3 09:01:15 overcloud-controller-0 logrotate-crond: kill -HUP 1575
        Jun 3 09:01:15 overcloud-controller-0 rsyslogd: [origin software="rsyslogd" swVersion="8.24.0" x-pid="1575" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
        Jun 3 09:01:15 overcloud-controller-0 logrotate-crond: kill -HUP 1575
        Jun 3 09:01:15 overcloud-controller-0 rsyslogd: [origin software="rsyslogd
    ...

    Adding sort -u in the pipeline of the postrotate script takes care of
    that.

    B) The logrotate_crond container should not rotate logs for services
    running on the host outside of containers (i.e. rsyslog has its own
    /etc/logrotate.d/rsyslog rules). Doing so violates the principle of
    least surprise.

    Using 'lsof ..+D /var/log/containers' takes care of this as we won't
    match any non containerized processes

    C) The find command matches older files to be deleted but the SIGHUP is
    never sent so we actually can end up in a situation where we remove a
    file but the new one never gets created because the service does not get
    a SIGHUP signal:

    ls -la /var/log/containers/httpd/*/*
    -rw-r--r--. 1 root root 52046652 May 29 14:10 /var/log/containers/httpd/aodh-api/aodh_wsgi_access.log.1
    -rw-r--r--. 1 root root 0 May 24 19:14 /var/log/containers/httpd/aodh-api/aodh_wsgi_error.log
    -rw-r--r--. 1 root root 5894 May 24 19:14 /var/log/containers/httpd/aodh-api/error_log
    -rw-r--r--. 1 root root 50755274 May 29 14:10 /var/log/containers/httpd/cinder-api/cinder_wsgi_access.log.1
    -rw-r--r--. 1 root root 4138 May 25 11:58 /var/log/containers/httpd/cinder-api/cinder_wsgi_error.log
    -rw-r--r--. 1 root root 5894 May 24 19:13 /var/log/containers/httpd/cinder-api/error_log

    Using 'lsof ..+D /var/log/containers' fixes this case as well because
    now we correctly match the processes that have a deleted file that is
    open and we send a proper SIGHUP to them.

    Tested by doing the following:
    1) Logging rotation of containerized services (B, C)
    1.1) Stopped the keystone container
    1.2) Made the /var/log/container/keystone/keystone.log file 21M large
    1.3) Started the keystone container and observed that it was logging
         correctly to /var/log/container/keystone/keystone.log
    1.4) Inside the logrotate_crond container we ran the following:
    /usr/sbin/logrotate -s /var/lib/logrotate/logrotate-crond.status /etc/logrotate-crond.conf
    1.5) We observed correct log rotation and keystone was notified via
    SIGHUP and started logging correctly:
    -rw-r--r--. 1 42425 42425 21628706 Jun 13 08:43 keystone.log.1
    -rw-r--r--. 1 42425 42425 999 Jun 13 08:43 keystone.log

    2) No SIGHUP to host processes (A)
    2.1) stopped rsyslog on the host and made one of its log files > 10M:
    -rw-r--r--. 1 root root 28M Jun 13 08:59 /var/log/messages
    2.2) restart rsyslog
    2.3) Ran the logrotation inside the container
    /usr/sbin/logrotate -s /var/lib/logrotate/logrotate-crond.status /etc/logrotate-crond.conf
    2.4) Observed that no SIGHUP was sent to rsyslog on the host

    Co-Authored-By: Damien Ciabrini <email address hidden>

    Change-Id: I5029a4b9c76268455812696290aaf82f1a0c2c23
    Closes-Bug: #1776533
    (cherry picked from commit 77d93f32873e740b0f04f323465ad14c044c896b)