Logstash logs are not rotated in Logstash 2.4

[Doug Szumski]

The Logstash 2.4 container has baked in config to rotate the Logstash logs:

(monasca-log-transformer)[logstash@sv-b16-u22 /var/log/kolla/logstash]$ cat /etc/logrotate.d/logstash
/var/log/logstash/*.log /var/log/logstash/*.err /var/log/logstash/*.stdout {
        daily
        rotate 7
        copytruncate
        compress
        delaycompress
        missingok
        notifempty
}

But since the log folder has been set to /var/log/kolla/logstash/ they are not rotated. This means that the logs can grow unbounded, potentially bringing down the host. Eg.

(monasca-log-transformer)[logstash@sv-1222 /var/log/kolla/logstash]$ ls -lah
total 412G
drwxr-sr-x 2 logstash kolla 4.0K Jun 21 2019 .
drwxrwsr-x 12 td-agent kolla 4.0K Oct 31 2019 ..
-rw-r--r-- 1 logstash kolla 94K Jun 22 12:30 monasca-log-metrics.log
-rw-r--r-- 1 logstash kolla 521K Jun 22 12:30 monasca-log-persister.log
-rw-r--r-- 1 logstash kolla 412G Jul 8 09:50 monasca-log-transformer.log

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/740946

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/740946
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=0e353d3f57c6f2cb45d993da8737b6e112dd1d45
Submitter: Zuul
Branch: stable/train

commit 0e353d3f57c6f2cb45d993da8737b6e112dd1d45
Author: Bartosz Bezak <email address hidden>
Date: Tue Jul 14 11:55:31 2020 +0000

    Fix Logstash 2.4 log rotation

    Logstash 2.4 uses integrated in container logrotate configuration
    which tries to rotate logs in /var/log/logstash and Kolla's Logstash
    logs are in /var/log/kolla/logstash/
    This patch adds kolla-ansible internal logrotate config for Logstash

    Change-Id: Ifbab7b673ba87925e52bb01c0996479eb1b76c44
    Closes-Bug: #1886787

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/741235

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/741250

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/741235
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=1aeed4343b7af37b7cdf932a0055e337227e1d8f
Submitter: Zuul
Branch: stable/rocky

commit 1aeed4343b7af37b7cdf932a0055e337227e1d8f
Author: Bartosz Bezak <email address hidden>
Date: Tue Jul 14 11:55:31 2020 +0000

    Fix Logstash 2.4 log rotation

    Logstash 2.4 uses integrated in container logrotate configuration
    which tries to rotate logs in /var/log/logstash and Kolla's Logstash
    logs are in /var/log/kolla/logstash/
    This patch adds kolla-ansible internal logrotate config for Logstash

    Change-Id: Ifbab7b673ba87925e52bb01c0996479eb1b76c44
    Closes-Bug: #1886787
    (cherry picked from commit 0e353d3f57c6f2cb45d993da8737b6e112dd1d45)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/741250
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=da1267e609f2724d94ef0a494a78a5591eb84d81
Submitter: Zuul
Branch: stable/stein

commit da1267e609f2724d94ef0a494a78a5591eb84d81
Author: Bartosz Bezak <email address hidden>
Date: Tue Jul 14 11:55:31 2020 +0000

    Fix Logstash 2.4 log rotation

    Logstash 2.4 uses integrated in container logrotate configuration
    which tries to rotate logs in /var/log/logstash and Kolla's Logstash
    logs are in /var/log/kolla/logstash/
    This patch adds kolla-ansible internal logrotate config for Logstash

    Change-Id: Ifbab7b673ba87925e52bb01c0996479eb1b76c44
    Closes-Bug: #1886787
    (cherry picked from commit 0e353d3f57c6f2cb45d993da8737b6e112dd1d45)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 8.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible rocky-eol release.