logrotate 3.14.0 does not rotate logs for broken symlinks in path

[Rabi Mishra]

Description of problem: Logs files of services such as nova, neutron keep getting bigger up to a few 100MBs but they are not being compressed and rotated.

This seems to be due to the swift broken symlink.

[root@overcloud-controller-0 containers]# cd neutron/
[root@overcloud-controller-0 neutron]# ls -tarlh
total 490M
drwxr-xr-x. 16 root root 221 Jun 12 14:04 ..
drwxr-xr-x. 2 42435 42435 24 Jun 12 14:04 .
-rw-r--r--. 1 42435 42435 234M Jun 12 14:09 server.log

[root@overcloud-controller-0 nova]# ls -tarlh
total 610M
-rw-------. 1 42436 42436 90K May 31 18:08 nova-manage.log
-rw-r--r--. 1 42436 42436 3.0K May 31 18:12 nova-conductor.log
-rw-r--r--. 1 42436 42436 1.9K May 31 18:12 nova-consoleauth.log
-rw-r--r--. 1 42436 42436 622 May 31 18:12 nova-novncproxy.log
-rw-r--r--. 1 42436 42436 8.5K May 31 18:12 nova-metadata-api.log
-rw-r--r--. 1 42436 42436 8.8K May 31 18:14 nova-scheduler.log
drwxr-xr-x. 2 42436 42436 241 Jun 1 00:26 .
-rw-r--r--. 1 42436 42436 0 Jun 1 00:26 nova-rowsflush.log
drwxr-xr-x. 16 root root 221 Jun 12 14:04 ..
-rw-r--r--. 1 42436 42436 8.5M Jun 12 14:09 nova-placement-api.log
-rw-r--r--. 1 42436 42436 346M Jun 12 14:09 nova-api.log

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/666808

[Rabi Mishra (rabi)]

So all swift logs are writtn to /var/log/swift/swift.log as configured in /etc/rsyslog.d/openstack-swift.conf and never rotated since we containerized logrotate as it has a borken link. strange no one complained about it yet:/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/666808
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d3a9614d8a631c1d29683d6426f501b88ecc2f85
Submitter: Zuul
Branch: master

commit d3a9614d8a631c1d29683d6426f501b88ecc2f85
Author: Rabi Mishra <email address hidden>
Date: Fri Jun 21 15:20:29 2019 +0530

    Don't create symlink for swift container logs

    logrotate 3.14.0 silently ignores all files when
    there is a broken symlink in the container.

    Change-Id: I52a74c7b5473340e2afd240d7d6af9ad5bc97544
    Closes-Bug: #1833690

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/666898

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/667005

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/666898
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1f6a24f6ade464ccd2d205f23560595b5e173739
Submitter: Zuul
Branch: stable/stein

commit 1f6a24f6ade464ccd2d205f23560595b5e173739
Author: Rabi Mishra <email address hidden>
Date: Fri Jun 21 15:20:29 2019 +0530

    Don't create symlink for swift container logs

    logrotate 3.14.0 silently ignores all files when
    there is a broken symlink in the container.

    Change-Id: I52a74c7b5473340e2afd240d7d6af9ad5bc97544
    Closes-Bug: #1833690
    (cherry picked from commit d3a9614d8a631c1d29683d6426f501b88ecc2f85)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/667005
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=90d05216b96adaf93dcf36680954328ed1cb053d
Submitter: Zuul
Branch: master

commit 90d05216b96adaf93dcf36680954328ed1cb053d
Author: Rabi Mishra <email address hidden>
Date: Mon Jun 24 08:47:36 2019 +0530

    Replace /var/log/containers/swift symlink with directories

    Once we remove the symlinks in update_staks, host_prep_tasks
    would create those directories.

    Assuming that minor update runs before upgrades, this should
    take care of upgrades too.

    Change-Id: Idabfec5f568a4e02900d103d23f4cb3c199e84c9
    Related-Bug: #1833690

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/667501

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/667526

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/667501
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=6f2877d986fd329777a1fb1bd8c0546cf23237cb
Submitter: Zuul
Branch: stable/stein

commit 6f2877d986fd329777a1fb1bd8c0546cf23237cb
Author: Rabi Mishra <email address hidden>
Date: Mon Jun 24 08:47:36 2019 +0530

    Replace /var/log/containers/swift symlink with directories

    Once we remove the symlinks in update_staks, host_prep_tasks
    would create those directories.

    Assuming that minor update runs before upgrades, this should
    take care of upgrades too.

    Change-Id: Idabfec5f568a4e02900d103d23f4cb3c199e84c9
    Related-Bug: #1833690
    (cherry picked from commit 90d05216b96adaf93dcf36680954328ed1cb053d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/667526
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b1486301f2c36838774fc5554637d93de9ab78e6
Submitter: Zuul
Branch: stable/rocky

commit b1486301f2c36838774fc5554637d93de9ab78e6
Author: Rabi Mishra <email address hidden>
Date: Fri Jun 21 15:20:29 2019 +0530

    Don't create symlink for swift container logs

    logrotate 3.14.0 silently ignores all files when
    there is a broken symlink in the container.

    Replaces /var/log/containers/swift symlink with directories

    Once we remove the symlinks in update_staks, host_prep_tasks
    would create those directories.

    Assuming that minor update runs before upgrades, this should
    take care of upgrades too.

    Conflicts:
     docker/services/swift-proxy.yaml
     docker/services/swift-storage.yaml

    Change-Id: I52a74c7b5473340e2afd240d7d6af9ad5bc97544
    Closes-Bug: #1833690
    (cherry picked from commit d3a9614d8a631c1d29683d6426f501b88ecc2f85
     and 90d05216b96adaf93dcf36680954328ed1cb053d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.6.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.4.1 release.