Allow logrotate configuration to control log retention based on age

[Juan Antonio Osorio Robles]

Our current logrotate configuration in puppet-tripleo handles rotation based off of log size and number of rotated logs (rotate at 10M size and retain 14 logs). For things such as GDPR, we need to be able to define the retention based on age as the logs may contain personally identifiable information that needs to be deleted on demand (within a window of time). Ideally, we would only retain logs for <1M (2 weeks might be appropriate), with the option to configure the policy differently via Director.

[Bogdan Dobrelya (bogdando)]

I think the goal is to delete all logs containing any info older than X days. Given that Linux will not show you a file creation time for the most of the cases, we only have a few options (and all seem have to belong to the logrotate's postrotate scripts only, as logrotate does not provide a clear interface for the size/maxsize/minsize vs time based rotation):

- Delete based on files' head -1 content based, trying to figure out expiration from timestamps (Given binary log files and numerous timestamp formats, is prolly bad idea, but gives the best approximation for the real "creation time" criterion)
- Delete based on the older of available files' attributes: -atime +X / -crime +X / -mtime +X (does not really purges old files created long time ago and accessed recently, which potentially leaves some logged user sensitive data in undetected files)
- Delete based on the older of available directories' mtime attribute (corresponds to the last file creation or deletion that happened) - not sure if that works better.

[Bogdan Dobrelya (bogdando)]

Let's split it into separate things. There is logrotate-crond container, which takes care of only log files living in the host's /var/log/containers. And there is journald/rsyslog on the host, and other logfiles, like pacemaker/ovs et al, sitting in /var/log. For the latter, containerized services' logs in /var/log/containers, I propose remove all files but compressed logs rotated in the postrotate step. And logrotate will take care of its compressed artifacts on its own. This is the only w/a to the creation time attribute not available for operators.

For the remaining steps, let's think of another solutions.

[Bogdan Dobrelya (bogdando)]

Note, I mean not the host logrotate, but logrotate-crond container in fact

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/569139

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/569154

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/569805

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/569806

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/569806
Reason: temporary abandoning it to restore once the master patch merged first

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/569805
Reason: temporary abandoning it to restore once the master patch merged first

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/569139
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=e13654504adb93f7313942b4d839577d3ccd072d
Submitter: Zuul
Branch: master

commit e13654504adb93f7313942b4d839577d3ccd072d
Author: Bogdan Dobrelya <email address hidden>
Date: Thu May 17 12:53:03 2018 +0200

    Force GDPR compliance of containers logs

    After purge_after_days, defaults to a 14, forcibly remove
    any rotated and compressed logs of containerized services
    in /var/log/containers. This overrides any related
    containerized logrotate configuration used for
    containerized services.

    Allow to alter rotation interval for log files managed
    via containerized logrotate. Defaults to 'daily'
    and rotate 14 (days).

    Use sharedscripts to clean up files in the postrotate
    script only once.

    Additionally, to enforce GDPR compliance of log files
    in /var/log/containers, put them under logrotate management
    (minsize 1) and always compress. Prohibit the size option
    as it does not honor time-based contstraints required by
    GDPR. Forcibly remove all files but those rotated and
    compressed logs, via the postscript section.

    Partial-bug: #1771543

    Change-Id: Id8e4717a5ecda53bc9cd39f1c2efaa80b56bd45e
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/569154
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=62cdc3949f733b726fc1e25708b755e1a21dd9f7
Submitter: Zuul
Branch: master

commit 62cdc3949f733b726fc1e25708b755e1a21dd9f7
Author: Bogdan Dobrelya <email address hidden>
Date: Thu May 17 17:16:29 2018 +0200

    Allow custom time constraints to rotate logs

    Time based constraints must prevail over size et al
    constraints applied for logs of containerized services.
    Time based constraints are needed in order to abide
    GDPR requirements.

    Depending on FS type, Linux may not allow system operators
    to access files creation time attribute (see the Birth
    attribute shown as '-' by the stat -c %w command).
    This is work-arouned by alternative strict time-based rules
    defined in the postrotate script, which purges files in
    /var/log/containers and ensures GDPR compliant system
    configuration, limited to the containerized services logs.

    Extended management of life time of journald and /var/log
    files residing on bare metal hosts, should be done in
    follow up patches.

    Partial-bug: #1771543

    Change-Id: I6f2f98aba103f83a4f64a435077f4de33f9692c0
    Depends-On: Id8e4717a5ecda53bc9cd39f1c2efaa80b56bd45e
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/569938

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: master
Review: https://review.openstack.org/569938

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/569976

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/570328

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/569806
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=f814414eda89c69c61cd48449326519be8a6bf16
Submitter: Zuul
Branch: stable/queens

commit f814414eda89c69c61cd48449326519be8a6bf16
Author: Bogdan Dobrelya <email address hidden>
Date: Thu May 17 12:53:03 2018 +0200

    Force GDPR compliance of containers logs

    After purge_after_days, defaults to a 14, forcibly remove
    any rotated and compressed logs of containerized services
    in /var/log/containers. This overrides any related
    containerized logrotate configuration used for
    containerized services.

    Allow to alter rotation interval for log files managed
    via containerized logrotate. Defaults to 'daily'
    and rotate 14 (days).

    Use sharedscripts to clean up files in the postrotate
    script only once.

    Additionally, to enforce GDPR compliance of log files
    in /var/log/containers, put them under logrotate management
    (minsize 1) and always compress. Prohibit the size option
    as it does not honor time-based contstraints required by
    GDPR. Forcibly remove all files but those rotated and
    compressed logs, via the postscript section.

    Partial-bug: #1771543

    Change-Id: Id8e4717a5ecda53bc9cd39f1c2efaa80b56bd45e
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit e13654504adb93f7313942b4d839577d3ccd072d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/569805
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c77ab579c4c0230bbeae7ca8d8b50348dbb22bd5
Submitter: Zuul
Branch: stable/queens

commit c77ab579c4c0230bbeae7ca8d8b50348dbb22bd5
Author: Bogdan Dobrelya <email address hidden>
Date: Thu May 17 17:16:29 2018 +0200

    Allow custom time constraints to rotate logs

    Time based constraints must prevail over size et al
    constraints applied for logs of containerized services.
    Time based constraints are needed in order to abide
    GDPR requirements.

    Depending on FS type, Linux may not allow system operators
    to access files creation time attribute (see the Birth
    attribute shown as '-' by the stat -c %w command).
    This is work-arouned by alternative strict time-based rules
    defined in the postrotate script, which purges files in
    /var/log/containers and ensures GDPR compliant system
    configuration, limited to the containerized services logs.

    Extended management of life time of journald and /var/log
    files residing on bare metal hosts, should be done in
    follow up patches.

    Partial-bug: #1771543

    Change-Id: I6f2f98aba103f83a4f64a435077f4de33f9692c0
    Depends-On: Id8e4717a5ecda53bc9cd39f1c2efaa80b56bd45e
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 62cdc3949f733b726fc1e25708b755e1a21dd9f7)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/570328
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=6de93743e7b85deaeaef075fa26209ced0e55ad1
Submitter: Zuul
Branch: stable/queens

commit 6de93743e7b85deaeaef075fa26209ced0e55ad1
Author: Bogdan Dobrelya <email address hidden>
Date: Tue May 22 14:53:27 2018 +0200

    Rework GDPR compliant logrotate config

    Set the logrotate maxage parameter to purge_after_days
    as well.

    Rework additional retention rules of files in
    /var/log/containers in the containerized logrotate
    postrotate script. The rules are based on any of the
    listed criteria met:

    * time of last access of contents (atime) exceeds
    purge_after_days,
    * time of last modification of contents (mtime) exceeds
    purge_after_days,
    * time of last modification of the inode (metadata, ctime)
    exceeds purge_after_days.

    Forcibly purge expired files with each containerized
    logrotate run triggered via cron. Note that the files creation
    time (the Birth attribute) is not taken into account as it
    cannot be accessed normally by system operators (depends on FS
    type). Retention policies based on the creation time must
    be managed elsewhere.

    Related-Bug: #1771543

    Change-Id: I9afa22f7dd344a29747206b286520a76d70d704b
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit d032e6d5fc89246865714909510e199d61e08fb0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/569976
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=5a58ca5d325ae3363cc3eb9d20d96910f9406691
Submitter: Zuul
Branch: master

commit 5a58ca5d325ae3363cc3eb9d20d96910f9406691
Author: Bogdan Dobrelya <email address hidden>
Date: Tue May 22 14:53:27 2018 +0200

    Rework GDPR compliant logrotate config

    Set the logrotate maxage parameter to purge_after_days
    as well.

    Rework additional retention rules of files in
    /var/log/containers in the containerized logrotate
    postrotate script. The rules are based on any of the
    listed criteria met:

    * time of last access of contents (atime) exceeds
    purge_after_days,
    * time of last modification of contents (mtime) exceeds
    purge_after_days,
    * time of last modification of the inode (metadata, ctime)
    exceeds purge_after_days.

    Forcibly purge expired files with each containerized
    logrotate run triggered via cron. Note that the files creation
    time (the Birth attribute) is not taken into account as it
    cannot be accessed normally by system operators (depends on FS
    type). Retention policies based on the creation time must
    be managed elsewhere.

    Related-Bug: #1771543

    Change-Id: I9afa22f7dd344a29747206b286520a76d70d704b
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[Bogdan Dobrelya (bogdando)]

@Juan could you please take it over for the remaining host-level logrotation fixes? The patches above only address that for the containerized services running undercloud and overcloud (and standalone). Feel free to reopen.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/601554

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/601554
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=13b3166e420a58be92c14d30910cbc8650053f41
Submitter: Zuul
Branch: stable/pike

commit 13b3166e420a58be92c14d30910cbc8650053f41
Author: Bogdan Dobrelya <email address hidden>
Date: Thu May 17 12:53:03 2018 +0200

    Force GDPR compliance of containers logs

    After purge_after_days, defaults to a 14, forcibly remove
    any rotated and compressed logs of containerized services
    in /var/log/containers. This overrides any related
    containerized logrotate configuration used for
    containerized services.

    Allow to alter rotation interval for log files managed
    via containerized logrotate. Defaults to 'daily'
    and rotate 14 (days).

    Use sharedscripts to clean up files in the postrotate
    script only once.

    Additionally, to enforce GDPR compliance of log files
    in /var/log/containers, put them under logrotate management
    (minsize 1) and always compress. Prohibit the size option
    as it does not honor time-based contstraints required by
    GDPR. Forcibly remove all files but those rotated and
    compressed logs, via the postscript section.

    Partial-bug: #1771543

    Change-Id: Id8e4717a5ecda53bc9cd39f1c2efaa80b56bd45e
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit e13654504adb93f7313942b4d839577d3ccd072d)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/606907

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/606907
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=0cbcf8dfbefd957d623c3dff5c5daca09fd96a27
Submitter: Zuul
Branch: stable/pike

commit 0cbcf8dfbefd957d623c3dff5c5daca09fd96a27
Author: Bogdan Dobrelya <email address hidden>
Date: Tue May 22 14:53:27 2018 +0200

    Rework GDPR compliant logrotate config

    Set the logrotate maxage parameter to purge_after_days
    as well.

    Rework additional retention rules of files in
    /var/log/containers in the containerized logrotate
    postrotate script. The rules are based on any of the
    listed criteria met:

    * time of last access of contents (atime) exceeds
    purge_after_days,
    * time of last modification of contents (mtime) exceeds
    purge_after_days,
    * time of last modification of the inode (metadata, ctime)
    exceeds purge_after_days.

    Forcibly purge expired files with each containerized
    logrotate run triggered via cron. Note that the files creation
    time (the Birth attribute) is not taken into account as it
    cannot be accessed normally by system operators (depends on FS
    type). Retention policies based on the creation time must
    be managed elsewhere.

    Related-Bug: #1771543

    Change-Id: I9afa22f7dd344a29747206b286520a76d70d704b
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit d032e6d5fc89246865714909510e199d61e08fb0)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/618550

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/618550
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=7781a3677160e56fe20b51ee37760442869786b7
Submitter: Zuul
Branch: stable/pike

commit 7781a3677160e56fe20b51ee37760442869786b7
Author: Bogdan Dobrelya <email address hidden>
Date: Thu May 17 17:16:29 2018 +0200

    Allow custom time constraints to rotate logs

    Time based constraints must prevail over size et al
    constraints applied for logs of containerized services.
    Time based constraints are needed in order to abide
    GDPR requirements.

    Depending on FS type, Linux may not allow system operators
    to access files creation time attribute (see the Birth
    attribute shown as '-' by the stat -c %w command).
    This is work-arouned by alternative strict time-based rules
    defined in the postrotate script, which purges files in
    /var/log/containers and ensures GDPR compliant system
    configuration, limited to the containerized services logs.

    Extended management of life time of journald and /var/log
    files residing on bare metal hosts, should be done in
    follow up patches.

    Partial-bug: #1771543

    Conflicts:
     docker/services/logrotate-crond.yaml
    Change-Id: I6f2f98aba103f83a4f64a435077f4de33f9692c0
    Depends-On: Id8e4717a5ecda53bc9cd39f1c2efaa80b56bd45e
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 62cdc3949f733b726fc1e25708b755e1a21dd9f7)