/var/lib/puppet/reports filling up the root filesystem

[Adriano Oliveira]

Brief Description
-----------------
The /var/lib/puppet/reports locations is reaching up to 5.8G of usage with files dating back to the installation of the system, increasing the root partition usage.
systemd-tmpfiles-clean.service is in failed state.

Severity
--------
Major

Steps to Reproduce
------------------
Leave the system running for several days and check that /var/lib/puppet/reports usage keep increasing and is not cleaned up.

Expected Behavior
------------------
The files are removed without requiring to clean them up manually.

Actual Behavior
------------------
/var/lib/puppet/reports is filling up the / filesystem and needs manual cleaning to free up the space.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Distributed Cloud (Issue was seen on subcloud but is a potential problem in any configuration)

Timestamp/Logs
--------------
##fm-event.log##

2021-11-22T20:29:55.000 controller-0 fmManager: info { "event_log_id" : "100.104", "reason_text" : "File System threshold exceeded ; threshold 90.00%, actual 99.92%", "entity_instance_id" : "region=schztxce-d697149-002.system=schztxce-d697149-002.host=controller-0.filesystem=/", "severity" : "critical", "state" : "set", "timestamp" : "2021-11-22 20:29:55.842715" }

Test Activity
-------------
Feature testing

Workaround
----------
Manual cleanup of /var/lib/puppet/reports

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config-files/+/825139

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config-files/+/825139
Committed: https://opendev.org/starlingx/config-files/commit/10957a714d697a7a192579e5321410a1d566ff93
Submitter: "Zuul (22348)"
Branch: master

commit 10957a714d697a7a192579e5321410a1d566ff93
Author: Adriano Oliveira <email address hidden>
Date: Tue Jan 18 11:37:01 2022 -0500

    Add automatic cleanup for puppet reports files

    The systemd-tmpfiles-clean.service is responsible for automatic
    cleanup of old files, however, this service was failed upon
    installation and configuration of the platform.
    The error was related to the kubernetes.conf file to have
    a reference to kube user, an override conf was added to
    systemd config files to reference root user.
    Also, the location /var/lib/puppet/reports was added to the
    systemd conf file.

    Closes-Bug: 1958269

    Test Plan:

    PASS: Verify systemd-tmpfiles-clean.service is active/loaded
    PASS: Verify /var/lib/puppet/reports is being cleaned automatically
    PASS: Verify cleanup on both AIO and Standard configurations
    Signed-off-by: Adriano Oliveira <email address hidden>
    Change-Id: I0d31b778c4c626a6aaf13dd3814607be38ad21ee