Logging inconsistency in 3.5

[Nobuto Murata]

maas: 3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1

Previously the doc explained cases for snap and deb respectively, and now there is a consolidated section for 3.5.
https://maas.io/docs/reference-release-notes-maas-3-5#logs-collapsed-into-system-log-files
https://maas.io/docs/about-maas-logging#logging-updates-in-maas-35

Issue 1: doc

> In 3.5, the MAAS snap uses the pebble service manager instead of
> supervisord. This means that the systemd component was renamed to
> snap.maas.pebble.service. It contains Pebble logs, as well as
> intercepted stdout of the services running under Pebble. The region and
> rack logs are cached there, as well (i.e., regiond.log and rackd.log
> are no more (supervisord was redirecting the stdout of the respective
> services).

I don't think this is applicable to deb.

Issue 2: empty log files

Log files are still created but empty. That confuses users.

$ dpkg -S /var/log/maas/
maas-common, maas-rack-controller, maas-proxy: /var/log/maas

$ wc -l /var/log/maas/regiond.log
0 /var/log/maas/regiond.log

Issue 3: log files are necessary for compliance and governance reasons

Mainly managing logs in journald is fine. However, collecting logs and forwarding those to the centralized management system is one of the important part of compliance and governance requirements in enterprise.

Not having anything under /var/log/ is problematic since most of the agents have rules like /var/log/* or /var/log/**/*log

e.g. our grafana-agent charm with a generic machine relation
https://github.com/canonical/grafana-agent-operator/blob/6eb1938895f5c31704917832d7d84ca9afaed799/src/charm.py#L429-L438

And Ubuntu itself still have rsyslogd in ubuntu-minimal to save logs as files by default. I expect MAAS saves logs to files in addition to journald.

[Jacopo Rota (r00ta)]

Hi Nobuto,

thank you for the feedback.

I'm marking this as low prio as the bug here is that we are still creating some logs folder.

Regarding to your feedback about logging in general it is more a feature request rather than a bug. I'm marking this with `bug-council` so that we can discuss this next thursday (and create a jira ticket in case). In case we will take actions on this we will create a ticket in our product feedback board to keep track of it and we will post the link here as well

[Jerzy Husakowski (jhusakowski)]

We'll take a look at the docs and fix inconsistencies and empty log files.

Regarding issue #3 - the grafana agent charm does collect journald by default and does not require files to be present in /var/log tree - see https://github.com/canonical/grafana-agent-operator/blob/6eb1938895f5c31704917832d7d84ca9afaed799/src/charm.py#L441-L449 If that does not work, please open a separate issue for it. Since the logs are collected, and according to the charm authors there's no hard requirement to output files to /var/log tree if the logs are already output to journald, we will not add log file output for now.