juju debug-log has line limit

[Carl Csaposs]

We're using juju debug-log to collect logs in our CI and noticed that logs are missing if there are too many logs

Increasing `agent-logfile-max-size` didn't help—it appears to be a limit specific to `juju debug-log`. We tried `--lines 18446744073709551615` and that also didn't help

This is on a localhost lxd cloud and the logs do exist in controller/0 /var/log/juju/logsink.log

Example CI run: https://github.com/canonical/opensearch-operator/actions/runs/8613054419/job/23603884248#step:25:5

Attached is a zip archive that includes `juju debug-log` output and logsink.log from the example CI run

Versions:
juju agent: 3.3.3

snaps:
Name Version Rev Tracking Publisher Notes
core20 20240111 2182 latest/stable canonical** base
core22 20240111 1122 latest/stable canonical** base
juju 3.3.3 26652 3.3/stable canonical** -
lxd 5.20-f3dd836 27049 latest/stable canonical** -
snapd 2.61.2 21184 latest/stable canonical** snapd

We're currently working around this by manually parsing logsink.log and using a regex to generate an output similar to juju debug-log: https://github.com/canonical/data-platform-workflows/blob/6208f2fcad102b4619af00659715a3b4c22aaa57/python/cli/data_platform_workflows_cli/convert_logsink_to_debug_log.py However, this only solves the issue for VM charms—not k8s charms

Originally reported on Matrix: https://matrix.to/#/!xzmWHtGpPfVCXKivIh:ubuntu.com/$DDSALIpreHYYoESReJ8noF739UHdHvulbixCJCDpQA4?via=ubuntu.com&via=matrix.org&via=mx.aouss.it

[Ian Booth (wallyworld)]

debug-log is "best effort" and is sourced from a capped collection in mongo. This is separate to any log files written to disk with the associated max file size setting.

The config you might want to try changing is this

// ModelLogsSize is the size of the capped collections used to hold the
// logs for the models, eg "20M". Size is per model.
ModelLogsSize = "model-logs-size"

I would be wary about setting that too high though.

[Joseph Phillips (manadart)]

Marked as incomplete pending feedback on suggested work-around.

[Carl Csaposs (carlcsaposs)]

Increasing `model-logs-size` seems to work

> I would be wary about setting that too high though.

Will this have any side effects other than increased disk usage?

Wondering if setting `model-logs-size` to `20G` (default is `20M`) in a CI environment (where we have more verbose logging than a production environment) will cause any issues

[John A Meinel (jameinel)]

Various invocations of `juju debug-log` such as `juju debug-log --replay` are going to have to iterate through all of that data. Things like the automatic pruning to keep the log table at reasonable sizes are also going to be impacted by larger collections.

IIRC, `model-logs-size` is per-model. So setting it to 20G means that you will have 20G * N models (eg in Prodstack that will be 20*300 = 1.2TB.)

But for CI that shouldn't be inherently problematic.

Note that you can already set a syslog forwarder and put it into a system for actually handling large amounts of logs, rather than just `juju debug-log`.