agent logging override does not work

[Peter Matulis]

I'm running 2.4-beta1-xenial-amd64 (--edge channel).

juju deploy mysql
juju ssh mysql/0
sudo vi /var/lib/juju/agents/unit-mysql-0/agent.conf

where the bottom of the file looks like:

============================
loggingconfig: <root>=WARNING;unit=DEBUG
values:
  CONTAINER_TYPE: ""
  NAMESPACE: ""
  LOGGING_OVERRIDE=juju=trace
mongoversion: "0.0"
============================

sudo systemctl restart jujud-unit-mysql-0.service

Back on the client,

juju deploy mediawiki
juju deploy wordpress
juju debug-log

Yet I'm getting lines that are prefixed with:

unit-mysql-0: 16:52:30 DEBUG
unit-mediawiki-0: 17:08:48 DEBUG
unit-wordpress-0: 17:10:09 DEBUG

Should I not just be getting mysql lines?

Secondary question: What is supposed to happen if I use LOGGING_OVERRIDE on multiple agents? Will it log all those agents?

[Ian Booth (wallyworld)]

The logging config has unit=DEBUG to I would expect to see debug logging for any unit in the debug-log output.

All logs from each agent are sent to the controller and stored, and this is what is output when using debug-log.

[Peter Matulis (petermatulis)]

So how is agent logging override configured then? I'm going by PR 7525 [1] where it says:

"edit the agent.conf file to specify 'LOGGING_OVERRIDE=juju=debug' in the values section."

It also says "agent can log at specified severity levels". So I chose the TRACE level.

[1]: https://github.com/juju/juju/pull/7525

[Ian Booth (wallyworld)]

Logging override provides a way to set logging levels for a particular agent different to the model itself. By default, unit agents log at debug level globally. To use your example, you would use the logging override to configure a specific agent (and only that agent) to log at trace level while the others remain at debug level. Regardless, all logs are collated in mongo for streaming out when debug-log is used. I think they are also stored locally in /var/log/juju on the machine running the agent.

[Peter Matulis (petermatulis)]

If all the agents log at DEBUG level by default then this override feature merely allows an agent to log at one level higher (TRACE), as reducing the level on a per-agent basis is an oddity.

The first sentence of PR 7525 certainly does not give this impression though:

"If there are many units in a model, it is not always desirable to enable debug mode for the entire model while attempting to resolve issues or work out why things aren't working."

That strongly suggests that the model default of DEBUG will *not* be used if this override config is applied, and only the specified level for a given agent will be used.

[John A Meinel (jameinel)]

Default logging is set globally with "logging-config" but the normal
default logging is "<root>=WARNING;juju.unit=DEBUG". You have both unit
agents and machine agents. All of the machine agent logging would be at
WARNING level.

Probably saying "unit" in the original PR was a bit of the confusion.
Though I will say that often TRACE is *much* more verbose than DEBUG, I
think the common things that we play with DEBUG settings tend to be more
around machine agents that might be failing to start, etc. In which case
having one machine agent logging at DEBUG vs all of them can certainly be
useful.

On Wed, Apr 11, 2018 at 5:48 PM, Peter Matulis <email address hidden>
wrote:

> If all the agents log at DEBUG level by default then this override
> feature merely allows an agent to log at one level higher (TRACE), as
> reducing the level on a per-agent basis is an oddity.
>
> The first sentence of PR 7525 certainly does not give this impression
> though:
>
> "If there are many units in a model, it is not always desirable to
> enable debug mode for the entire model while attempting to resolve
> issues or work out why things aren't working."
>
> That strongly suggests that the model default of DEBUG will *not* be
> used if this override config is applied, and only the specified level
> for a given agent will be used.
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1762489
>
> Title:
> agent logging override does not work
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1762489/+subscriptions
>

[Peter Matulis (petermatulis)]

What would happen if one edits the value of 'loggingconfig' on a per-agent basis (agent.conf)?

[Tim Penhey (thumper)]

Peter, the purpose of the logging override is primarily to aid in debugging production issues on large deployments.

The model-wide logging configuration is set using the model config 'logging-config'. There is a worker on every agent that makes sure that the agent respects the model configuration.

If there is an override set in the agent.conf file, the model config is ignored and what is set in the agent.conf file is used.

It is designed to be used during the debugging process on-site, and not for general use.

The config can be set on as many agents as you like.

[Peter Matulis (petermatulis)]

@Tim Alrighty. I'll invalidate.