Standard way of configuring logging

[Jonathan Lange]

It's useful to be able to set:
    - log file location
    - severity level cutoff
    - rotation policy
      - how often
      - how are files renamed
      - how many old files are kept

in a configuration file that can be easily edited by sysadmins, rather than in Python code that constructs and assembles logging.Handler objects.

[Barry Warsaw (barry)]

Jono, let me understand better what you're thinking about. I have some similar code in Mailman, but I'm trying to think about how to generalize this. In my schema.conf file I have the following:

{{{
[logging.template]
# This defines various log settings. The options available are:
#
# - level -- Overrides the default level; this may be any of the
# standard Python logging levels, case insensitive.
# - format -- Overrides the default format string
# - datefmt -- Overrides the default date format string
# - path -- Overrides the default logger path. This may be a relative
# path name, in which case it is relative to Mailman's LOG_DIR,
# or it may be an absolute path name. You cannot change the
# handler class that will be used.
# - propagate -- Boolean specifying whether to propagate log message from this
# logger to the root "mailman" logger. You cannot override
# settings for the root logger.
#
# In this section, you can define defaults for all loggers, which will be
# prefixed by 'mailman.'. Use subsections to override settings for specific
# loggers. The names of the available loggers are:
#
# - archiver -- All archiver output
# - bounce -- All bounce processing logs go here
# - config -- Configuration issues
# - debug -- Only used for development
# - error -- All exceptions go to this log
# - fromusenet -- Information related to the Usenet to Mailman gateway
# - http -- Internal wsgi-based web interface
# - locks -- Lock state changes
# - mischief -- Various types of hostile activity
# - post -- Information about messages posted to mailing lists
# - qrunner -- qrunner start/stops
# - smtp -- Successful SMTP activity
# - smtp-failure -- Unsuccessful SMTP activity
# - subscribe -- Information about leaves/joins
# - vette -- Information related to admindb activity
format: %(asctime)s (%(process)d) %(message)s
datefmt: %b %d %H:%M:%S %Y
propagate: no
level: info
path: mailman

[logging.root]

[logging.archiver]

[logging.bounce]
path: bounce

[logging.config]

[logging.debug]
path: debug
level: debug

[logging.error]

[logging.fromusenet]

[logging.http]

[logging.locks]

[logging.mischief]

[logging.qrunner]

[logging.smtp]
path: smtp

[logging.subscribe]

[logging.vette]
}}}

then in my logger initialize code, I've got something like this:

{{{
    # First, find the root logger and configure the logging subsystem.
    # Initialize the root logger, then create a formatter for all the sublogs.
    logging.basicConfig(format=config.logging.root.format,
                        datefmt=config.logging.root.datefmt,
                        level=as_log_level(config.logging.root.level))
    # Create the subloggers
    for logger_config in config.logger_configs:
        logger_name = 'mailman.' + logger_config.name.split('.')[-1]
        log = logging.getLogger(logger_name)
        # Get settings from log configuration file (or defaults).
        log_format = logger_config.format
        log_dat...

[Jonathan Lange (jml)]

Looks good.

The schema you provide doesn't have any way of specifying things like log rotation though. I think it would be good to be able to configure those.

I guess http://docs.python.org/library/logging.html#configuration is also relevant here.

[Jonathan Lange (jml)]

I now no longer care about configuring log rotation. I have been enlightened.

[Curtis Hovey (sinzui)]

I think we can claim this is fix released since the crucial issue here was loglevels, nothing special is needed for log locations, rename, or rotation.

[Jonathan Lange (jml)]

Sure.

[Curtis Hovey (sinzui)]

Log file location is already supported as a string. Log rotation is commonly done by other tools; it is not the concern of lazr.config. The log level was implemented by Barry, so I think the work that should be done is complete.

[Jonathan Lange (jml)]

We still need a log hander that responds to SIGUSR1 and rotates the log. That should be a separate bug though.

[Curtis Hovey (sinzui)]

On Tue, 2009-02-03 at 21:42 +0000, Jonathan Lange wrote:
> We still need a log hander that responds to SIGUSR1 and rotates the log.
> That should be a separate bug though.

I don't understand what this has to do with configuration. Are you
requesting that lazr.config define the SIGs as enums for the application
to use? That may be useful. I don't know of any apps that configure
actions to SIGs.

--

__C U R T I S C. H O V E Y_______
Guilty of stealing everything I am.

[Jonathan Lange (jml)]

On Wed, Feb 4, 2009 at 9:01 AM, Curtis Hovey <email address hidden> wrote:
> On Tue, 2009-02-03 at 21:42 +0000, Jonathan Lange wrote:
>> We still need a log hander that responds to SIGUSR1 and rotates the log.
>> That should be a separate bug though.
>
> I don't understand what this has to do with configuration. Are you
> requesting that lazr.config define the SIGs as enums for the application
> to use? That may be useful. I don't know of any apps that configure
> actions to SIGs.
>

It's not part of config. It's part of doing logging right.

Handling SIGHUP (not SIGUSR1, my bad) is the Right Way to do log rotation.

jml