oslo.log

missing SYSLOG_FACILITY using journal fills up kern.log

Bug #1871840 reported by Benjamin Reichel
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.log
Fix Released
Undecided
Benjamin Reichel

Bug Description

We use journal logging for keystone and found all the logs in kern.log which led to rather big file because it is not rotated so often. I could also find out that the log messages coming from OpenStack journal logs lack the SYSLOG_FACILITY:

Tue 2020-04-07 11:28:54.307881 UTC [s=7d06979fce01459ea38f7c77ff0d0441;i=bdc534;b=20dba9fb32dc447d8841660817aa35fb;m=79fb17f102;t=5a2b1b04d1ddd;x=90f590abec5f37f5]
    CODE_FILE=/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py
    CODE_LINE=220
    CODE_FUNC=__call__
    PROCESS_NAME=MainProcess
    PRIORITY=6
    LOGGER_NAME=keystone.common.wsgi
    LOGGER_LEVEL=INFO
    _TRANSPORT=journal
    _UID=116
    _GID=121
    _COMM=uwsgi
    _EXE=/usr/bin/uwsgi-core
    _CAP_EFFECTIVE=0
    _SYSTEMD_SLICE=system.slice
    _BOOT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    _MACHINE_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    _HOSTNAME=hostname
    MESSAGE=2020-04-07 11:28:54.307 27764 INFO keystone.common.wsgi [-] GET https://backend-keystone_main-5000//
    THREAD_NAME=uWSGIWorker12Core0
    SYSLOG_IDENTIFIER=keystone-wsgi-public
    _PID=27764
    _CMDLINE=/usr/bin/uwsgi --ini /etc/keystone/uwsgi.keystone-main.ini
    _SYSTEMD_CGROUP=/system.slice/keystone-main.service
    _SYSTEMD_UNIT=keystone-main.service
    _SOURCE_REALTIME_TIMESTAMP=1586258934307881

If SYSLOG_FACILITY is present, the message does not end up in kern.log anymore.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.log (master)

Fix proposed to branch: master
Review: https://review.opendev.org/718710

Changed in oslo.log:
assignee: nobody → Benjamin Reichel (breichel)
status: New → In Progress
OpenStack Infra (hudson-openstack)
Changed in oslo.log:
assignee: Benjamin Reichel (breichel) → Stephen Finucane (stephenfinucane)
OpenStack Infra (hudson-openstack)
Changed in oslo.log:
assignee: Stephen Finucane (stephenfinucane) → Benjamin Reichel (breichel)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.log (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/726156

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.log (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/726157

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.log (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/726159

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.log (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/726173

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.log (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/726174

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.log (stable/rocky)

Change abandoned by Benjamin Reichel (<email address hidden>) on branch: stable/rocky
Review: https://review.opendev.org/726174
Reason: no new release for rocky

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.log (master)

Reviewed: https://review.opendev.org/718710
Committed: https://git.openstack.org/cgit/openstack/oslo.log/commit/?id=5b12e66f6283962d96884e4aa3fa7db7d4e062fa
Submitter: Zuul
Branch: master

commit 5b12e66f6283962d96884e4aa3fa7db7d4e062fa
Author: Benjamin Reichel <email address hidden>
Date: Thu Apr 9 15:02:20 2020 +0200

    Add missing SYSLOG_FACILITY to JournalHandler

    Without SYSLOG_FACILITY log messages from journald forwards the message
    to kern.log. This seems to be default behaviour as it treats the message
    as kernel message.

    This option provides a proper facility to the log message and kern.log
    will not be filled up anymore.

    Change-Id: I62a32ed46a400b62ead8c1e6e64acee658e2a769
    Closes-Bug: #1871840

Changed in oslo.log:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.log (stable/ussuri)

Reviewed: https://review.opendev.org/726156
Committed: https://git.openstack.org/cgit/openstack/oslo.log/commit/?id=a67f7b5ce5b956918640784d9a180873676d5820
Submitter: Zuul
Branch: stable/ussuri

commit a67f7b5ce5b956918640784d9a180873676d5820
Author: Benjamin Reichel <email address hidden>
Date: Thu Apr 9 15:02:20 2020 +0200

    Add missing SYSLOG_FACILITY to JournalHandler

    Without SYSLOG_FACILITY log messages from journald forwards the message
    to kern.log. This seems to be default behaviour as it treats the message
    as kernel message.

    This option provides a proper facility to the log message and kern.log
    will not be filled up anymore.

    Cherry-picked from commit 5b12e66f6283962d96884e4aa3fa7db7d4e062fa
    Change-Id: I62a32ed46a400b62ead8c1e6e64acee658e2a769
    Closes-Bug: #1871840

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.log (stable/train)

Reviewed: https://review.opendev.org/726157
Committed: https://git.openstack.org/cgit/openstack/oslo.log/commit/?id=65ee5f5bbb3ec759bfa1bf1dbccb02c3b348bad0
Submitter: Zuul
Branch: stable/train

commit 65ee5f5bbb3ec759bfa1bf1dbccb02c3b348bad0
Author: Benjamin Reichel <email address hidden>
Date: Thu Apr 9 15:02:20 2020 +0200

    Add missing SYSLOG_FACILITY to JournalHandler

    Without SYSLOG_FACILITY log messages from journald forwards the message
    to kern.log. This seems to be default behaviour as it treats the message
    as kernel message.

    This option provides a proper facility to the log message and kern.log
    will not be filled up anymore.

    Cherry-picked from commit 5b12e66f6283962d96884e4aa3fa7db7d4e062fa
    Cherry-picked from commit 82751ce2a2b913a5d107133a15960c9c6d8fd594
    Change-Id: I62a32ed46a400b62ead8c1e6e64acee658e2a769
    Closes-Bug: #1871840

tags: added: in-stable-train
tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.log (stable/stein)

Reviewed: https://review.opendev.org/726159
Committed: https://git.openstack.org/cgit/openstack/oslo.log/commit/?id=19fd9a2e2a283705552cb2c1f53c229b6ecf85e1
Submitter: Zuul
Branch: stable/stein

commit 19fd9a2e2a283705552cb2c1f53c229b6ecf85e1
Author: Benjamin Reichel <email address hidden>
Date: Thu Apr 9 15:02:20 2020 +0200

    Add missing SYSLOG_FACILITY to JournalHandler

    Without SYSLOG_FACILITY log messages from journald forwards the message
    to kern.log. This seems to be default behaviour as it treats the message
    as kernel message.

    This option provides a proper facility to the log message and kern.log
    will not be filled up anymore.

    Cherry-picked from commit 5b12e66f6283962d96884e4aa3fa7db7d4e062fa
    Cherry-picked from commit 82751ce2a2b913a5d107133a15960c9c6d8fd594
    Cherry-picked from commit 911a45401c1292b1946c200aa0f6a98b9e246fd8
    Change-Id: I62a32ed46a400b62ead8c1e6e64acee658e2a769
    Closes-Bug: #1871840

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.log (stable/queens)

Change abandoned by Benjamin Reichel (<email address hidden>) on branch: stable/queens
Review: https://review.opendev.org/726173
Reason: no new release for queens

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to oslo.log (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/732457

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to oslo.log (master)

Reviewed: https://review.opendev.org/732457
Committed: https://git.openstack.org/cgit/openstack/oslo.log/commit/?id=184235c26517598d6e60f4db72e095ba061a0d6e
Submitter: Zuul
Branch: master

commit 184235c26517598d6e60f4db72e095ba061a0d6e
Author: Brian Haley <email address hidden>
Date: Mon Jun 1 17:25:56 2020 -0400

    Default facility to None in OSJournalHandler class

    Use a default value of None for facility in OSJournalHandler
    class, as is done in the OSSysLogHandler class.

    Neutron started failing with a:

     E1120: No value for argument 'facility' in constructor call

    With a recent release of oslo.log.

    Change-Id: I5269b82d219fd2377535120d9d266238d50431b3
    Related-Bug: #1871840

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to oslo.log (stable/ussuri)

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/733084

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to oslo.log (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/733085

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to oslo.log (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/733086

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to oslo.log (stable/ussuri)

Reviewed: https://review.opendev.org/733084
Committed: https://git.openstack.org/cgit/openstack/oslo.log/commit/?id=c77f3c9aa82364c482be9b37e6d29634273bb90f
Submitter: Zuul
Branch: stable/ussuri

commit c77f3c9aa82364c482be9b37e6d29634273bb90f
Author: Brian Haley <email address hidden>
Date: Mon Jun 1 17:25:56 2020 -0400

    Default facility to None in OSJournalHandler class

    Use a default value of None for facility in OSJournalHandler
    class, as is done in the OSSysLogHandler class.

    Neutron started failing with a:

     E1120: No value for argument 'facility' in constructor call

    With a recent release of oslo.log.

    Change-Id: I5269b82d219fd2377535120d9d266238d50431b3
    Related-Bug: #1871840
    (cherry picked from commit 184235c26517598d6e60f4db72e095ba061a0d6e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to oslo.log (stable/train)

Reviewed: https://review.opendev.org/733085
Committed: https://git.openstack.org/cgit/openstack/oslo.log/commit/?id=e19c4076b1f0d7fdbd6d68a09c973934029926a1
Submitter: Zuul
Branch: stable/train

commit e19c4076b1f0d7fdbd6d68a09c973934029926a1
Author: Brian Haley <email address hidden>
Date: Mon Jun 1 17:25:56 2020 -0400

    Default facility to None in OSJournalHandler class

    Use a default value of None for facility in OSJournalHandler
    class, as is done in the OSSysLogHandler class.

    Neutron started failing with a:

     E1120: No value for argument 'facility' in constructor call

    With a recent release of oslo.log.

    Change-Id: I5269b82d219fd2377535120d9d266238d50431b3
    Related-Bug: #1871840
    (cherry picked from commit 184235c26517598d6e60f4db72e095ba061a0d6e)
    (cherry picked from commit c77f3c9aa82364c482be9b37e6d29634273bb90f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to oslo.log (stable/stein)

Reviewed: https://review.opendev.org/733086
Committed: https://git.openstack.org/cgit/openstack/oslo.log/commit/?id=ef23f7085059a7844be308db454a20168fbc97e1
Submitter: Zuul
Branch: stable/stein

commit ef23f7085059a7844be308db454a20168fbc97e1
Author: Brian Haley <email address hidden>
Date: Mon Jun 1 17:25:56 2020 -0400

    Default facility to None in OSJournalHandler class

    Use a default value of None for facility in OSJournalHandler
    class, as is done in the OSSysLogHandler class.

    Neutron started failing with a:

     E1120: No value for argument 'facility' in constructor call

    With a recent release of oslo.log.

    Change-Id: I5269b82d219fd2377535120d9d266238d50431b3
    Related-Bug: #1871840
    (cherry picked from commit 184235c26517598d6e60f4db72e095ba061a0d6e)
    (cherry picked from commit c77f3c9aa82364c482be9b37e6d29634273bb90f)
    (cherry picked from commit e19c4076b1f0d7fdbd6d68a09c973934029926a1)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.