Fluentd log parsing cannot grok Horizon logs

[Stig Telfer]

It appears that all logs from Horizon are throwing parse errors from fluentd of this form:

got incomplete line before first line from /var/log/kolla/horizon/horizon.log: "[Wed Sep 30 17:02:00.196771 2020] [:error] [pid 16] Logging out user \\"\\".\\n"

Horizon logs take this form:

[Wed Sep 30 17:02:00.196771 2020] [:error] [pid 16] Logging out user "".

This is not the same format as logs from other OpenStack services, but is apparently being parsed as though it is.

This issue was seen on Rocky and Train OpenStack releases.

[Doug Szumski (dszumski)]

A quick fix could be to exclude the Horizon log from 00-global.conf and process this log with other Apache logs. The format isn't the same as the access logs, but we're not currently doing much parsing on them anyway.

[Seunghun Lee (seunghunlee)]

This bug is still present. I found warning messages with this issue. https://zuul.opendev.org/t/openstack/build/b2e4a443a1d541028dfa9340cbd9e282/log/primary/logs/kolla/fluentd/fluentd.txt#18787

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/901985

[Doug Szumski (dszumski)]

Fix: https://review.opendev.org/c/openstack/kolla-ansible/+/901985

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/902066

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/901985
Committed: https://opendev.org/openstack/kolla-ansible/commit/4168b46cffc6297bd069400eb76abb38d28cfb5a
Submitter: "Zuul (22348)"
Branch: master

commit 4168b46cffc6297bd069400eb76abb38d28cfb5a
Author: Doug Szumski <email address hidden>
Date: Mon Nov 27 15:41:29 2023 +0000

    Fix Horizon WSGI application log parsing

    Like other WSGI services in Kolla Ansible, the Horizon WSGI application
    handles log output via the `wsgi.errors` object. See [1] for further
    information. The problem is that this log output is written to a file called
    `horizon.log`, causing it to processed as an 'Oslo log' in the Fluentd
    processing pipeline. Since the log format doesn't match the expected format,
    this results in parsing errors.

    This fix renames the log file and adjusts the format to match other WSGI
    applications. The logs are then processed in the same way as other WSGI
    application logs, resolving the issue.

    [1] https://modwsgi.readthedocs.io/en/master/user-guides/debugging-techniques.html

    Change-Id: I93777d1c53920f5470c78356e6b3a4064fbe04b4
    Closes-Bug: #1898174

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/902069

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/902070

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/902066
Committed: https://opendev.org/openstack/kolla-ansible/commit/ccd04bde68173fce44b244d9c31d790adad9ddb0
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit ccd04bde68173fce44b244d9c31d790adad9ddb0
Author: Doug Szumski <email address hidden>
Date: Mon Nov 27 15:41:29 2023 +0000

    Fix Horizon WSGI application log parsing

    Like other WSGI services in Kolla Ansible, the Horizon WSGI application
    handles log output via the `wsgi.errors` object. See [1] for further
    information. The problem is that this log output is written to a file called
    `horizon.log`, causing it to processed as an 'Oslo log' in the Fluentd
    processing pipeline. Since the log format doesn't match the expected format,
    this results in parsing errors.

    This fix renames the log file and adjusts the format to match other WSGI
    applications. The logs are then processed in the same way as other WSGI
    application logs, resolving the issue.

    [1] https://modwsgi.readthedocs.io/en/master/user-guides/debugging-techniques.html

    Change-Id: I93777d1c53920f5470c78356e6b3a4064fbe04b4
    Closes-Bug: #1898174
    (cherry picked from commit 4168b46cffc6297bd069400eb76abb38d28cfb5a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/902069
Committed: https://opendev.org/openstack/kolla-ansible/commit/04695d49da68333c462cad40ff94bee687d33e3e
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 04695d49da68333c462cad40ff94bee687d33e3e
Author: Doug Szumski <email address hidden>
Date: Mon Nov 27 15:41:29 2023 +0000

    Fix Horizon WSGI application log parsing

    Like other WSGI services in Kolla Ansible, the Horizon WSGI application
    handles log output via the `wsgi.errors` object. See [1] for further
    information. The problem is that this log output is written to a file called
    `horizon.log`, causing it to processed as an 'Oslo log' in the Fluentd
    processing pipeline. Since the log format doesn't match the expected format,
    this results in parsing errors.

    This fix renames the log file and adjusts the format to match other WSGI
    applications. The logs are then processed in the same way as other WSGI
    application logs, resolving the issue.

    [1] https://modwsgi.readthedocs.io/en/master/user-guides/debugging-techniques.html

    Change-Id: I93777d1c53920f5470c78356e6b3a4064fbe04b4
    Closes-Bug: #1898174
    (cherry picked from commit 4168b46cffc6297bd069400eb76abb38d28cfb5a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/902070
Committed: https://opendev.org/openstack/kolla-ansible/commit/da5d7c4b442cdd6ce2cddaeebaadfa31b2a8dce2
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit da5d7c4b442cdd6ce2cddaeebaadfa31b2a8dce2
Author: Doug Szumski <email address hidden>
Date: Mon Nov 27 15:41:29 2023 +0000

    Fix Horizon WSGI application log parsing

    Like other WSGI services in Kolla Ansible, the Horizon WSGI application
    handles log output via the `wsgi.errors` object. See [1] for further
    information. The problem is that this log output is written to a file called
    `horizon.log`, causing it to processed as an 'Oslo log' in the Fluentd
    processing pipeline. Since the log format doesn't match the expected format,
    this results in parsing errors.

    This fix renames the log file and adjusts the format to match other WSGI
    applications. The logs are then processed in the same way as other WSGI
    application logs, resolving the issue.

    [1] https://modwsgi.readthedocs.io/en/master/user-guides/debugging-techniques.html

    Change-Id: I93777d1c53920f5470c78356e6b3a4064fbe04b4
    Closes-Bug: #1898174
    (cherry picked from commit 4168b46cffc6297bd069400eb76abb38d28cfb5a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 17.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 16.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 14.11.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 15.4.0 release.