[fluentd] unable to parse several logs

[Viorel-Cosmin Miron]

Reporting issues from fluentd:

got incomplete line before first line from /var/log/kolla/cinder/cinder-api.log: "2020-04-11 18:21:03.327555 /var/lib/kolla/venv/lib/python3.6/site-packages/webob/acceptparse.py:1051: DeprecationWarning: The behavior of AcceptValidHeader.best_match is currently being maintained for backward compatibility, but it will be deprecated in the future, as it does not conform to the RFC.\n"

got incomplete line before first line from /var/log/kolla/cinder/cinder-api.log: "2020-04-11 18:21:03.327572 DeprecationWarning,\n"

got incomplete line before first line from /var/log/kolla/barbican/barbican-api.log: "{address space usage: 259821568 bytes/247MB} {rss usage: 102064128 bytes/97MB} [pid: 24|app: 0|req: 3158/34710] 172.22.0.1 () {30 vars in 723 bytes} [Sat Apr 11 18:20:55 2020] GET /v1/secrets/65402c51-b41d-41a9-963c-9a96320c2a8a/payload => generated 1089 bytes in 17 msecs (HTTP/1.1 200) 4 headers in 166 bytes (1 switches on core 0)\n"

got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: "2020-04-11 18:20:22,701 [136] INFO gnocchi.cli.metricd: 0 measurements bundles across 0 metrics wait to be processed.\n"

This generate 90/95 warnings / 30 seconds.

Latest kolla-ansible with train stable source on ubuntu 18.04.

[Viorel-Cosmin Miron (uhl-hosting)]

got incomplete line before first line from /var/log/kolla/barbican/barbican-api.log: "{address space usage: 262696960 bytes/250MB} {rss usage: 104644608 bytes/99MB} [pid: 25|app: 0|req: 8834/43740] 172.22.0.1 () {30 vars in 723 bytes} [Fri Apr 17 12:10:43 2020] GET /v1/secrets/7f441dee-04de-47f0-8240-e37ddffcb4cb/payload => generated 1093 bytes in 15 msecs (HTTP/1.1 200) 4 headers

Is this fitting here?

[Viorel-Cosmin Miron (uhl-hosting)]

got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: "2020-04-17 12:10:18,534 [100] DEBUG gnocchi.chef: Processing measures for sack incoming128-124\n"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/876785

[Graeme Moss (gramimoss)]

The review https://review.opendev.org/c/openstack/kolla-ansible/+/876785 seems to be a quick fix we need to have a permeant solution.

The default value of logging_context_format_string and logging_user_identity_format has changed [1][2]

[1]https://opendev.org/openstack/oslo.log/src/branch/master/oslo_log/_options.py#L147
[2]https://opendev.org/openstack/oslo.log/src/branch/master/oslo_log/_options.py#L169

the review to follow is https://review.opendev.org/c/openstack/kolla-ansible/+/876039 which uses the correct logging context to regex the logs.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/876039
Committed: https://opendev.org/openstack/kolla-ansible/commit/900c2d20bb461111a37d37349b693bf8f8cb7135
Submitter: "Zuul (22348)"
Branch: master

commit 900c2d20bb461111a37d37349b693bf8f8cb7135
Author: lishuliang <[e-mail]>
Date: Thu Mar 2 16:46:44 2023 +0800

    Fix the fluentd regexp to collect the logs

    The default value of logging_context_format_string and
    logging_user_identity_format has changed [1][2][3].
    This is to correct fluentd to use the same format.

    [1]https://opendev.org/openstack/oslo.log/commit/ebdee7f39920ad5b4268ee296952432b0d41a375
    [2]https://opendev.org/openstack/oslo.log/commit/7ed065f38df0595881d07eb91cec9e5c07cb38be

    [3]https://review.opendev.org/c/openstack/oslo.log/+/838185

    Closes-Bug: #1872220

    Change-Id: Ide0079ef4d28e3a3fa60288a74b936531c52ce60

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Dr. Jens Harbott <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/876785
Reason: this was fixed in https://review.opendev.org/c/openstack/kolla-ansible/+/876039 in a better way

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/884539

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/883915
Committed: https://opendev.org/openstack/kolla-ansible/commit/5ed259ec07e89cdc1c4102fb33aa3fee30490170
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 5ed259ec07e89cdc1c4102fb33aa3fee30490170
Author: lishuliang <[e-mail]>
Date: Thu Mar 2 16:46:44 2023 +0800

    Fix the fluentd regexp to collect the logs

    The default value of logging_context_format_string and
    logging_user_identity_format has changed [1][2][3].
    This is to correct fluentd to use the same format.

    [1]https://opendev.org/openstack/oslo.log/commit/ebdee7f39920ad5b4268ee296952432b0d41a375
    [2]https://opendev.org/openstack/oslo.log/commit/7ed065f38df0595881d07eb91cec9e5c07cb38be

    [3]https://review.opendev.org/c/openstack/oslo.log/+/838185

    Closes-Bug: #1872220

    Change-Id: Ide0079ef4d28e3a3fa60288a74b936531c52ce60
    (cherry picked from commit 900c2d20bb461111a37d37349b693bf8f8cb7135)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 15.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 16.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 17.0.0.0rc1 release candidate.