fluentd rabbitmq input: pattern not found

[joek-office]

What happened:
In Kibana/Elasticsearch comes many failures from fluentd with "pattern not found" message. This comes from an change in the log format of rabbitmq. Because of this, the parser pattern for the fluentd input isn't matching the new format of that messages and throw the error.
Example Messages:
pattern not matched: "2023-04-03 16:09:13.014794+02:00 [info] <0.17024.434> connection <0.17024.434> (10.10.10.12:59258 -> 10.10.10.12:5672) has a client-provided name: mod_wsgi:35:14771eaa-32f2-476b-a650-b22edfc51179"

pattern not matched: "2023-04-03 16:09:06.435201+02:00 [error] <0.16363.388> missed heartbeats from client, timeout: 60s"

What you expected to happen:
No "pattern not matched" error messages from fluentd

How to reproduce it (minimal and precise):

**Environment**:
* Host OS:
Rocky Linux 9.1 (Blue Onyx)

* Kernel (e.g. `uname -a`):
Linux ctrlhost2 5.14.0-162.12.1.el9_1.0.2.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Jan 30 22:14:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

* Docker version if applicable (e.g. `docker version`):
23.0.1

* Kolla-Ansible version (e.g. `git head or tag or stable branch` or pip package version if using release):
master

* Are you using official images from Docker Hub or self built?
Images from Quay.io
fluentd:zed-rocky-9

[Sven Kieske (s-kieske)]

is this the upstream change which is referenced here?

https://github.com/rabbitmq/rabbitmq-server/pull/2861

[joek-office (joek-office)]

@Sven Kieske
Yes, seems that this is the reason of the new log format.
I have filed a patch a few days ago.

https://review.opendev.org/c/openstack/kolla-ansible/+/879426

Think the patch would respect the old and new log format of rabbit.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/879426
Committed: https://opendev.org/openstack/kolla-ansible/commit/b9f2ad94a7bbd2d02d8ba69106e4d7a64d04f82a
Submitter: "Zuul (22348)"
Branch: master

commit b9f2ad94a7bbd2d02d8ba69106e4d7a64d04f82a
Author: joek-office <email address hidden>
Date: Tue Apr 4 09:44:40 2023 +0200

    Change regex for the multiline parser at the fluentd rabbitmq input

    Related-Bug: #2015184

    Closes-Bug: #2015184

    Change-Id: I6ea0949d58e38e2cda41c9702001e7bd500f23f1

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/901707

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/901708

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/901709

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 17.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/901707
Committed: https://opendev.org/openstack/kolla-ansible/commit/dbacd038f21dc15967f31e46d37177d264236f4b
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit dbacd038f21dc15967f31e46d37177d264236f4b
Author: joek-office <email address hidden>
Date: Tue Apr 4 09:44:40 2023 +0200

    Change regex for the multiline parser at the fluentd rabbitmq input

    Related-Bug: #2015184

    Closes-Bug: #2015184

    Change-Id: I6ea0949d58e38e2cda41c9702001e7bd500f23f1
    (cherry picked from commit b9f2ad94a7bbd2d02d8ba69106e4d7a64d04f82a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/901708
Committed: https://opendev.org/openstack/kolla-ansible/commit/e4bd422d4322282fd9f26932711c3b4af81fd97a
Submitter: "Zuul (22348)"
Branch: stable/zed

commit e4bd422d4322282fd9f26932711c3b4af81fd97a
Author: joek-office <email address hidden>
Date: Tue Apr 4 09:44:40 2023 +0200

    Change regex for the multiline parser at the fluentd rabbitmq input

    Related-Bug: #2015184

    Closes-Bug: #2015184

    Change-Id: I6ea0949d58e38e2cda41c9702001e7bd500f23f1
    (cherry picked from commit b9f2ad94a7bbd2d02d8ba69106e4d7a64d04f82a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/901709
Committed: https://opendev.org/openstack/kolla-ansible/commit/f03d2bc0447e2140c7b3ed509a6cbe79f1f42aec
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit f03d2bc0447e2140c7b3ed509a6cbe79f1f42aec
Author: joek-office <email address hidden>
Date: Tue Apr 4 09:44:40 2023 +0200

    Change regex for the multiline parser at the fluentd rabbitmq input

    Related-Bug: #2015184

    Closes-Bug: #2015184

    Change-Id: I6ea0949d58e38e2cda41c9702001e7bd500f23f1
    (cherry picked from commit b9f2ad94a7bbd2d02d8ba69106e4d7a64d04f82a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 16.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 14.11.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 15.4.0 release.