Fluentd config incorrectly parses MariaDB logs

[Isaac Prior]

Fluentd parses mariadb.log in the MariaDB container when Monasca/Fluentd is enabled.
Two different log formats are used in mariadb.log (mysqld 10.1.20-MariaDB).
Log messages from the mysqld_safe process are parsed by fluentd correctly but others are not.

An example of the two log formats:
"""
190913 10:00:32 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
190913 10:00:32 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql//wsrep_recovery.8Bt4Pg' --pid-file='/var/lib/mysql//operator.local-recover.pid'
2019-09-13 10:00:33 140183735036096 [Note] /usr/libexec/mysqld (mysqld 10.1.20-MariaDB) starting as process 194 ...
190913 10:00:36 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
2019-09-13 10:00:36 139854442735808 [Note] /usr/libexec/mysqld (mysqld 10.1.20-MariaDB) starting as process 229 ...
"""

As viewed in Kibana the '2019-09-13' format messages are interpretted as continuations of the '190913' format ones. The result is a multi-line log event that captures all messages until the next short-form time format.

The correct behaviour is to parse all messages that begin with a timestamp (of both types) as individual log messages.

Centos7 - 3.10.0-862.2.3.el7.x86_64
Docker version 17.05.0-ce, build 89658be
Tested in Kolla-Ansible Vagrant environment - Stable/Rocky.
Kolla tag 8.0.0.0rc1
globals.yml includes:
"""
enable_monasca: "yes"
monasca_install_type: "source"
"""

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/686428

[Doug Szumski (dszumski)]

Another example of logs which are not collected (WSREP_SST):

```
2019-10-29 18:12:34 140571024795392 [Note] WSREP: Member 0.0 (foo_node) synced with group.
WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20191029 18:12:34.325)
WSREP_SST: [INFO] Galera co-ords from recovery: 41f28e99-020a-11e9-bfd1-cfa08f7a3e8e:72402191 0 (20191029 18:12:34.329)
WSREP_SST: [INFO] Total time on joiner: 0 seconds (20191029 18:12:34.332)
WSREP_SST: [INFO] Removing the sst_in_progress file (20191029 18:12:34.334)
2019-10-29 18:12:34 140571371378880 [Note] WSREP: SST complete, seqno: 72402191
```

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/686428
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=233a187a76c69e4c892a4ad70e710ee67836dfdd
Submitter: Zuul
Branch: master

commit 233a187a76c69e4c892a4ad70e710ee67836dfdd
Author: Isaac Prior <email address hidden>
Date: Thu Oct 3 16:29:39 2019 +0100

    Parse MariaDB log messages in different formats into Fluentd

    MariaDB logs contain two different log message formats, one output
    from mysqld and one from mysqld_safe. This patch splits the message
    formats by tag and parses them separately.

    Change-Id: I58857be67ae387eeda7487811a6af85b0f95970c
    Closes-Bug: #1845629

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/695392

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/695393

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/695394

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/695393
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=8fdc2e140b0d202cb72f7805203bdea78c93db58
Submitter: Zuul
Branch: stable/stein

commit 8fdc2e140b0d202cb72f7805203bdea78c93db58
Author: Isaac Prior <email address hidden>
Date: Thu Oct 3 16:29:39 2019 +0100

    Parse MariaDB log messages in different formats into Fluentd

    MariaDB logs contain two different log message formats, one output
    from mysqld and one from mysqld_safe. This patch splits the message
    formats by tag and parses them separately.

    Change-Id: I58857be67ae387eeda7487811a6af85b0f95970c
    Closes-Bug: #1845629
    (cherry picked from commit 233a187a76c69e4c892a4ad70e710ee67836dfdd)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/695394
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=8ebe4a632181438c9878ac988697f547706d0100
Submitter: Zuul
Branch: stable/rocky

commit 8ebe4a632181438c9878ac988697f547706d0100
Author: Isaac Prior <email address hidden>
Date: Thu Oct 3 16:29:39 2019 +0100

    Parse MariaDB log messages in different formats into Fluentd

    MariaDB logs contain two different log message formats, one output
    from mysqld and one from mysqld_safe. This patch splits the message
    formats by tag and parses them separately.

    Change-Id: I58857be67ae387eeda7487811a6af85b0f95970c
    Closes-Bug: #1845629
    (cherry picked from commit 233a187a76c69e4c892a4ad70e710ee67836dfdd)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/695392
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=debbd789ab3b59ba9cad792554b2c5d48ea25cde
Submitter: Zuul
Branch: stable/train

commit debbd789ab3b59ba9cad792554b2c5d48ea25cde
Author: Isaac Prior <email address hidden>
Date: Thu Oct 3 16:29:39 2019 +0100

    Parse MariaDB log messages in different formats into Fluentd

    MariaDB logs contain two different log message formats, one output
    from mysqld and one from mysqld_safe. This patch splits the message
    formats by tag and parses them separately.

    Change-Id: I58857be67ae387eeda7487811a6af85b0f95970c
    Closes-Bug: #1845629
    (cherry picked from commit 233a187a76c69e4c892a4ad70e710ee67836dfdd)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 7.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 8.1.0 release.