OpenStack service logs missing in Elasticsearch

[Doug Szumski]

As reported by Bernd Bausch on the mailing list there is an issue with forwarding logs to Elasticsearch or the Monasca API.

We see some logs arrive. When Fluentd is restarted we see the missing logs arrive in a burst, as if the local buffers have been flushed. An example
is apache_access logs, although people have also reported missing openstack service logs.

The issue is related to an upgrade of the td-agent package.

Fluentd 1.11.2 (comes via td-agent-4.0.1-1.el8.x86_64 on Centos) is the final working version

Fluentd 1.12.2/1.12.3/13.0.0 exhibits the issue described here

Currently this package is not pinned.

[Doug Szumski (dszumski)]

More specifically I can upgrade to Fluentd release 1.11.5 before I see this issue.

```
(fluentd)[root@control01 /]# td-agent-gem install fluentd -v 1.11.5
```

The issue appears in 1.12.0 and doesn't go away in 1.13.0.

Some OpenStack service logs still arrive without changing the match tags from *.** to **.

[Doug Szumski (dszumski)]

Fix: https://review.opendev.org/c/openstack/kolla/+/795086

[Radosław Piliszek (yoctozepto)]

Only Victoria+ we use td-agent 4.x. Ussuri- we used td-agent 3.x

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/795086
Committed: https://opendev.org/openstack/kolla/commit/6d0882ef799bb531d007c3e6839ef318fd512d5c
Submitter: "Zuul (22348)"
Branch: master

commit 6d0882ef799bb531d007c3e6839ef318fd512d5c
Author: Doug Szumski <email address hidden>
Date: Mon Jun 7 10:14:55 2021 +0000

    Pin td-agent to 4.0.* to fix missing logs

    The recent inclusion of Fluentd 1.12.x in Centos/Ubuntu packages
    causes a significant number of logs to fail to appear in the
    Elasticsearch or Monasca output plugins. The logs appear to
    sit in a local buffer, and are only flushed when Fluentd is
    restarted. The same issue persists in the most recent release
    (1.13.0), so for now we pin the td-agent package to 4.0.* which
    should restrict Fluentd to the 1.11.x release where we don't
    see this issue.

    Backport to Victoria.

    Change-Id: Iefcdd3100b7e3c5320bc5f1286a18251bdeab885
    Closes-Bug: #1930867

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla/+/796438

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/kolla/+/796439

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/796439
Committed: https://opendev.org/openstack/kolla/commit/d52bb8e7b94e532a3b8c7e9c5ce4560a8824a26d
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit d52bb8e7b94e532a3b8c7e9c5ce4560a8824a26d
Author: Doug Szumski <email address hidden>
Date: Mon Jun 7 10:14:55 2021 +0000

    Pin td-agent to 4.0.* to fix missing logs

    The recent inclusion of Fluentd 1.12.x in Centos/Ubuntu packages
    causes a significant number of logs to fail to appear in the
    Elasticsearch or Monasca output plugins. The logs appear to
    sit in a local buffer, and are only flushed when Fluentd is
    restarted. The same issue persists in the most recent release
    (1.13.0), so for now we pin the td-agent package to 4.0.* which
    should restrict Fluentd to the 1.11.x release where we don't
    see this issue.

    Backport to Victoria.

    Change-Id: Iefcdd3100b7e3c5320bc5f1286a18251bdeab885
    Closes-Bug: #1930867
    (cherry picked from commit 6d0882ef799bb531d007c3e6839ef318fd512d5c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/796438
Committed: https://opendev.org/openstack/kolla/commit/11d63e07c5168d9479f3fb542566c1fc6e4af9a3
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 11d63e07c5168d9479f3fb542566c1fc6e4af9a3
Author: Doug Szumski <email address hidden>
Date: Mon Jun 7 10:14:55 2021 +0000

    Pin td-agent to 4.0.* to fix missing logs

    The recent inclusion of Fluentd 1.12.x in Centos/Ubuntu packages
    causes a significant number of logs to fail to appear in the
    Elasticsearch or Monasca output plugins. The logs appear to
    sit in a local buffer, and are only flushed when Fluentd is
    restarted. The same issue persists in the most recent release
    (1.13.0), so for now we pin the td-agent package to 4.0.* which
    should restrict Fluentd to the 1.11.x release where we don't
    see this issue.

    Backport to Victoria.

    Change-Id: Iefcdd3100b7e3c5320bc5f1286a18251bdeab885
    Closes-Bug: #1930867
    (cherry picked from commit 6d0882ef799bb531d007c3e6839ef318fd512d5c)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 12.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 11.1.0 release.

[Guillermoadrianmolina (guillermoadrianmolina)]

This bug also affects Ubuntu (and Debian) as stated, but the patches do not cover the pinning on the correct version of td-agent on deb based distros.

I've manually installed it doing:

root@docker1:~# docker exec -u root -it fluentd bash
(fluentd)[root@docker1 /]# curl --location http://packages.treasuredata.com.s3.amazonaws.com/4/debian/buster/pool/contrib/t/td-agent/td-agent_4.0.1-1_amd64.deb -o /tmp/td-agent_4.0.1-1_amd64.deb
(fluentd)[root@docker1 /]# dpkg -i /tmp/td-agent_4.0.1-1_amd64.deb
root@docker1:~# docker restart fluentd

Can you also include Ubuntu/Debian in the patch?

[Radosław Piliszek (yoctozepto)]

I checked and Ubuntu images have the proper td-agent version.
However, Debian ones do not.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla/+/807475

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/807475
Committed: https://opendev.org/openstack/kolla/commit/5253e031c6e62e89b03a3756d44c58da007f1cbd
Submitter: "Zuul (22348)"
Branch: master

commit 5253e031c6e62e89b03a3756d44c58da007f1cbd
Author: Radosław Piliszek <email address hidden>
Date: Sun Sep 5 08:50:52 2021 +0000

    Pin td-agent to 4.0.* also on Debian

    Debian was missed in the original patch. [1]

    Backport to Victoria.

    [1] 6d0882ef799bb531d007c3e6839ef318fd512d5c
    aka Iefcdd3100b7e3c5320bc5f1286a18251bdeab885

    Closes-Bug: #1930867
    Change-Id: Id5938697b62ba040062475fcf5215d64e1ba8c6f

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla/+/807382

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/kolla/+/807383

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/807382
Committed: https://opendev.org/openstack/kolla/commit/1ec1d8b467f8eca44182f2e0e366a3245b2dd1db
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 1ec1d8b467f8eca44182f2e0e366a3245b2dd1db
Author: Radosław Piliszek <email address hidden>
Date: Sun Sep 5 08:50:52 2021 +0000

    Pin td-agent to 4.0.* also on Debian

    Debian was missed in the original patch. [1]

    Backport to Victoria.

    [1] 6d0882ef799bb531d007c3e6839ef318fd512d5c
    aka Iefcdd3100b7e3c5320bc5f1286a18251bdeab885

    Closes-Bug: #1930867
    Change-Id: Id5938697b62ba040062475fcf5215d64e1ba8c6f
    (cherry picked from commit 5253e031c6e62e89b03a3756d44c58da007f1cbd)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/807383
Committed: https://opendev.org/openstack/kolla/commit/16b501fa8a4df1b4588669e9c7c446d63ac86d64
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 16b501fa8a4df1b4588669e9c7c446d63ac86d64
Author: Radosław Piliszek <email address hidden>
Date: Sun Sep 5 08:50:52 2021 +0000

    Pin td-agent to 4.0.* also on Debian

    Debian was missed in the original patch. [1]

    Backport to Victoria.

    [1] 6d0882ef799bb531d007c3e6839ef318fd512d5c
    aka Iefcdd3100b7e3c5320bc5f1286a18251bdeab885

    Closes-Bug: #1930867
    Change-Id: Id5938697b62ba040062475fcf5215d64e1ba8c6f
    (cherry picked from commit 5253e031c6e62e89b03a3756d44c58da007f1cbd)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 13.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 11.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 12.1.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/821695
Committed: https://opendev.org/openstack/kolla/commit/a8e6924beaea3e268435156f6bbc34812a69a01c
Submitter: "Zuul (22348)"
Branch: master

commit a8e6924beaea3e268435156f6bbc34812a69a01c
Author: Dr. Jens Harbott <email address hidden>
Date: Tue Dec 14 13:10:39 2021 +0100

    Unpin td-agent and cap elasticsearch gem

    With [0] the version of td-agent being installed was pinned, which isn't
    sustainable in the long run, so we drop the pin.

    Latest version of the elasticsearch gem no longer works with older
    (OSS) versions of Elasticsearch. This is fixed by downgrading the version
    of the elasticsearch gems.

    [0] Iefcdd3100b7e3c5320bc5f1286a18251bdeab885
    Related-Bug: 1930867
    Closes-Bug: 1954759
    Signed-off-by: Dr. Jens Harbott <email address hidden>
    Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/823155
    Change-Id: I3045786e863b098d7339d1066aef6c857aa5f97f

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla/+/828560

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla/+/828563

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/828560
Committed: https://opendev.org/openstack/kolla/commit/5c6eb173973a7781c8955115a6573ae59e45a479
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 5c6eb173973a7781c8955115a6573ae59e45a479
Author: Dr. Jens Harbott <email address hidden>
Date: Tue Dec 14 13:10:39 2021 +0100

    Unpin td-agent and cap elasticsearch gem

    With [0] the version of td-agent being installed was pinned, which isn't
    sustainable in the long run, so we drop the pin.

    Latest version of the elasticsearch gem no longer works with older
    (OSS) versions of Elasticsearch. This is fixed by downgrading the version
    of the elasticsearch gems.

    Backport includes 711ec0829a95a12ae9e290aeb3837b457e1dd215 [1].

    [0] Iefcdd3100b7e3c5320bc5f1286a18251bdeab885
    [1] I353ef9b6b8da0e726c17ad2e06995d3690731e1d
    Related-Bug: 1930867
    Closes-Bug: 1954759
    Closes-Bug: 1960444
    Signed-off-by: Dr. Jens Harbott <email address hidden>
    Co-Authored-By: Radosław Piliszek <email address hidden>
    Change-Id: I3045786e863b098d7339d1066aef6c857aa5f97f
    (cherry picked from commit a8e6924beaea3e268435156f6bbc34812a69a01c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/828563
Committed: https://opendev.org/openstack/kolla/commit/3bbc5b3299c69ec56e57d63bdb487121742a2730
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 3bbc5b3299c69ec56e57d63bdb487121742a2730
Author: Dr. Jens Harbott <email address hidden>
Date: Tue Dec 14 13:10:39 2021 +0100

    Unpin td-agent and cap elasticsearch gem

    With [0] the version of td-agent being installed was pinned, which isn't
    sustainable in the long run, so we drop the pin.

    Latest version of the elasticsearch gem no longer works with older
    (OSS) versions of Elasticsearch. This is fixed by downgrading the version
    of the elasticsearch gems.

    Backport includes 711ec0829a95a12ae9e290aeb3837b457e1dd215 [1].

    [0] Iefcdd3100b7e3c5320bc5f1286a18251bdeab885
    [1] I353ef9b6b8da0e726c17ad2e06995d3690731e1d
    Related-Bug: 1930867
    Closes-Bug: 1954759
    Closes-Bug: 1960444
    Signed-off-by: Dr. Jens Harbott <email address hidden>
    Co-Authored-By: Radosław Piliszek <email address hidden>
    Change-Id: I3045786e863b098d7339d1066aef6c857aa5f97f
    (cherry picked from commit a8e6924beaea3e268435156f6bbc34812a69a01c)