Enable monasca broken fluentd container

[Chiawei Xie]

kolla-ansible version: stein
kolla version: stein
source: ubuntu source

When enable monasca, the fluentd container got error messages:

[warn]: 'utc' is deperecated for output plugin. This parameter is used for formatter plugin in compatibility layer. If you want to use same feature, use timekey_use_utc parameter in <buffer> directive instead

[error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Unknown output plugin 'monasca'. Run 'gem search -rd fluent-plugin' to find plugins"

[Mark Goddard (mgoddard)]

I cracked open the kolla/ubuntu-source-fluentd:stein image, and found the following gem installed via 'gem list':

fluentd-monasca-output (0.0.1)

I see the same output in the kolla/centos-source-fluentd:stein image. However, the CentOS image also includes a bunch of other fluentd plugins that don't appear in the ubuntu image:

fluent-plugin-elasticsearch (1.9.5.1)
fluent-plugin-grok-parser (0.3.1)
fluent-plugin-kubernetes_metadata_filter (0.27.0)
fluent-plugin-rewrite-tag-filter (1.5.6)
fluent-plugin-secure-forward (0.4.5)

The main difference is that the ubuntu plugins are installed via apt, whereas the centos plugins are installed via gem.

I've now found a command called td-agent-gem in the ubuntu image (fluent is referred to as td-agent in ubuntu for legacy reasons). This lists the expected plugins, other than the monasca one:

fluent-config-regexp-type (1.0.0)
fluent-logger (0.8.0)
fluent-plugin-elasticsearch (3.5.1)
fluent-plugin-grep (0.3.4)
fluent-plugin-grok-parser (2.1.4)
fluent-plugin-kafka (0.9.4)
fluent-plugin-kubernetes_metadata_filter (2.1.6)
fluent-plugin-parser (0.6.1)
fluent-plugin-prometheus (1.4.0)
fluent-plugin-record-modifier (2.0.1)
fluent-plugin-rewrite-tag-filter (2.2.0, 2.0.0)
fluent-plugin-s3 (1.1.10)
fluent-plugin-secure-forward (0.4.5)
fluent-plugin-td (1.0.0)
fluent-plugin-td-monitoring (0.2.4)
fluent-plugin-webhdfs (1.2.3)

When installing the monasca plugin, we currently use 'fluentd-gem install' on ubuntu, we probably need to run 'td-agent-gem install'.

[Mark Goddard (mgoddard)]

Fix proposed: https://review.opendev.org/660953

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/660953

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/660953
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=c585ca5055c7a92fdbe5826030d3529e5c397540
Submitter: Zuul
Branch: master

commit c585ca5055c7a92fdbe5826030d3529e5c397540
Author: Mark Goddard <email address hidden>
Date: Thu May 23 10:25:28 2019 +0100

    Use td-agent-gem to install fluentd monasca plugin on ubuntu

    If monasca is enabled on ubuntu, we see the following error in fluentd:

    [error]: config error file="/etc/td-agent/td-agent.conf"
    error_class=Fluent::ConfigError error="Unknown output plugin 'monasca'.
    Run 'gem search -rd fluent-plugin' to find plugins"

    This is because we use 'fluent-gem install' to install the gem, which
    works on CentOS but on Ubuntu it is necessary to use 'td-agent-gem
    install'.

    Change-Id: I515f9764ca93a41a871e520696193a9298856d08
    Closes-Bug: #1830147

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/666074

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/666075

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/666075
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=40104a393e11cee0f2d37a6b2eea0cbbd388e71b
Submitter: Zuul
Branch: stable/rocky

commit 40104a393e11cee0f2d37a6b2eea0cbbd388e71b
Author: Mark Goddard <email address hidden>
Date: Thu May 23 10:25:28 2019 +0100

    Use td-agent-gem to install fluentd monasca plugin on ubuntu

    If monasca is enabled on ubuntu, we see the following error in fluentd:

    [error]: config error file="/etc/td-agent/td-agent.conf"
    error_class=Fluent::ConfigError error="Unknown output plugin 'monasca'.
    Run 'gem search -rd fluent-plugin' to find plugins"

    This is because we use 'fluent-gem install' to install the gem, which
    works on CentOS but on Ubuntu it is necessary to use 'td-agent-gem
    install'.

    Change-Id: I515f9764ca93a41a871e520696193a9298856d08
    Closes-Bug: #1830147
    (cherry picked from commit c585ca5055c7a92fdbe5826030d3529e5c397540)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/666074
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=588c1076c3da583e9f654dcfbed5fde13788dc0d
Submitter: Zuul
Branch: stable/stein

commit 588c1076c3da583e9f654dcfbed5fde13788dc0d
Author: Mark Goddard <email address hidden>
Date: Thu May 23 10:25:28 2019 +0100

    Use td-agent-gem to install fluentd monasca plugin on ubuntu

    If monasca is enabled on ubuntu, we see the following error in fluentd:

    [error]: config error file="/etc/td-agent/td-agent.conf"
    error_class=Fluent::ConfigError error="Unknown output plugin 'monasca'.
    Run 'gem search -rd fluent-plugin' to find plugins"

    This is because we use 'fluent-gem install' to install the gem, which
    works on CentOS but on Ubuntu it is necessary to use 'td-agent-gem
    install'.

    Change-Id: I515f9764ca93a41a871e520696193a9298856d08
    Closes-Bug: #1830147
    (cherry picked from commit c585ca5055c7a92fdbe5826030d3529e5c397540)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 8.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 7.0.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 9.0.0.0rc1 release candidate.