kolla

fluentd crash - elasticsearch version mismatch

Bug #2018463 reported by Nicolas Froger
26
This bug affects 4 people
Affects Status Importance Assigned to Milestone
kolla
Fix Released
High
Unassigned
Xena
Fix Released
Undecided
Unassigned
Yoga
Fix Released
Undecided
Unassigned

Bug Description

Hello,

fluentd fails to start on Xena on latest images because of what seems to be an ElasticSearch version mismatch. I have seen someone report a similar bug on the openstack-discuss mailing list on Yoga.

Here are the logs of the fluentd container :

```
Running command: '/usr/sbin/td-agent -o /var/log/kolla/fluentd/fluentd.log'
The client is unable to verify that the server is Elasticsearch. Some functionality may not be compatible if the server is running an unsupported product.
2023-05-04 11:07:14 +0200 [error]: fluent/log.rb:372:error: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Using Elasticsearch client 8.7.0 is not compatible for your Elasticsearch server. Please check your using elasticsearch gem version and Elasticsearch server."
```

With the fluentd image :

```
()[td-agent@4bb5e4e878e6 /]$ td-agent-gem list | grep elastic
elastic-transport (8.1.0)
elasticsearch (8.7.0, 7.13.0)
elasticsearch-api (8.7.0, 7.13.0)
elasticsearch-transport (7.13.0)
fluent-plugin-elasticsearch (5.3.0, 5.2.4)
```

From the elasticsearch container :
```
(elasticsearch)[elasticsearch@monitor-1 /]$ /usr/share/elasticsearch/bin/elasticsearch -V
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Version: 7.10.2, Build: oss/deb/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z, JVM: 11.0.18
```

Images used :
* quay.io/openstack.kolla/ubuntu-source-fluentd@sha256:17f7d31fd74ee3ccc23475f897612d95d5bbca8c07e361336ddda2cb8a7d34f3
* quay.io/openstack.kolla/ubuntu-source-elasticsearch@sha256:aa59e760943a2f7b829abe9d3ef15ddefab639832b48d6630d64e816256d3444

Regards,

Nicolas

Michal Nasiadka (mnasiadka)
Changed in kolla:
status: New → Triaged
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla/+/882289

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/882289
Committed: https://opendev.org/openstack/kolla/commit/8e8ef4c38999b95cd989fc4ba7580874807e706d
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 8e8ef4c38999b95cd989fc4ba7580874807e706d
Author: Michal Nasiadka <email address hidden>
Date: Thu May 4 15:09:17 2023 +0200

    fluentd: cap elasticsearch again

    Closes-Bug: #2018463
    Change-Id: If5bec23cb4945fea7eba87a4c9252b1ae201ce05

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla/+/882772

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/882772
Committed: https://opendev.org/openstack/kolla/commit/fed3e69d6c8b366c7698f6ffc9a322ebaf437e34
Submitter: "Zuul (22348)"
Branch: stable/xena

commit fed3e69d6c8b366c7698f6ffc9a322ebaf437e34
Author: Michal Nasiadka <email address hidden>
Date: Thu May 4 15:09:17 2023 +0200

    fluentd: cap elasticsearch again

    Closes-Bug: #2018463
    Change-Id: If5bec23cb4945fea7eba87a4c9252b1ae201ce05
    (cherry picked from commit 8e8ef4c38999b95cd989fc4ba7580874807e706d)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla/+/882776

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/882776
Committed: https://opendev.org/openstack/kolla/commit/6f3fe16dc2d8b1198aff0f1ba32fa5622821ed6f
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 6f3fe16dc2d8b1198aff0f1ba32fa5622821ed6f
Author: Michal Nasiadka <email address hidden>
Date: Thu May 4 15:09:17 2023 +0200

    fluentd: cap elasticsearch again

    Closes-Bug: #2018463
    Change-Id: If5bec23cb4945fea7eba87a4c9252b1ae201ce05
    (cherry picked from commit 8e8ef4c38999b95cd989fc4ba7580874807e706d)
    (cherry picked from commit fed3e69d6c8b366c7698f6ffc9a322ebaf437e34)

tags: added: in-stable-wallaby
Revision history for this message
Nicolas Froger (nfrog) wrote :
Download full text (10.0 KiB)

On Xena running the latest fluentd image, I'm having this issue now:

Running command: '/usr/sbin/td-agent -o /var/log/kolla/fluentd/fluentd.log'
/opt/td-agent/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.3.0, because faraday-1.10.3 conflicts with faraday (>= 2.0.0), faraday-excon-1.1.0 conflicts with faraday-excon (>= 2.0.0) (Gem::ConflictError)
 from /opt/td-agent/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
 from /opt/td-agent/lib/ruby/2.7.0/rubygems.rb:217:in `rescue in try_activate'
 from /opt/td-agent/lib/ruby/2.7.0/rubygems.rb:210:in `try_activate'
 from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
 from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:20:in `<top (required)>'
 from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
 from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:103:in `block in search'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:100:in `each'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:100:in `search'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/registry.rb:44:in `lookup'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:169:in `new_impl'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:114:in `new_output'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:108:in `block in configure'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:99:in `each'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/multi_output.rb:99:in `configure'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin/out_copy.rb:39:in `configure'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:187:in `configure'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:132:in `add_match'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:74:in `block in configure'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in `each'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in `configure'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:149:in `configure'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:105:in `configure'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:80:in `run_configure'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:571:in `run_supervisor'
 from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/command/fl...

Revision history for this message
Mesut Muhammet Şahin (messah) wrote :

Hi,

Is there any update? When we upgrade our cluster from wallaby to xena over quay.io container images, we also had the same problem.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 14.10.0

This issue was fixed in the openstack/kolla 14.10.0 release.

Mesut Muhammet Şahin (messah)
no longer affects: zed (Ubuntu)
Revision history for this message
Mesut Muhammet Şahin (messah) wrote (last edit ):

Hi,

This issue also happens after zed upgrade. Zed relase not seem affected project in task. When I want to add zed release in task, I made a little mistake :(

we are not deploying ElasticSearch with Kolla Ansible, we use external elasticsearch that we created an elasticsearch cluster outside of openstack. we defined our cluster with `elasticsearch_address` parameter. Our elasticsearch cluster version is 7.x but fluentd needs elasticsearch 8.x version.

Here are the logs of the fluentd container :

```
2023-08-04 09:15:36 +0300 [info]: adding match pattern="syslog.local1.**" type="copy"
2023-08-04 09:15:36 +0300 [error]: #0 config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Using Elasticsearch client 8.7.0 is not compatible for your Elasticsearch server. Please check your using elasticsearch gem version and Elasticsearch server."
2023-08-04 09:15:36 +0300 [error]: Worker 0 exited unexpectedly with status 2
2023-08-04 09:15:36 +0300 [info]: Received graceful stop
```

Images used :

quay.io/openstack.kolla/fluentd:zed-ubuntu-jammy

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla wallaby-eol

This issue was fixed in the openstack/kolla wallaby-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla xena-eol

This issue was fixed in the openstack/kolla xena-eol release.

Michal Nasiadka (mnasiadka)
Changed in kolla:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.