fluentd crash - Gem ConflictError

[Wodel Youchi]

Hi,

After updating to the latest images of the 10th of May. fluentd is crashing with this error :

Running command: '/usr/sbin/td-agent -o /var/log/kolla/fluentd/fluentd.log'
/opt/td-agent/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.3.0, because faraday-1.10.3 conflicts with faraday (>= 2.0.0), faraday-excon-1.1.0 conflicts with faraday-excon (>= 2.0.0) (Gem::ConflictError)

Regards.

[Wodel Youchi (wodel)]

Hi,

I did some tests with td-agent-gem
I tried to uninstall faraday-2.7.4, but it turns out its needed by fluent-plugin-elasticsearch-5.3.0.

I think we have to downgrade to fluent-plugin-elasticsearch-5.2.5 to be able to uninstall faraday > 2.0.0 and so on to avoid the conflict.

Regards.

[Michal Nasiadka (mnasiadka)]

What branch/release are you using? What is the distribution?

[Wodel Youchi (wodel)]

Hi,

I am using Yoga with CentOS containers and Rocky linux 8 as os for my nodes.

Regards.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla/+/883358

[Wodel Youchi (wodel)]

Hi,

I tried the proposed fix https://review.opendev.org/c/openstack/kolla/+/883358, fluent-plugin-elasticsearch-5.3.0 is still being installed, I did confirm that using dive.

I tried this fix :

{# NOTE(frickler): Downgrading elasticsearch gems for compatibility with OSS versions of ES #}
{{ macros.install_fluent_plugins(fluentd_plugins | customizable("plugins")) }} \
    && td-agent-gem install elasticsearch:7.13.0 \
    && td-agent-gem install "fluent-plugin-elasticsearch:5.2.5" \ <---------------------
    && td-agent-gem uninstall "elasticsearch:>7.13.0" \
    && td-agent-gem uninstall "elasticsearch-api:>7.13.0" \
    && td-agent-gem uninstall "elasticsearch-transport:>7.13.0" \
    && td-agent-gem uninstall "elastic-transport:>7.13.0" \
    && td-agent-gem uninstall "fluent-plugin-elasticsearch:>=5.3.0" <----------------------

Using this, I got a working fluentd container after deployment.

Regards.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/883358
Committed: https://opendev.org/openstack/kolla/commit/64ab2e983de8bec2da394110334c88a7b35d5773
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 64ab2e983de8bec2da394110334c88a7b35d5773
Author: Michal Nasiadka <email address hidden>
Date: Wed May 17 13:47:50 2023 +0200

    fluentd: pin fluent-plugin-elasticsearch to <5.3.0

    5.3.0 requires faraday>2.0, let's pin stable branches requiring
    ElasticSearch support.

    Closes-Bug: #2019557
    Change-Id: I13fe5b46fed3f1ad14eab9fae513617a3d2053c8

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla/+/883408

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla/+/883536

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/883408
Committed: https://opendev.org/openstack/kolla/commit/39d7bc0959a6b697cfceac01fd79705ad72a10e5
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 39d7bc0959a6b697cfceac01fd79705ad72a10e5
Author: Michal Nasiadka <email address hidden>
Date: Wed May 17 13:47:50 2023 +0200

    fluentd: pin fluent-plugin-elasticsearch to <5.3.0

    5.3.0 requires faraday>2.0, let's pin stable branches requiring
    ElasticSearch support.

    Closes-Bug: #2019557
    Change-Id: I13fe5b46fed3f1ad14eab9fae513617a3d2053c8
    (cherry picked from commit 64ab2e983de8bec2da394110334c88a7b35d5773)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/883536
Committed: https://opendev.org/openstack/kolla/commit/4fccf664b4b64f2711c4eea616114ae60aa4f78c
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 4fccf664b4b64f2711c4eea616114ae60aa4f78c
Author: Michal Nasiadka <email address hidden>
Date: Wed May 17 13:47:50 2023 +0200

    fluentd: pin fluent-plugin-elasticsearch to <5.3.0

    5.3.0 requires faraday>2.0, let's pin stable branches requiring
    ElasticSearch support.

    Closes-Bug: #2019557
    Change-Id: I13fe5b46fed3f1ad14eab9fae513617a3d2053c8
    (cherry picked from commit 64ab2e983de8bec2da394110334c88a7b35d5773)

[Wodel Youchi (wodel)]

Hi,

The patch does not work, fluent-plugin-elasticsearch version 5.3.0 is still being installed and the error still there.

Latest image tested 29 May 2023.

I used this instead :

{# NOTE(frickler): Downgrading elasticsearch gems for compatibility with OSS versions of ES #}
{{ macros.install_fluent_plugins(fluentd_plugins | customizable("plugins")) }} \
    && td-agent-gem install elasticsearch:7.13.0 \
    && td-agent-gem install "fluent-plugin-elasticsearch:5.2.5" \ <---------------------
    && td-agent-gem uninstall "elasticsearch:>7.13.0" \
    && td-agent-gem uninstall "elasticsearch-api:>7.13.0" \
    && td-agent-gem uninstall "elasticsearch-transport:>7.13.0" \
    && td-agent-gem uninstall "elastic-transport:>7.13.0" \
    && td-agent-gem uninstall "fluent-plugin-elasticsearch:>=5.3.0" <----------------------

Regards.

[Sven Kieske (s-kieske)]

I have also seen other reports that the fix is incomplete, see:

https://github.com/osism/issues/issues/531

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla/+/884856

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/884856
Committed: https://opendev.org/openstack/kolla/commit/59a5de343ac3adf9463d7231fa0f5ff5f1f0a433
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 59a5de343ac3adf9463d7231fa0f5ff5f1f0a433
Author: Sven Kieske <email address hidden>
Date: Wed May 31 12:02:53 2023 +0200

    fluentd: pin td-agent to 4.4.2

    Closes-Bug: #2019557

    Change-Id: I3723e418c061c387c7f18cd33556ebb74ba1bd95
    Signed-off-by: Sven Kieske <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 14.10.0 release.

[Mesut Muhammet Şahin (messah)]

Hi,

Is there any update for Xena? When we upgrade our cluster from wallaby to xena over quay.io container images, we also had the same problem.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla wallaby-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla xena-eol release.