Fluentd stops working on Ubuntu deploys

[Dincer Celik]

Fluentd stops working with error below. Using stable/stein branch on Ubuntu 18.04.

  2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/fileutils.rb:230:in `mkdir' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/fileutils.rb:230:in `fu_mkdir' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/fileutils.rb:208:in `block (2 levels) in mkdir_p' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/fileutils.rb:206:in `reverse_each' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/fileutils.rb:206:in `block in mkdir_p' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/fileutils.rb:191:in `each' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/fileutils.rb:191:in `mkdir_p' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_tail.rb:192:in `start' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/root_agent.rb:203:in `block in start' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/root_agent.rb:192:in `block (2 levels) in lifecycle' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/root_agent.rb:191:in `each' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/root_agent.rb:191:in `block in lifecycle' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/root_agent.rb:178:in `each' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/root_agent.rb:178:in `lifecycle' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/root_agent.rb:202:in `start' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/engine.rb:274:in `start' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/engine.rb:219:in `run' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/supervisor.rb:805:in `run_engine' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/supervisor.rb:549:in `block in run_worker' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/supervisor.rb:730:in `main_process' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/supervisor.rb:544:in `run_worker' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/command/fluentd.rb:316:in `<top (required)>' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/bin/fluentd:8:in `<top (required)>' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in `load' 2019-07-20 05:12:57 +0000 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>' 2019-07-20 05:12:57 +0000 [error]: #0 /usr/sbin/td-agent:7:in `load' 2019-07-20 05:12:57 +0000 [error]: #0 /usr/sbin/td-agent:7:in `<main>'2019-07-20 05:12:57 +0000 [error]: #0 unexpected error error_class=Errno::EACCES error="Permission denied @ dir_s_mkdir - /var/run/fluentd" 2019-07-20 05:12:57 +0000 [error]: #0 suppressed same stacktrace2019-07-20 05:12:57 +0000 [info]: Worker 0 finished unexpectedly with status 1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/671888

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/673479

[Radosław Piliszek (yoctozepto)]

@Dincer - you sure it stopped fluentd completely? Seems like this error message is harmless here. The k-a fix made it go.

[Dincer Celik (dincercelik)]

@Radoslaw yes the containers has exited. I'll re-check and will let you know.

[Radosław Piliszek (yoctozepto)]

Asking because I reproduced the error message including the 'finished unexpectedly with status 1' part but it does not exit the container - it works fine despite these messages. Could be there is some other issue too.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/673479
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=19b345de3eae2266f13dcb3c3efa1b3c49fe892a
Submitter: Zuul
Branch: master

commit 19b345de3eae2266f13dcb3c3efa1b3c49fe892a
Author: Radosław Piliszek <email address hidden>
Date: Tue Jul 30 12:43:20 2019 +0200

    Fix fluentd monasca pos path for Debian/Ubuntu x86_64

    Change-Id: I6d205fe327f198e699519ebe9d589b9ee77a62d2
    Closes-Bug: #1837274
    Signed-off-by: Radosław Piliszek <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/673897

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/673897
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=da20cc978c1bf59c60a71779e1814125c43754ae
Submitter: Zuul
Branch: stable/stein

commit da20cc978c1bf59c60a71779e1814125c43754ae
Author: Radosław Piliszek <email address hidden>
Date: Tue Jul 30 12:43:20 2019 +0200

    Fix fluentd monasca pos path for Debian/Ubuntu x86_64

    Change-Id: I6d205fe327f198e699519ebe9d589b9ee77a62d2
    Closes-Bug: #1837274
    Signed-off-by: Radosław Piliszek <email address hidden>
    (cherry picked from commit 19b345de3eae2266f13dcb3c3efa1b3c49fe892a)

[Dincer Celik (dincercelik)]

@Radoslaw worked like a charm. Thanks.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 8.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc1 release candidate.