Fluentd config files are unreadable

[Doug Szumski]

Change ID I28f63ec81f1ea5bc4a213d053bfb2c04388d5925 sets Fluentd config folder permissions to 0600 which prevents Fluentd from reading the files in this directory. We want the folder to be 0700.

Someone observing this bug will see the Fluentd container start, but then not load any config. It will just hang indefinitely, not doing anything.

[Doug Szumski (dszumski)]

Currently affects master, stein and train.

[Radosław Piliszek (yoctozepto)]

How could we prevent that in the future?

Stein pending, change not yet landed.

Other branches not release - please do no reno change.

[Doug Szumski (dszumski)]

Indeed, finally a change which doesn't need a reno :)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/713417

[Doug Szumski (dszumski)]

You could have some check to make sure logs actually end up somewhere in CI. The Monasca CI job is one path. Checking fluent writes some logs to file could be another easy one.

[Mark Goddard (mgoddard)]

We discussed a while ago having an output plugin in CI that writes out anything ingested by fluentd to files. We could then check for the existence of these files, and possibly even match input and output log data.

Alternatively we know that some logs are gathered via syslog then written to a file. We could check for the existence of the file as a lighter weight check.

[Mark Goddard (mgoddard)]

HAProxy for example. We don't see a log file for it in current jobs: https://428315e8239f3b7f7084-227e8e509ae944c51d029ecb9a7574d9.ssl.cf5.rackcdn.com/713134/7/check/kolla-ansible-centos8-source/85d2ad9/primary/logs/kolla/haproxy/. With the proposed fix we do: https://de3c4139cd3b60376541-1634c0891e365f9cbd205121e56f3649.ssl.cf2.rackcdn.com/713417/2/check/kolla-ansible-centos8-source/365485c/primary/logs/kolla/haproxy/

[Radosław Piliszek (yoctozepto)]

Mark or Doug, please propose checking HAProxy log existence (and mark this bug related).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/713417
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=c92378d7881927d46ff5d485edcf25989c0c4170
Submitter: Zuul
Branch: master

commit c92378d7881927d46ff5d485edcf25989c0c4170
Author: Doug Szumski <email address hidden>
Date: Tue Mar 17 11:16:38 2020 +0000

    Make Fluentd config folders readable

    Currently, config folders lack the execute bit so Fluentd
    cannot read the config and just does nothing when it starts up. This
    change explicitly sets the execute bit on folders which need it,
    rather than doing it in a more generic way which is more risky from
    a security perspective.

    Change-Id: Ia840f4b67043df4eaa654f47673dcdc973f13d9c
    Closes-Bug: #1867754

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/713513

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/713519

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/713513
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=f80a10435448f5a5cdc16b5aab99ca84fec1d076
Submitter: Zuul
Branch: stable/train

commit f80a10435448f5a5cdc16b5aab99ca84fec1d076
Author: Doug Szumski <email address hidden>
Date: Tue Mar 17 11:16:38 2020 +0000

    Make Fluentd config folders readable

    Currently, config folders lack the execute bit so Fluentd
    cannot read the config and just does nothing when it starts up. This
    change explicitly sets the execute bit on folders which need it,
    rather than doing it in a more generic way which is more risky from
    a security perspective.

    Change-Id: Ia840f4b67043df4eaa654f47673dcdc973f13d9c
    Closes-Bug: #1867754
    (cherry picked from commit c92378d7881927d46ff5d485edcf25989c0c4170)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/713519
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=06fbffe44025919761252cba50dbaef74f95df98
Submitter: Zuul
Branch: stable/stein

commit 06fbffe44025919761252cba50dbaef74f95df98
Author: Will Szumski <email address hidden>
Date: Thu Feb 6 17:10:06 2020 +0000

    Combined fluentd fixes

    1. Delete stale fluent config on restart

    We already only include .conf files in fluent.conf:

    (fluentd)[fluentd@cpu-e-1041 /etc/fluentd]$ cat fluent.conf
    @include input/*.conf
    @include filter/*.conf
    @include format/*.conf
    @include output/*.conf

    so this change should not cause ill effect. This works because of the
    merge option in config files:

    merge: merges the source directory into the target directory instead of
    replacing it. Boolean, defaults to false.

    see https://docs.openstack.org/kolla/latest/admin/kolla_api.html#kolla-api-external-config

    Original Change-Id: I28f63ec81f1ea5bc4a213d053bfb2c04388d5925
    Closes-Bug: #1862211
    (cherry picked from commit e7870e9df9e416b1b342ad6264bb66378fe62b3f)

    2. Make Fluentd config folders readable

    Currently, config folders lack the execute bit so Fluentd
    cannot read the config and just does nothing when it starts up. This
    change explicitly sets the execute bit on folders which need it,
    rather than doing it in a more generic way which is more risky from
    a security perspective.

    Change-Id: Ia840f4b67043df4eaa654f47673dcdc973f13d9c
    Closes-Bug: #1867754
    (cherry picked from commit c92378d7881927d46ff5d485edcf25989c0c4170)