Logs sent to Monasca can be lost in Fluent
Bug #1855700 reported by
Doug Szumski
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
kolla-ansible |
Fix Released
|
Medium
|
Doug Szumski | ||
Rocky |
Won't Fix
|
Medium
|
Mark Goddard | ||
Stein |
Won't Fix
|
Medium
|
Mark Goddard | ||
Train |
Won't Fix
|
Medium
|
Mark Goddard | ||
Ussuri |
Fix Released
|
Medium
|
Doug Szumski |
Bug Description
Monasca Fluentd output plugin is configured to use an in memory buffer. If a Fluentd container is shut down, killed, crashes, etc. then logs buffered to memory may not be sent to the Monasca Log API. We should buffer to file to fix this.
Changed in kolla-ansible: | |
assignee: | nobody → Doug Szumski (dszumski) |
status: | New → In Progress |
Changed in kolla-ansible: | |
importance: | Undecided → Medium |
To post a comment you must log in.
Reviewed: https:/ /review. opendev. org/666271 /git.openstack. org/cgit/ openstack/ kolla-ansible/ commit/ ?id=5293b1294f5 a67107f247f4628 40ef145bf2ef44
Committed: https:/
Submitter: Zuul
Branch: master
commit 5293b1294f5a671 07f247f462840ef 145bf2ef44
Author: Doug Szumski <email address hidden>
Date: Wed Jun 19 11:27:14 2019 +0000
Enable buffering to file for Monasca logs
This enables buffering to file, rather than memory for Monasca logs.
A dedicated docker volume is used for the file buffer. If a post
to the Monasca Log API fails, retries will be made using an exponential
backoff algorithm with a maximum retry interval of 30mins. The maximum
interval is set relatively low to try and reduce the risk of large
buffers accumulating, and therefore the risk of overloading the Monasca
Log API.
Closes-Bug: #1855700 92d2f4960b21312 23ab5dbdbec
Change-Id: Ib5286e9dbaf2bc