Even though Kolla services are configured to log output to file rather than
stdout, some stdout still occurs when for example the container re(starts).
Since the Docker logs are not constrained in size, they can fill up the
docker volumes drive and bring down the host. One example of when this is
particularly problematic is when Fluentd cannot parse a log message. The
warning output is written to the Docker log and in production we have seen
it eat 100GB of disk space in less than a day. We could configure Fluentd
not to do this, but the problem may still occur via another mechanism.
Change-Id: Ia6d3935263a5909c71750b34eb69e72e6e558b7a
Closes-Bug: #1794249
(cherry picked from commit bd54b99132d019b747dcd98af9371628625c8a1b)
Reviewed: https:/ /review. opendev. org/659821 /git.openstack. org/cgit/ openstack/ kolla-ansible/ commit/ ?id=354aa1d19b2 b8fbeaeda35bed1 5626bab4acfa8b
Committed: https:/
Submitter: Zuul
Branch: stable/rocky
commit 354aa1d19b2b8fb eaeda35bed15626 bab4acfa8b
Author: Doug Szumski <email address hidden>
Date: Thu Oct 11 10:12:41 2018 +0100
Constrain the size of Docker logs
Even though Kolla services are configured to log output to file rather than
stdout, some stdout still occurs when for example the container re(starts).
Since the Docker logs are not constrained in size, they can fill up the
docker volumes drive and bring down the host. One example of when this is
particularly problematic is when Fluentd cannot parse a log message. The
warning output is written to the Docker log and in production we have seen
it eat 100GB of disk space in less than a day. We could configure Fluentd
not to do this, but the problem may still occur via another mechanism.
Change-Id: Ia6d3935263a590 9c71750b34eb69e 72e6e558b7a 747dcd98af93716 28625c8a1b)
Closes-Bug: #1794249
(cherry picked from commit bd54b99132d019b