Even though Kolla services are configured to log output to file rather than
stdout, some stdout still occurs when for example the container re(starts).
Since the Docker logs are not constrained in size, they can fill up the
docker volumes drive and bring down the host. One example of when this is
particularly problematic is when Fluentd cannot parse a log message. The
warning output is written to the Docker log and in production we have seen
it eat 100GB of disk space in less than a day. We could configure Fluentd
not to do this, but the problem may still occur via another mechanism.
Reviewed: https:/ /review. openstack. org/609670 /git.openstack. org/cgit/ openstack/ kolla-ansible/ commit/ ?id=bd54b99132d 019b747dcd98af9 371628625c8a1b
Committed: https:/
Submitter: Zuul
Branch: master
commit bd54b99132d019b 747dcd98af93716 28625c8a1b
Author: Doug Szumski <email address hidden>
Date: Thu Oct 11 10:12:41 2018 +0100
Constrain the size of Docker logs
Even though Kolla services are configured to log output to file rather than
stdout, some stdout still occurs when for example the container re(starts).
Since the Docker logs are not constrained in size, they can fill up the
docker volumes drive and bring down the host. One example of when this is
particularly problematic is when Fluentd cannot parse a log message. The
warning output is written to the Docker log and in production we have seen
it eat 100GB of disk space in less than a day. We could configure Fluentd
not to do this, but the problem may still occur via another mechanism.
Change-Id: Ia6d3935263a590 9c71750b34eb69e 72e6e558b7a
Closes-Bug: #1794249