Add option to specify log-opt max-size value for docker containers

[Eric Miller]

Using Rocky, 7.0.0.0rc1 release of Kolla Ansible...

This is really a wish list item, and maybe there is a way to do it already, so I thought I'd ask.

Docker logs will fill up disks pretty quickly, especially with fluentd and swift-object-server containers (from what we have seen so far).

There is an option "--log opt max-size=XXXX" as describe here:
https://docs.docker.com/config/containers/logging/json-file/

This would allow logs to roll and keep them at bay.

I think this we be added to the containers' configs at the kolla-build stage? Or is there an on-demand method for adding this option after completing a deployment?

Eric

[Eric Miller (erickmiller)]

I hadn't scrolled up to the top of that page, which indicates that Docker has a global option for enabling the JSON file logging driver by creating /etc/docker/daemon.json with:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "10"
  }
}

in it - so I am trying that now. If this works, I have answered my own question. :)

It doesn't seem to have taken effect on existing containers, after a container restart - so this may need to be added to the kolla-ansible baremetal bootstrap component here (before the "Start docker" task):
https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/baremetal/tasks/post-install.yml

Eric

[Eric Miller (erickmiller)]

This worked perfectly, resulting in rotation of logs such as the json.log files found with:
find / | grep "\-json.log" | xargs ls -al

cat << EOF > /usr/share/kolla-ansible/ansible/roles/baremetal/templates/docker_daemon_config.j2
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "10"
  }
}
EOF

and this patch to:
/usr/share/kolla-ansible/ansible/roles/baremetal/tasks/post-install.yml

88a89,102
> - name: Ensure docker config directory exists
> file:
> path: /etc/docker
> state: directory
> recurse: yes
> become: True
>
> - name: Configure docker daemon
> become: True
> template:
> src: docker_daemon_config.j2
> dest: /etc/docker/daemon.json
> register: docker_configured
>

Eric

[Eric Miller (erickmiller)]

Hi Doug, if you can tell me the proper procedure for creating a patch file for uploading to a ticket like this, I will use that from now on when submitting tickets where I have a fix.

Thanks!

Eric

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/609670

[Doug Szumski (dszumski)]

Hi Eric, nice job on the fix. Unfortunately I created a patch before I'd found your bug report which I've now submitted. Please feel free to comment on it. I think there are some merits to your approach as the config in the unit file is pretty long now.

https://review.openstack.org/#/c/609670/

Anyhow, in future, there are a couple of useful links, first the contributor guide (one of the things you will need to do is sign the contributor agreement):

https://docs.openstack.org/contributors/code-and-documentation/index.html

And secondly the first contact support list:

https://wiki.openstack.org/wiki/First_Contact_SIG

Hopefully that helps.

Cheers,
Doug

[Eric Miller (erickmiller)]

Thanks Doug! I'll read and sign and see how I can help! :)

Eric

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/609670
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=bd54b99132d019b747dcd98af9371628625c8a1b
Submitter: Zuul
Branch: master

commit bd54b99132d019b747dcd98af9371628625c8a1b
Author: Doug Szumski <email address hidden>
Date: Thu Oct 11 10:12:41 2018 +0100

    Constrain the size of Docker logs

    Even though Kolla services are configured to log output to file rather than
    stdout, some stdout still occurs when for example the container re(starts).
    Since the Docker logs are not constrained in size, they can fill up the
    docker volumes drive and bring down the host. One example of when this is
    particularly problematic is when Fluentd cannot parse a log message. The
    warning output is written to the Docker log and in production we have seen
    it eat 100GB of disk space in less than a day. We could configure Fluentd
    not to do this, but the problem may still occur via another mechanism.

    Change-Id: Ia6d3935263a5909c71750b34eb69e72e6e558b7a
    Closes-Bug: #1794249

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 8.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/659821

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/659821
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=354aa1d19b2b8fbeaeda35bed15626bab4acfa8b
Submitter: Zuul
Branch: stable/rocky

commit 354aa1d19b2b8fbeaeda35bed15626bab4acfa8b
Author: Doug Szumski <email address hidden>
Date: Thu Oct 11 10:12:41 2018 +0100

    Constrain the size of Docker logs

    Even though Kolla services are configured to log output to file rather than
    stdout, some stdout still occurs when for example the container re(starts).
    Since the Docker logs are not constrained in size, they can fill up the
    docker volumes drive and bring down the host. One example of when this is
    particularly problematic is when Fluentd cannot parse a log message. The
    warning output is written to the Docker log and in production we have seen
    it eat 100GB of disk space in less than a day. We could configure Fluentd
    not to do this, but the problem may still occur via another mechanism.

    Change-Id: Ia6d3935263a5909c71750b34eb69e72e6e558b7a
    Closes-Bug: #1794249
    (cherry picked from commit bd54b99132d019b747dcd98af9371628625c8a1b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 7.1.1 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/678869

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/678869
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=403cc371cfdc852c52abf44cb82bf1b0e15bcb53
Submitter: Zuul
Branch: stable/queens

commit 403cc371cfdc852c52abf44cb82bf1b0e15bcb53
Author: Doug Szumski <email address hidden>
Date: Thu Oct 11 10:12:41 2018 +0100

    Constrain the size of Docker logs

    Even though Kolla services are configured to log output to file rather than
    stdout, some stdout still occurs when for example the container re(starts).
    Since the Docker logs are not constrained in size, they can fill up the
    docker volumes drive and bring down the host. One example of when this is
    particularly problematic is when Fluentd cannot parse a log message. The
    warning output is written to the Docker log and in production we have seen
    it eat 100GB of disk space in less than a day. We could configure Fluentd
    not to do this, but the problem may still occur via another mechanism.

    Change-Id: Ia6d3935263a5909c71750b34eb69e72e6e558b7a
    Closes-Bug: #1794249
    (cherry picked from commit bd54b99132d019b747dcd98af9371628625c8a1b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 6.2.2 release.