EKS Ubuntu docker log rotation

[suman M]

We are using AWS EKS and Ubuntu optimized AMI for EKS, ami-0fbeb45a81e0e242f. what we noticed is docker container logs are not rotated or cleaned and because of that we are seeing disk pressure on nodes. we also don't see any /etc/docker/daemon.json file for this AMI. Is this issue addressed in another version of AMI or is there any workaround. Please any thoughts would be helpful. Thanks

[Tomáš Virtus (virtustom)]

Hello Suman. Can you please share more details?

- Which region is the AMI ami-0fbeb45a81e0e242f registered to?

- Can you double-check this AMI ID is really a Ubuntu for EKS AMI?

- Is this EKS 1.22?

[suman M (sumanubuntu01)]

Thanks Tomas for getting back, please see below for more information

EKS version : 1.21
Region: US-EAST-1
AMI: ami-0fbeb45a81e0e242f
AMI NAME: ubuntu-eks/k8s_1.21/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20220201

what I see is that daemon configuration is not set that means it is using default logger which means no log rotation, which causing logs to grow and causing the disk pressure on the nodes. Please see here for docker daemon settings
https://docs.docker.com/config/containers/logging/configure/

attaching details regarding ami and logs

[suman M (sumanubuntu01)]

Hi Tom,

  Any update on this? Thanks

[Thomas Bechtold (toabctl)]

This should be fixed with the AMIs having serial 20220518.1 . Please reopen if you still have this problem.

[suman M (sumanubuntu01)]

Thank you Thomas, we will get this tested and get back to you. Appreciate your help here.