increase inotify limits for kubelet/cAdvisor
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Kubernetes Worker Charm |
Fix Released
|
Medium
|
Mike Wilson |
Bug Description
One of our k8s clusters, running v1.12.8, has a lot of cron jobs running, and therefore spawns a lot of pods. This seems to provoke an inotify leak somewhere in k8s that eventually causes our nodes to stop working and become NotReady. kubelet was logging this at the end of each attempt to start:
May 12 06:25:43 juju-66cffb-
May 12 06:25:43 juju-66cffb-
May 12 06:25:43 juju-66cffb-
May 12 06:25:45 juju-66cffb-
fs.inotify.
Various third parties increased these limits:
* https:/
* https:/
and it seems that cAdvisor has fixed it, and possibly the change has made it into some version of Kubernetes itself, although the precise status of the k8s issue is not entirely clear to me
* https:/
* https:/
If the fixed cAdvisor has not yet made it to all current releases, then the Juju charm should probably bump the inotify limits in the meantime.
tags: | added: sts |
Changed in charm-kubernetes-worker: | |
milestone: | none → 1.16 |
Changed in charm-kubernetes-worker: | |
status: | Fix Committed → Fix Released |
This is a configurable thing after the next stable release. It was fixed in https:/ /github. com/charmed- kubernetes/ layer-kubernete s-master- worker- base/pull/ 3.
The way it will work is `juju config sysctl="{ fs.inotify. max_user_ watches= 1048576 }"`