increase inotify watch limits

[James Page]

Quite a few apps in k8s use inotify to watch a directory or file including traefik:

2023-07-12T10:37:47.536Z [traefik] time="2023-07-12T10:37:47Z" level=debug msg="*file.Provider provider configuration: {\"directory\":\"/opt/traefik/juju\",\"watch\":true}"
2023-07-12T10:37:47.536Z [traefik] time="2023-07-12T10:37:47Z" level=error msg="Cannot start the provider *file.Provider: error creating file watcher: too many open files"

with lots of containers its really easy to hit the default inotify watch limit.

[James Page (james-page)]

Blocking limit was actually the max number of inotify instances per user - which defaults to 128

Increasing this to 512 resulted in service being restored and traefik correctly picking up configuration.

[James Page (james-page)]

sudo sysctl fs.inotify.max_user_instances=512

[James Page (james-page)]

INotify instances per user (e.g. limits specified by fs.inotify.max_user_instances):

INSTANCES USER
----------- ------------------
128 root
4 ubuntu
1 systemd-ti+
1 systemd-re+
1 nobody
1 messagebus

looks like we're hovering right on that threshold of 128 user instances

[James Page (james-page)]

reference for useful script - https://github.com/fatso83/dotfiles/blob/master/utils/scripts/inotify-consumers

[James Page (james-page)]

https://github.com/openstack-snaps/snap-openstack/pull/177

[James Page (james-page)]

Discussed with microk8s folk as well and agreed they would bump to 1024 - so we may be able to drop this delta at some point in time.