max_user_instances in inotify runs low on juju-db units - version 1.25.6

[Drew Freiberger]

I found on one of our clouds that is back-rev at 1.25.6 that there is an issue with running many services and running out of inotify user instances.

Issue was found when running tail -f /var/log/juju/all-machines, I got dreaded error:
        tail: inotify cannot be used, reverting to polling: Too many open files
Tried upping max_user_watches from 1024 to 1048576 (per suggestion on https://askubuntu.com/questions/250369/why-does-upstart-fail-to-start-my-juju-service-with-unknown-job-juju-ubuntu-0
Didn't help at that point, still tail -f showed the too many open files.

I upped the fs.inotify.max_user_instances from 128 to 2048 and restarted juju-db to great success.

I think juju-db may need to come with a raised max_user_instances setting to account for large cloud installs.

running ubuntu trusty with all updates (other than juju agent versions)

List of processes and filehandles pointing to inotify:

root@lexisnexis-os-2:/home/ubuntu# for foo in /proc/*/fd/*; do readlink -f $foo; done | grep inotify | sort | uniq -c | sort -nr
      8 /proc/8567/fd/anon_inode:inotify
      8 /proc/8565/fd/anon_inode:inotify
      8 /proc/7829/fd/anon_inode:inotify
      8 /proc/6676/fd/anon_inode:inotify
      8 /proc/6606/fd/anon_inode:inotify
      8 /proc/1356295/fd/anon_inode:inotify
      8 /proc/1182922/fd/anon_inode:inotify
      7 /proc/8560/fd/anon_inode:inotify
      6 /proc/8376/fd/anon_inode:inotify
      6 /proc/7601/fd/anon_inode:inotify
      6 /proc/2676260/fd/anon_inode:inotify
      6 /proc/1/fd/anon_inode:inotify
      6 /proc/1787829/fd/anon_inode:inotify
      2 /proc/7763/fd/anon_inode:inotify
      2 /proc/6737/fd/anon_inode:inotify
      1 /proc/5996/fd/anon_inode:inotify
      1 /proc/4348/fd/anon_inode:inotify
      1 /proc/4146642/fd/anon_inode:inotify
      1 /proc/4055301/fd/anon_inode:inotify
      1 /proc/4038967/fd/anon_inode:inotify
      1 /proc/37736/fd/anon_inode:inotify
      1 /proc/36633/fd/anon_inode:inotify
      1 /proc/36592/fd/anon_inode:inotify
      1 /proc/35544/fd/anon_inode:inotify
      1 /proc/34577/fd/anon_inode:inotify
      1 /proc/33171/fd/anon_inode:inotify
      1 /proc/32643/fd/anon_inode:inotify
      1 /proc/31746/fd/anon_inode:inotify
      1 /proc/31610/fd/anon_inode:inotify
      1 /proc/30852/fd/anon_inode:inotify
      1 /proc/29809/fd/anon_inode:inotify
      1 /proc/2678148/fd/anon_inode:inotify
      1 /proc/2251006/fd/anon_inode:inotify
      1 /proc/2189625/fd/anon_inode:inotify
      1 /proc/2187232/fd/anon_inode:inotify
      1 /proc/2150373/fd/anon_inode:inotify
      1 /proc/2053725/fd/anon_inode:inotify
      1 /proc/2052087/fd/anon_inode:inotify
      1 /proc/2013083/fd/anon_inode:inotify
      1 /proc/2008976/fd/anon_inode:inotify
      1 /proc/1999224/fd/anon_inode:inotify
      1 /proc/1985370/fd/anon_inode:inotify
      1 /proc/1982901/fd/anon_inode:inotify
      1 /proc/1789709/fd/anon_inode:inotify
      1 /proc/1359607/fd/anon_inode:inotify
      1 /proc/1184707/fd/anon_inode:inotify

[JuanJo Ciarlante (jjo)]

FYI smells like lp#1631038 , which looks like fixed juju 2.0+