dnsmasq hits inotify max_user_instances limit in busy neutron deployments

[Narinder Gupta]

I deployed openstack HA with openvswitch. During creating the instances we found few instances get the ip address from dhcp agent. But after some time we are seeing issue with agent with the following error.

dnsmasq: failed to create inotify: Too many open files

2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for 1346711a-66b8-4d24-81b8-e9d7a2e28248.
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/dhcp/agent.py", line 112, in call_driver
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent getattr(driver, action)(**action_kwargs)
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 210, in enable
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent self.spawn_process()
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 424, in spawn_process
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent self._spawn_or_reload_process(reload_with_HUP=False)
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 438, in _spawn_or_reload_process
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent pm.enable(reload_cfg=reload_with_HUP)
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/external_process.py", line 92, in enable
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent run_as_root=self.run_as_root)
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 927, in execute
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent log_fail_as_error=log_fail_as_error, **kwargs)
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 140, in execute
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent raise RuntimeError(msg)
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent RuntimeError: Exit code: 5; Stdin: ; Stdout: ; Stderr:
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent dnsmasq: failed to create inotify: Too many open files
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent
2016-06-16 02:57:22.712 35607 ERROR neutron.agent.dhcp.agent

root@rack-6-m1:/var/log/neutron# pastebinit neutron-dhcp-agent.log
http://paste.ubuntu.com/17389205/

pastebinit ../juju/unit-neutron-gateway-0.log
http://paste.ubuntu.com/17389255/

When i tried the same bundle with trusty then it works fine.

[Narinder Gupta (narindergupta)]

after making this change it is ok how can i do it using charm and which charm can i include in?

[Narinder Gupta (narindergupta)]

echo 256 > /proc/sys/fs/inotify/max_user_instances

[Robin Cernin (rcernin)]

Note that wont persist across reboots, and you will hit the same issue.

The following is the right way to have it runtime + persistent across reboots.

sysctl -w fs.inotify.max_user_instances=256 >> /etc/sysctl.conf

[James Page (james-page)]

The neutron-gateway charm can also do this via configuration:

  juju config neutron-gateway sysctl="{ fs.inotify.max_user_instances: 256}"

This will be persistent to disk for reboot durability.

[James Page (james-page)]

https://bugzilla.redhat.com/show_bug.cgi?id=1474515 looks similar

[James Page (james-page)]

Checking one of my own installations:

cat /proc/sys/fs/inotify/max_user_instances
1024

so I'm not quite sure how a default gets set here.

[James Page (james-page)]

and another:

$ cat /proc/sys/fs/inotify/max_user_instances
128

[James Page (james-page)]

Increasing this by default in the charm sounds sensible to me either way.

[James Page (james-page)]

That first install has up-to-date LXD:

ubuntu@juju-b88663-nova-lxd-demo-8:/etc/sysctl.d$ cat 10-lxd-inotify.conf
# Increase the user inotify instance limit to allow for about
# 100 containers to run before the limit is hit again
fs.inotify.max_user_instances = 1024

second one does not; so I think increasing this value during charm installation makes huge sense.

[James Page (james-page)]

Also https://bugzilla.redhat.com/show_bug.cgi?id=1491505

[James Page (james-page)]

Supporting change to charmhelpers:

  https://github.com/juju/charm-helpers/pull/15

[Nobuto Murata (nobuto)]

FWIW, the bump of fs.inotify.max_user_instances has been done by 2.0.10-0ubuntu1~16.04.2 for xenial on 2017-08-31.
https://launchpad.net/ubuntu/+source/lxd/2.0.10-0ubuntu1~16.04.2