[cs rev=261] "sysctl -p" fails on lxd

[Alvaro Uria]

On unprivileged containers (default), neutron-gateway goes into error state because it can't change kernel parameters.

root@juju-27cb94-18:~# cat /etc/sysctl.d/50-quantum-gateway.conf
net.ipv4.neigh.default.gc_thresh1=128
net.ipv4.neigh.default.gc_thresh2=28672
net.ipv4.neigh.default.gc_thresh3=32768
net.ipv6.neigh.default.gc_thresh1=128
net.ipv6.neigh.default.gc_thresh2=28672
net.ipv6.neigh.default.gc_thresh3=32768
net.nf_conntrack_max=1000000
net.netfilter.nf_conntrack_buckets=204800
net.netfilter.nf_conntrack_max=1000000

root@juju-27cb94-18:~# sysctl -p /etc/sysctl.d/50-quantum-gateway.conf
sysctl: cannot stat /proc/sys/net/ipv4/neigh/default/gc_thresh1: No such file or directory
sysctl: cannot stat /proc/sys/net/ipv4/neigh/default/gc_thresh2: No such file or directory
sysctl: cannot stat /proc/sys/net/ipv4/neigh/default/gc_thresh3: No such file or directory
sysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh1: No such file or directory
sysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh2: No such file or directory
sysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh3: No such file or directory
sysctl: cannot stat /proc/sys/net/nf_conntrack_max: No such file or directory
sysctl: cannot stat /proc/sys/net/netfilter/nf_conntrack_buckets: No such file or directory
sysctl: cannot stat /proc/sys/net/netfilter/nf_conntrack_max: No such file or directory

2019-04-25 08:04:17 DEBUG juju-log Updating sysctl_file: /etc/sysctl.d/50-quantum-gateway.conf values: {'net.ipv4.neigh.default.gc_thresh1': 128, 'net.ipv4.neigh.default.gc_thresh2': 28672, 'net.ipv4.neigh.defau
lt.gc_thresh3': 32768, 'net.ipv6.neigh.default.gc_thresh1': 128, 'net.ipv6.neigh.default.gc_thresh2': 28672, 'net.ipv6.neigh.default.gc_thresh3': 32768, 'net.nf_conntrack_max': 1000000, 'net.netfilter.nf_conntra
ck_buckets': 204800, 'net.netfilter.nf_conntrack_max': 1000000}
2019-04-25 08:04:17 DEBUG config-changed sysctl: cannot stat /proc/sys/net/ipv4/neigh/default/gc_thresh1: No such file or directory
2019-04-25 08:04:17 DEBUG config-changed sysctl: cannot stat /proc/sys/net/ipv4/neigh/default/gc_thresh2: No such file or directory
2019-04-25 08:04:17 DEBUG config-changed sysctl: cannot stat /proc/sys/net/ipv4/neigh/default/gc_thresh3: No such file or directory
2019-04-25 08:04:17 DEBUG config-changed sysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh1: No such file or directory
2019-04-25 08:04:17 DEBUG config-changed sysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh2: No such file or directory
2019-04-25 08:04:17 DEBUG config-changed sysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh3: No such file or directory
2019-04-25 08:04:17 DEBUG config-changed sysctl: cannot stat /proc/sys/net/nf_conntrack_max: No such file or directory
2019-04-25 08:04:17 DEBUG config-changed sysctl: cannot stat /proc/sys/net/netfilter/nf_conntrack_buckets: No such file or directory
2019-04-25 08:04:17 DEBUG config-changed sysctl: cannot stat /proc/sys/net/netfilter/nf_conntrack_max: No such file or directory
2019-04-25 08:04:17 DEBUG config-changed Traceback (most recent call last):
2019-04-25 08:04:17 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/config-changed", line 362, in <module>
2019-04-25 08:04:17 DEBUG config-changed hooks.execute(sys.argv)
2019-04-25 08:04:17 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/core/hookenv.py", line 914, in execute
2019-04-25 08:04:17 DEBUG config-changed self._hooks[hook_name]()
2019-04-25 08:04:17 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1546, in wrapped_f
2019-04-25 08:04:17 DEBUG config-changed stopstart, restart_functions)
2019-04-25 08:04:17 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/core/host.py", line 741, in restart_on_change_helper
2019-04-25 08:04:17 DEBUG config-changed r = lambda_f()
2019-04-25 08:04:17 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1545, in <lambda>
2019-04-25 08:04:17 DEBUG config-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache'],
2019-04-25 08:04:17 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 93, in _harden_inner2
2019-04-25 08:04:17 DEBUG config-changed return f(*args, **kwargs)
2019-04-25 08:04:17 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/config-changed", line 128, in config_changed
2019-04-25 08:04:17 DEBUG config-changed '/etc/sysctl.d/50-quantum-gateway.conf')
2019-04-25 08:04:17 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/core/sysctl.py", line 65, in create
2019-04-25 08:04:17 DEBUG config-changed check_call(call)
2019-04-25 08:04:17 DEBUG config-changed File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
2019-04-25 08:04:17 DEBUG config-changed raise CalledProcessError(retcode, cmd)
2019-04-25 08:04:17 DEBUG config-changed subprocess.CalledProcessError: Command '['sysctl', '-p', '/etc/sysctl.d/50-quantum-gateway.conf']' returned non-zero exit status 255.
2019-04-25 08:04:17 ERROR juju.worker.uniter.operation runhook.go:132 hook "config-changed" failed: exit status 1

[Peter Sabaini (peter-sabaini)]

This is a regression resulting from Bug #1780348

[Alvaro Uria (aluria)]

I can be worked around by setting:
juju config neutron-gateway sysctl=''

However, I think host.is_container() should validate if such default is possible. FWIW, I have tried to enable security.privileged and security.nesting on the container running neutron-gateway, and it didn't work (so it seems the unit should get blocked if sysctl is not empty).

[Peter Sabaini (peter-sabaini)]

Also note the workaround mentioned in:
https://ask.openstack.org/en/question/121359/error-neutron-gateway-in-openstack-on-lxd/?answer=121385#post-id-121385

-- setting sysctl to the empty string explicitly.

juju config neutron-gateway sysctl= juju resolve neutron-gateway/0

[Peter Sabaini (peter-sabaini)]

Heh, crossed wires. Agreed the charm should handle this situation. Possibly it could include an lxd profile as per https://docs.jujucharms.com/2.5/en/clouds-lxd-advanced , however the default sysctls don't include the ones set in bug #1780348 so ideally that should be extended

[Alex Kavanagh (ajkavanagh)]

This sounds like a documentation issue and config validation issue associated with config('sysctl'). Marking as low priority as there is a workaround.