nf_conntrack sysctl settings are not applied on reboot

[Mauricio Faria de Oliveira]

The neutron-gateway charm correctly configures nf_conntrack sysctl settings
in /etc/sysctl.d/, but does not guarantee the kernel module is loaded early.

Problem: if the kernel module that provides a sysctl option(s) is not loaded,
its sysctl option(s) are not available in /proc/sys/, and thus cannot be set.

This is the case for the nf_conntrack module on boot time: it is only loaded
_after_ the system-sysctl.service (or /etc/init/kmod.conf service, on Trusty),
so any sysctl settings related to nf_conntrack are _not_ applied on reboot.

On 'juju deploy' time, its sysctl settings are applied correctly, since the
nf_conntrack module has been loaded previously (probably by firewall tools.)

But once the unit/machine reboots, they are not set again on boot.

The proposed patch introduces the 'modules' config option, with the
default value of 'nf_conntrack' and loads the modules automatically
on boot, and also right before the sysctl setting on config-changed
(should an user-specified sysctl also need kernel modules load then.)

Test-case:
---------

Deploy and relate just neutron-gateway and rabbitmq-server:

 $ juju add-model conntrack-sysctl
 $ juju deploy neutron-gateway
 $ juju deploy rabbitmq-server
 $ juju add-relation neutron-gateway:amqp rabbitmq-server

Example sysctl:

 $ juju config neutron-gateway sysctl | grep net.nf_conntrack_max
   net.nf_conntrack_max : 1000000,

Verify that the sysctl setting is correct on deploy time,
but not after reboot (it goes back to the default value.)

 $ juju run -u neutron-gateway/0 'cat /proc/sys/net/nf_conntrack_max'
 1000000

 $ juju run -u neutron-gateway/0 'reboot' # and wait for agent idle

 $ juju run -u neutron-gateway/0 'cat /proc/sys/net/nf_conntrack_max'
 65536

With the proposed patch, it continues correct after reboot:

 $ juju upgrade-charm --path ./charm-neutron-gateway/ neutron-gateway

 $ juju run -u neutron-gateway/0 'cat /proc/sys/net/nf_conntrack_max'
 1000000

 $ juju run -u neutron-gateway/0 'reboot' # and wait for agent idle

 $ juju run -u neutron-gateway/0 'cat /proc/sys/net/nf_conntrack_max'
 1000000

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/738116

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738116
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-gateway/commit/?id=4bf26683d1c503f06dbf4776ece67f90970c0c21
Submitter: Zuul
Branch: master

commit 4bf26683d1c503f06dbf4776ece67f90970c0c21
Author: Mauricio Faria de Oliveira <email address hidden>
Date: Thu Jun 25 13:01:03 2020 -0300

    Add option to auto-load kernel modules for sysctl

    The nf_conntrack module is not loaded early enough on boot,
    thus when sysctl options are applied, its settings are not.

    This results in the correct sysctl settings seen on deploy
    time (because nf_conntrack was loaded previously by others)
    but not after reboot, despite configured in /etc/sysctl.d/.

    So, insert it in /etc/modules for module auto-load on boot
    (available on Trusty via /etc/init/kmod.conf, then Xenial+
    via systemd-sysctl.service).

    Since users can configure the sysctl option and thus need
    more modules, introduce the config option 'kernel-modules'
    (with 'nf_conntrack' as default.)

    It's handled before sysctl in the config-changed hook in
    case some sysctl option(s) needs not yet loaded module(s).

    In case of failure to load modules, log a warning message.

    Closes-Bug: #1885192
    Change-Id: I661a4fe2d9284455e536b073dc93696355baf122
    Signed-off-by: Mauricio Faria de Oliveira <email address hidden>