role openstack_hosts fails to add br_netfilter to /etc/modules

[Michael Gugino]

When running the role openstack_hosts against a target host, br_netfilter is not loaded. This appears to be due to a commit in v3.17 which changed the kernel module netfilter's behavior detailed in http://askubuntu.com/a/645643
The text of this commit message is "netfilter: bridge: move br_netfilter out of the core"

Ubuntu 14.04 was originally released with kernel 3.13, however the most recent iteration ships with kernel 3.19. This bug most likely only effects Ubuntu releases (and other releases potentially in the future) kernel version 3.19 and later.

As a result, the task 'Adding new system tuning' fails with the following messages:
---------------
  failed: [compute1] => (item={'value': 0, 'key': 'net.bridge.bridge-nf-call-ip6tables'}) => {"failed": true, "item": {"key": "net.bridge.bridge-nf-call-ip6tables", "value": 0}}
  msg: setting net.bridge.bridge-nf-call-ip6tables failed: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory

  ...ignoring
  ok: [controller1] => (item={'value': 0, 'key': 'net.bridge.bridge-nf-call-ip6tables'})
  failed: [compute1] => (item={'value': 0, 'key': 'net.bridge.bridge-nf-call-iptables'}) => {"failed": true, "item": {"key": "net.bridge.bridge-nf-call-iptables", "value": 0}}
  msg: setting net.bridge.bridge-nf-call-iptables failed: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory

  ...ignoring
  ok: [controller1] => (item={'value': 0, 'key': 'net.bridge.bridge-nf-call-iptables'})
  failed: [compute1] => (item={'value': 0, 'key': 'net.bridge.bridge-nf-call-arptables'}) => {"failed": true, "item": {"key": "net.bridge.bridge-nf-call-arptables", "value": 0}}
  msg: setting net.bridge.bridge-nf-call-arptables failed: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-arptables: No such file or directory

  ...ignoring
---------------

The potential impact of this bug is that by default, the aforementioned values default to 1 instead of 0. Thus, if the module is loaded after the system calls 'sysctl -p', the correct values of 0 will not be applied. This will result in the lxc_host's iptables filtering lxc_container traffic inadvertently. More details of potential impact can be found here: http://wiki.libvirt.org/page/Net.bridge-nf-call_and_sysctl.conf

Since this module was previously loaded automatically when a bridge was created, I believe we should add this module to roles and playbooks as necessary to correct the condition.

I believe this bug may also be a candidate for back-porting.

[Michael Gugino (gugino-michael)]

I submitted proposed fix https://review.openstack.org/#/c/266021/ for review.

[Jean-Philippe Evrard (jean-philippe-evrard)]

A colleague of mine claims to have the same issue on liberty. Maybe it should be worth a cherry pick in other branches.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/266021
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-openstack_hosts/commit/?id=45d89c2bb9fbc478eb4223516f572cf34ca1ca40
Submitter: Jenkins
Branch: master

commit 45d89c2bb9fbc478eb4223516f572cf34ca1ca40
Author: Michael Gugino <email address hidden>
Date: Mon Jan 11 14:59:00 2016 -0500

    Add missing kernel module br_netfilter to openstack_hosts role

    The openstack_hosts role contains a list of several kernel
    modules which are necessary for the successful deployment of lxc
    containers. The role also applies certain kernel parameters via
    sysctl, including 3 parameters related to to the kernel module
    br_netfilter. These parameters are:

    - { key: 'net.bridge.bridge-nf-call-ip6tables', value: 0 }
    - { key: 'net.bridge.bridge-nf-call-iptables', value: 0 }
    - { key: 'net.bridge.bridge-nf-call-arptables', value: 0 }

    The list of modules and parameters can be found in the file
    openstack-ansible-openstack_hosts/defaults/main.yml

    A kernel change committed in version 3.17 disabled the module
    br_netfilter from being loaded automatically. The most recent release
    of Ubuntu 14.04 LTS utilizes kernel version 3.19. This results in
    the aforementioned parameters failing to apply successfully, as well
    as sysctl failing to properly parse /etc/sysctl.conf.

    As a result, the task 'Adding new system tuning' fails.

    This change adds br_netfilter to the variable
    openstack_host_kernel_modules in
    openstack-ansible-openstack_hosts/defaults/main.yml to correct the bug
    introduced by a change in kernel behavior upstream.

    This change will allow the module br_netfilter to be loaded via
    modprobe, as well as inserted into /etc/modules. This will allow the
    aforementioned parameters to be applied correctly.

    Change-Id: I76bfd447e0f9747679362775c44c9f84793a78f5
    Closes-Bug: #1532917

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/267013
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=cb43901c5aaa3872fba5055c0f600a290cd267a4
Submitter: Jenkins
Branch: liberty

commit cb43901c5aaa3872fba5055c0f600a290cd267a4
Author: Michael Gugino <email address hidden>
Date: Wed Jan 13 11:31:41 2016 -0500

    Add missing kernel module br_netfilter to openstack_hosts role

    The openstack_hosts role contains a list of several kernel
    modules which are necessary for the successful deployment of lxc
    containers. The role also applies certain kernel parameters via
    sysctl, including 3 parameters related to to the kernel module
    br_netfilter. These parameters are:

    - { key: 'net.bridge.bridge-nf-call-ip6tables', value: 0 }
    - { key: 'net.bridge.bridge-nf-call-iptables', value: 0 }
    - { key: 'net.bridge.bridge-nf-call-arptables', value: 0 }

    The list of modules and parameters can be found in the file
    openstack-ansible-openstack_hosts/defaults/main.yml

    A kernel change committed in version 3.17 disabled the module
    br_netfilter from being loaded automatically. The most recent release
    of Ubuntu 14.04 LTS utilizes kernel version 3.19. This results in
    the aforementioned parameters failing to apply successfully, as well
    as sysctl failing to properly parse /etc/sysctl.conf.

    As a result, the task 'Adding new system tuning' fails.

    This change adds br_netfilter to the variable
    openstack_host_kernel_modules in
    openstack-ansible-openstack_hosts/defaults/main.yml to correct the bug
    introduced by a change in kernel behavior upstream.

    This change will allow the module br_netfilter to be loaded via
    modprobe, as well as inserted into /etc/modules. This will allow the
    aforementioned parameters to be applied correctly.

    The br_netfilter kernel module is only available for kernels above 3.18.
    This patch implements a check for the kernel version and only adds that
    module if the current kernel is at the correct version or above.

    This is a combined backport of:
    - https://review.openstack.org/266021
    - https://review.openstack.org/266530

    Change-Id: Iab644667fe879e43b2e3c7d4c2c65cd15ca2c714
    Closes-Bug: #1532917

[Jesse Pretorius (jesse-pretorius)]

Removing Kilo as a target series. This won't be backported to Kilo unless specifically requested.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 12.0.8 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 12.0.9 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 12.0.11 release.