Security groups may not work on neutron+CentOS due to wrong sysctl.conf settings

[Aleksandr Didenko]

api: '1.0'
astute_sha: ef8aa0fd0e3ce20709612906f1f0551b5682a6ce
auth_required: true
build_id: 2014-12-03_01-07-36
build_number: '48'
feature_groups:
- mirantis
- experimental
fuellib_sha: a3043477337b4a0a8fd166dc83d6cd5d504f5da8
fuelmain_sha: 7626c5aeedcde77ad22fc081c25768944697d404
nailgun_sha: 500e36d08a45dbb389bf2bd97673d9bff48ee84d
ostf_sha: 64cb59c681658a7a55cc2c09d079072a41beb346
production: docker
release: 5.1.1

Steps to reproduce:
1) Deploy HA env on CentOS with neutron-vlan
2) Create security group with custom rules
3) Run 'sysctl -p' on compute nodes
4) Start instances and check if your rules work

Actual result: they do not work. iptables also shows zero counters for bridge devices on compute node:

iptables -L -n -v | grep PHYSDEV

Similar bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=997941
https://bugs.launchpad.net/openstack-manuals/+bug/1359691

So the problem is caused by the following sysctl settings we have in /etc/sysctl.conf file for CentOS:

# Disable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables=0
net.bridge.bridge-nf-call-iptables=0
net.bridge.bridge-nf-call-arptables=0

After usual boot, when 'bridge' kernel module is loaded, it sets those net.bridge.bridge-nf-* options to '1' and everything works fine. But if someone runs 'sysctl -p' on a CentOS compute node, this will break neutron security rules.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/140401

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/140410

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140401
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=d839b1c07c72f637b3614820d251d2dda0f60885
Submitter: Jenkins
Branch: master

commit d839b1c07c72f637b3614820d251d2dda0f60885
Author: Aleksandr Didenko <email address hidden>
Date: Tue Dec 9 18:40:32 2014 +0200

    Fix sysctl settings for neutron on CentOS

    We should make sure the following settings are configured
    /etc/sysctl.conf file on compute nodes in case we use neutron:

    net.bridge.bridge-nf-call-arptables = 1
    net.bridge.bridge-nf-call-iptables = 1
    net.bridge.bridge-nf-call-ip6tables = 1

    Otherwise security groups functionality may be broken after
    executing 'sysctl -p' on compute nodes.

    DocImpact
    Closes-bug: #1400787
    Change-Id: I8582c24706c3a7253e00569eef275f116d765bca

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/140422

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140422
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=f677606638f1e39605ca3d1eb056cbb98799fd24
Submitter: Jenkins
Branch: stable/5.1

commit f677606638f1e39605ca3d1eb056cbb98799fd24
Author: Meg McRoberts <email address hidden>
Date: Tue Dec 9 10:11:56 2014 -0800

    5.1.1 Known Issue about CentOS/neutron netfilter/Security Groups

    /etc/sysctl netfilter parameters are set to 0 rather than 1
    on CentOS. It's too late to apply a fix to 5.1.1 so adding
    to the Known Issues list.

    Change-Id: I4f4e11e207bb1ce6e66e6bc526a8f62fbafa4563
    Partial-Bug: 1400787

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140410
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=b9c875a4f0bb2678efbbe8171fc117e0aab99a4a
Submitter: Jenkins
Branch: stable/5.1

commit b9c875a4f0bb2678efbbe8171fc117e0aab99a4a
Author: Aleksandr Didenko <email address hidden>
Date: Tue Dec 9 18:40:32 2014 +0200

    Fix sysctl settings for neutron on CentOS

    We should make sure the following settings are configured
    /etc/sysctl.conf file on compute nodes in case we use neutron:

    net.bridge.bridge-nf-call-arptables = 1
    net.bridge.bridge-nf-call-iptables = 1
    net.bridge.bridge-nf-call-ip6tables = 1

    Otherwise security groups functionality may be broken after
    executing 'sysctl -p' on compute nodes.

    DocImpact
    Closes-bug: #1400787
    Change-Id: I8582c24706c3a7253e00569eef275f116d765bca

[Dennis Dmitriev (ddmitriev)]

Released on 6.0:
{"build_id": "2014-12-09_22-41-06", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "49", "auth_required": true, "api": "1.0", "nailgun_sha": "22bd43b89a17843f9199f92d61fc86cb0f8772f1", "production": "docker", "fuelmain_sha": "3aab16667f47dd8384904e27f70f7a87ba15f4ee", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-09_22-41-06", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "49", "api": "1.0", "nailgun_sha": "22bd43b89a17843f9199f92d61fc86cb0f8772f1", "production": "docker", "fuelmain_sha": "3aab16667f47dd8384904e27f70f7a87ba15f4ee", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "2c99931072d951301d395ebd5bf45c8d401301bb"}}}, "fuellib_sha": "2c99931072d951301d395ebd5bf45c8d401301bb"}

[Dennis Dmitriev (ddmitriev)]

Confirmed for 5.1.1 (build #48 RC3)

[Dennis Dmitriev (ddmitriev)]

Confirmed for 5.1.2 ISO from the CI job http://jenkins-product.srt.mirantis.net:8080/view/5.1.2/job/5.1.2.all/5/

[Aleksandr Didenko (adidenko)]

On 5.1.2 #68
    "api": "1.0",
    "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce",
    "auth_required": true,
    "build_id": "2015-02-05_20-51-08",
    "build_number": "68",
    "feature_groups": [
        "mirantis"
    ],
    "fuellib_sha": "c3912b24e58e3d3a86ba77c25ee7b1ade2ea572c",
    "fuelmain_sha": "2d15e4e6da8970d1c61eebbfdfbd5f49d14b23ac",
    "nailgun_sha": "ac0b2eca001750178b3305e4a958050be5b2634a",
    "ostf_sha": "df7ea052abd77148cfd0edd453bc5ff572b82cdc",
    "production": "docker",
    "release": "5.1.2",

I see no problems with required settings on compute node:

[root@node-5 ~]# sysctl -a | grep net.bridge.bridge-nf-call
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1

And I can see non-zero counters in iptables:

   72 7167 neutron-openvswi-sg-chain all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tape4efce61-a1 --physdev-is-bridged
  104 8899 neutron-openvswi-sg-chain all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tape4efce61-a1 --physdev-is-bridged
    0 0 neutron-openvswi-oe4efce61-a all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tape4efce61-a1 --physdev-is-bridged
   72 7167 neutron-openvswi-ie4efce61-a all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tape4efce61-a1 --physdev-is-bridged
  104 8899 neutron-openvswi-oe4efce61-a all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tape4efce61-a1 --physdev-is-bridged

So please, re-check on 5.1.2