neutron does not create the necessary iptables rules for l3 and dhcp agents when linuxbridge used

[Nikita Gerasimov]

Version: pike
openstack-neutron-11.0.0-3.el7

Config: according to https://docs.openstack.org/neutron/pike/install/install-rdo.html
ml2 linuxbridge vxlan

neutron creates rules in neutron-linuxbri-FORWARD chain only for compute ports but router and dhcp ports have no mention at all. So router and dhcp traffic remains within host bridge.

Expected:
neutron creates rules like -A neutron-linuxbri-FORWARD -m physdev --physdev-out tapb48c914e-20 --physdev-is-bridged for all agents ports in bridge.

# iptables-save
# Generated by iptables-save v1.4.21 on Thu Sep 28 18:16:57 2017
*nat
:PREROUTING ACCEPT [23760:1495817]
:INPUT ACCEPT [22739:1402147]
:OUTPUT ACCEPT [1778:116606]
:POSTROUTING ACCEPT [2260:170214]
COMMIT
# Completed on Thu Sep 28 18:16:57 2017
# Generated by iptables-save v1.4.21 on Thu Sep 28 18:16:57 2017
*mangle
:PREROUTING ACCEPT [922003:1129881715]
:INPUT ACCEPT [906034:1128976690]
:FORWARD ACCEPT [20488:1851370]
:OUTPUT ACCEPT [774093:3908358570]
:POSTROUTING ACCEPT [793969:3910141934]
COMMIT
# Completed on Thu Sep 28 18:16:57 2017
# Generated by iptables-save v1.4.21 on Thu Sep 28 18:16:57 2017
*raw
:PREROUTING ACCEPT [922261:1129974352]
:OUTPUT ACCEPT [774348:3908396136]
:neutron-linuxbri-OUTPUT - [0:0]
:neutron-linuxbri-PREROUTING - [0:0]
-A PREROUTING -j neutron-linuxbri-PREROUTING
-A OUTPUT -j neutron-linuxbri-OUTPUT
COMMIT
# Completed on Thu Sep 28 18:16:57 2017
# Generated by iptables-save v1.4.21 on Thu Sep 28 18:16:57 2017
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [27196:421070402]
:neutron-filter-top - [0:0]
:neutron-linuxbri-FORWARD - [0:0]
:neutron-linuxbri-INPUT - [0:0]
:neutron-linuxbri-OUTPUT - [0:0]
:neutron-linuxbri-local - [0:0]
:neutron-linuxbri-sg-chain - [0:0]
:neutron-linuxbri-sg-fallback - [0:0]
-A INPUT -j neutron-linuxbri-INPUT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-linuxbri-FORWARD
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-linuxbri-OUTPUT
-A neutron-filter-top -j neutron-linuxbri-local
-A neutron-linuxbri-FORWARD -m physdev --physdev-out tapb48c914e-20 --physdev-is-bridged -m comment --comment "Accept all packets when port security is disabled." -j ACCEPT
-A neutron-linuxbri-FORWARD -m physdev --physdev-in tapb48c914e-20 --physdev-is-bridged -m comment --comment "Accept all packets when port security is disabled." -j ACCEPT
-A neutron-linuxbri-INPUT -m physdev --physdev-in tapb48c914e-20 --physdev-is-bridged -m comment --comment "Accept all packets when port security is disabled." -j ACCEPT
-A neutron-linuxbri-sg-chain -j ACCEPT
-A neutron-linuxbri-sg-fallback -m comment --comment "Default drop rule for unmatched traffic." -j DROP
COMMIT
# Completed on Thu Sep 28 18:16:57 2017

# brctl show
bridge name bridge id STP enabled interfaces
brq76f218a0-55 8000.1a1da1c5730b no tap5015bfe4-c5
                                                        tapa6d0f381-b7
                                                        tapb48c914e-20
                                                        vxlan-1006
brq8856ee40-24 8000.921ccb87ce25 no tap8d487e05-d8
                                                        vxlan-1043

[Brian Haley (brian-haley)]

The l3-agent and dhcp-agent create iptables rules inside network namespaces, so won't be visible in the "root" namespace.

Is there a specific problem you're seeing?

[Nikita Gerasimov (nikita-gerasimov)]

The problem is not inside namespaces they looks fine.

In the example above tapb48c914e-20 is nova port, tap5015bfe4-c5 and tapa6d0f381-b7 dhcp and ruter ports.
Default security rule "-A FORWARD -j REJECT --reject-with icmp-host-prohibited" prevent non local traffic. So by default traffic localized inside brq76f218a0-55 and not go through vxlan-1006.
Nova port bridge this by rules in neutron-linuxbri-FORWARD chain:
-A neutron-linuxbri-FORWARD -m physdev --physdev-out tapb48c914e-20 --physdev-is-bridged -m comment --comment "Accept all packets when port security is disabled." -j ACCEPT
-A neutron-linuxbri-FORWARD -m physdev --physdev-in tapb48c914e-20 --physdev-is-bridged -m comment --comment "Accept all packets when port security is disabled." -j ACCEPT

In current state servers behind vxlan-1006 can't reach tap5015bfe4-c5 tapa6d0f381-b7 agents and tapb48c914e-20 VM didn't receive answers from remote agents which are isolated on their hosts.

I believe that agent ports connected bridge have to have same security rules as nova ports respecting to brq76f218a0-55.

I think it was work in ocata thanks to legacy nova net chains and the issue was applicable only for router HA net see 1717927 , but in pike all agent traffic locked in the host bridges.
Unfortunately I have no ocata setup to check it more closely for now.

[Brian Haley (brian-haley)]

Yes, I guess this looks like a problem, I just don't have a multi-node linuxbridge agent configuration to test with as my typical deployment is with OVS.

Do you have any time to work on a possible patch?

[Nikita Gerasimov (nikita-gerasimov)]

Unfortunately we have no resources for this in the near future.

You don't have to have multi-node setup to debug this issue. In single-node guests will not face with unreachable agents but you still could observe missing rules in neutron-linuxbri-FORWARD chain for appropriate agent ports.

[Slawek Kaplonski (slaweq)]

Hi,

I dig little bit more into this issue and I found that ports like routed or DHCP ports are filtered in neutron-server in https://github.com/openstack/neutron/blob/master/neutron/api/rpc/handlers/securitygroups_rpc.py#L88 because such ports are marked as trusted there.
Then neutron-linuxbridge agent don't get any info about such ports and is not even trying to iterate over them in https://github.com/openstack/neutron/blob/master/neutron/agent/securitygroups_rpc.py#L120
But as I'm not SG expert I don't know exactly what could be the best way to try to fix it.
@Brian: maybe You have any idea about that?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/525607

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/525607
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=97b30494a9263db684e5901113b53c25e55d1854
Submitter: Zuul
Branch: master

commit 97b30494a9263db684e5901113b53c25e55d1854
Author: Sławek Kapłoński <email address hidden>
Date: Tue Dec 5 14:37:50 2017 +0100

    Iptables firewall driver adds forward rules for trusted ports

    Iptables firewall driver can now add process trusted ports and
    adds rules for them to FORWARD chain.

    Change-Id: I67d0f17b4b56671fc2e2dd6e2fc4518dc42cd131
    Closes-Bug: #1720205

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/550782

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/550782
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4e5fb5240c90c90c3a7416f59745fa7adf1ed2a8
Submitter: Zuul
Branch: stable/queens

commit 4e5fb5240c90c90c3a7416f59745fa7adf1ed2a8
Author: Sławek Kapłoński <email address hidden>
Date: Tue Dec 5 14:37:50 2017 +0100

    Iptables firewall driver adds forward rules for trusted ports

    Iptables firewall driver can now add process trusted ports and
    adds rules for them to FORWARD chain.

    Change-Id: I67d0f17b4b56671fc2e2dd6e2fc4518dc42cd131
    Closes-Bug: #1720205
    (cherry picked from commit 97b30494a9263db684e5901113b53c25e55d1854)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/575677

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/575677
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e123bd7e1d60d399072828391534096e50b2d1cb
Submitter: Zuul
Branch: stable/pike

commit e123bd7e1d60d399072828391534096e50b2d1cb
Author: Sławek Kapłoński <email address hidden>
Date: Tue Dec 5 14:37:50 2017 +0100

    Iptables firewall driver adds forward rules for trusted ports

    Iptables firewall driver can now add process trusted ports and
    adds rules for them to FORWARD chain.

    Change-Id: I67d0f17b4b56671fc2e2dd6e2fc4518dc42cd131
    Closes-Bug: #1720205
    (cherry picked from commit 97b30494a9263db684e5901113b53c25e55d1854)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.6 release.