FWaaSv2 configures iptables with invalid port name

[Eric Miller]

This might be a duplicate of:
https://bugs.launchpad.net/neutron/+bug/1798577

however, I see the same issue with floating IPs and DVR, not just SNAT. This breaks the FWaaSv2 service, resulting in no filtering.

Deployment uses Kolla Ansible 8.0.1 (Stein) on CentOS, which installs neutron-fwaas from here:
http://tarballs.openstack.org/neutron-fwaas/neutron-fwaas-15.0.0.tar.gz

DVR is being used, but the issue appears to exist on both distributed routers and centralized routers on the network nodes (both qrouter and snat namespaces).

I am attaching a small bash script that creates a firewall rule, policy, and group to block TCP port 25 outbound. When this didn't work (outbound SMTP traffic was not blocked), I looked at the iptables rules on the network node and compute nodes.

These are the interfaces in the qrouter namespace of a test router that has the issue with fwaasv2:

2: rfp-3f6273be-2@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether be:8b:2c:40:dc:e5 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.125.2/31 scope global rfp-3f6273be-2
       valid_lft forever preferred_lft forever
    inet6 fe80::bc8b:2cff:fe40:dce5/64 scope link
       valid_lft forever preferred_lft forever

1118: qr-a418f15b-fa: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:6f:fe:2c brd ff:ff:ff:ff:ff:ff
    inet 192.168.99.254/24 brd 192.168.99.255 scope global qr-a418f15b-fa
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe6f:fe2c/64 scope link
       valid_lft forever preferred_lft forever

The is the iptables config (iptables -S), with some notes injected:

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N neutron-filter-top
-N neutron-l3-agent-FORWARD
-N neutron-l3-agent-INPUT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-accepted
-N neutron-l3-agent-dropped
-N neutron-l3-agent-fwaas-defau
-N neutron-l3-agent-iv4ff1b0489
-N neutron-l3-agent-local
-N neutron-l3-agent-ov4ff1b0489
-N neutron-l3-agent-rejected
-N neutron-l3-agent-scope
-A INPUT -j neutron-l3-agent-INPUT
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-l3-agent-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A neutron-filter-top -j neutron-l3-agent-local
-A neutron-l3-agent-FORWARD -j neutron-l3-agent-scope

# NOTE: These 4 rules have the wrong interface - note that "a418f15b-f" is
# the name of the qr interface, NOT the rfp interface (see above interface list):
-A neutron-l3-agent-FORWARD -o rfp-a418f15b-f -j neutron-l3-agent-iv4ff1b0489
-A neutron-l3-agent-FORWARD -i rfp-a418f15b-f -j neutron-l3-agent-ov4ff1b0489
-A neutron-l3-agent-FORWARD -o rfp-a418f15b-f -j neutron-l3-agent-fwaas-defau
-A neutron-l3-agent-FORWARD -i rfp-a418f15b-f -j neutron-l3-agent-fwaas-defau

-A neutron-l3-agent-INPUT -m mark --mark 0x1/0xffff -j ACCEPT
-A neutron-l3-agent-INPUT -p tcp -m tcp --dport 9697 -j DROP
-A neutron-l3-agent-accepted -j ACCEPT
-A neutron-l3-agent-dropped -j DROP
-A neutron-l3-agent-fwaas-defau -j neutron-l3-agent-dropped
-A neutron-l3-agent-iv4ff1b0489 -m state --state INVALID -j neutron-l3-agent-dropped
-A neutron-l3-agent-iv4ff1b0489 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-iv4ff1b0489 -p tcp -m tcp --dport 25 -j neutron-l3-agent-dropped
-A neutron-l3-agent-ov4ff1b0489 -m state --state INVALID -j neutron-l3-agent-dropped
-A neutron-l3-agent-ov4ff1b0489 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-ov4ff1b0489 -p tcp -m tcp --dport 25 -j neutron-l3-agent-dropped
-A neutron-l3-agent-rejected -j REJECT --reject-with icmp-port-unreachable

# NOTE: These two rules use the correct interface names:
-A neutron-l3-agent-scope -o rfp-3f6273be-2 -m mark ! --mark 0x4000000/0xffff0000 -j DROP
-A neutron-l3-agent-scope -o qr-a418f15b-fa -m mark ! --mark 0x4000000/0xffff0000 -j DROP

This obviously breaks the FWaaSv2 pretty severely (iptables simply ignores the rules with the incorrect interface names).

The same issue occurs in the SNAT namespace...

The SNAT namespace interfaces:

1115: ha-8632c48f-29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9150 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:ee:dd:91 brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-8632c48f-29
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:feee:dd91/64 scope link
       valid_lft forever preferred_lft forever

1143: sg-8f9c085a-be: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:3a:31:f8 brd ff:ff:ff:ff:ff:ff

1155: qg-8e38e9f5-28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:f9:3a:d3 brd ff:ff:ff:ff:ff:ff

The iptables (iptables -S) output - with notes injected:

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N neutron-filter-top
-N neutron-l3-agent-FORWARD
-N neutron-l3-agent-INPUT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-accepted
-N neutron-l3-agent-dropped
-N neutron-l3-agent-fwaas-defau
-N neutron-l3-agent-iv4ff1b0489
-N neutron-l3-agent-local
-N neutron-l3-agent-ov4ff1b0489
-N neutron-l3-agent-rejected
-N neutron-l3-agent-scope
-A INPUT -j neutron-l3-agent-INPUT
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-l3-agent-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A neutron-filter-top -j neutron-l3-agent-local
-A neutron-l3-agent-FORWARD -j neutron-l3-agent-scope

# NOTE: These 4 rules have the wrong interface - note that "a418f15b-fa" is
# the name of the qr interface, NOT the sg interface. It is actually the "rfp"
# interface from the qrouter namespace!
-A neutron-l3-agent-FORWARD -o sg-a418f15b-fa -j neutron-l3-agent-iv4ff1b0489
-A neutron-l3-agent-FORWARD -i sg-a418f15b-fa -j neutron-l3-agent-ov4ff1b0489
-A neutron-l3-agent-FORWARD -o sg-a418f15b-fa -j neutron-l3-agent-fwaas-defau
-A neutron-l3-agent-FORWARD -i sg-a418f15b-fa -j neutron-l3-agent-fwaas-defau
-A neutron-l3-agent-accepted -j ACCEPT
-A neutron-l3-agent-dropped -j DROP
-A neutron-l3-agent-fwaas-defau -j neutron-l3-agent-dropped
-A neutron-l3-agent-iv4ff1b0489 -m state --state INVALID -j neutron-l3-agent-dropped
-A neutron-l3-agent-iv4ff1b0489 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-iv4ff1b0489 -p tcp -m tcp --dport 25 -j neutron-l3-agent-dropped
-A neutron-l3-agent-ov4ff1b0489 -m state --state INVALID -j neutron-l3-agent-dropped
-A neutron-l3-agent-ov4ff1b0489 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-ov4ff1b0489 -p tcp -m tcp --dport 25 -j neutron-l3-agent-dropped
-A neutron-l3-agent-rejected -j REJECT --reject-with icmp-port-unreachable

# NOTE: This rule has the correct interface name:
-A neutron-l3-agent-scope -o sg-8f9c085a-be -m mark ! --mark 0x4000000/0xffff0000 -j DROP

FWaaSv2 is a pretty critical component, and I haven't found a patch that has been made after Stein (maybe I just didn't find it though).

Eric

[Eric Miller (erickmiller)]

Attaching the script I mentioned that includes the commands that create the firewall rule, policy, and group.

[Eric Miller (erickmiller)]

Sorry - correction on the version of FWaaS:

Deployment uses Kolla Ansible 8.0.1 (Stein) on CentOS, which installs neutron-fwaas from here:
http://tarballs.openstack.org/neutron-fwaas/neutron-fwaas-14.0.0.tar.gz

[Eric Miller (erickmiller)]

Adding the Kolla Ansible project since it appears that FWaaSv2 is broken when using DVR. Kolla Ansible may want to have a flag to prevent FWaaSv2 from installing if DVR is also being used.

Hopefully there is a quick fix, but it doesn't look like it after reviewing the code.

[Ryan Tidwell (ryan-tidwell)]

I don't think this is a duplicate of https://bugs.launchpad.net/neutron/+bug/1798577, but it is very similar and the two issues may be linked. This could be the more generic case of the issue.

[Michal Nasiadka (mnasiadka)]

bug report is for Stein and Stein is EOL - please test out on newer version.