neutron fwaas2 l3 - inconsistent order of jump rules

[Victor Serbu]

Scenario:
I have a router with 2 internal interfaces.
each port of router is assigned to a different firewall group (port1 in firewall group fg1, port2 in firewall group fg2).
If I remove port1 from fg1, l3 agent remove port1 rules from iptables neutron-l3-agent-FORWARD chain.
But then if I insert again port1 in fg1, port1 firewall rules are inserted at the end of iptables neutron-l3-agent-FORWARD chain.

#initial
#port1 in fg1, port2 in fg2
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:neutron-filter-top - [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-local - [0:0]
:neutron-l3-agent-scope - [0:0]
:neutron-l3-agent-accepted - [0:0]
:neutron-l3-agent-dropped - [0:0]
:neutron-l3-agent-rejected - [0:0]
:neutron-l3-agent-fwaas-defau - [0:0]
:neutron-l3-agent-iv4a43875cf - [0:0]
:neutron-l3-agent-ov4a43875cf - [0:0]
:neutron-l3-agent-iv451003ddf - [0:0]
:neutron-l3-agent-ov451003ddf - [0:0]
-A INPUT -j neutron-l3-agent-INPUT
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-l3-agent-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A neutron-filter-top -j neutron-l3-agent-local
-A neutron-l3-agent-FORWARD -j neutron-l3-agent-scope
-A neutron-l3-agent-FORWARD -o qr-4fd4f4f5-60 -j neutron-l3-agent-iv4a43875cf
-A neutron-l3-agent-FORWARD -i qr-4fd4f4f5-60 -j neutron-l3-agent-ov4a43875cf
-A neutron-l3-agent-FORWARD -o qr-16bfccec-5c -j neutron-l3-agent-iv451003ddf
-A neutron-l3-agent-FORWARD -i qr-16bfccec-5c -j neutron-l3-agent-ov451003ddf
-A neutron-l3-agent-FORWARD -o qr-16bfccec-5c -j neutron-l3-agent-fwaas-defau
-A neutron-l3-agent-FORWARD -i qr-16bfccec-5c -j neutron-l3-agent-fwaas-defau
-A neutron-l3-agent-INPUT -m mark --mark 0x1/0xffff -j ACCEPT
-A neutron-l3-agent-INPUT -p tcp -m tcp --dport 9697 -j DROP
-A neutron-l3-agent-scope -o qr-16bfccec-5c -m mark ! --mark 0x4000000/0xffff0000 -j DROP
-A neutron-l3-agent-scope -o qr-4fd4f4f5-60 -m mark ! --mark 0x4000000/0xffff0000 -j DROP
-A neutron-l3-agent-accepted -j ACCEPT
-A neutron-l3-agent-dropped -j DROP
-A neutron-l3-agent-rejected -j REJECT --reject-with icmp-port-unreachable
-A neutron-l3-agent-fwaas-defau -j neutron-l3-agent-dropped
-A neutron-l3-agent-iv4a43875cf -m state --state INVALID -j neutron-l3-agent-dropped
-A neutron-l3-agent-iv4a43875cf -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-iv4a43875cf -s 10.170.0.0/24 -d 10.160.0.0/24 -j neutron-l3-agent-accepted
-A neutron-l3-agent-ov4a43875cf -m state --state INVALID -j neutron-l3-agent-dropped
-A neutron-l3-agent-ov4a43875cf -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-iv451003ddf -m state --state INVALID -j neutron-l3-agent-dropped
-A neutron-l3-agent-iv451003ddf -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-ov451003ddf -m state --state INVALID -j neutron-l3-agent-dropped
-A neutron-l3-agent-ov451003ddf -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-ov451003ddf -s 10.170.0.128/25 -d 10.160.0.128/25 -j neutron-l3-agent-accepted
COMMIT

#remove port1 from fg1
iptables-save
....
-A neutron-filter-top -j neutron-l3-agent-local
-A neutron-l3-agent-FORWARD -j neutron-l3-agent-scope
-A neutron-l3-agent-FORWARD -o qr-16bfccec-5c -j neutron-l3-agent-iv451003ddf
-A neutron-l3-agent-FORWARD -i qr-16bfccec-5c -j neutron-l3-agent-ov451003ddf
-A neutron-l3-agent-INPUT -m mark --mark 0x1/0xffff -j ACCEPT
-A neutron-l3-agent-INPUT -p tcp -m tcp --dport 9697 -j DROP
-A neutron-l3-agent-scope -o qr-16bfccec-5c -m mark ! --mark 0x4000000/0xffff0000 -j DROP
-A neutron-l3-agent-scope -o qr-4fd4f4f5-60 -m mark ! --mark 0x4000000/0xffff0000 -j DROP
-A neutron-l3-agent-accepted -j ACCEPT
-A neutron-l3-agent-dropped -j DROP
-A neutron-l3-agent-rejected -j REJECT --reject-with icmp-port-unreachable
-A neutron-l3-agent-iv451003ddf -m state --state INVALID -j neutron-l3-agent-dropped
-A neutron-l3-agent-iv451003ddf -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-ov451003ddf -m state --state INVALID -j neutron-l3-agent-dropped
-A neutron-l3-agent-ov451003ddf -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-ov451003ddf -s 10.170.0.128/25 -d 10.160.0.128/25 -j neutron-l3-agent-accepted
COMMIT

#insert again port1 in fg1
iptables-save
....
-A neutron-l3-agent-FORWARD -j neutron-l3-agent-scope
-A neutron-l3-agent-FORWARD -o qr-16bfccec-5c -j neutron-l3-agent-iv451003ddf
-A neutron-l3-agent-FORWARD -i qr-16bfccec-5c -j neutron-l3-agent-ov451003ddf
-A neutron-l3-agent-FORWARD -o qr-4fd4f4f5-60 -j neutron-l3-agent-iv4a43875cf
-A neutron-l3-agent-FORWARD -i qr-4fd4f4f5-60 -j neutron-l3-agent-ov4a43875cf
-A neutron-l3-agent-FORWARD -o qr-4fd4f4f5-60 -j neutron-l3-agent-fwaas-defau
-A neutron-l3-agent-FORWARD -i qr-4fd4f4f5-60 -j neutron-l3-agent-fwaas-defau
-A neutron-l3-agent-INPUT -m mark --mark 0x1/0xffff -j ACCEPT
-A neutron-l3-agent-INPUT -p tcp -m tcp --dport 9697 -j DROP
-A neutron-l3-agent-scope -o qr-16bfccec-5c -m mark ! --mark 0x4000000/0xffff0000 -j DROP
-A neutron-l3-agent-scope -o qr-4fd4f4f5-60 -m mark ! --mark 0x4000000/0xffff0000 -j DROP
-A neutron-l3-agent-accepted -j ACCEPT
-A neutron-l3-agent-dropped -j DROP
-A neutron-l3-agent-rejected -j REJECT --reject-with icmp-port-unreachable
-A neutron-l3-agent-iv451003ddf -m state --state INVALID -j neutron-l3-agent-dropped
-A neutron-l3-agent-iv451003ddf -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-ov451003ddf -m state --state INVALID -j neutron-l3-agent-dropped
-A neutron-l3-agent-ov451003ddf -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-ov451003ddf -s 10.170.0.128/25 -d 10.160.0.128/25 -j neutron-l3-agent-accepted
-A neutron-l3-agent-fwaas-defau -j neutron-l3-agent-dropped
-A neutron-l3-agent-iv4a43875cf -m state --state INVALID -j neutron-l3-agent-dropped
-A neutron-l3-agent-iv4a43875cf -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-ov4a43875cf -m state --state INVALID -j neutron-l3-agent-dropped
-A neutron-l3-agent-ov4a43875cf -m state --state RELATED,ESTABLISHED -j ACCEPT

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Victor:

Can you provide information about the versions you are using? And the OS and the iptables version.

This is a bug for FWaaS, that is no longer maintained [1].

Regards.

[1]https://opendev.org/openstack/neutron-fwaas/src/branch/master/README.rst

[Victor Serbu (victor2709)]

Hello
I have openstack train on centos8 installed using kayobe

(neutron-l3-agent)[root@testnetwork /]# rpm -qa | grep neutron
python3-neutron-15.3.0-1.el8.noarch
python3-neutron-fwaas-15.0.1-1.el8.noarch
python3-neutron-lib-1.29.1-1.el8.noarch
python3-neutron-dynamic-routing-15.0.0-1.el8.noarch
openstack-neutron-vpnaas-15.0.0-1.el8.noarch
openstack-neutron-fwaas-15.0.1-1.el8.noarch
openstack-neutron-ml2-15.3.0-1.el8.noarch
openstack-neutron-common-15.3.0-1.el8.noarch
python3-neutron-vpnaas-15.0.0-1.el8.noarch
openstack-neutron-15.3.0-1.el8.noarch
python3-neutronclient-6.14.1-1.el8.noarch

(neutron-l3-agent)[root@testnetwork /]# cat /etc/redhat-release
CentOS Linux release 8.3.2011
(neutron-l3-agent)[root@testnetwork /]# iptables --version
iptables v1.8.4 (nf_tables)