firewall rules on DVR FIP fails to work for ingress traffic

[ZongKai LI]

=====
my env
=====
controller +network node(dvr_snat) + 2 compute nodes(dvr)
DVR: enable DVR when using devstack to deploy this env
FWaaS: manually git clone neutron-fwaas and to configure, using iptables as driver

====
steps
====
1) create net, subnet, boot VM-1 on CN-1, VM-2 on CN-2, create router, and attach subnet onto router.
2) create external network, set as router gateway net, create 2 floating IPs and associate to two VMs.
3) confirm DVR FIP works: fip ns created, iptable rules updated in qrouter ns, two VMs are pingable by floating IP.
floating IP like: 192.168.0.4 and 192.168.0.5
4) create firewall rules, firewall policy and create firewall on router.
firewall rule like:
    fw-r1: ICMP, source: 192.168.0.184/29(none), dest: 192.168.0.0/28(none), allow
    fw-r2: ICMP, source: 192.168.0.0/28(none), dest: 192.168.0.184/29(none), allow
5) confirm firewall rules updated in qrouter ns.
6) on host who has IP like 192.168.0.190, try to ping floating IPs mentioned in step 3.
expected: floating IPs should be pingable (for IP 192.168.0.190 is in 192.168.0.184/29, and two firewall rules allows)
observed: no response, "100% packet loss" from ping command. floating IP fail to ping.

========
more details
========
----------------------------
firewall iptable rules:
----------------------------
-A INPUT -j neutron-l3-agent-INPUT
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-l3-agent-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A neutron-filter-top -j neutron-l3-agent-local
-A neutron-l3-agent-FORWARD -o rfp-+ -j neutron-l3-agent-iv4322a9b15
-A neutron-l3-agent-FORWARD -i rfp-+ -j neutron-l3-agent-ov4322a9b15
-A neutron-l3-agent-FORWARD -o rfp-+ -j neutron-l3-agent-fwaas-defau
-A neutron-l3-agent-FORWARD -i rfp-+ -j neutron-l3-agent-fwaas-defau
-A neutron-l3-agent-INPUT -m mark --mark 0x1/0xffff -j ACCEPT
-A neutron-l3-agent-INPUT -p tcp -m tcp --dport 9697 -j DROP
-A neutron-l3-agent-fwaas-defau -j DROP
-A neutron-l3-agent-iv4322a9b15 -m state --state INVALID -j DROP
-A neutron-l3-agent-iv4322a9b15 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-iv4322a9b15 -s 192.168.0.0/28 -d 192.168.0.184/29 -p icmp -j ACCEPT
-A neutron-l3-agent-iv4322a9b15 -s 192.168.0.184/29 -d 192.168.0.0/28 -p icmp -j ACCEPT
-A neutron-l3-agent-ov4322a9b15 -m state --state INVALID -j DROP
-A neutron-l3-agent-ov4322a9b15 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-ov4322a9b15 -s 192.168.0.0/28 -d 192.168.0.184/29 -p icmp -j ACCEPT
-A neutron-l3-agent-ov4322a9b15 -s 192.168.0.184/29 -d 192.168.0.0/28 -p icmp -j ACCEPT

-----------------------------------
DVR FIP nat iptable rules:
-----------------------------------
1) for 192.168.0.4:
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 192.168.0.4/32 -j DNAT --to-destination 20.0.1.7
-A neutron-l3-agent-POSTROUTING ! -i rfp-4bf3186c-d ! -o rfp-4bf3186c-d -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 192.168.0.4/32 -j DNAT --to-destination 20.0.1.7
-A neutron-l3-agent-float-snat -s 20.0.1.7/32 -j SNAT --to-source 192.168.0.4
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat

2) for 192.168.0.5:
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 192.168.0.5/32 -j DNAT --to-destination 20.0.1.6
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 192.168.0.5/32 ! -i qr-+ -j DNAT --to-destination 20.0.1.6
-A neutron-l3-agent-float-snat -s 20.0.1.6/32 -j SNAT --to-source 192.168.0.5
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat

----------------------------------------------------------------------
tcpdump result: (192.168.0.190 ping 192.168.0.4)
----------------------------------------------------------------------
1) on fg in fip ns, ingress traffic caught:
fa:16:3e:b3:3e:8c > fa:16:3e:9d:ea:ed, ethertype IPv4 (0x0800), length 98: 192.168.0.190 > 192.168.0.4: ICMP echo request, id 28356, seq 31, length 64
and fg:
40: fg-59c9ce49-3a: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether fa:16:3e:9d:ea:ed brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.133/24 brd 192.168.0.255 scope global fg-59c9ce49-3a
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe9d:eaed/64 scope link
       valid_lft forever preferred_lft forever

2) on fpr in fip ns, ingress traffic caught:
a2:c4:79:bf:c0:1a > aa:ab:39:2f:ac:df, ethertype IPv4 (0x0800), length 98: 192.168.0.190 > 192.168.0.4: ICMP echo request, id 28356, seq 159, length 64
and fpr:
13: fpr-4bf3186c-d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether a2:c4:79:bf:c0:1a brd ff:ff:ff:ff:ff:ff
    inet 169.254.31.143/31 scope global fpr-4bf3186c-d
       valid_lft forever preferred_lft forever
    inet6 fe80::a0c4:79ff:febf:c01a/64 scope link
       valid_lft forever preferred_lft forever
and rfp:
10: rfp-4bf3186c-d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether aa:ab:39:2f:ac:df brd ff:ff:ff:ff:ff:ff
    inet 169.254.31.142/31 scope global rfp-4bf3186c-d
       valid_lft forever preferred_lft forever
    inet 192.168.0.4/32 brd 192.168.0.4 scope global rfp-4bf3186c-d
       valid_lft forever preferred_lft forever
    inet6 fe80::a8ab:39ff:fe2f:acdf/64 scope link
       valid_lft forever preferred_lft forever

3) on rfp in qrouter ns, ingress traffic caught:
a2:c4:79:bf:c0:1a > aa:ab:39:2f:ac:df, ethertype IPv4 (0x0800), length 98: 192.168.0.190 > 192.168.0.4: ICMP echo request, id 28356, seq 372, length 64

4) on qr in qrouter ns, no traffic caught :(

========
idea/discuss
========
in my understanding, we have DNAT rules in PREROUTING chain, like:
-A neutron-l3-agent-PREROUTING -d 192.168.0.4/32 -j DNAT --to-destination 20.0.1.7
will change ingress traffic destination IP from 192.168.0.4(firewall know and allow) to 20.0.1.7(firewall dont know and deny by neutron-l3-agent-fwaas-defau).
So before firewall rules in FORWARD related chains like:
-A neutron-l3-agent-iv4322a9b15 -s 192.168.0.0/28 -d 192.168.0.184/29 -p icmp -j ACCEPT
-A neutron-l3-agent-iv4322a9b15 -s 192.168.0.184/29 -d 192.168.0.0/28 -p icmp -j ACCEPT
have chance to accept, the ingress traffic has already been changed, and will go to neutron-l3-agent-fwaas-defau chain and be dropped.

an idea is try to move firewall rules into fip ns(but this will raise other potential problems, for firewall rules will no longer be isolated by namespace), I tested this, it can work.

[ZongKai LI (zongkai)]

I tested this issue for ingress traffic, not sure whether will happen to egress traffic.

[Ryan Moats (rmoats)]

Was this test with tip of tree or with some other version?

[Ritesh Anand (ritesh-anand)]

Observed on master Nov 9 build.

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Since FWaaS is moving to v2 implementation and port based, this issue should get resolved with that.

[Miguel Angel Ajo (mangelajo)]

Given comment #4, and if this is not considered to be a supported feature of DVR + FWaaS at this moment, I'd say this could be moved to Won't fix.

As per @armax comments, this bug should be discussed in the DVR meeting.

Cheers.

[Ryan Moats (rmoats)]

This was discussed during the 11/12 DVR meeting: http://eavesdrop.openstack.org/meetings/neutron_dvr/2015/neutron_dvr.2015-11-11-15.00.log.html (search for the bug number). The opinion there is that this should be addressed once FWaaS V2 is implemented.

[Carl Baldwin (carl-baldwin)]

At this point, given that fwaas is totally up in the air, I don't think we're going to take on anymore DVR / FWaaS bugs.