FWaaS: FIP namespace created after/before Firewall creation doesn't contain FW rules

[Bharath]

L3 agent is set to "dvr_snat" mode.

Steps to reproduce:
1) Create security group rules
2) Boot nova instance
3) Create floating ip on public network and associate it to the nova instance
4) Create firewall rules
5) Create firewall policy with the above rules
6) Create firewall with the above policy

Expected Result:
Both SNAT and FIP namespaces should contain the FW rules

Observed Result:
Only SNAT namespace contains the FW rules while the FIP namespace doesn't

Impact:
Due to this, the packets transferred over the external network that are destined to this instance could bypass the firewall rules using the floating ip of the instance.

Following are the commands and their output:

demofw@devstack:~/devstack$ neutron router-show router1
+-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| external_gateway_info | {"network_id": "b162e6a0-ab7b-4a2f-8239-e06f1e13ccb5", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "4743e969-7a36-427e-86e0-908c767c9d12", "ip_address": "172.24.4.2"}, {"subnet_id": "8f59a96f-2a06-4973-bbf9-40ae143ec1df", "ip_address": "2001:db8::3"}]} |
| id | fb2f3983-7a9d-4bc7-812c-20f0c182ef1a |
| name | router1 |
| routes | |
| status | ACTIVE |
| tenant_id | b8e6948ab2394672b2fa603c75d02eda |
+-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

demofw@devstack:~/devstack$ neutron floatingip-create public --port-id 0d283e44-96e9-4f98-9db1-5f464f8163b8
Created a new floatingip:
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| fixed_ip_address | 10.0.0.4 |
| floating_ip_address | 172.24.4.5 |
| floating_network_id | b162e6a0-ab7b-4a2f-8239-e06f1e13ccb5 |
| id | 278316a4-3aa4-4414-812c-b909823c5915 |
| port_id | 0d283e44-96e9-4f98-9db1-5f464f8163b8 |
| router_id | fb2f3983-7a9d-4bc7-812c-20f0c182ef1a |
| status | DOWN |
| tenant_id | b8e6948ab2394672b2fa603c75d02eda |
+---------------------+--------------------------------------+

demofw@devstack:~/devstack$ nova list
+--------------------------------------+------+--------+------------+-------------+-------------------------------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+-------------------------------------------------------------------+
| 9194e3a1-4d15-4a13-a2bc-32609c1a5f23 | vm1 | ACTIVE | - | Running | private=fd54:320b:956:0:f816:3eff:fe85:8d8d, 10.0.0.4, 172.24.4.5 |
+--------------------------------------+------+--------+------------+-------------+-------------------------------------------------------------------+

demofw@devstack:~/devstack$ neutron firewall-rule-list
+--------------------------------------+-----------+--------------------------------------+----------------------+---------+
| id | name | firewall_policy_id | summary | enabled |
+--------------------------------------+-----------+--------------------------------------+----------------------+---------+
| 4056da20-96f4-4504-91ac-252dd4b86c76 | deny-icmp | 75599732-0995-43d6-a859-6995b31a5115 | ICMP, | True |
| | | | source: none(none), | |
| | | | dest: none(none), | |
| | | | deny | |
| b1f3b83f-6e34-49d6-92b0-f707e9e8ee1e | deny-http | 75599732-0995-43d6-a859-6995b31a5115 | TCP, | True |
| | | | source: none(none), | |
| | | | dest: none(80), | |
| | | | deny | |
| cba48aea-35ee-4c3b-80f0-d3799f13407f | allow-ssh | 75599732-0995-43d6-a859-6995b31a5115 | TCP, | True |
| | | | source: none(none), | |
| | | | dest: none(22), | |
| | | | allow | |
+--------------------------------------+-----------+--------------------------------------+----------------------+---------+

demofw@devstack:~/devstack$ neutron firewall-policy-list
+--------------------------------------+-----------+----------------------------------------+
| id | name | firewall_rules |
+--------------------------------------+-----------+----------------------------------------+
| 75599732-0995-43d6-a859-6995b31a5115 | policy-fw | [4056da20-96f4-4504-91ac-252dd4b86c76, |
| | | b1f3b83f-6e34-49d6-92b0-f707e9e8ee1e, |
| | | cba48aea-35ee-4c3b-80f0-d3799f13407f] |
+--------------------------------------+-----------+----------------------------------------+

demofw@devstack:~/devstack$ neutron firewall-show demo-fw
+--------------------+--------------------------------------+
| Field | Value |
+--------------------+--------------------------------------+
| admin_state_up | True |
| description | |
| firewall_policy_id | 75599732-0995-43d6-a859-6995b31a5115 |
| id | 66560c40-9fe1-410b-98ea-9367145d6692 |
| name | demo-fw |
| router_ids | fb2f3983-7a9d-4bc7-812c-20f0c182ef1a |
| status | ACTIVE |
| tenant_id | b8e6948ab2394672b2fa603c75d02eda |
+--------------------+--------------------------------------+

demofw@devstack:~/devstack$ ip netns
fip-b162e6a0-ab7b-4a2f-8239-e06f1e13ccb5
snat-fb2f3983-7a9d-4bc7-812c-20f0c182ef1a
qrouter-fb2f3983-7a9d-4bc7-812c-20f0c182ef1a
qdhcp-855d0284-89d6-4045-8355-367f97d408f3

demofw@devstack:~/devstack$ sudo ip netns exec snat-fb2f3983-7a9d-4bc7-812c-20f0c182ef1a iptables -L -n -v
Chain INPUT (policy ACCEPT 4 packets, 1284 bytes)
 pkts bytes target prot opt in out source destination
    4 1284 neutron-l3-agent-INPUT all -- * * 0.0.0.0/0 0.0.0.0/0

Chain FORWARD (policy ACCEPT 2 packets, 168 bytes)
 pkts bytes target prot opt in out source destination
    2 168 neutron-filter-top all -- * * 0.0.0.0/0 0.0.0.0/0
    2 168 neutron-l3-agent-FORWARD all -- * * 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target prot opt in out source destination
    0 0 neutron-filter-top all -- * * 0.0.0.0/0 0.0.0.0/0
    0 0 neutron-l3-agent-OUTPUT all -- * * 0.0.0.0/0 0.0.0.0/0

Chain neutron-filter-top (2 references)
 pkts bytes target prot opt in out source destination
    2 168 neutron-l3-agent-local all -- * * 0.0.0.0/0 0.0.0.0/0

Chain neutron-l3-agent-FORWARD (1 references)
 pkts bytes target prot opt in out source destination
    0 0 neutron-l3-agent-iv466560c40 all -- * sg-+ 0.0.0.0/0 0.0.0.0/0
    0 0 neutron-l3-agent-ov466560c40 all -- sg-+ * 0.0.0.0/0 0.0.0.0/0
    0 0 neutron-l3-agent-fwaas-defau all -- * sg-+ 0.0.0.0/0 0.0.0.0/0
    0 0 neutron-l3-agent-fwaas-defau all -- sg-+ * 0.0.0.0/0 0.0.0.0/0

Chain neutron-l3-agent-INPUT (1 references)
 pkts bytes target prot opt in out source destination

Chain neutron-l3-agent-OUTPUT (1 references)
 pkts bytes target prot opt in out source destination

Chain neutron-l3-agent-fwaas-defau (2 references)
 pkts bytes target prot opt in out source destination
    0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0

Chain neutron-l3-agent-iv466560c40 (1 references)
 pkts bytes target prot opt in out source destination
    0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
    0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
    0 0 DROP icmp -- * * 0.0.0.0/0 0.0.0.0/0
    0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
    0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22

Chain neutron-l3-agent-local (1 references)
 pkts bytes target prot opt in out source destination

Chain neutron-l3-agent-ov466560c40 (1 references)
 pkts bytes target prot opt in out source destination
    0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
    0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
    0 0 DROP icmp -- * * 0.0.0.0/0 0.0.0.0/0
    0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
    0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22

demofw@devstack:~/devstack$ sudo ip netns exec fip-b162e6a0-ab7b-4a2f-8239-e06f1e13ccb5 iptables -L -n -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target prot opt in out source destination
    0 0 neutron-l3-agent-INPUT all -- * * 0.0.0.0/0 0.0.0.0/0

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target prot opt in out source destination
    0 0 neutron-filter-top all -- * * 0.0.0.0/0 0.0.0.0/0
    0 0 neutron-l3-agent-FORWARD all -- * * 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target prot opt in out source destination
    0 0 neutron-filter-top all -- * * 0.0.0.0/0 0.0.0.0/0
    0 0 neutron-l3-agent-OUTPUT all -- * * 0.0.0.0/0 0.0.0.0/0

Chain neutron-filter-top (2 references)
 pkts bytes target prot opt in out source destination
    0 0 neutron-l3-agent-local all -- * * 0.0.0.0/0 0.0.0.0/0

Chain neutron-l3-agent-FORWARD (1 references)
 pkts bytes target prot opt in out source destination

Chain neutron-l3-agent-INPUT (1 references)
 pkts bytes target prot opt in out source destination

Chain neutron-l3-agent-OUTPUT (1 references)
 pkts bytes target prot opt in out source destination

Chain neutron-l3-agent-local (1 references)
 pkts bytes target prot opt in out source destination

[Sridar Kandaswamy (skandasw)]

With the earlier model and the tight coupling btwn FW and L3Agent - we had a call into the FW Agent to process namespace create/deletion. With the L3 Agent refactor, instead of hacking this in to the fip handling code - the fix should be to implement the observer hierarchy to handle notifications in fwaas.

[ZongKai LI (zongkai)]

@Sridar Kandaswamy
hi Sridar, are you still working on this, anything new?

[Sridar Kandaswamy (skandasw)]

Hi ZongKai: Yes we have this as an agenda for Mitaka, rather than deal with this just as a bug fix - we plan to refactor the code around this area on L3Agent and FWaaSAgent interaction.

[ZongKai LI (zongkai)]

Hi, Sridar.
Well, beside your plan, I did some investigation. To iptables driver scenario, seems firewall rules for DVR floating IP will be added into qrouter namespace, not fip namespace. And that will make firewall rules fail to work.

FW rules will build on FORWARD chain, but DNAT rules built on PREROUTING chain will change ingress traffic destination IP address first.

And relative code I allocated is https://github.com/openstack/neutron-fwaas/blob/master/neutron_fwaas/services/firewall/drivers/linux/iptables_fwaas.py#L89-L92 ,

        if router_info.dist_fip_count:
            # handle the fip case on n/w or compute node.
            ipt_mgrs.append({'ipt': router_info.iptables_manager,
                             'if_prefix': ROUTER_2_FIP_DEV_PREFIX})

while I think it should be:

        if router_info.dist_fip_count:
            # handle the fip case on n/w or compute node.
            ipt_mgrs.append({'ipt': router_info.fip_ns.iptables_manager,
                             'if_prefix': FIP_EXT_DEV_PREFIX}) # 'fg-', we need add this constant in this file.

I did test that in my env, works. Steps I did:
1) associate floating IP to VM,
2) create firewall rule(action=allow, protocol=icmp) for floating IP address and other firewall staff.
3) enter fip namespace, can find firewall rule for floating IP.
4) floating IP is pingable, expect result.
5) update firewall rule(action=deny), enter fip namespace, iptable rules updated.
6) floating IP is not pingable, expect result.

But I didn't test cases like:
1) create firewall before floating IP created.
2) after firewall and floating IP created, restart l3-agent.

Hope that can help you.
but I didn't test cases like restart l3-agent

[Bharath (bharathm)]

Hi Zongkai:

It's strange that you need to do the mentioned code changes to get the FIP working. When I was testing in DevStack & multi-node env without any code changes, FIP namespaces use to get the iptables rules accurately if the FW was created after the FIP namespace creation.

However this bug doesn't just affect FIP but also impact any namespace that's created after the FW creation because of the refactored L3 agent code (which, afaik, in the past use to notify FWaaS of the new namespaces created so that the iptables rules could be written to them).

As Sridhar mentioned, from discussions in IRC FWaaS meetings, we are going to implement the observer hierarchy in FWaaS that could catch the new namespace notifications and update them accordingly.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/278863

[Armando Migliaccio (armando-migliaccio)]

Please resume work or let someone else pick it up.

[bharath (bharath-7)]

will finish it off this week

[bharath (bharath-7)]

ETA : 20th-Mar

[Armando Migliaccio (armando-migliaccio)]

This could be fixed for backporting purposes, but fixing it in master does not seem worthwhile as the FWaaS v2 API means the implementation must be revised. It's now over a month since the last expiration. At some point we must admit when too late is too late.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/278863
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.