neutron

FWaaS: Ip tables rules do not get updated in case of distributed virtual routers (DVR)

Bug #1716401 reported by Christoph Fiehe
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
neutron
New
Undecided
Sridar Kandaswamy

Bug Description

I have set up an HA/DVR deployment of OpenStack Pike on Ubuntu 16.04 and enabled FWaaS v1. After applying the Fix from Bug #1715395, firewall rules get created in case of HA/DVR, but updates do not have any effect, e.g. when you disassociate a firewall from a distributed router.

Use Case:
1. Set up an HA/DVP deployment of OpenStack Pike.

2. Create a firewall rule.
$ neutron firewall-rule-create --name test-rule --protocol icmp --action reject
Created a new firewall_rule:
+------------------------+--------------------------------------+
| Field | Value |
+------------------------+--------------------------------------+
| action | reject |
| description | |
| destination_ip_address | |
| destination_port | |
| enabled | True |
| firewall_policy_id | |
| id | 6c2516cb-b69d-46b6-958e-e47c1cf1709e |
| ip_version | 4 |
| name | test-rule |
| position | |
| project_id | ed2d2efd86dd40e7a45491d8502318d3 |
| protocol | icmp |
| shared | False |
| source_ip_address | |
| source_port | |
| tenant_id | ed2d2efd86dd40e7a45491d8502318d3 |
+------------------------+--------------------------------------+

3. Create a firewall policy.
$ neutron firewall-policy-create --firewall-rules test-rule test-policy
Created a new firewall_policy:
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| audited | False |
| description | |
| firewall_rules | 6c2516cb-b69d-46b6-958e-e47c1cf1709e |
| id | 53a8d733-e81c-4113-9354-d40b5b426e00 |
| name | test-policy |
| project_id | ed2d2efd86dd40e7a45491d8502318d3 |
| shared | False |
| tenant_id | ed2d2efd86dd40e7a45491d8502318d3 |
+----------------+--------------------------------------+

4. Create a firewall.
$ neutron firewall-create --name test-firewall test-policy
Created a new firewall:
+--------------------+--------------------------------------+
| Field | Value |
+--------------------+--------------------------------------+
| admin_state_up | True |
| description | |
| firewall_policy_id | 53a8d733-e81c-4113-9354-d40b5b426e00 |
| id | a468caca-c555-4f89-adbc-bcdbb06a3fca |
| name | test-firewall |
| project_id | ed2d2efd86dd40e7a45491d8502318d3 |
| router_ids | |
| status | INACTIVE |
| tenant_id | ed2d2efd86dd40e7a45491d8502318d3 |
+--------------------+--------------------------------------+

5. Assign the firewall to a distributed router.
$ neutron firewall-update --router demo-router test-firewall
Updated firewall: test-firewall

6. Spawn a virtual machine and assign a floating ip.

7. Check namespaces on the compute node hosting the virtual machine.
$ ip netns
fip-4a3959c3-b011-4bd0-8f4f-f405be92d9ac
qrouter-09a379b5-907f-4e3e-b29a-8701b82f2641

8. Check ip tables rules in the router's namespace.
$ ip netns exec qrouter-09a379b5-907f-4e3e-b29a-8701b82f2641 iptables -n -L -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target prot opt in out source destination
    0 0 neutron-l3-agent-INPUT all -- * * 0.0.0.0/0 0.0.0.0/0

Chain FORWARD (policy ACCEPT 40 packets, 2400 bytes)
 pkts bytes target prot opt in out source destination
  185 11100 neutron-filter-top all -- * * 0.0.0.0/0 0.0.0.0/0
  185 11100 neutron-l3-agent-FORWARD all -- * * 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target prot opt in out source destination
    0 0 neutron-filter-top all -- * * 0.0.0.0/0 0.0.0.0/0
    0 0 neutron-l3-agent-OUTPUT all -- * * 0.0.0.0/0 0.0.0.0/0

Chain neutron-filter-top (2 references)
 pkts bytes target prot opt in out source destination
  185 11100 neutron-l3-agent-local all -- * * 0.0.0.0/0 0.0.0.0/0

Chain neutron-l3-agent-FORWARD (1 references)
 pkts bytes target prot opt in out source destination
  185 11100 neutron-l3-agent-scope all -- * * 0.0.0.0/0 0.0.0.0/0
    0 0 neutron-l3-agent-iv465d8c835 all -- * rfp-+ 0.0.0.0/0 0.0.0.0/0
   39 2340 neutron-l3-agent-ov465d8c835 all -- rfp-+ * 0.0.0.0/0 0.0.0.0/0
    0 0 neutron-l3-agent-fwaas-defau all -- * rfp-+ 0.0.0.0/0 0.0.0.0/0
    0 0 neutron-l3-agent-fwaas-defau all -- rfp-+ * 0.0.0.0/0 0.0.0.0/0

Chain neutron-l3-agent-INPUT (1 references)
 pkts bytes target prot opt in out source destination
    0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x1/0xffff
    0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:9697

Chain neutron-l3-agent-OUTPUT (1 references)
 pkts bytes target prot opt in out source destination

Chain neutron-l3-agent-fwaas-defau (2 references)
 pkts bytes target prot opt in out source destination
    0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0

Chain neutron-l3-agent-iv465d8c835 (1 references)
 pkts bytes target prot opt in out source destination
    0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
    0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
    0 0 REJECT icmp -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable

Chain neutron-l3-agent-local (1 references)
 pkts bytes target prot opt in out source destination

Chain neutron-l3-agent-ov465d8c835 (1 references)
 pkts bytes target prot opt in out source destination
    0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
    0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
   39 2340 REJECT icmp -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable

Chain neutron-l3-agent-scope (1 references)
 pkts bytes target prot opt in out source destination
    0 0 DROP all -- * rfp-09a379b5-9 0.0.0.0/0 0.0.0.0/0 mark match ! 0x4000000/0xffff0000
    0 0 DROP all -- * qr-2cd58562-ad 0.0.0.0/0 0.0.0.0/0 mark match ! 0x4000000/0xffff0000

9. Disassociate the firewall from the router.
$ neutron firewall-update --no-routers test-firewall
Updated firewall: test-firewall

10. Recheck ip tables rules within router's namespace.
$ ip netns exec qrouter-09a379b5-907f-4e3e-b29a-8701b82f2641 iptables -n -L -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target prot opt in out source destination
    0 0 neutron-l3-agent-INPUT all -- * * 0.0.0.0/0 0.0.0.0/0

Chain FORWARD (policy ACCEPT 127 packets, 7668 bytes)
 pkts bytes target prot opt in out source destination
  698 41976 neutron-filter-top all -- * * 0.0.0.0/0 0.0.0.0/0
  698 41976 neutron-l3-agent-FORWARD all -- * * 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target prot opt in out source destination
    0 0 neutron-filter-top all -- * * 0.0.0.0/0 0.0.0.0/0
    0 0 neutron-l3-agent-OUTPUT all -- * * 0.0.0.0/0 0.0.0.0/0

Chain neutron-filter-top (2 references)
 pkts bytes target prot opt in out source destination
  698 41976 neutron-l3-agent-local all -- * * 0.0.0.0/0 0.0.0.0/0

Chain neutron-l3-agent-FORWARD (1 references)
 pkts bytes target prot opt in out source destination
  698 41976 neutron-l3-agent-scope all -- * * 0.0.0.0/0 0.0.0.0/0
   99 5988 neutron-l3-agent-iv465d8c835 all -- * rfp-+ 0.0.0.0/0 0.0.0.0/0
  366 21960 neutron-l3-agent-ov465d8c835 all -- rfp-+ * 0.0.0.0/0 0.0.0.0/0
   99 5988 neutron-l3-agent-fwaas-defau all -- * rfp-+ 0.0.0.0/0 0.0.0.0/0
    0 0 neutron-l3-agent-fwaas-defau all -- rfp-+ * 0.0.0.0/0 0.0.0.0/0

Chain neutron-l3-agent-INPUT (1 references)
 pkts bytes target prot opt in out source destination
    0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x1/0xffff
    0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:9697

Chain neutron-l3-agent-OUTPUT (1 references)
 pkts bytes target prot opt in out source destination

Chain neutron-l3-agent-fwaas-defau (2 references)
 pkts bytes target prot opt in out source destination
   99 5988 DROP all -- * * 0.0.0.0/0 0.0.0.0/0

Chain neutron-l3-agent-iv465d8c835 (1 references)
 pkts bytes target prot opt in out source destination
    0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
    0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
    0 0 REJECT icmp -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable

Chain neutron-l3-agent-local (1 references)
 pkts bytes target prot opt in out source destination

Chain neutron-l3-agent-ov465d8c835 (1 references)
 pkts bytes target prot opt in out source destination
    0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
    0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
  366 21960 REJECT icmp -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable

Chain neutron-l3-agent-scope (1 references)
 pkts bytes target prot opt in out source destination
    0 0 DROP all -- * rfp-09a379b5-9 0.0.0.0/0 0.0.0.0/0 mark match ! 0x4000000/0xffff0000
    0 0 DROP all -- * qr-2cd58562-ad 0.0.0.0/0 0.0.0.0/0 mark match ! 0x4000000/0xffff0000

11. The iptables rules "0 0 REJECT icmp -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable" is still active, although the firewall has been successfully disassociated from the distributed router.
$ neutron firewall-show test-firewall
+--------------------+--------------------------------------+
| Field | Value |
+--------------------+--------------------------------------+
| admin_state_up | True |
| description | |
| firewall_policy_id | 53a8d733-e81c-4113-9354-d40b5b426e00 |
| id | a468caca-c555-4f89-adbc-bcdbb06a3fca |
| name | test-firewall |
| project_id | ed2d2efd86dd40e7a45491d8502318d3 |
| router_ids | |
| status | INACTIVE |
| tenant_id | ed2d2efd86dd40e7a45491d8502318d3 |
+--------------------+--------------------------------------+

12. The "neutron-l3-agent.log" on the compute node does not contain any entries regarding FWaaS. Any updates of a firewall or of its policies and rules do not have any effect when using distributed virtual routers (DVR).

May someone please have a look?

Christoph Fiehe (fiehe)
summary: - FWaaS: Ip tables rules do not get updated in case of distributed routers
- (DVR)
+ FWaaS: Ip tables rules do not get updated in case of distributed virtual
+ routers (DVR)
Reedip (reedip-banerjee-deactivatedaccount)
Changed in neutron:
assignee: nobody → Reedip (reedip-banerjee)
Revision history for this message
Reedip (reedip-banerjee-deactivatedaccount) wrote :

Actually its the other way round

Revision history for this message
Christoph Fiehe (fiehe) wrote :

Ok. I have removed the duplicate flag from this bug and marked Bug #171619 as duplicate.

Revision history for this message
Swaminathan Vasudevan (swaminathan-vasudevan) wrote :

We might have to check the DVR agent side code for changes in fip disassociate. There were lot more changes that went in this area, and the make sure that the firewall agent is polling on the right event for cleaning up the rules.

Revision history for this message
Swaminathan Vasudevan (swaminathan-vasudevan) wrote :

I knew that the firewall agent was dependent on the router_update on floatingip configuration.
Recent changes in the DVR agent were, we do create the fip-namespace based on the gateway add and gateway remove and the connection to the router-namespace is also created at that time.
Only the floatingip rules are added when the floatingip is configured.
So you should always see the rfp port until the gateway is not cleared.

Revision history for this message
Reedip (reedip-banerjee-deactivatedaccount) wrote :

Thanks Swami.
I might need some more information, so will ping you on the IRC :)

Revision history for this message
Swaminathan Vasudevan (swaminathan-vasudevan) wrote :

Are you at the PTG, if so we can meet to discuss.

Revision history for this message
Reedip (reedip-banerjee-deactivatedaccount) wrote :

No, but Sridar , German and Yushiro are. We also have a meeting today ( FWaaS Project discussion) at 2030 IST / 0900 Mountain Time, so maybe we can have a discussion then ?

Revision history for this message
Swaminathan Vasudevan (swaminathan-vasudevan) wrote :

Any more update on this. Was the links to the arch. usefull.
Most of the handling by FWaaS is done on router_update event.
If you can give me the pointer to where the FWaaS is setting the iptable rules and clearing it, I can take a look at it.

Revision history for this message
Swaminathan Vasudevan (swaminathan-vasudevan) wrote :

@Reedip any update on your findings.

Revision history for this message
Reedip (reedip-banerjee-deactivatedaccount) wrote :

Sorry, couldnt do much on this as of now

Changed in neutron:
assignee: Reedip (reedip-banerjee) → nobody
Revision history for this message
Swaminathan Vasudevan (swaminathan-vasudevan) wrote :

Reedip, thanks for your update.

Revision history for this message
Swaminathan Vasudevan (swaminathan-vasudevan) wrote :

Needs triaging.

Revision history for this message
Sridar Kandaswamy (skandasw) wrote :

Thanks Swami for the discussion and the info - on my first impression I need to check on how FWaaS programs the Router namespace - perhaps we are not in sync with some of the changes on DVR side. Let me do some more investigation and then reconnect.

Changed in neutron:
assignee: nobody → Sridar Kandaswamy (skandasw)
Revision history for this message
Swaminathan Vasudevan (swaminathan-vasudevan) wrote :

This bug is also a duplicate of
https://bugs.launchpad.net/neutron/+bug/1845557

Revision history for this message
Swaminathan Vasudevan (swaminathan-vasudevan) wrote :

This patch fixes the issue.
https://review.opendev.org/#/c/686029/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.