openvswitch firewall flows cause flooding on integration bridge

Bug #1732067 reported by James Denton on 2017-11-14
392
This bug affects 23 people
Affects Status Importance Assigned to Milestone
OpenStack Security Advisory
Undecided
Unassigned
OpenStack Security Notes
Undecided
Unassigned
neutron
High
LIU Yulong

Bug Description

Environment: OpenStack Newton
Driver: ML2 w/ OVS
Firewall: openvswitch

In this environment, we have observed OVS flooding network traffic across all ports in a given VLAN on the integration bridge due to the lack of a FDB entry for the destination MAC address. Across the large fleet of 240+ nodes, this is causing a considerable amount of noise on any given node.

In this test, we have 3 machines:

Client: fa:16:3e:e8:59:00 (10.10.60.2)
Server: fa:16:3e:80:cb:0a (10.10.60.9)
Bystander: fa:16:3e:a0:ee:02 (10.10.60.10)

The server is running a web server using netcat:

while true ; do sudo nc -l -p 80 < index.html ; done

Client requests page using curl:

ip netns exec qdhcp-b07e6cb3-0943-45a2-b5ff-efb7e99e4d3d curl http://10.10.60.9/

We should expect to see the communication limited to the client and server. However, the captures below reflect the server->client responses being broadcast out all tap interfaces connected to br-int in the same local vlan:

root@osa-newton-ovs-compute01:~# tcpdump -i tap5f03424d-1c -ne port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap5f03424d-1c, link-type EN10MB (Ethernet), capture size 262144 bytes
02:20:30.190675 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 74: 10.10.60.2.54796 > 10.10.60.9.80: Flags [S], seq 213484442, win 29200, options [mss 1460,sackOK,TS val 140883559 ecr 0,nop,wscale 7], length 0
02:20:30.191926 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 74: 10.10.60.9.80 > 10.10.60.2.54796: Flags [S.], seq 90006557, ack 213484443, win 14480, options [mss 1460,sackOK,TS val 95716 ecr 140883559,nop,wscale 4], length 0
02:20:30.192837 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 66: 10.10.60.2.54796 > 10.10.60.9.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 140883560 ecr 95716], length 0
02:20:30.192986 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 140: 10.10.60.2.54796 > 10.10.60.9.80: Flags [P.], seq 1:75, ack 1, win 229, options [nop,nop,TS val 140883560 ecr 95716], length 74: HTTP: GET / HTTP/1.1
02:20:30.195806 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 79: 10.10.60.9.80 > 10.10.60.2.54796: Flags [P.], seq 1:14, ack 1, win 905, options [nop,nop,TS val 95717 ecr 140883560], length 13: HTTP
02:20:30.196207 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 66: 10.10.60.2.54796 > 10.10.60.9.80: Flags [.], ack 14, win 229, options [nop,nop,TS val 140883561 ecr 95717], length 0
02:20:30.197481 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 66: 10.10.60.9.80 > 10.10.60.2.54796: Flags [.], ack 75, win 905, options [nop,nop,TS val 95717 ecr 140883560], length 0

^^^ On the server tap we see the bi-directional traffic

root@osa-newton-ovs-compute01:/home/ubuntu# tcpdump -i tapb8051da9-60 -ne port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tapb8051da9-60, link-type EN10MB (Ethernet), capture size 262144 bytes
02:20:30.192165 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 74: 10.10.60.9.80 > 10.10.60.2.54796: Flags [S.], seq 90006557, ack 213484443, win 14480, options [mss 1460,sackOK,TS val 95716 ecr 140883559,nop,wscale 4], length 0
02:20:30.195827 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 79: 10.10.60.9.80 > 10.10.60.2.54796: Flags [P.], seq 1:14, ack 1, win 905, options [nop,nop,TS val 95717 ecr 140883560], length 13: HTTP
02:20:30.197500 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 66: 10.10.60.9.80 > 10.10.60.2.54796: Flags [.], ack 75, win 905, options [nop,nop,TS val 95717 ecr 140883560], length 0

^^^ On the bystander tap we see the flooded traffic

The FDB tables reflect the lack of CAM entry for the client on br-int bridge. I would expect to see the MAC address on the patch uplink:

root@osa-newton-ovs-compute01:/home/ubuntu# ovs-appctl fdb/show br-int | grep 'fa:16:3e:e8:59:00'
root@osa-newton-ovs-compute01:/home/ubuntu# ovs-appctl fdb/show br-provider | grep 'fa:16:3e:e8:59:00'
    2 850 fa:16:3e:e8:59:00 3

Sources[1] point to the fact that an 'output' action negates the MAC learning mechanism in OVS. Related Table 82 entries are below, and code is here[2]:

cookie=0x94ebb7913c37a0ec, duration=415.490s, table=82, n_packets=5, n_bytes=424, idle_age=31, priority=70,ct_state=+est-rel-rpl,tcp,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a,tp_dst=80 actions=strip_vlan,output:13
cookie=0x94ebb7913c37a0ec, duration=415.489s, table=82, n_packets=354, n_bytes=35229, idle_age=154, priority=70,ct_state=+est-rel-rpl,tcp,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a,tp_dst=22 actions=strip_vlan,output:13
cookie=0x94ebb7913c37a0ec, duration=415.489s, table=82, n_packets=1, n_bytes=78, idle_age=154, priority=70,ct_state=+new-est,tcp,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a,tp_dst=80 actions=ct(commit,zone=NXM_NX_REG6[0..15]),strip_vlan,output:13
cookie=0x94ebb7913c37a0ec, duration=415.489s, table=82, n_packets=1, n_bytes=78, idle_age=415, priority=70,ct_state=+new-est,tcp,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a,tp_dst=22 actions=ct(commit,zone=NXM_NX_REG6[0..15]),strip_vlan,output:13
cookie=0x94ebb7913c37a0ec, duration=415.491s, table=82, n_packets=120, n_bytes=7920, idle_age=305, priority=50,ct_state=+est-rel+rpl,ct_zone=4,ct_mark=0,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a actions=strip_vlan,output:13
cookie=0x94ebb7913c37a0ec, duration=415.491s, table=82, n_packets=0, n_bytes=0, idle_age=415, priority=50,ct_state=-new-est+rel-inv,ct_zone=4,ct_mark=0,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a actions=strip_vlan,output:13

My testing shows that massaging the flow rules to remove the 'output' action and instead use a 'mod_vlan_vid' action (for the sake of getting it working) results in expected behavior:

cookie=0x85cd1a977dd54be0, duration=0.359s, table=82, n_packets=0, n_bytes=0, idle_age=2110, priority=70,ct_state=+est-rel-rpl,tcp,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a,tp_dst=80 actions=mod_vlan_vid:4,NORMAL
cookie=0x85cd1a977dd54be0, duration=0.359s, table=82, n_packets=0, n_bytes=0, idle_age=518, priority=70,ct_state=+est-rel-rpl,tcp,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a,tp_dst=22 actions=mod_vlan_vid:4,NORMAL
cookie=0x85cd1a977dd54be0, duration=0.359s, table=82, n_packets=0, n_bytes=0, idle_age=392, priority=70,ct_state=+new-est,tcp,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a,tp_dst=80 actions=ct(commit,zone=NXM_NX_REG6[0..15]),mod_vlan_vid:4,NORMAL
cookie=0x85cd1a977dd54be0, duration=0.359s, table=82, n_packets=0, n_bytes=0, idle_age=185, priority=70,ct_state=+new-est,tcp,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a,tp_dst=22 actions=ct(commit,zone=NXM_NX_REG6[0..15]),mod_vlan_vid:4,NORMAL
cookie=0x85cd1a977dd54be0, duration=0.361s, table=82, n_packets=0, n_bytes=0, idle_age=5263, priority=50,ct_state=+est-rel+rpl,ct_zone=4,ct_mark=0,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a actions=strip_vlan,output:13
cookie=0x85cd1a977dd54be0, duration=0.361s, table=82, n_packets=0, n_bytes=0, idle_age=5373, priority=50,ct_state=-new-est+rel-inv,ct_zone=4,ct_mark=0,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a actions=strip_vlan,output:13

The MAC of the client shows up now on br-int FDB:

root@osa-newton-ovs-compute01:/home/ubuntu# ovs-appctl fdb/show br-int | grep 'fa:16:3e:e8:59:00'
    1 4 fa:16:3e:e8:59:00 2

The test below shows that traffic is only seen on server tap and not bystander tap:

root@osa-newton-ovs-compute01:/home/ubuntu# tcpdump -i tap5f03424d-1c -ne port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap5f03424d-1c, link-type EN10MB (Ethernet), capture size 262144 bytes
03:46:52.606940 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 74: 10.10.60.2.55808 > 10.10.60.9.80: Flags [S], seq 3645914146, win 29200, options [mss 1460,sackOK,TS val 142179163 ecr 0,nop,wscale 7], length 0
03:46:52.608880 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 74: 10.10.60.9.80 > 10.10.60.2.55808: Flags [S.], seq 3531519972, ack 3645914147, win 14480, options [mss 1460,sackOK,TS val 1391324 ecr 142179163,nop,wscale 4], length 0
03:46:52.610175 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 66: 10.10.60.2.55808 > 10.10.60.9.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 142179164 ecr 1391324], length 0
03:46:52.610273 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 140: 10.10.60.2.55808 > 10.10.60.9.80: Flags [P.], seq 1:75, ack 1, win 229, options [nop,nop,TS val 142179164 ecr 1391324], length 74: HTTP: GET / HTTP/1.1
03:46:52.613851 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 66: 10.10.60.9.80 > 10.10.60.2.55808: Flags [.], ack 75, win 905, options [nop,nop,TS val 1391325 ecr 142179164], length 0
03:46:52.614007 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 79: 10.10.60.9.80 > 10.10.60.2.55808: Flags [P.], seq 1:14, ack 75, win 905, options [nop,nop,TS val 1391325 ecr 142179164], length 13: HTTP
03:46:52.614314 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 66: 10.10.60.2.55808 > 10.10.60.9.80: Flags [.], ack 14, win 229, options [nop,nop,TS val 142179165 ecr 1391325], length 0

root@osa-newton-ovs-compute01:/home/ubuntu# tcpdump -i tapb8051da9-60 -ne port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tapb8051da9-60, link-type EN10MB (Ethernet), capture size 262144 bytes

>> Nothing! As expected.

I need to build out an environment using the master branch, but the code at [3] seems to indicate the 'output' action is still specified.

Thanks for taking a look and let me know if you have any questions.

[1] https://mail.openvswitch.org/pipermail/ovs-discuss/2016-August/042276.html
[2] https://github.com/openstack/neutron/blob/newton-eol/neutron/agent/linux/openvswitch_firewall/rules.py#L73
[3] https://github.com/openstack/neutron/blob/master/neutron/agent/linux/openvswitch_firewall/rules.py#L80

James Denton (james-denton) wrote :
Download full text (4.1 KiB)

Just an update -- I was able to replicate this using neutron-openvswitch-agent 11.0.0.0rc2.dev368.

Server:

root@osa-master-ovs:/home/ubuntu# tcpdump -i tap849fa737-65 -ne port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap849fa737-65, link-type EN10MB (Ethernet), capture size 262144 bytes
04:29:21.667498 fa:16:3e:32:40:3d > fa:16:3e:f3:2c:61, ethertype IPv4 (0x0800), length 74: 192.168.1.2.42444 > 192.168.1.3.80: Flags [S], seq 364460666, win 29200, options [mss 1460,sackOK,TS val 2559349669 ecr 0,nop,wscale 7], length 0
04:29:21.668573 fa:16:3e:f3:2c:61 > fa:16:3e:32:40:3d, ethertype IPv4 (0x0800), length 74: 192.168.1.3.80 > 192.168.1.2.42444: Flags [S.], seq 4137876027, ack 364460667, win 14480, options [mss 1460,sackOK,TS val 4294931608 ecr 2559349669,nop,wscale 4], length 0
04:29:21.669023 fa:16:3e:32:40:3d > fa:16:3e:f3:2c:61, ethertype IPv4 (0x0800), length 66: 192.168.1.2.42444 > 192.168.1.3.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 2559349670 ecr 4294931608], length 0
04:29:21.669077 fa:16:3e:32:40:3d > fa:16:3e:f3:2c:61, ethertype IPv4 (0x0800), length 141: 192.168.1.2.42444 > 192.168.1.3.80: Flags [P.], seq 1:76, ack 1, win 229, options [nop,nop,TS val 2559349670 ecr 4294931608], length 75: HTTP: GET / HTTP/1.1
04:29:21.669979 fa:16:3e:f3:2c:61 > fa:16:3e:32:40:3d, ethertype IPv4 (0x0800), length 66: 192.168.1.3.80 > 192.168.1.2.42444: Flags [.], ack 76, win 905, options [nop,nop,TS val 4294931609 ecr 2559349670], length 0
04:29:21.672658 fa:16:3e:f3:2c:61 > fa:16:3e:32:40:3d, ethertype IPv4 (0x0800), length 90: 192.168.1.3.80 > 192.168.1.2.42444: Flags [P.], seq 1:25, ack 76, win 905, options [nop,nop,TS val 4294931610 ecr 2559349670], length 24: HTTP
04:29:21.672710 fa:16:3e:32:40:3d > fa:16:3e:f3:2c:61, ethertype IPv4 (0x0800), length 66: 192.168.1.2.42444 > 192.168.1.3.80: Flags [.], ack 25, win 229, options [nop,nop,TS val 2559349671 ecr 4294931610], length 0
04:29:55.543336 fa:16:3e:f3:2c:61 > fa:16:3e:32:40:3d, ethertype IPv4 (0x0800), length 66: 192.168.1.3.80 > 192.168.1.2.42444: Flags [F.], seq 25, ack 76, win 905, options [nop,nop,TS val 4294940076 ecr 2559349671], length 0
04:29:55.579959 fa:16:3e:32:40:3d > fa:16:3e:f3:2c:61, ethertype IPv4 (0x0800), length 66: 192.168.1.2.42444 > 192.168.1.3.80: Flags [F.], seq 76, ack 26, win 229, options [nop,nop,TS val 2559358148 ecr 4294940076], length 0
04:29:55.580711 fa:16:3e:f3:2c:61 > fa:16:3e:32:40:3d, ethertype IPv4 (0x0800), length 66: 192.168.1.3.80 > 192.168.1.2.42444: Flags [.], ack 77, win 905, options [nop,nop,TS val 4294940087 ecr 2559358148], length 0
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel

Bystander:

root@osa-master-ovs:/home/ubuntu# tcpdump -i tap2bceb97c-0b -ne port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap2bceb97c-0b, link-type EN10MB (Ethernet), capture size 262144 bytes
04:29:21.668768 fa:16:3e:f3:2c:61 > fa:16:3e:32:40:3d, ethertype IPv4 (0x0800), length 74: 192.168.1.3.80 > 192.168.1.2.42444: Flags [S.], seq 4137876027, ack 364460667, win 14480, options [mss 1460,sackOK,TS val 4294931608 ecr 2559349669...

Read more...

tags: added: ovs-fw
Peter Slovak (slovak-peto) wrote :

@james-denton, do you by any chance use such IP addresses inside your VMs that Neutron isn't aware of? We spotted the exact same behavior in our setup when we created Neutron ports with IP addresses A, B, C and additionally used IPs X, Y, Z inside the VMs.

The X, Y and Z IPs were managed by Corosync and to be able to communicate with them, we used a somewhat lazy allowed_address_pair 0.0.0.0/0 on the three ports. As we realized later, the OVS firewall driver takes the port IPs and allowed_address_pairs into account when creating some egress flows [1].

Then we ended up with flow groups with criteria "nw_src=10.10.1.1" (the IP on Neutron port) and "nw_src=0.0.0.0" (IP from allowed_address_pairs). The first IP never had any problems with flooding and when using it as source, the target MAC was learned by ovs and appeared in the br-int fdb. I'm guessing adding IPs X, Y and Z to allowed_address_pairs will solve this, but have to verify this yet.

[1] https://github.com/openstack/neutron/blob/newton-eol/neutron/agent/linux/openvswitch_firewall/firewall.py#L405

Peter Slovak (slovak-peto) wrote :

Nevermind, the fix with allowed_address_pairs only worked intermittently in our setup. I haven't successfully traced why it worked in some occasions and didn't work in others. Consider my previous advice invalid.

Yang Li (yang-li) wrote :

I also get the same problem, always no mac in fdb tables that cause tcp ack packet flood to other port, is there a solution for this problem?

Anton Kurbatov (akurbatov) wrote :

Also get the same. It looks like VM hosted on the same node as a victim VM may sniff all traffic and password/cookies/etc may then be leaked.

James Denton (james-denton) wrote :

This may have been resolved around the Pike timeframe, IIRC. Can you please confirm which version of OpenStack you’re using?

Yang Li (yang-li) wrote :

My openstack version is Newton, can you confirm which patch solved this problem, thanks a lot !!
BTW,Not only same node‘s vm, the vms in different nodes also has this problem。

Yang Li (yang-li) wrote :

I modified the openflow, and it worked, no more flood packets.
cookie=0x85cd1a977dd54be0, duration=0.359s, table=82, n_packets=0, n_bytes=0, idle_age=518, priority=70,ct_state=+est-rel-rpl,tcp,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a,tp_dst=22 actions=NORMAL

I'm not sure is there any problem for this modification, it's a workaround, not compatible with later version of neutron

Jesse (jesse-5) wrote :

After tested, It seems that this issue is caused by ovs-fw ingress flows with strip_vlan,output=<port_id> in table=81 and table=82.
When I ping VM's floatingip from outside. for example, floatingip for VM is 172.24.0.157, internal ip is 192.168.111.18/fa:16:3e:b2:c2:84, outside host IP is 172.24.0.3/e6:73:51:97:74:4e.
If outside host has no arp entry for 172.24.0.157, it will send arp broadcast, and br-int on host which VM on will update it's fdb entry.

[root@node-2 ~]# ovs-appctl fdb/show br-int | grep e6:73:51:97:74:4e
    2 3 e6:73:51:97:74:4e 3

This fdb entry timeout is 300s, even though VM ping continues, this entry will not be updated because strip_vlan,output=<port_id> flow. This flow will forward ping package to VM and do not update fdb entry in br-int.
After 300s, this entry disappear, The reply icmp package will flood in br-int bridge (This do not affect the original ping, but flood will affect other VMs on this host. If other VMs has ingress QoS, this flood will affect other VM's network connection).
If you delete 172.24.0.157 fdb entry on outside host by `arp -d 172.24.0.157`, the fdb entry in br-int will come back, and flood is stopped.

To solve this problem, comment #8 is a solution, for ingress package, we just do actions=NORMAL at last to let br-int update fdb entry.

Anton Kurbatov (akurbatov) wrote :

I am also agree flooding is caused due to
- for egress trffic there is action=NORMAL
- for ingress traffic there is direct action=output:PORT
in the openflow rules.

NORMAL action causes openvswitch to flood traffic if there is no dst mac entry in fdb table
(https://github.com/openvswitch/ovs/blob/master/ofproto/ofproto-dpif-xlate.c#L3109)
At the same time MAC address is put into fdb table only if NORMAL action is performed (note, no mac learning is performed for direct output:PORT action)

E.g.
- a VM with m_mac sends packets to a node with node_mac -> it forces an ovs to learn vm_mac and remember this mac in br-int fdb table, then packet is flooded to all ovs ports;
- the node responses to vm_mac using output:VM_PORT rule -> the packet is sent directly to the VM port (without node_mac learning)
- the VM again sends packet to the node -> there is no node_mac in the fdb table and it forces ovs another packet flooding to all ports.

Fix proposed to branch: master
Review: https://review.openstack.org/639009

Changed in neutron:
assignee: nobody → Yang Li (yang-li)
status: New → In Progress
Changed in neutron:
assignee: Yang Li (yang-li) → zhengyong (zhengy23)
Changed in neutron:
assignee: zhengyong (zhengy23) → Zhengdong Wu (zhengdong.wu)
Junien Fridrick (axino) wrote :

Note : an ARP request with broadcast ethernet address (ff:ff:ff:ff:ff:ff) will trigger MAC learning. This is used by VMs when the IP is not already in their MAC table. After what, they switch to unicast ARP requests, which _don't_ trigger MAC learning - so the FDB entry expires after 5 min (by default).

Junien Fridrick (axino) wrote :

This is also GREATLY impacting VM throughput if you have multiple VMs in the same neutron network on a hypervisor.

Hongbin Lu (hongbin.lu) on 2019-04-19
Changed in neutron:
importance: Undecided → High
Junien Fridrick (axino) wrote :

Note : I'm also seeing this behaviour with the iptables_hybrid firewall driver

Changed in neutron:
assignee: Zhengdong Wu (zhengdong.wu) → yangjianfeng (yangjianfeng)

Fix proposed to branch: master
Review: https://review.opendev.org/666991

Changed in neutron:
assignee: yangjianfeng (yangjianfeng) → LIU Yulong (dragon889)
Jeremy Stanley (fungi) wrote :

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

(The duplicate bug 1813439 was previously being tracked as a potential vulnerability report.)

Changed in ossa:
status: New → Incomplete
information type: Public → Public Security
Jeremy Stanley (fungi) wrote :

Is this related to bug 1837252? The symptoms seem very similar.

sean mooney (sean-k-mooney) wrote :

no this is not related to bug 1837252

ml2/ovs with ovs firewall does not use hybrid plug so there is no linux bridge between ovs
and the tap device so they are complete different bugs caused by different issues.

the effect is similar but not the same underlying cause or network backend.

Hello:

We are currently testing this bug in our development systems. ASAP we'll provide some feedback.

Regards.

Miguel Lavalle (minsel) wrote :

@fungi,

The testing Rodolfo is talking about in #19 will help decide the scope of any potential vulnerability. will keep you posted

LIU Yulong (dragon889) wrote :

We have an updated fix locally which can cover both enable/disable openflow security group (firewall).
I will update it to this patch set: https://review.opendev.org/#/c/666991/

And also I added a new bug for the new solution, it will add a flow table to do something like a switch FDB table. The accepted egress flows will be take care in that. For more information please see here:
https://bugs.launchpad.net/neutron/+bug/1841622

Changed in neutron:
assignee: LIU Yulong (dragon889) → Brian Haley (brian-haley)

This bug has had a related patch abandoned and has been automatically un-assigned due to inactivity. Please re-assign yourself if you are continuing work or adjust the state as appropriate if it is no longer valid.

Changed in neutron:
assignee: Brian Haley (brian-haley) → nobody
status: In Progress → New
tags: added: timeout-abandon

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/639009
Reason: This review is > 4 weeks without comment and currently blocked by a core reviewer with a -2. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and contacting the reviewer with the -2 on this review to ensure you address their concerns.

Jeremy Stanley (fungi) wrote :

It seems like bug 1841622 was opened for an alternative fix for this even though all the discussion about the problem happened here. The code change associated with that duplicate bug is currently active, but the bug is not marked as a suspected vulnerability like this one.

Is there still any indication that this bug could represent an exploitable security vulnerability in deployments, which can be patched safely in supported stable branches as well? If not, then we should probably convert this to a regular bug report and close the security advisory task associated with it.

Regardless, I suspect bug 1841622 should be folded back into this one to limit confusion since it's not reporting a bug but rather describing a potential fix for this existing bug.

Slawek Kaplonski (slaweq) wrote :

Hi Jeremy,

I marked bug 1841622 as duplicate of this one.
Regarding Your question about potential vulnerability, I don't think there is any here. This bug is about broadcasting packets to other ports from the same tenant network. Other networks are always using another vlan tag locally in br-int so there is no way that such packets can be seen on interfaces which belongs to other networks.

James Denton (james-denton) wrote :

There are cases where the same network is shared across multiple tenants relying only on security groups to filter. So, I would say the broadcast of packets to all instances in the same vlan presented by this bug is less than ideal.

Changed in neutron:
milestone: none → ussuri-2
LIU Yulong (dragon889) wrote :

@James, yes, that's we called shared network. It can be used from different projects.

Changed in neutron:
assignee: nobody → LIU Yulong (dragon889)
status: New → In Progress

Reviewed: https://review.opendev.org/666991
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=efa8dd08957b5b6b1a05f0ed412ff00462a9f216
Submitter: Zuul
Branch: master

commit efa8dd08957b5b6b1a05f0ed412ff00462a9f216
Author: LIU Yulong <email address hidden>
Date: Mon Jun 24 00:08:12 2019 +0800

    Add accepted egress direct flow

    Do not flood the packets to bridge, since we have the
    bridge port list, we can add a simple direct flow to
    the right port only.

    Closes-Bug: #1732067
    Related-Bug: #1841622
    Change-Id: I14fefe289a19b718b247bf0740ca9bc47f8903f4

Changed in neutron:
status: In Progress → Fix Released
Jeremy Stanley (fungi) wrote :

Is there any plan for backporting this fix to supported stable branches?

Slawek Kaplonski (slaweq) wrote :

Hi fungi,

I proposed backport to stable/train as it was without any conflicts. For older branches it seems it may require some more work and maybe some other patches to be backported too but I don't think there is any problem with backporting this fix.
It introduced new config option, but it has default value that guarantee old behavior so I think we still can backport it.

Jeremy Stanley (fungi) wrote :

If mitigating this on some supported stable branches is going to require the operator to make configuration changes, then this is probably better handled as a security note than an advisory.

Changed in ossa:
status: Incomplete → Won't Fix
Jeremy Stanley (fungi) wrote :

Once the stable branch changes merge, this will be a class B1 report (default config value is insecure): https://security.openstack.org/vmt-process.html#incident-report-taxonomy

Reviewed: https://review.opendev.org/704506
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6c1cf886671df66f6790045268b1394a97caabcc
Submitter: Zuul
Branch: stable/train

commit 6c1cf886671df66f6790045268b1394a97caabcc
Author: LIU Yulong <email address hidden>
Date: Mon Jun 24 00:08:12 2019 +0800

    Add accepted egress direct flow

    Do not flood the packets to bridge, since we have the
    bridge port list, we can add a simple direct flow to
    the right port only.

    Closes-Bug: #1732067
    Related-Bug: #1841622
    Change-Id: I14fefe289a19b718b247bf0740ca9bc47f8903f4
    (cherry picked from commit efa8dd08957b5b6b1a05f0ed412ff00462a9f216)

tags: added: in-stable-train

This issue was fixed in the openstack/neutron 15.0.2 release.

This issue was fixed in the openstack/neutron 16.0.0.0b1 development milestone.

Reviewed: https://review.opendev.org/709410
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0255239ed5affd65568c33ba07cd72fb0b594449
Submitter: Zuul
Branch: stable/queens

commit 0255239ed5affd65568c33ba07cd72fb0b594449
Author: LIU Yulong <email address hidden>
Date: Mon Jun 24 00:08:12 2019 +0800

    Add accepted egress direct flow

    Do not flood the packets to bridge, since we have the
    bridge port list, we can add a simple direct flow to
    the right port only.

    Conflicts:
     neutron/agent/linux/openvswitch_firewall/firewall.py
     neutron/conf/plugins/ml2/drivers/ovs_conf.py
     neutron/tests/unit/agent/linux/openvswitch_firewall/test_firewall.py
     neutron/tests/unit/plugins/ml2/drivers/openvswitch/agent/openflow/native/test_br_int.py

    Closes-Bug: #1732067
    Related-Bug: #1841622
    Change-Id: I14fefe289a19b718b247bf0740ca9bc47f8903f4
    (cherry picked from commit efa8dd08957b5b6b1a05f0ed412ff00462a9f216)

tags: added: in-stable-queens
Yi Yang (yangyi01) wrote :

Hi, Yulong, has this fix patch completely fixed this bug or just partially fixed it? My colleague tried this patch, but br-int can't learn MAC yet, what's wrong?

In addition, I noticed this fix patch just changed agent code, does this mean we just apply it to ovs agent in compute node, neutron server and neutron database needn't do any change, right?

I files another bug, https://bugs.launchpad.net/neutron/+bug/1866445, I'm not sure if this fix patch is effective to firewall_driver=iptables_hybrid.

Peter Slovak (slovak-peto) wrote :

Yi Yang, I don't think the purpose of this patch is learning the MAC address. The fix is implemented by adding a flow that sends traffic destined to a MAC directly to a specific interface, thus short-circuiting the default NORMAL action which (correctly) broadcasts the traffic to all interfaces except the incoming one.

So from the high-level point of view, after this fix is applied, maybe you don't actually need br-int to learn the destination MAC address. It's only necessary if there still existed traffic that didn't match the new direct flow and instead would fall back to the flow with NORMAL action. If you witnessed such a case, you're probably right to create a new bug report.

I don't think neutron-server has anything to do with this change. Flows are populated by agents, so only neutron-openvswitch-agent would have to be restarted in my opinion.

I also don't think that firewall_driver setting is relevant to this change. The flow added in this patch, as I see it (correct me if I'm wrong), sits before any firewall rules whether you're using iptables_hybrid or ovs. This patch is related to traffic forwarding, not traffic filtering.

Reviewed: https://review.opendev.org/710184
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5578c7073fd454f2df43e0d728ba2a64de87fccf
Submitter: Zuul
Branch: stable/stein

commit 5578c7073fd454f2df43e0d728ba2a64de87fccf
Author: LIU Yulong <email address hidden>
Date: Mon Jun 24 00:08:12 2019 +0800

    Add accepted egress direct flow

    Do not flood the packets to bridge, since we have the
    bridge port list, we can add a simple direct flow to
    the right port only.

    Conflicts:
     neutron/conf/plugins/ml2/drivers/ovs_conf.py

    Closes-Bug: #1732067
    Related-Bug: #1841622
    Change-Id: I14fefe289a19b718b247bf0740ca9bc47f8903f4
    (cherry picked from commit efa8dd08957b5b6b1a05f0ed412ff00462a9f216)

tags: added: in-stable-stein
Yi Yang (yangyi01) wrote :

@Peter, I have confirmed it doesn't work for the case firewall_driver=iptables_hybrid. I also find it can't learn local MAC after applied this fix patch. In addition, I also find iperf3 pps test with 16 bytes UDP packet has noticeable packet loss rate.

My issue is this fix patch is just a workaround or a real fix? What issue makes br-int not to be able to learn MAC addresses? Why can we not fix this MAC learning issue? We find if one subnet isn't added into a router, MAC learning is ok, but once it is added a router, br-int won't learn VMs' MACs in other compute nodes, I think it is an ovs openflow pipeline design defect, why can't we fix this design defect and make sure br-int can learn MACs as expected.

Yi Yang (yangyi01) wrote :

Hi, guys

I submitted a gerrit commit https://review.opendev.org/#/c/712640/, it can make sure Yulong's fix patch can work for firewall_driver=iptables_hybrid, please help review if it is ok, thanks a lot.

Son Do Xuan (sondx25) wrote :

- I execute to patch the code according to the commit:
https://review.opendev.org/#/q/efa8dd08957b5b6b1a05f0ed412ff00462a9f216

- "patch-tun" port of "br-int" switch learn MAC according to the ARP reply message. But timeout of "MAC address table" is only 300 seconds. After this time, packet continues flooding on "br-int" switch.

I can reproduce with firewall_driver=iptables_hybrid and dvr. I attached a patch that directs only packets from the gateway to the vm ports. All other packets from tunnels will hit the NORMAL flow in table 60.

Without patch:
table=60, priority=20,dl_vlan=1,dl_dst=fa:16:3e:5b:72:d8 actions=strip_vlan,output:"qvod638e8da-88"
table=60, priority=3 actions=NORMAL

With patch (gateway_mac is fa:16:3e:f1:f3:06):
table=60, priority=20,dl_vlan=1,dl_src=fa:16:3e:f1:f3:06,dl_dst=fa:16:3e:5b:72:d8 actions=strip_vlan,output:"qvod638e8da-88"
table=60, priority=3 actions=NORMAL

@sondx25 - can you try this patch?

Reviewed: https://review.opendev.org/710183
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fdc8987f791b24119ef22c434f8877df13bcdded
Submitter: Zuul
Branch: stable/rocky

commit fdc8987f791b24119ef22c434f8877df13bcdded
Author: LIU Yulong <email address hidden>
Date: Mon Jun 24 00:08:12 2019 +0800

    Add accepted egress direct flow

    Do not flood the packets to bridge, since we have the
    bridge port list, we can add a simple direct flow to
    the right port only.

    Conflicts:
        neutron/agent/linux/openvswitch_firewall/firewall.py
        neutron/conf/plugins/ml2/drivers/ovs_conf.py

    Closes-Bug: #1732067
    Related-Bug: #1841622
    Change-Id: I14fefe289a19b718b247bf0740ca9bc47f8903f4
    (cherry picked from commit efa8dd08957b5b6b1a05f0ed412ff00462a9f216)

tags: added: in-stable-rocky
tags: added: neutron-proactive-backport-potential
Arjun Baindur (abaindur) wrote :

Is this truly fixed? The patch doesn't seem to fix the underlying issue at hand - ingress MACs (coming from br-vlan or br-tun) are NOT learned by br-int.

Also what are the implications of turning on this setting? The commit states:

"A new config option ``explicitly_egress_direct``, with default value False,
    was added for the aim of distinguishing clouds which are running the
    network node mixed with compute services, upstream neutron CI should be
    an example. In such situation, this ``explicitly_egress_direct`` should be
    set to False, because there are numerous cases from HA routers which can
    not be covered, particularly when you have centralized floating IPs running
    in such mixed hosts.
    Otherwise, set ``explicitly_egress_direct`` to True to avoid the flooding.
    One more note is if your network nodes are for networing services only, we
    recommand you disable all the security_group to get a higher performance."

I take that to say on a network node mixed with compute services it should be set to False. But isn't that the case for every node in DVR? The compute node hosts qrouter, fip router, and can also host dhcp services.

Or does it mean we not supposed to enable this setting on hosts with dvr_snat mode for l3-agent? The message states something about centralized fips - which are hosting on the snat router/node.

Arjun Baindur (abaindur) wrote :

This fix seems to break certain other scenarios by causing flooding in opposite direction: https://bugs.launchpad.net/neutron/+bug/1884708

Arjun Baindur (abaindur) wrote :

We are seeing ingress flooding occur due to enabling this fix on other networks. We have a VLAN based network, both attached and unattached to a DVR router. In both cases, on ingress the NORMAL action at table=60 is hit.

This is because for a provider network, there are no dvr MAC rules with an explicit output: action to a local port. In DVR subnet, these rules are added with the EXTERNAL vlan/segmentation ID. Not with the local vlan. However, there is a table=0, priority=3 rule which converts from the external VLAN to host local VLAN.

NORMAL action at table=60 for ingress, requires the local VM MACs to be learned. But these are never learned, because the fix resubmits egress packets to table 61, which has a direct output: action northbound.

So now we have flooding in opposite direction. But we still need this fix because in other scenarios, we see the egress flooding.

https://bugs.launchpad.net/neutron/+bug/1884708

Related fix proposed to branch: master
Review: https://review.opendev.org/752672

Related fix proposed to branch: master
Review: https://review.opendev.org/752676

Change abandoned by Rodolfo Alonso Hernandez (<email address hidden>) on branch: master
Review: https://review.opendev.org/752672

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/738397
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/712640
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Changed in neutron:
status: Fix Released → In Progress
tags: removed: timeout-abandon

Reviewed: https://review.opendev.org/738551
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=959d8b6d73e2a6ab1a45c9a7b0b05ae163e650fc
Submitter: Zuul
Branch: master

commit 959d8b6d73e2a6ab1a45c9a7b0b05ae163e650fc
Author: LIU Yulong <email address hidden>
Date: Fri Jul 10 17:25:15 2020 +0800

    Local mac direct flow for non-openflow firewall

    When there is no openflow firewall, aka the ovs agent security group
    is disabled or Noop/HybridIptable, this patch will introduce a different
    ingress pipeline for bridge ports which will avoid ingress flood:
    (1) table=0, in_port=patch_bridge,dl_vlan=physical_vlan action=mod_vlan:local_vlan,goto:60 (original)
    (2) table=60, in_port=patch_bridge action=goto:61 (new)
    (3) table=61, dl_dst=local_port_mac,dl_vlan=local_vlan, action=strip_vlan,output:<ofport> (changes)

    And changes the local ports pipeline:
    (1) table=0, in_port=local_ofport action=goto:25 (original)
    (2) table=25, in_port=local_ofport,dl_src=local_port_mac action=goto:60 (original)
    (3) table=60, in_port=local_ofport,dl_src=local_port_mac action=local_vlan->reg6,goto:61 (changes)
    (4) table=61, dl_dst=local_port_mac,reg6=local_vlan, action=output:<ofport> (changes)

    Closes-Bug: #1884708
    Closes-Bug: #1881070
    Related-Bug: #1732067
    Related-Bug: #1866445
    Related-Bug: #1883321

    Change-Id: Iecf9cffaf02616342f1727ad7db85545d8adbec2

Change abandoned by Yi Yang (<email address hidden>) on branch: master
Review: https://review.opendev.org/712640
Reason: https://review.opendev.org/#/c/738551/ has covered this, so abandon it.

Reviewed: https://review.opendev.org/759364
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ef14d258eea91ef563c63334b2da1623d93418f3
Submitter: Zuul
Branch: stable/ussuri

commit ef14d258eea91ef563c63334b2da1623d93418f3
Author: LIU Yulong <email address hidden>
Date: Fri Jul 10 17:25:15 2020 +0800

    Local mac direct flow for non-openflow firewall

    When there is no openflow firewall, aka the ovs agent security group
    is disabled or Noop/HybridIptable, this patch will introduce a different
    ingress pipeline for bridge ports which will avoid ingress flood:
    (1) table=0, in_port=patch_bridge,dl_vlan=physical_vlan action=mod_vlan:local_vlan,goto:60 (original)
    (2) table=60, in_port=patch_bridge action=goto:61 (new)
    (3) table=61, dl_dst=local_port_mac,dl_vlan=local_vlan, action=strip_vlan,output:<ofport> (changes)

    And changes the local ports pipeline:
    (1) table=0, in_port=local_ofport action=goto:25 (original)
    (2) table=25, in_port=local_ofport,dl_src=local_port_mac action=goto:60 (original)
    (3) table=60, in_port=local_ofport,dl_src=local_port_mac action=local_vlan->reg6,goto:61 (changes)
    (4) table=61, dl_dst=local_port_mac,reg6=local_vlan, action=output:<ofport> (changes)

    Closes-Bug: #1884708
    Closes-Bug: #1881070
    Related-Bug: #1732067
    Related-Bug: #1866445
    Related-Bug: #1883321

    Change-Id: Iecf9cffaf02616342f1727ad7db85545d8adbec2
    (cherry picked from commit 959d8b6d73e2a6ab1a45c9a7b0b05ae163e650fc)

tags: added: in-stable-ussuri

Reviewed: https://review.opendev.org/759365
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c06895e8e78de06c25d36cb347313240432953cf
Submitter: Zuul
Branch: stable/train

commit c06895e8e78de06c25d36cb347313240432953cf
Author: LIU Yulong <email address hidden>
Date: Fri Jul 10 17:25:15 2020 +0800

    Local mac direct flow for non-openflow firewall

    When there is no openflow firewall, aka the ovs agent security group
    is disabled or Noop/HybridIptable, this patch will introduce a different
    ingress pipeline for bridge ports which will avoid ingress flood:
    (1) table=0, in_port=patch_bridge,dl_vlan=physical_vlan action=mod_vlan:local_vlan,goto:60 (original)
    (2) table=60, in_port=patch_bridge action=goto:61 (new)
    (3) table=61, dl_dst=local_port_mac,dl_vlan=local_vlan, action=strip_vlan,output:<ofport> (changes)

    And changes the local ports pipeline:
    (1) table=0, in_port=local_ofport action=goto:25 (original)
    (2) table=25, in_port=local_ofport,dl_src=local_port_mac action=goto:60 (original)
    (3) table=60, in_port=local_ofport,dl_src=local_port_mac action=local_vlan->reg6,goto:61 (changes)
    (4) table=61, dl_dst=local_port_mac,reg6=local_vlan, action=output:<ofport> (changes)

    Closes-Bug: #1884708
    Closes-Bug: #1881070
    Related-Bug: #1732067
    Related-Bug: #1866445
    Related-Bug: #1883321

    Change-Id: Iecf9cffaf02616342f1727ad7db85545d8adbec2
    (cherry picked from commit 959d8b6d73e2a6ab1a45c9a7b0b05ae163e650fc)

Reviewed: https://review.opendev.org/759366
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7c757ad3372b5fe015ae4c5e3949c804e8515d20
Submitter: Zuul
Branch: stable/stein

commit 7c757ad3372b5fe015ae4c5e3949c804e8515d20
Author: LIU Yulong <email address hidden>
Date: Fri Jul 10 17:25:15 2020 +0800

    Local mac direct flow for non-openflow firewall

    When there is no openflow firewall, aka the ovs agent security group
    is disabled or Noop/HybridIptable, this patch will introduce a different
    ingress pipeline for bridge ports which will avoid ingress flood:
    (1) table=0, in_port=patch_bridge,dl_vlan=physical_vlan action=mod_vlan:local_vlan,goto:60 (original)
    (2) table=60, in_port=patch_bridge action=goto:61 (new)
    (3) table=61, dl_dst=local_port_mac,dl_vlan=local_vlan, action=strip_vlan,output:<ofport> (changes)

    And changes the local ports pipeline:
    (1) table=0, in_port=local_ofport action=goto:25 (original)
    (2) table=25, in_port=local_ofport,dl_src=local_port_mac action=goto:60 (original)
    (3) table=60, in_port=local_ofport,dl_src=local_port_mac action=local_vlan->reg6,goto:61 (changes)
    (4) table=61, dl_dst=local_port_mac,reg6=local_vlan, action=output:<ofport> (changes)

    Closes-Bug: #1884708
    Closes-Bug: #1881070
    Related-Bug: #1732067
    Related-Bug: #1866445
    Related-Bug: #1883321

    Change-Id: Iecf9cffaf02616342f1727ad7db85545d8adbec2
    (cherry picked from commit 959d8b6d73e2a6ab1a45c9a7b0b05ae163e650fc)

To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Other bug subscribers