openvswitch firewall flows cause flooding on integration bridge

[James Denton]

Environment: OpenStack Newton
Driver: ML2 w/ OVS
Firewall: openvswitch

In this environment, we have observed OVS flooding network traffic across all ports in a given VLAN on the integration bridge due to the lack of a FDB entry for the destination MAC address. Across the large fleet of 240+ nodes, this is causing a considerable amount of noise on any given node.

In this test, we have 3 machines:

Client: fa:16:3e:e8:59:00 (10.10.60.2)
Server: fa:16:3e:80:cb:0a (10.10.60.9)
Bystander: fa:16:3e:a0:ee:02 (10.10.60.10)

The server is running a web server using netcat:

while true ; do sudo nc -l -p 80 < index.html ; done

Client requests page using curl:

ip netns exec qdhcp-b07e6cb3-0943-45a2-b5ff-efb7e99e4d3d curl http://10.10.60.9/

We should expect to see the communication limited to the client and server. However, the captures below reflect the server->client responses being broadcast out all tap interfaces connected to br-int in the same local vlan:

root@osa-newton-ovs-compute01:~# tcpdump -i tap5f03424d-1c -ne port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap5f03424d-1c, link-type EN10MB (Ethernet), capture size 262144 bytes
02:20:30.190675 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 74: 10.10.60.2.54796 > 10.10.60.9.80: Flags [S], seq 213484442, win 29200, options [mss 1460,sackOK,TS val 140883559 ecr 0,nop,wscale 7], length 0
02:20:30.191926 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 74: 10.10.60.9.80 > 10.10.60.2.54796: Flags [S.], seq 90006557, ack 213484443, win 14480, options [mss 1460,sackOK,TS val 95716 ecr 140883559,nop,wscale 4], length 0
02:20:30.192837 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 66: 10.10.60.2.54796 > 10.10.60.9.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 140883560 ecr 95716], length 0
02:20:30.192986 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 140: 10.10.60.2.54796 > 10.10.60.9.80: Flags [P.], seq 1:75, ack 1, win 229, options [nop,nop,TS val 140883560 ecr 95716], length 74: HTTP: GET / HTTP/1.1
02:20:30.195806 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 79: 10.10.60.9.80 > 10.10.60.2.54796: Flags [P.], seq 1:14, ack 1, win 905, options [nop,nop,TS val 95717 ecr 140883560], length 13: HTTP
02:20:30.196207 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 66: 10.10.60.2.54796 > 10.10.60.9.80: Flags [.], ack 14, win 229, options [nop,nop,TS val 140883561 ecr 95717], length 0
02:20:30.197481 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 66: 10.10.60.9.80 > 10.10.60.2.54796: Flags [.], ack 75, win 905, options [nop,nop,TS val 95717 ecr 140883560], length 0

^^^ On the server tap we see the bi-directional traffic

root@osa-newton-ovs-compute01:/home/ubuntu# tcpdump -i tapb8051da9-60 -ne port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tapb8051da9-60, link-type EN10MB (Ethernet), capture size 262144 bytes
02:20:30.192165 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 74: 10.10.60.9.80 > 10.10.60.2.54796: Flags [S.], seq 90006557, ack 213484443, win 14480, options [mss 1460,sackOK,TS val 95716 ecr 140883559,nop,wscale 4], length 0
02:20:30.195827 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 79: 10.10.60.9.80 > 10.10.60.2.54796: Flags [P.], seq 1:14, ack 1, win 905, options [nop,nop,TS val 95717 ecr 140883560], length 13: HTTP
02:20:30.197500 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 66: 10.10.60.9.80 > 10.10.60.2.54796: Flags [.], ack 75, win 905, options [nop,nop,TS val 95717 ecr 140883560], length 0

^^^ On the bystander tap we see the flooded traffic

The FDB tables reflect the lack of CAM entry for the client on br-int bridge. I would expect to see the MAC address on the patch uplink:

root@osa-newton-ovs-compute01:/home/ubuntu# ovs-appctl fdb/show br-int | grep 'fa:16:3e:e8:59:00'
root@osa-newton-ovs-compute01:/home/ubuntu# ovs-appctl fdb/show br-provider | grep 'fa:16:3e:e8:59:00'
    2 850 fa:16:3e:e8:59:00 3

Sources[1] point to the fact that an 'output' action negates the MAC learning mechanism in OVS. Related Table 82 entries are below, and code is here[2]:

cookie=0x94ebb7913c37a0ec, duration=415.490s, table=82, n_packets=5, n_bytes=424, idle_age=31, priority=70,ct_state=+est-rel-rpl,tcp,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a,tp_dst=80 actions=strip_vlan,output:13
cookie=0x94ebb7913c37a0ec, duration=415.489s, table=82, n_packets=354, n_bytes=35229, idle_age=154, priority=70,ct_state=+est-rel-rpl,tcp,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a,tp_dst=22 actions=strip_vlan,output:13
cookie=0x94ebb7913c37a0ec, duration=415.489s, table=82, n_packets=1, n_bytes=78, idle_age=154, priority=70,ct_state=+new-est,tcp,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a,tp_dst=80 actions=ct(commit,zone=NXM_NX_REG6[0..15]),strip_vlan,output:13
cookie=0x94ebb7913c37a0ec, duration=415.489s, table=82, n_packets=1, n_bytes=78, idle_age=415, priority=70,ct_state=+new-est,tcp,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a,tp_dst=22 actions=ct(commit,zone=NXM_NX_REG6[0..15]),strip_vlan,output:13
cookie=0x94ebb7913c37a0ec, duration=415.491s, table=82, n_packets=120, n_bytes=7920, idle_age=305, priority=50,ct_state=+est-rel+rpl,ct_zone=4,ct_mark=0,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a actions=strip_vlan,output:13
cookie=0x94ebb7913c37a0ec, duration=415.491s, table=82, n_packets=0, n_bytes=0, idle_age=415, priority=50,ct_state=-new-est+rel-inv,ct_zone=4,ct_mark=0,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a actions=strip_vlan,output:13

My testing shows that massaging the flow rules to remove the 'output' action and instead use a 'mod_vlan_vid' action (for the sake of getting it working) results in expected behavior:

cookie=0x85cd1a977dd54be0, duration=0.359s, table=82, n_packets=0, n_bytes=0, idle_age=2110, priority=70,ct_state=+est-rel-rpl,tcp,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a,tp_dst=80 actions=mod_vlan_vid:4,NORMAL
cookie=0x85cd1a977dd54be0, duration=0.359s, table=82, n_packets=0, n_bytes=0, idle_age=518, priority=70,ct_state=+est-rel-rpl,tcp,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a,tp_dst=22 actions=mod_vlan_vid:4,NORMAL
cookie=0x85cd1a977dd54be0, duration=0.359s, table=82, n_packets=0, n_bytes=0, idle_age=392, priority=70,ct_state=+new-est,tcp,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a,tp_dst=80 actions=ct(commit,zone=NXM_NX_REG6[0..15]),mod_vlan_vid:4,NORMAL
cookie=0x85cd1a977dd54be0, duration=0.359s, table=82, n_packets=0, n_bytes=0, idle_age=185, priority=70,ct_state=+new-est,tcp,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a,tp_dst=22 actions=ct(commit,zone=NXM_NX_REG6[0..15]),mod_vlan_vid:4,NORMAL
cookie=0x85cd1a977dd54be0, duration=0.361s, table=82, n_packets=0, n_bytes=0, idle_age=5263, priority=50,ct_state=+est-rel+rpl,ct_zone=4,ct_mark=0,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a actions=strip_vlan,output:13
cookie=0x85cd1a977dd54be0, duration=0.361s, table=82, n_packets=0, n_bytes=0, idle_age=5373, priority=50,ct_state=-new-est+rel-inv,ct_zone=4,ct_mark=0,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a actions=strip_vlan,output:13

The MAC of the client shows up now on br-int FDB:

root@osa-newton-ovs-compute01:/home/ubuntu# ovs-appctl fdb/show br-int | grep 'fa:16:3e:e8:59:00'
    1 4 fa:16:3e:e8:59:00 2

The test below shows that traffic is only seen on server tap and not bystander tap:

root@osa-newton-ovs-compute01:/home/ubuntu# tcpdump -i tap5f03424d-1c -ne port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap5f03424d-1c, link-type EN10MB (Ethernet), capture size 262144 bytes
03:46:52.606940 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 74: 10.10.60.2.55808 > 10.10.60.9.80: Flags [S], seq 3645914146, win 29200, options [mss 1460,sackOK,TS val 142179163 ecr 0,nop,wscale 7], length 0
03:46:52.608880 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 74: 10.10.60.9.80 > 10.10.60.2.55808: Flags [S.], seq 3531519972, ack 3645914147, win 14480, options [mss 1460,sackOK,TS val 1391324 ecr 142179163,nop,wscale 4], length 0
03:46:52.610175 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 66: 10.10.60.2.55808 > 10.10.60.9.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 142179164 ecr 1391324], length 0
03:46:52.610273 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 140: 10.10.60.2.55808 > 10.10.60.9.80: Flags [P.], seq 1:75, ack 1, win 229, options [nop,nop,TS val 142179164 ecr 1391324], length 74: HTTP: GET / HTTP/1.1
03:46:52.613851 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 66: 10.10.60.9.80 > 10.10.60.2.55808: Flags [.], ack 75, win 905, options [nop,nop,TS val 1391325 ecr 142179164], length 0
03:46:52.614007 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 79: 10.10.60.9.80 > 10.10.60.2.55808: Flags [P.], seq 1:14, ack 75, win 905, options [nop,nop,TS val 1391325 ecr 142179164], length 13: HTTP
03:46:52.614314 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 66: 10.10.60.2.55808 > 10.10.60.9.80: Flags [.], ack 14, win 229, options [nop,nop,TS val 142179165 ecr 1391325], length 0

root@osa-newton-ovs-compute01:/home/ubuntu# tcpdump -i tapb8051da9-60 -ne port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tapb8051da9-60, link-type EN10MB (Ethernet), capture size 262144 bytes

>> Nothing! As expected.

I need to build out an environment using the master branch, but the code at [3] seems to indicate the 'output' action is still specified.

Thanks for taking a look and let me know if you have any questions.

[1] https://mail.openvswitch.org/pipermail/ovs-discuss/2016-August/042276.html
[2] https://github.com/openstack/neutron/blob/newton-eol/neutron/agent/linux/openvswitch_firewall/rules.py#L73
[3] https://github.com/openstack/neutron/blob/master/neutron/agent/linux/openvswitch_firewall/rules.py#L80

[James Denton (james-denton)]

Just an update -- I was able to replicate this using neutron-openvswitch-agent 11.0.0.0rc2.dev368.

Server:

root@osa-master-ovs:/home/ubuntu# tcpdump -i tap849fa737-65 -ne port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap849fa737-65, link-type EN10MB (Ethernet), capture size 262144 bytes
04:29:21.667498 fa:16:3e:32:40:3d > fa:16:3e:f3:2c:61, ethertype IPv4 (0x0800), length 74: 192.168.1.2.42444 > 192.168.1.3.80: Flags [S], seq 364460666, win 29200, options [mss 1460,sackOK,TS val 2559349669 ecr 0,nop,wscale 7], length 0
04:29:21.668573 fa:16:3e:f3:2c:61 > fa:16:3e:32:40:3d, ethertype IPv4 (0x0800), length 74: 192.168.1.3.80 > 192.168.1.2.42444: Flags [S.], seq 4137876027, ack 364460667, win 14480, options [mss 1460,sackOK,TS val 4294931608 ecr 2559349669,nop,wscale 4], length 0
04:29:21.669023 fa:16:3e:32:40:3d > fa:16:3e:f3:2c:61, ethertype IPv4 (0x0800), length 66: 192.168.1.2.42444 > 192.168.1.3.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 2559349670 ecr 4294931608], length 0
04:29:21.669077 fa:16:3e:32:40:3d > fa:16:3e:f3:2c:61, ethertype IPv4 (0x0800), length 141: 192.168.1.2.42444 > 192.168.1.3.80: Flags [P.], seq 1:76, ack 1, win 229, options [nop,nop,TS val 2559349670 ecr 4294931608], length 75: HTTP: GET / HTTP/1.1
04:29:21.669979 fa:16:3e:f3:2c:61 > fa:16:3e:32:40:3d, ethertype IPv4 (0x0800), length 66: 192.168.1.3.80 > 192.168.1.2.42444: Flags [.], ack 76, win 905, options [nop,nop,TS val 4294931609 ecr 2559349670], length 0
04:29:21.672658 fa:16:3e:f3:2c:61 > fa:16:3e:32:40:3d, ethertype IPv4 (0x0800), length 90: 192.168.1.3.80 > 192.168.1.2.42444: Flags [P.], seq 1:25, ack 76, win 905, options [nop,nop,TS val 4294931610 ecr 2559349670], length 24: HTTP
04:29:21.672710 fa:16:3e:32:40:3d > fa:16:3e:f3:2c:61, ethertype IPv4 (0x0800), length 66: 192.168.1.2.42444 > 192.168.1.3.80: Flags [.], ack 25, win 229, options [nop,nop,TS val 2559349671 ecr 4294931610], length 0
04:29:55.543336 fa:16:3e:f3:2c:61 > fa:16:3e:32:40:3d, ethertype IPv4 (0x0800), length 66: 192.168.1.3.80 > 192.168.1.2.42444: Flags [F.], seq 25, ack 76, win 905, options [nop,nop,TS val 4294940076 ecr 2559349671], length 0
04:29:55.579959 fa:16:3e:32:40:3d > fa:16:3e:f3:2c:61, ethertype IPv4 (0x0800), length 66: 192.168.1.2.42444 > 192.168.1.3.80: Flags [F.], seq 76, ack 26, win 229, options [nop,nop,TS val 2559358148 ecr 4294940076], length 0
04:29:55.580711 fa:16:3e:f3:2c:61 > fa:16:3e:32:40:3d, ethertype IPv4 (0x0800), length 66: 192.168.1.3.80 > 192.168.1.2.42444: Flags [.], ack 77, win 905, options [nop,nop,TS val 4294940087 ecr 2559358148], length 0
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel

Bystander:

root@osa-master-ovs:/home/ubuntu# tcpdump -i tap2bceb97c-0b -ne port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap2bceb97c-0b, link-type EN10MB (Ethernet), capture size 262144 bytes
04:29:21.668768 fa:16:3e:f3:2c:61 > fa:16:3e:32:40:3d, ethertype IPv4 (0x0800), length 74: 192.168.1.3.80 > 192.168.1.2.42444: Flags [S.], seq 4137876027, ack 364460667, win 14480, options [mss 1460,sackOK,TS val 4294931608 ecr 2559349669...

[Peter Slovak (slovak-peto)]

@james-denton, do you by any chance use such IP addresses inside your VMs that Neutron isn't aware of? We spotted the exact same behavior in our setup when we created Neutron ports with IP addresses A, B, C and additionally used IPs X, Y, Z inside the VMs.

The X, Y and Z IPs were managed by Corosync and to be able to communicate with them, we used a somewhat lazy allowed_address_pair 0.0.0.0/0 on the three ports. As we realized later, the OVS firewall driver takes the port IPs and allowed_address_pairs into account when creating some egress flows [1].

Then we ended up with flow groups with criteria "nw_src=10.10.1.1" (the IP on Neutron port) and "nw_src=0.0.0.0" (IP from allowed_address_pairs). The first IP never had any problems with flooding and when using it as source, the target MAC was learned by ovs and appeared in the br-int fdb. I'm guessing adding IPs X, Y and Z to allowed_address_pairs will solve this, but have to verify this yet.

[1] https://github.com/openstack/neutron/blob/newton-eol/neutron/agent/linux/openvswitch_firewall/firewall.py#L405

[Peter Slovak (slovak-peto)]

Nevermind, the fix with allowed_address_pairs only worked intermittently in our setup. I haven't successfully traced why it worked in some occasions and didn't work in others. Consider my previous advice invalid.

[Yang Li (yang-li)]

I also get the same problem, always no mac in fdb tables that cause tcp ack packet flood to other port, is there a solution for this problem?

[Anton Kurbatov (akurbatov)]

Also get the same. It looks like VM hosted on the same node as a victim VM may sniff all traffic and password/cookies/etc may then be leaked.

[James Denton (james-denton)]

This may have been resolved around the Pike timeframe, IIRC. Can you please confirm which version of OpenStack you’re using?

[Yang Li (yang-li)]

My openstack version is Newton, can you confirm which patch solved this problem, thanks a lot !!
BTW，Not only same node‘s vm， the vms in different nodes also has this problem。

[Yang Li (yang-li)]

I modified the openflow, and it worked, no more flood packets.
cookie=0x85cd1a977dd54be0, duration=0.359s, table=82, n_packets=0, n_bytes=0, idle_age=518, priority=70,ct_state=+est-rel-rpl,tcp,reg5=0xd,dl_dst=fa:16:3e:80:cb:0a,tp_dst=22 actions=NORMAL

I'm not sure is there any problem for this modification, it's a workaround, not compatible with later version of neutron

[Jesse (jesse-5)]

After tested, It seems that this issue is caused by ovs-fw ingress flows with strip_vlan,output=<port_id> in table=81 and table=82.
When I ping VM's floatingip from outside. for example, floatingip for VM is 172.24.0.157, internal ip is 192.168.111.18/fa:16:3e:b2:c2:84, outside host IP is 172.24.0.3/e6:73:51:97:74:4e.
If outside host has no arp entry for 172.24.0.157, it will send arp broadcast, and br-int on host which VM on will update it's fdb entry.

[root@node-2 ~]# ovs-appctl fdb/show br-int | grep e6:73:51:97:74:4e
    2 3 e6:73:51:97:74:4e 3

This fdb entry timeout is 300s, even though VM ping continues, this entry will not be updated because strip_vlan,output=<port_id> flow. This flow will forward ping package to VM and do not update fdb entry in br-int.
After 300s, this entry disappear, The reply icmp package will flood in br-int bridge (This do not affect the original ping, but flood will affect other VMs on this host. If other VMs has ingress QoS, this flood will affect other VM's network connection).
If you delete 172.24.0.157 fdb entry on outside host by `arp -d 172.24.0.157`, the fdb entry in br-int will come back, and flood is stopped.

To solve this problem, comment #8 is a solution, for ingress package, we just do actions=NORMAL at last to let br-int update fdb entry.

[Anton Kurbatov (akurbatov)]

I am also agree flooding is caused due to
- for egress trffic there is action=NORMAL
- for ingress traffic there is direct action=output:PORT
in the openflow rules.

NORMAL action causes openvswitch to flood traffic if there is no dst mac entry in fdb table
(https://github.com/openvswitch/ovs/blob/master/ofproto/ofproto-dpif-xlate.c#L3109)
At the same time MAC address is put into fdb table only if NORMAL action is performed (note, no mac learning is performed for direct output:PORT action)

E.g.
- a VM with m_mac sends packets to a node with node_mac -> it forces an ovs to learn vm_mac and remember this mac in br-int fdb table, then packet is flooded to all ovs ports;
- the node responses to vm_mac using output:VM_PORT rule -> the packet is sent directly to the VM port (without node_mac learning)
- the VM again sends packet to the node -> there is no node_mac in the fdb table and it forces ovs another packet flooding to all ports.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/639009

[Junien F (axino)]

Note : an ARP request with broadcast ethernet address (ff:ff:ff:ff:ff:ff) will trigger MAC learning. This is used by VMs when the IP is not already in their MAC table. After what, they switch to unicast ARP requests, which _don't_ trigger MAC learning - so the FDB entry expires after 5 min (by default).

[Junien F (axino)]

This is also GREATLY impacting VM throughput if you have multiple VMs in the same neutron network on a hypervisor.

[Junien F (axino)]

Note : I'm also seeing this behaviour with the iptables_hybrid firewall driver

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/666991

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

(The duplicate bug 1813439 was previously being tracked as a potential vulnerability report.)

[Jeremy Stanley (fungi)]

Is this related to bug 1837252? The symptoms seem very similar.

[sean mooney (sean-k-mooney)]

no this is not related to bug 1837252

ml2/ovs with ovs firewall does not use hybrid plug so there is no linux bridge between ovs
and the tap device so they are complete different bugs caused by different issues.

the effect is similar but not the same underlying cause or network backend.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

We are currently testing this bug in our development systems. ASAP we'll provide some feedback.

Regards.

[Miguel Lavalle (minsel)]

@fungi,

The testing Rodolfo is talking about in #19 will help decide the scope of any potential vulnerability. will keep you posted

[LIU Yulong (dragon889)]

We have an updated fix locally which can cover both enable/disable openflow security group (firewall).
I will update it to this patch set: https://review.opendev.org/#/c/666991/

And also I added a new bug for the new solution, it will add a flow table to do something like a switch FDB table. The accepted egress flows will be take care in that. For more information please see here:
https://bugs.launchpad.net/neutron/+bug/1841622

[Slawek Kaplonski (slaweq)]

This bug has had a related patch abandoned and has been automatically un-assigned due to inactivity. Please re-assign yourself if you are continuing work or adjust the state as appropriate if it is no longer valid.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/639009
Reason: This review is > 4 weeks without comment and currently blocked by a core reviewer with a -2. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and contacting the reviewer with the -2 on this review to ensure you address their concerns.

[Jeremy Stanley (fungi)]

It seems like bug 1841622 was opened for an alternative fix for this even though all the discussion about the problem happened here. The code change associated with that duplicate bug is currently active, but the bug is not marked as a suspected vulnerability like this one.

Is there still any indication that this bug could represent an exploitable security vulnerability in deployments, which can be patched safely in supported stable branches as well? If not, then we should probably convert this to a regular bug report and close the security advisory task associated with it.

Regardless, I suspect bug 1841622 should be folded back into this one to limit confusion since it's not reporting a bug but rather describing a potential fix for this existing bug.

[Slawek Kaplonski (slaweq)]

Hi Jeremy,

I marked bug 1841622 as duplicate of this one.
Regarding Your question about potential vulnerability, I don't think there is any here. This bug is about broadcasting packets to other ports from the same tenant network. Other networks are always using another vlan tag locally in br-int so there is no way that such packets can be seen on interfaces which belongs to other networks.

[James Denton (james-denton)]

There are cases where the same network is shared across multiple tenants relying only on security groups to filter. So, I would say the broadcast of packets to all instances in the same vlan presented by this bug is less than ideal.

[LIU Yulong (dragon889)]

@James, yes, that's we called shared network. It can be used from different projects.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/666991
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=efa8dd08957b5b6b1a05f0ed412ff00462a9f216
Submitter: Zuul
Branch: master

commit efa8dd08957b5b6b1a05f0ed412ff00462a9f216
Author: LIU Yulong <email address hidden>
Date: Mon Jun 24 00:08:12 2019 +0800

    Add accepted egress direct flow

    Do not flood the packets to bridge, since we have the
    bridge port list, we can add a simple direct flow to
    the right port only.

    Closes-Bug: #1732067
    Related-Bug: #1841622
    Change-Id: I14fefe289a19b718b247bf0740ca9bc47f8903f4

[Jeremy Stanley (fungi)]

Is there any plan for backporting this fix to supported stable branches?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/704506

[Slawek Kaplonski (slaweq)]

Hi fungi,

I proposed backport to stable/train as it was without any conflicts. For older branches it seems it may require some more work and maybe some other patches to be backported too but I don't think there is any problem with backporting this fix.
It introduced new config option, but it has default value that guarantee old behavior so I think we still can backport it.

[Jeremy Stanley (fungi)]

If mitigating this on some supported stable branches is going to require the operator to make configuration changes, then this is probably better handled as a security note than an advisory.

[Jeremy Stanley (fungi)]

Once the stable branch changes merge, this will be a class B1 report (default config value is insecure): https://security.openstack.org/vmt-process.html#incident-report-taxonomy

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/704506
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6c1cf886671df66f6790045268b1394a97caabcc
Submitter: Zuul
Branch: stable/train

commit 6c1cf886671df66f6790045268b1394a97caabcc
Author: LIU Yulong <email address hidden>
Date: Mon Jun 24 00:08:12 2019 +0800

    Add accepted egress direct flow

    Do not flood the packets to bridge, since we have the
    bridge port list, we can add a simple direct flow to
    the right port only.

    Closes-Bug: #1732067
    Related-Bug: #1841622
    Change-Id: I14fefe289a19b718b247bf0740ca9bc47f8903f4
    (cherry picked from commit efa8dd08957b5b6b1a05f0ed412ff00462a9f216)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/709410

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/710183

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/710184

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/709410
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0255239ed5affd65568c33ba07cd72fb0b594449
Submitter: Zuul
Branch: stable/queens

commit 0255239ed5affd65568c33ba07cd72fb0b594449
Author: LIU Yulong <email address hidden>
Date: Mon Jun 24 00:08:12 2019 +0800

    Add accepted egress direct flow

    Do not flood the packets to bridge, since we have the
    bridge port list, we can add a simple direct flow to
    the right port only.

    Conflicts:
     neutron/agent/linux/openvswitch_firewall/firewall.py
     neutron/conf/plugins/ml2/drivers/ovs_conf.py
     neutron/tests/unit/agent/linux/openvswitch_firewall/test_firewall.py
     neutron/tests/unit/plugins/ml2/drivers/openvswitch/agent/openflow/native/test_br_int.py

    Closes-Bug: #1732067
    Related-Bug: #1841622
    Change-Id: I14fefe289a19b718b247bf0740ca9bc47f8903f4
    (cherry picked from commit efa8dd08957b5b6b1a05f0ed412ff00462a9f216)

[Yi Yang (yangyi01)]

Hi, Yulong, has this fix patch completely fixed this bug or just partially fixed it? My colleague tried this patch, but br-int can't learn MAC yet, what's wrong?

In addition, I noticed this fix patch just changed agent code, does this mean we just apply it to ovs agent in compute node, neutron server and neutron database needn't do any change, right?

I files another bug, https://bugs.launchpad.net/neutron/+bug/1866445, I'm not sure if this fix patch is effective to firewall_driver=iptables_hybrid.

[Peter Slovak (slovak-peto)]

Yi Yang, I don't think the purpose of this patch is learning the MAC address. The fix is implemented by adding a flow that sends traffic destined to a MAC directly to a specific interface, thus short-circuiting the default NORMAL action which (correctly) broadcasts the traffic to all interfaces except the incoming one.

So from the high-level point of view, after this fix is applied, maybe you don't actually need br-int to learn the destination MAC address. It's only necessary if there still existed traffic that didn't match the new direct flow and instead would fall back to the flow with NORMAL action. If you witnessed such a case, you're probably right to create a new bug report.

I don't think neutron-server has anything to do with this change. Flows are populated by agents, so only neutron-openvswitch-agent would have to be restarted in my opinion.

I also don't think that firewall_driver setting is relevant to this change. The flow added in this patch, as I see it (correct me if I'm wrong), sits before any firewall rules whether you're using iptables_hybrid or ovs. This patch is related to traffic forwarding, not traffic filtering.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/710184
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5578c7073fd454f2df43e0d728ba2a64de87fccf
Submitter: Zuul
Branch: stable/stein

commit 5578c7073fd454f2df43e0d728ba2a64de87fccf
Author: LIU Yulong <email address hidden>
Date: Mon Jun 24 00:08:12 2019 +0800

    Add accepted egress direct flow

    Do not flood the packets to bridge, since we have the
    bridge port list, we can add a simple direct flow to
    the right port only.

    Conflicts:
     neutron/conf/plugins/ml2/drivers/ovs_conf.py

    Closes-Bug: #1732067
    Related-Bug: #1841622
    Change-Id: I14fefe289a19b718b247bf0740ca9bc47f8903f4
    (cherry picked from commit efa8dd08957b5b6b1a05f0ed412ff00462a9f216)

[Yi Yang (yangyi01)]

@Peter, I have confirmed it doesn't work for the case firewall_driver=iptables_hybrid. I also find it can't learn local MAC after applied this fix patch. In addition, I also find iperf3 pps test with 16 bytes UDP packet has noticeable packet loss rate.

My issue is this fix patch is just a workaround or a real fix? What issue makes br-int not to be able to learn MAC addresses? Why can we not fix this MAC learning issue? We find if one subnet isn't added into a router, MAC learning is ok, but once it is added a router, br-int won't learn VMs' MACs in other compute nodes, I think it is an ovs openflow pipeline design defect, why can't we fix this design defect and make sure br-int can learn MACs as expected.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/712640

[Yi Yang (yangyi01)]

Hi, guys

I submitted a gerrit commit https://review.opendev.org/#/c/712640/, it can make sure Yulong's fix patch can work for firewall_driver=iptables_hybrid, please help review if it is ok, thanks a lot.

[Son Do Xuan (sondx25)]

- I execute to patch the code according to the commit:
https://review.opendev.org/#/q/efa8dd08957b5b6b1a05f0ed412ff00462a9f216

- "patch-tun" port of "br-int" switch learn MAC according to the ARP reply message. But timeout of "MAC address table" is only 300 seconds. After this time, packet continues flooding on "br-int" switch.

[Darragh O'Reilly (darragh-oreilly)]

I can reproduce with firewall_driver=iptables_hybrid and dvr. I attached a patch that directs only packets from the gateway to the vm ports. All other packets from tunnels will hit the NORMAL flow in table 60.

Without patch:
table=60, priority=20,dl_vlan=1,dl_dst=fa:16:3e:5b:72:d8 actions=strip_vlan,output:"qvod638e8da-88"
table=60, priority=3 actions=NORMAL

With patch (gateway_mac is fa:16:3e:f1:f3:06):
table=60, priority=20,dl_vlan=1,dl_src=fa:16:3e:f1:f3:06,dl_dst=fa:16:3e:5b:72:d8 actions=strip_vlan,output:"qvod638e8da-88"
table=60, priority=3 actions=NORMAL

@sondx25 - can you try this patch?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/710183
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fdc8987f791b24119ef22c434f8877df13bcdded
Submitter: Zuul
Branch: stable/rocky

commit fdc8987f791b24119ef22c434f8877df13bcdded
Author: LIU Yulong <email address hidden>
Date: Mon Jun 24 00:08:12 2019 +0800

    Add accepted egress direct flow

    Do not flood the packets to bridge, since we have the
    bridge port list, we can add a simple direct flow to
    the right port only.

    Conflicts:
        neutron/agent/linux/openvswitch_firewall/firewall.py
        neutron/conf/plugins/ml2/drivers/ovs_conf.py

    Closes-Bug: #1732067
    Related-Bug: #1841622
    Change-Id: I14fefe289a19b718b247bf0740ca9bc47f8903f4
    (cherry picked from commit efa8dd08957b5b6b1a05f0ed412ff00462a9f216)

[Arjun Baindur (abaindur)]

Is this truly fixed? The patch doesn't seem to fix the underlying issue at hand - ingress MACs (coming from br-vlan or br-tun) are NOT learned by br-int.

Also what are the implications of turning on this setting? The commit states:

"A new config option ``explicitly_egress_direct``, with default value False,
    was added for the aim of distinguishing clouds which are running the
    network node mixed with compute services, upstream neutron CI should be
    an example. In such situation, this ``explicitly_egress_direct`` should be
    set to False, because there are numerous cases from HA routers which can
    not be covered, particularly when you have centralized floating IPs running
    in such mixed hosts.
    Otherwise, set ``explicitly_egress_direct`` to True to avoid the flooding.
    One more note is if your network nodes are for networing services only, we
    recommand you disable all the security_group to get a higher performance."

I take that to say on a network node mixed with compute services it should be set to False. But isn't that the case for every node in DVR? The compute node hosts qrouter, fip router, and can also host dhcp services.

Or does it mean we not supposed to enable this setting on hosts with dvr_snat mode for l3-agent? The message states something about centralized fips - which are hosting on the snat router/node.

[Arjun Baindur (abaindur)]

This fix seems to break certain other scenarios by causing flooding in opposite direction: https://bugs.launchpad.net/neutron/+bug/1884708

[Arjun Baindur (abaindur)]

We are seeing ingress flooding occur due to enabling this fix on other networks. We have a VLAN based network, both attached and unattached to a DVR router. In both cases, on ingress the NORMAL action at table=60 is hit.

This is because for a provider network, there are no dvr MAC rules with an explicit output: action to a local port. In DVR subnet, these rules are added with the EXTERNAL vlan/segmentation ID. Not with the local vlan. However, there is a table=0, priority=3 rule which converts from the external VLAN to host local VLAN.

NORMAL action at table=60 for ingress, requires the local VM MACs to be learned. But these are never learned, because the fix resubmits egress packets to table 61, which has a direct output: action northbound.

So now we have flooding in opposite direction. But we still need this fix because in other scenarios, we see the egress flooding.

https://bugs.launchpad.net/neutron/+bug/1884708

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/738397

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/752672

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/752676

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rodolfo Alonso Hernandez (<email address hidden>) on branch: master
Review: https://review.opendev.org/752672

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/738397
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/712640
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738551
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=959d8b6d73e2a6ab1a45c9a7b0b05ae163e650fc
Submitter: Zuul
Branch: master

commit 959d8b6d73e2a6ab1a45c9a7b0b05ae163e650fc
Author: LIU Yulong <email address hidden>
Date: Fri Jul 10 17:25:15 2020 +0800

    Local mac direct flow for non-openflow firewall

    When there is no openflow firewall, aka the ovs agent security group
    is disabled or Noop/HybridIptable, this patch will introduce a different
    ingress pipeline for bridge ports which will avoid ingress flood:
    (1) table=0, in_port=patch_bridge,dl_vlan=physical_vlan action=mod_vlan:local_vlan,goto:60 (original)
    (2) table=60, in_port=patch_bridge action=goto:61 (new)
    (3) table=61, dl_dst=local_port_mac,dl_vlan=local_vlan, action=strip_vlan,output:<ofport> (changes)

    And changes the local ports pipeline:
    (1) table=0, in_port=local_ofport action=goto:25 (original)
    (2) table=25, in_port=local_ofport,dl_src=local_port_mac action=goto:60 (original)
    (3) table=60, in_port=local_ofport,dl_src=local_port_mac action=local_vlan->reg6,goto:61 (changes)
    (4) table=61, dl_dst=local_port_mac,reg6=local_vlan, action=output:<ofport> (changes)

    Closes-Bug: #1884708
    Closes-Bug: #1881070
    Related-Bug: #1732067
    Related-Bug: #1866445
    Related-Bug: #1883321

    Change-Id: Iecf9cffaf02616342f1727ad7db85545d8adbec2

[OpenStack Infra (hudson-openstack)]

Change abandoned by Yi Yang (<email address hidden>) on branch: master
Review: https://review.opendev.org/712640
Reason: https://review.opendev.org/#/c/738551/ has covered this, so abandon it.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/759363

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/759364

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/759365

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/759366

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/759367

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/759369

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/759364
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ef14d258eea91ef563c63334b2da1623d93418f3
Submitter: Zuul
Branch: stable/ussuri

commit ef14d258eea91ef563c63334b2da1623d93418f3
Author: LIU Yulong <email address hidden>
Date: Fri Jul 10 17:25:15 2020 +0800

    Local mac direct flow for non-openflow firewall

    When there is no openflow firewall, aka the ovs agent security group
    is disabled or Noop/HybridIptable, this patch will introduce a different
    ingress pipeline for bridge ports which will avoid ingress flood:
    (1) table=0, in_port=patch_bridge,dl_vlan=physical_vlan action=mod_vlan:local_vlan,goto:60 (original)
    (2) table=60, in_port=patch_bridge action=goto:61 (new)
    (3) table=61, dl_dst=local_port_mac,dl_vlan=local_vlan, action=strip_vlan,output:<ofport> (changes)

    And changes the local ports pipeline:
    (1) table=0, in_port=local_ofport action=goto:25 (original)
    (2) table=25, in_port=local_ofport,dl_src=local_port_mac action=goto:60 (original)
    (3) table=60, in_port=local_ofport,dl_src=local_port_mac action=local_vlan->reg6,goto:61 (changes)
    (4) table=61, dl_dst=local_port_mac,reg6=local_vlan, action=output:<ofport> (changes)

    Closes-Bug: #1884708
    Closes-Bug: #1881070
    Related-Bug: #1732067
    Related-Bug: #1866445
    Related-Bug: #1883321

    Change-Id: Iecf9cffaf02616342f1727ad7db85545d8adbec2
    (cherry picked from commit 959d8b6d73e2a6ab1a45c9a7b0b05ae163e650fc)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/759365
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c06895e8e78de06c25d36cb347313240432953cf
Submitter: Zuul
Branch: stable/train

commit c06895e8e78de06c25d36cb347313240432953cf
Author: LIU Yulong <email address hidden>
Date: Fri Jul 10 17:25:15 2020 +0800

    Local mac direct flow for non-openflow firewall

    When there is no openflow firewall, aka the ovs agent security group
    is disabled or Noop/HybridIptable, this patch will introduce a different
    ingress pipeline for bridge ports which will avoid ingress flood:
    (1) table=0, in_port=patch_bridge,dl_vlan=physical_vlan action=mod_vlan:local_vlan,goto:60 (original)
    (2) table=60, in_port=patch_bridge action=goto:61 (new)
    (3) table=61, dl_dst=local_port_mac,dl_vlan=local_vlan, action=strip_vlan,output:<ofport> (changes)

    And changes the local ports pipeline:
    (1) table=0, in_port=local_ofport action=goto:25 (original)
    (2) table=25, in_port=local_ofport,dl_src=local_port_mac action=goto:60 (original)
    (3) table=60, in_port=local_ofport,dl_src=local_port_mac action=local_vlan->reg6,goto:61 (changes)
    (4) table=61, dl_dst=local_port_mac,reg6=local_vlan, action=output:<ofport> (changes)

    Closes-Bug: #1884708
    Closes-Bug: #1881070
    Related-Bug: #1732067
    Related-Bug: #1866445
    Related-Bug: #1883321

    Change-Id: Iecf9cffaf02616342f1727ad7db85545d8adbec2
    (cherry picked from commit 959d8b6d73e2a6ab1a45c9a7b0b05ae163e650fc)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/759366
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7c757ad3372b5fe015ae4c5e3949c804e8515d20
Submitter: Zuul
Branch: stable/stein

commit 7c757ad3372b5fe015ae4c5e3949c804e8515d20
Author: LIU Yulong <email address hidden>
Date: Fri Jul 10 17:25:15 2020 +0800

    Local mac direct flow for non-openflow firewall

    When there is no openflow firewall, aka the ovs agent security group
    is disabled or Noop/HybridIptable, this patch will introduce a different
    ingress pipeline for bridge ports which will avoid ingress flood:
    (1) table=0, in_port=patch_bridge,dl_vlan=physical_vlan action=mod_vlan:local_vlan,goto:60 (original)
    (2) table=60, in_port=patch_bridge action=goto:61 (new)
    (3) table=61, dl_dst=local_port_mac,dl_vlan=local_vlan, action=strip_vlan,output:<ofport> (changes)

    And changes the local ports pipeline:
    (1) table=0, in_port=local_ofport action=goto:25 (original)
    (2) table=25, in_port=local_ofport,dl_src=local_port_mac action=goto:60 (original)
    (3) table=60, in_port=local_ofport,dl_src=local_port_mac action=local_vlan->reg6,goto:61 (changes)
    (4) table=61, dl_dst=local_port_mac,reg6=local_vlan, action=output:<ofport> (changes)

    Closes-Bug: #1884708
    Closes-Bug: #1881070
    Related-Bug: #1732067
    Related-Bug: #1866445
    Related-Bug: #1883321

    Change-Id: Iecf9cffaf02616342f1727ad7db85545d8adbec2
    (cherry picked from commit 959d8b6d73e2a6ab1a45c9a7b0b05ae163e650fc)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/759363
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=88bbb58c80b9c888371e25267715b155851d9278
Submitter: Zuul
Branch: stable/victoria

commit 88bbb58c80b9c888371e25267715b155851d9278
Author: LIU Yulong <email address hidden>
Date: Fri Jul 10 17:25:15 2020 +0800

    Local mac direct flow for non-openflow firewall

    When there is no openflow firewall, aka the ovs agent security group
    is disabled or Noop/HybridIptable, this patch will introduce a different
    ingress pipeline for bridge ports which will avoid ingress flood:
    (1) table=0, in_port=patch_bridge,dl_vlan=physical_vlan action=mod_vlan:local_vlan,goto:60 (original)
    (2) table=60, in_port=patch_bridge action=goto:61 (new)
    (3) table=61, dl_dst=local_port_mac,dl_vlan=local_vlan, action=strip_vlan,output:<ofport> (changes)

    And changes the local ports pipeline:
    (1) table=0, in_port=local_ofport action=goto:25 (original)
    (2) table=25, in_port=local_ofport,dl_src=local_port_mac action=goto:60 (original)
    (3) table=60, in_port=local_ofport,dl_src=local_port_mac action=local_vlan->reg6,goto:61 (changes)
    (4) table=61, dl_dst=local_port_mac,reg6=local_vlan, action=output:<ofport> (changes)

    Closes-Bug: #1884708
    Closes-Bug: #1881070
    Related-Bug: #1732067
    Related-Bug: #1866445
    Related-Bug: #1883321

    Change-Id: Iecf9cffaf02616342f1727ad7db85545d8adbec2
    (cherry picked from commit 959d8b6d73e2a6ab1a45c9a7b0b05ae163e650fc)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/752676
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7842344a774b9cd098d07b65e410440e91cc8f3c
Submitter: Zuul
Branch: master

commit 7842344a774b9cd098d07b65e410440e91cc8f3c
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Oct 7 10:43:48 2020 +0000

    Add property "noopfirewall_or_firewall_disabled"

    Added new property to "SecurityGroupAgentRpc" class,
    "noopfirewall_or_firewall_disabled". If the security group API is not
    enabled or the firewall configured is "NoopFirewallDriver", the property
    will return "True".

    Related-Bug: #1732067

    Change-Id: I64db2f259462570e4c1d8df7396d188519071431

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/759367
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7ce65c94786d2d144a49cb991575534d0771bb20
Submitter: Zuul
Branch: stable/rocky

commit 7ce65c94786d2d144a49cb991575534d0771bb20
Author: LIU Yulong <email address hidden>
Date: Fri Jul 10 17:25:15 2020 +0800

    Local mac direct flow for non-openflow firewall

    When there is no openflow firewall, aka the ovs agent security group
    is disabled or Noop/HybridIptable, this patch will introduce a different
    ingress pipeline for bridge ports which will avoid ingress flood:
    (1) table=0, in_port=patch_bridge,dl_vlan=physical_vlan action=mod_vlan:local_vlan,goto:60 (original)
    (2) table=60, in_port=patch_bridge action=goto:61 (new)
    (3) table=61, dl_dst=local_port_mac,dl_vlan=local_vlan, action=strip_vlan,output:<ofport> (changes)

    And changes the local ports pipeline:
    (1) table=0, in_port=local_ofport action=goto:25 (original)
    (2) table=25, in_port=local_ofport,dl_src=local_port_mac action=goto:60 (original)
    (3) table=60, in_port=local_ofport,dl_src=local_port_mac action=local_vlan->reg6,goto:61 (changes)
    (4) table=61, dl_dst=local_port_mac,reg6=local_vlan, action=output:<ofport> (changes)

    Closes-Bug: #1884708
    Closes-Bug: #1881070
    Related-Bug: #1732067
    Related-Bug: #1866445
    Related-Bug: #1883321

    Conflicts:
        neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
        neutron/tests/unit/plugins/ml2/drivers/openvswitch/agent/openflow/native/test_br_int.py

    Change-Id: Iecf9cffaf02616342f1727ad7db85545d8adbec2
    (cherry picked from commit 959d8b6d73e2a6ab1a45c9a7b0b05ae163e650fc)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/759369
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7fe3e3d1e04d206245480d28534d9ee11949a9fa
Submitter: Zuul
Branch: stable/queens

commit 7fe3e3d1e04d206245480d28534d9ee11949a9fa
Author: LIU Yulong <email address hidden>
Date: Fri Jul 10 17:25:15 2020 +0800

    Local mac direct flow for non-openflow firewall

    When there is no openflow firewall, aka the ovs agent security group
    is disabled or Noop/HybridIptable, this patch will introduce a different
    ingress pipeline for bridge ports which will avoid ingress flood:
    (1) table=0, in_port=patch_bridge,dl_vlan=physical_vlan action=mod_vlan:local_vlan,goto:60 (original)
    (2) table=60, in_port=patch_bridge action=goto:61 (new)
    (3) table=61, dl_dst=local_port_mac,dl_vlan=local_vlan, action=strip_vlan,output:<ofport> (changes)

    And changes the local ports pipeline:
    (1) table=0, in_port=local_ofport action=goto:25 (original)
    (2) table=25, in_port=local_ofport,dl_src=local_port_mac action=goto:60 (original)
    (3) table=60, in_port=local_ofport,dl_src=local_port_mac action=local_vlan->reg6,goto:61 (changes)
    (4) table=61, dl_dst=local_port_mac,reg6=local_vlan, action=output:<ofport> (changes)

    Closes-Bug: #1884708
    Closes-Bug: #1881070
    Related-Bug: #1732067
    Related-Bug: #1866445
    Related-Bug: #1883321

    Conflicts:
        neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
        neutron/tests/unit/plugins/ml2/drivers/openvswitch/agent/openflow/native/test_br_int.py

    Change-Id: Iecf9cffaf02616342f1727ad7db85545d8adbec2
    (cherry picked from commit 959d8b6d73e2a6ab1a45c9a7b0b05ae163e650fc)

[Rascal (rascal07)]

Hi

What is the planned release date of the package with fixes for ussuri on ubuntu focal?

[Jeremy Stanley (fungi)]

At least from an upstream openstack/neutron source code perspective, the fix was included in the 16.3.0 tag on the stable/ussuri branch two weeks ago.

[Eugene Nikanorov (enikanorov)]

With all the backports, neutron/Queens still exhibits unicast flooding, although, of egress traffic that is headed towards snat port (be it centralized or DVR).
E.g., other ports on br-int, sharing the same vlan are getting egress traffic from port, which is sending packets towards MAC that is not in fdb of br-int.

I believe, we need a similar fix to address egress side of the problem

[Slawek Kaplonski (slaweq)]

@Eugene, can You open new bug for that ingress traffic issue? And also can You check if that is also the thing in the master branch?

[Hua Zhang (zhhuabj)]

setting explicitly_egress_direct=true on l3ha+dvr env will introduce a regression issue, see lp 1945306

[LIU Yulong (dragon889)]

The patch of https://review.opendev.org/c/openstack/neutron/+/666991 which introduced the config option ``explicitly_egress_direct=True/False`` had fixed the following problems:
1. the egress flooding issue on br-int when enable openvswitch(openflow) security group driver
https://bugs.launchpad.net/neutron/+bug/1732067

2. fix the east-west traffic broken of dvr
https://bugs.launchpad.net/neutron/+bug/1831534 (this bug is for VLAN network, but the issue is not vlan only).

3. fix some potential ingress flood issue on br-int

And I had put some issues here as well:
https://bugs.launchpad.net/neutron/+bug/1934666/comments/5

So, not use explicitly_egress_direct=True, you have to face these issues.

Another thing is that as I said in the release note before, do not use ``explicitly_egress_direct=True`` in host which enable dvr_snat and compute service. There are too many cases need to cover, please try to combine the following cases for DVR:
1. vlan/vxlan
2. dvr/dvr+ha
3. agent mode(dvr, dvr_snat, dvr_no_external)
4. east-west traffic and north-south traffic with the Scenario of src and dest in or not in same host
5. IPv6
6. allowed_address_pair
7. enable/disabl openflow firewall
8. HA router failover
The final cases is too many to cover.

And FYI, we had mark that dvr_snat + compute services is not supported.
https://review.opendev.org/c/openstack/neutron/+/801503

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/814733

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron rocky-eol release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/814733
Reason: Please, feel free to restore the patch, address the comments and rebase the patch.