After I realized and corrected two mistakes in my original bug report I managed to fully reproduce this bug. This is still not a minimal reproduction, but at least it works in an upstream environment. mistake #1: The firewall_driver should be openvswitch, not noop. mistake #2: The explicitly_egress_direct config option should be in section [agent], not [ovs]. The reproduction below actually demonstrates two bugs, this one and another one. To track the latter I'll open a new bug report, but will point out the 2nd bug here anyways. I also had to realize that this bug has security implications, because there is crosstalk between networks. In particular traffic meant for a trunk subport may be forwarded on a trunk's parent's network too. In most trunk port use cases I believe the severity of this crosstalk is limited, because these networks most of time belong to the same tenant. However if the parent network belongs to a different tenant or is a shared network then the crosstalk may be overheard by a different tenant too. The amount of traffic potentially overheard is also limited by the state of the trunk bridge's fdb. # environment two host devstack setup with ml2/ovs devstack0 - all in one host devstack0a - compute host (most commands are issued here) # devstack0a's correct config ml2_conf.ini: [securitygroup] firewall_driver = openvswitch [agent] explicitly_egress_direct = True [ovs] bridge_mappings = physnet0:br-physnet0,... # on devstack0a # arbitrary IP that we can ping from our vms on net1 sudo ip link set up dev br-physnet0 sudo ip link add link br-physnet0 name br-physnet0.101 type vlan id 101 sudo ip link set up dev br-physnet0.101 sudo ip address add dev br-physnet0.101 10.0.101.1/24 # code devstack 6b0f055b neutron dce8c34dd2 openvswitch 2.17.8-0ubuntu0.22.04.1 linux 5.15.0-91-generic # clean up, if not the first run openstack server delete vma --wait openstack network trunk delete trunka openstack port delete port1a port0a openstack network delete net1 net0 # build the environment openstack network create net0 --provider-network-type vlan --provider-physical-network physnet0 --provider-segment 100 openstack network create net1 --provider-network-type vlan --provider-physical-network physnet0 --provider-segment 101 openstack subnet create --network net0 --subnet-range 10.0.100.0/24 subnet0 openstack subnet create --network net1 --subnet-range 10.0.101.0/24 subnet1 openstack port create --no-security-group --disable-port-security --network net0 --fixed-ip ip-address=10.0.100.10 port0a port0a_mac="$( openstack port show port0a -f value -c mac_address )" openstack port create --no-security-group --disable-port-security --mac-address "$port0a_mac" --network net1 --fixed-ip ip-address=10.0.101.10 port1a # we know these two ports have the same mac (on different networks though) therefore later I may use port0a's mac where port1a's mac would be more logical openstack network trunk create --parent-port port0a trunka openstack network trunk set --subport port=port1a,segmentation-type=vlan,segmentation-id=101 trunka openstack server create --flavor ds1G --image u1804 --nic port-id=port0a --availability-zone :devstack0a --wait vma # mac addresses for reference $ openstack port show port0a -f value -c mac_address fa:16:3e:2a:0f:f5 # on devstack0a $ ifdata -ph br-physnet0 82:E8:18:67:7E:40 # on devstack0a sudo virsh console "$( openstack server show vma -f value -c OS-EXT-SRV-ATTR:instance_name )" # on vma # test trunk parent port ping -c3 10.0.100.2 # configure subport vlan subinterface in vm ip link add link ens3 name ens3.101 type vlan id 101 ip link set up dev ens3.101 dhclient -v ens3.101 # test trunk subport ping -c3 10.0.101.2 # on vma # kill all traffic that would refresh fdb entries iptables -A OUTPUT -o ens3 -j DROP ip6tables -A OUTPUT -o ens3 -j DROP iptables -A OUTPUT -o ens3.101 -j DROP ip6tables -A OUTPUT -o ens3.101 -j DROP # on devstack0a # flush fdbs sudo ovs-appctl fdb/flush br-physnet0 sudo ovs-appctl fdb/flush br-int sudo ovs-appctl fdb/flush "tbr-$( openstack network trunk show trunka -f value -c id | cut -b1-10 )" # on devstack0a # double check fdb content for all fdbs and relevant macs sudo ovs-appctl fdb/show br-physnet0 | egrep -i "$( openstack port show port0a -f value -c mac_address )|$( ifdata -ph br-physnet0 )" sudo ovs-appctl fdb/show br-int | egrep -i "$( openstack port show port0a -f value -c mac_address )|$( ifdata -ph br-physnet0 )" sudo ovs-appctl fdb/show "tbr-$( openstack network trunk show trunka -f value -c id | cut -b1-10 )" | egrep -i "$( openstack port show port0a -f value -c mac_address )|$( ifdata -ph br-physnet0 )" # all empty as expected # on devstack0a # scenario: subport and physnet bridge mac not in any fdbs, egress == vma trunka vnic -> physnet bridge $ sudo ovs-appctl ofproto/trace "tbr-$( openstack network trunk show trunka -f value -c id | cut -b1-10 )" in_port="$( sudo ovs-vsctl -- --columns=ofport find Interface name=$( echo "tap$( openstack port show port0a -f value -c id )" | cut -b1-14 ) | awk '{ print $3 }' )",dl_vlan=101,dl_dst=$( ifdata -ph br-physnet0 ),dl_src=$( openstack port show port0a -f value -c mac_address ) Flow: in_port=1,dl_vlan=101,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=fa:16:3e:2a:0f:f5,dl_dst=82:e8:18:67:7e:40,dl_type=0x0000 bridge("tbr-7f1eda7f-2") ------------------------ 0. priority 0 NORMAL -> no learned MAC for destination, flooding bridge("br-int") ---------------- 0. priority 0, cookie 0x9dba29ce7d7981b7 goto_table:58 58. priority 0, cookie 0x9dba29ce7d7981b7 goto_table:60 60. in_port=38, priority 100, cookie 0x9dba29ce7d7981b7 set_field:0x26->reg5 set_field:0x2->reg6 resubmit(,73) 73. reg5=0x26, priority 80, cookie 0x9dba29ce7d7981b7 resubmit(,94) 94. reg6=0x2,dl_src=fa:16:3e:2a:0f:f5,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00, priority 10, cookie 0x9dba29ce7d7981b7 push_vlan:0x8100 set_field:4098->vlan_vid output:1 bridge("br-physnet0") --------------------- 0. in_port=1,dl_vlan=2, priority 4, cookie 0xadc17b675a0a3114 set_field:4196->vlan_vid NORMAL -> no learned MAC for destination, flooding bridge("br-int") ---------------- 0. priority 0, cookie 0x9dba29ce7d7981b7 goto_table:58 58. priority 0, cookie 0x9dba29ce7d7981b7 goto_table:60 60. in_port=39, priority 100, cookie 0x9dba29ce7d7981b7 set_field:0x27->reg5 set_field:0x1->reg6 resubmit(,73) 73. reg5=0x27, priority 80, cookie 0x9dba29ce7d7981b7 resubmit(,94) 94. reg6=0x1,dl_src=fa:16:3e:2a:0f:f5,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00, priority 10, cookie 0x9dba29ce7d7981b7 push_vlan:0x8100 set_field:4097->vlan_vid output:1 bridge("br-physnet0") --------------------- 0. in_port=1,dl_vlan=1, priority 4, cookie 0xadc17b675a0a3114 set_field:4197->vlan_vid NORMAL -> no learned MAC for destination, flooding Final flow: unchanged Megaflow: recirc_id=0,eth,in_port=1,dl_vlan=101,dl_vlan_pcp=0,dl_src=fa:16:3e:2a:0f:f5,dl_dst=82:e8:18:67:7e:40,dl_type=0x0000 Datapath actions: 11,push_vlan(vid=100,pcp=0),2,pop_vlan,2 # The output of the empty-fdb-egress scenario means we forward the same packet twice from the tbr to br-int, once over the proper subport patch cable and a second time incorrectly over the parent patch cable. # The latter one will be incorrectly tagged and may be overheard by parties who should not hear this traffic. # This happens because the tpt port is incorrectly left in the default trunk configuration instead of being configured as an access port. # scenario: subport and physnet bridge mac not in any fdbs, ingress == physnet bridge -> vma trunka vnic $ sudo ovs-appctl ofproto/trace br-physnet0 in_port=LOCAL,dl_vlan=101,dl_src=$( ifdata -ph br-physnet0 ),dl_dst=$( openstack port show port0a -f value -c mac_address ) Flow: in_port=LOCAL,dl_vlan=101,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=82:e8:18:67:7e:40,dl_dst=fa:16:3e:2a:0f:f5,dl_type=0x0000 bridge("br-physnet0") --------------------- 0. priority 0, cookie 0xadc17b675a0a3114 NORMAL -> no learned MAC for destination, flooding bridge("br-int") ---------------- 0. in_port=1,dl_vlan=101, priority 3, cookie 0x9dba29ce7d7981b7 set_field:4097->vlan_vid goto_table:58 58. priority 0, cookie 0x9dba29ce7d7981b7 goto_table:60 60. priority 3, cookie 0x9dba29ce7d7981b7 NORMAL -> no learned MAC for destination, flooding bridge("br-tun") ---------------- 0. in_port=1, priority 1, cookie 0xc01333f352b6863b goto_table:2 2. dl_dst=00:00:00:00:00:00/01:00:00:00:00:00, priority 0, cookie 0xc01333f352b6863b goto_table:20 20. priority 0, cookie 0xc01333f352b6863b goto_table:22 22. priority 0, cookie 0xc01333f352b6863b drop bridge("tbr-7f1eda7f-2") ------------------------ 0. priority 0 NORMAL -> no learned MAC for destination, flooding bridge("br-int") ---------------- 0. priority 0, cookie 0x9dba29ce7d7981b7 goto_table:58 58. priority 0, cookie 0x9dba29ce7d7981b7 goto_table:60 60. in_port=38, priority 100, cookie 0x9dba29ce7d7981b7 set_field:0x26->reg5 set_field:0x2->reg6 resubmit(,73) 73. reg5=0x26, priority 80, cookie 0x9dba29ce7d7981b7 resubmit(,94) 94. reg6=0x2,dl_dst=fa:16:3e:2a:0f:f5, priority 12, cookie 0x9dba29ce7d7981b7 output:38 >> skipping output to input port Final flow: unchanged Megaflow: recirc_id=0,eth,in_port=LOCAL,dl_vlan=101,dl_vlan_pcp=0,dl_src=82:e8:18:67:7e:40,dl_dst=fa:16:3e:2a:0f:f5,dl_type=0x0000 Datapath actions: pop_vlan,push_vlan(vid=1,pcp=0),3,pop_vlan,10,push_vlan(vid=101,pcp=0),11,12 # The empty-fdb-ingress scenario demonstrates the forwarding loop my downstream colleagues originally observed (br-int -> tbr -> br-int). # In more complicated setups this can also lead to duplicated traffic. # It is not a "clear loop" though, because the packet enters br-int for the 2nd time as if it was coming from a different network. # on devstack0a sudo virsh console "$( openstack server show vma -f value -c OS-EXT-SRV-ATTR:instance_name )" # on vma # allow and generate traffic that will keep fdb entries always fresh, but to avoid confusion between the same mac on trunk parent and subport keep the parent port's traffic blocked iptables -D OUTPUT -o ens3.101 -j DROP ip6tables -D OUTPUT -o ens3.101 -j DROP ping 10.0.101.1 # on devstack0a # clear again all past junk sudo ovs-appctl fdb/flush br-physnet0 sudo ovs-appctl fdb/flush br-int sudo ovs-appctl fdb/flush "tbr-$( openstack network trunk show trunka -f value -c id | cut -b1-10 )" # on devstack0a # port VLAN MAC Age <-- header for the output below $ sudo ovs-appctl fdb/show br-physnet0 | egrep -i "$( openstack port show port0a -f value -c mac_address )|$( ifdata -ph br-physnet0 )" 1 100 fa:16:3e:2a:0f:f5 1 LOCAL 101 82:e8:18:67:7e:40 0 1 101 fa:16:3e:2a:0f:f5 0 $ sudo ovs-appctl fdb/show br-int | egrep -i "$( openstack port show port0a -f value -c mac_address )|$( ifdata -ph br-physnet0 )" 1 1 82:e8:18:67:7e:40 0 $ sudo ovs-appctl fdb/show "tbr-$( openstack network trunk show trunka -f value -c id | cut -b1-10 )" | egrep -i "$( openstack port show port0a -f value -c mac_address )|$( ifdata -ph br-physnet0 )" 3 101 82:e8:18:67:7e:40 0 1 101 fa:16:3e:2a:0f:f5 0 # In br-physnet0's fdb we clearly see that it has seen and learned the duplicated packet in the wrong network. # Remember the parent port is still in a killed state in the guest, br-physnet0 could not have learned from the traffic originating from the parent port. # Please also note, that br-int never learned about port1a's mac (which is the same as port0a's mac). # This is the root cause of the 2nd bug I mentioned at the top. # Using explicitly_egress_direct = True means that we have an asymmetric forwarding setup. # In the ingress direction we have a normal action (trying to use the fdb). # But in the egress direction we have a direct output action, never updating the fdb. # Therefore the normal action of the ingress direction must always flood, even if we have constant traffic from which br-int could learn. # This problem is not fixed by gerrit change #905125. # Even with https://review.opendev.org/c/openstack/neutron/+/905125 ps5 applied, the problem remains: $ sudo ovs-appctl fdb/flush br-int $ sleep 3 # while ping 10.0.101.1 is running $ sudo ovs-appctl fdb/show br-int | egrep -i "$( openstack port show port0a -f value -c mac_address )" [nothing] # br-int's fdb clearly does not learn this mac because of the ongoing ping traffic. # However I believe different (at the moment unidentified) traffic may occasionally hit a normal action in br-int and then br-int learns port0a's mac, but this is rare compared to the ping traffic we generate here. # I do not consider this problem to be a security bug. # For the sake of completeness I added below the outputs for the cases when all fdbs learned all relevant macs. # I believe there are no further problems in these parts. # scenario: subport and physnet bridge mac in all fdbs, egress == vma trunka vnic -> physnet bridge $ sudo ovs-appctl ofproto/trace "tbr-$( openstack network trunk show trunka -f value -c id | cut -b1-10 )" in_port="$( sudo ovs-vsctl -- --columns=ofport find Interface name=$( echo "tap$( openstack port show port0a -f value -c id )" | cut -b1-14 ) | awk '{ print $3 }' )",dl_vlan=101,dl_dst=$( ifdata -ph br-physnet0 ),dl_src=$( openstack port show port0a -f value -c mac_address ) Flow: in_port=1,dl_vlan=101,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=fa:16:3e:2a:0f:f5,dl_dst=82:e8:18:67:7e:40,dl_type=0x0000 bridge("tbr-7f1eda7f-2") ------------------------ 0. priority 0 NORMAL -> forwarding to learned port bridge("br-int") ---------------- 0. priority 0, cookie 0x9dba29ce7d7981b7 goto_table:58 58. priority 0, cookie 0x9dba29ce7d7981b7 goto_table:60 60. in_port=39, priority 100, cookie 0x9dba29ce7d7981b7 set_field:0x27->reg5 set_field:0x1->reg6 resubmit(,73) 73. reg5=0x27, priority 80, cookie 0x9dba29ce7d7981b7 resubmit(,94) 94. reg6=0x1,dl_src=fa:16:3e:2a:0f:f5,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00, priority 10, cookie 0x9dba29ce7d7981b7 push_vlan:0x8100 set_field:4097->vlan_vid output:1 bridge("br-physnet0") --------------------- 0. in_port=1,dl_vlan=1, priority 4, cookie 0xadc17b675a0a3114 set_field:4197->vlan_vid NORMAL -> forwarding to learned port Final flow: unchanged Megaflow: recirc_id=0,eth,in_port=1,dl_vlan=101,dl_src=fa:16:3e:2a:0f:f5,dl_dst=82:e8:18:67:7e:40,dl_type=0x0000 Datapath actions: 2 # scenario: subport and physnet bridge mac in all fdbs, ingress == physnet bridge -> vma trunka vnic $ sudo ovs-appctl ofproto/trace br-physnet0 in_port=LOCAL,dl_vlan=101,dl_src=$( ifdata -ph br-physnet0 ),dl_dst=$( openstack port show port0a -f value -c mac_address ) Flow: in_port=LOCAL,dl_vlan=101,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=82:e8:18:67:7e:40,dl_dst=fa:16:3e:2a:0f:f5,dl_type=0x0000 bridge("br-physnet0") --------------------- 0. priority 0, cookie 0xadc17b675a0a3114 NORMAL -> forwarding to learned port bridge("br-int") ---------------- 0. in_port=1,dl_vlan=101, priority 3, cookie 0x9dba29ce7d7981b7 set_field:4097->vlan_vid goto_table:58 58. priority 0, cookie 0x9dba29ce7d7981b7 goto_table:60 60. priority 3, cookie 0x9dba29ce7d7981b7 NORMAL -> no learned MAC for destination, flooding bridge("br-tun") ---------------- 0. in_port=1, priority 1, cookie 0xc01333f352b6863b goto_table:2 2. dl_dst=00:00:00:00:00:00/01:00:00:00:00:00, priority 0, cookie 0xc01333f352b6863b goto_table:20 20. priority 0, cookie 0xc01333f352b6863b goto_table:22 22. priority 0, cookie 0xc01333f352b6863b drop bridge("tbr-7f1eda7f-2") ------------------------ 0. priority 0 NORMAL -> forwarding to learned port Final flow: unchanged Megaflow: recirc_id=0,eth,in_port=LOCAL,dl_vlan=101,dl_vlan_pcp=0,dl_src=82:e8:18:67:7e:40,dl_dst=fa:16:3e:2a:0f:f5,dl_type=0x0000 Datapath actions: pop_vlan,push_vlan(vid=1,pcp=0),3,pop_vlan,10,push_vlan(vid=101,pcp=0),12