[ovn] ovs bridge included as destination port in ovs flows preventing HW offload for traffic between VMs on different provider-vlan networks

[Itai Levy]

Platform: Canonical Charmed OpenStack ("Yoga" release)

OS: 22.04 (Jammy), Kernel 5.15.0-39-generic, inbox network drivers (no MOFED)

OVS 2.17.0
OVN 22.03.0-0ubuntu1
Setup: CX6Dx, LAG (LACP MLAG), OVN as SDN

Issue:

Trying to run the following use case: traffic between VMs on different hosts, different subnets per VM, over vlan provider network. VM1 is iperf client, VM2 is iperf server.

According to OVS dumps, the flows created by OVN to handle the Tx/Rx traffic on node 1 hosting VM1 (which includes also the routing function between the networks) includes for some reason the SW bridge "br-nvda" as action destination port in addition to the physical port (bond0).

HW Offload will not work in this case - as we can see those flows are not offloaded.

ufid:15f82551-5cf9-4151-a60a-50e8258bab8a, skb_priority(0/0),skb_mark(0/0),ct_state(0/0x3f),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0),dp_hash(0/0),in_port(bond0),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:9c:5e:e0,dst=fa:16:3e:2b:a0:15),eth_type(0x8100),vlan(vid=101,pcp=0),encap(eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=0/0)), packets:9397051, bytes:14700492497, used:0.000s, dp:tc, actions:br-nvda,pop_vlan,ct(zone=7),recirc(0x754)

ufid:899dee5f-8841-4115-a37b-3585f7640cc9, skb_priority(0/0),skb_mark(0/0),ct_state(0x2a/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0x754),dp_hash(0/0),in_port(bond0),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=fa:16:3e:2b:a0:15),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=11.11.11.197,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:7588713, bytes:394647240, used:0.000s, offloaded:yes, dp:tc, actions:ens1f1npf1vf7

ufid:4f7f5d31-c54c-4752-af6c-80f373230d81, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ens1f1npf1vf7),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:2b:a0:15,dst=fa:16:3e:9c:5e:e0),eth_type(0x0800),ipv4(src=11.11.11.197,dst=22.22.22.128/255.255.255.128,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=0/0), packets:117335475, bytes:1053706955718, used:0.130s, offloaded:yes, dp:tc, actions:ct(zone=7),recirc(0x753)

ufid:5b3a1a8e-3d23-4cff-80ce-d9b04d5983e0, skb_priority(0/0),skb_mark(0/0),ct_state(0x22/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0x753),dp_hash(0/0),in_port(ens1f1npf1vf7),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:2b:a0:15,dst=fa:16:3e:9c:5e:e0),eth_type(0x0800),ipv4(src=11.11.11.192/255.255.255.224,dst=22.22.22.163,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0), packets:88045088, bytes:786251663770, used:0.000s, dp:tc, actions:ct_clear,set(eth(src=fa:16:3e:8f:6d:cf,dst=fa:16:3e:96:33:df)),set(ipv4(ttl=63)),push_vlan(vid=102,pcp=0),bond0,br-nvda

OVS config:

root@node4:/home/ubuntu# ovs-vsctl show
6d6b9dfe-0015-46d3-acaf-36395e1f43f1
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-nvda
        fail_mode: standalone
        datapath_type: system
        Port veth73ed544e
            tag: 9
            Interface veth73ed544e
        Port patch-provnet-58aabff2-1e12-42d5-9ac8-14309fff218e-to-br-int
            Interface patch-provnet-58aabff2-1e12-42d5-9ac8-14309fff218e-to-br-int
                type: patch
                options: {peer=patch-br-int-to-provnet-58aabff2-1e12-42d5-9ac8-14309fff218e}
        Port vethbc6d678e
            tag: 10
            Interface vethbc6d678e
        Port veth133ebbad
            tag: 9
            Interface veth133ebbad
        Port vethf2a1825b
            tag: 9
            Interface vethf2a1825b
        Port veth10230c73
            tag: 9
            Interface veth10230c73
        Port vethcbb2d467
            tag: 10
            Interface vethcbb2d467
        Port br-nvda.10
            tag: 10
            Interface br-nvda.10
                type: internal
        Port veth3f7b64b1
            tag: 10
            Interface veth3f7b64b1
        Port veth7ab5b2ed
            tag: 10
            Interface veth7ab5b2ed
        Port veth139bcf0d
            tag: 9
            Interface veth139bcf0d
        Port vethfb8fc976
            tag: 9
            Interface vethfb8fc976
        Port veth5ae35405
            tag: 10
            Interface veth5ae35405
        Port veth1b146461
            tag: 9
            Interface veth1b146461
        Port patch-provnet-ec0f96b1-404b-43a2-b817-aaf79420165b-to-br-int
            Interface patch-provnet-ec0f96b1-404b-43a2-b817-aaf79420165b-to-br-int
                type: patch
                options: {peer=patch-br-int-to-provnet-ec0f96b1-404b-43a2-b817-aaf79420165b}
        Port br-nvda.40
            tag: 40
            Interface br-nvda.40
                type: internal
        Port bond0
            Interface bond0
                type: system
        Port veth11ba1236
            tag: 9
            Interface veth11ba1236
        Port vethecac9608
            tag: 10
            Interface vethecac9608
        Port veth9d28bc1e
            tag: 10
            Interface veth9d28bc1e
        Port br-nvda
            Interface br-nvda
                type: internal
        Port patch-provnet-6393739e-d829-49cb-a51a-e90d44ac9fc4-to-br-int
            Interface patch-provnet-6393739e-d829-49cb-a51a-e90d44ac9fc4-to-br-int
                type: patch
                options: {peer=patch-br-int-to-provnet-6393739e-d829-49cb-a51a-e90d44ac9fc4}
        Port vethbe67c807
            tag: 10
            Interface vethbe67c807
        Port veth27a196f8
            tag: 10
            Interface veth27a196f8
        Port vethfb237fb7
            tag: 9
            Interface vethfb237fb7
        Port br-nvda.9
            tag: 9
            Interface br-nvda.9
                type: internal
    Bridge br-int
        fail_mode: secure
        datapath_type: system
        Port tap143edacb-e0
            Interface tap143edacb-e0
        Port patch-br-int-to-provnet-58aabff2-1e12-42d5-9ac8-14309fff218e
            Interface patch-br-int-to-provnet-58aabff2-1e12-42d5-9ac8-14309fff218e
                type: patch
                options: {peer=patch-provnet-58aabff2-1e12-42d5-9ac8-14309fff218e-to-br-int}
        Port ovn-node3.-0
            Interface ovn-node3.-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="172.16.0.10"}
                bfd_status: {diagnostic="Neighbor Signaled Session Down", flap_count="25", forwarding="true", remote_diagnostic="Control Detection Time Expired", remote_state=up, state=up}
        Port ens1f1npf1vf7
            Interface ens1f1npf1vf7
        Port patch-br-int-to-provnet-6393739e-d829-49cb-a51a-e90d44ac9fc4
            Interface patch-br-int-to-provnet-6393739e-d829-49cb-a51a-e90d44ac9fc4
                type: patch
                options: {peer=patch-provnet-6393739e-d829-49cb-a51a-e90d44ac9fc4-to-br-int}
        Port br-int
            Interface br-int
                type: internal
        Port patch-br-int-to-provnet-ec0f96b1-404b-43a2-b817-aaf79420165b
            Interface patch-br-int-to-provnet-ec0f96b1-404b-43a2-b817-aaf79420165b
                type: patch
                options: {peer=patch-provnet-ec0f96b1-404b-43a2-b817-aaf79420165b-to-br-int}
    ovs_version: "2.17.0"

[Itai Levy (etlvnvda)]

To reproduce:

openstack network create vlan_data --provider-physical-network tenantvlan --provider-network-type vlan --provider-segment 101 --share
openstack subnet create vlan_data_subnet --dhcp --network vlan_data --subnet-range 11.11.11.0/24 --allocation-pool start=11.11.11.100,end=11.11.11.200
openstack port create direct_vlan1 --vnic-type=direct --network vlan_data --binding-profile '{"capabilities":["switchdev"]}' --security-group my_policy

openstack network create vlan_data2 --provider-physical-network tenantvlan --provider-network-type vlan --provider-segment 102 --share
openstack subnet create vlan_data2_subnet --dhcp --network vlan_data2 --subnet-range 22.22.22.0/24 --allocation-pool start=22.22.22.100,end=22.22.22.200
openstack port create direct_vlan2 --vnic-type=direct --network vlan_data2 --binding-profile '{"capabilities":["switchdev"]}' --security-group my_policy

openstack router create vlan_router
openstack router add subnet vlan_router vlan_data_subnet
openstack router add subnet vlan_router vlan_data2_subnet

openstack server create --key-name bastion --flavor d1.demo --image perf --port direct_vlan1 vm1 --availability-zone nova:node3.maas
openstack server create --key-name bastion --flavor d1.demo --image perf --port direct_vlan2 vm2 --availability-zone nova:node4.maas