offloading disabled when using linuxbrige or bonding

[Junien F]

Hi,

I'm running Ubuntu 18.04 kernel 4.15.0-47-generic, openvswitch 2.10.0-0ubuntu2~cloud0 set up by neutron (from openstack rocky) - VXLAN overlay.

I've found out that as soon as a linuxbrige or bond is involved the network path of openvswitch, the offloading features of the NIC (checksumming, segmentation, VXLAN encap) get disabled, which reduces the throughput a lot.

You can find the flows below. Examples of setup used :

"Normal" setup :

br-bond0 (192.168.11.11)
 bond0
  enp50s0f0
  enp113s0f0 (handling traffic)

Only interface in the bond :

br-bond0 (192.168.11.11)
 bond0
  enp50s0f0 (handling traffic)

Interface directly in the bridge :

br-bond0 (192.168.11.11)
 enp50s0f0

With the setups above, there is NO offloading happening (tcpdump reports packet sizes of the MTU (~9k)). If the IP address is directly on the enp50s0f0 interface, then offloading works.

For what it's worth, the NICs are Mellanox MT27800.

Thanks !

OVS flows :

$ sudo ovs-vsctl show
6f7e2040-e13b-4029-86b7-4f360aad2efe
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge "br-physnet1"
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "phy-br-physnet1"
            Interface "phy-br-physnet1"
                type: patch
                options: {peer="int-br-physnet1"}
        Port "br-physnet1"
            Interface "br-physnet1"
                type: internal
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "sg-eecf32f7-da"
            tag: 2
            Interface "sg-eecf32f7-da"
                type: internal
        Port "qvoc654e76e-de"
            tag: 2
            Interface "qvoc654e76e-de"
        Port "qr-84debef4-b7"
            tag: 2
            Interface "qr-84debef4-b7"
                type: internal
        Port "qr-3e9ad9a7-02"
            tag: 3
            Interface "qr-3e9ad9a7-02"
                type: internal
        Port "fg-997d43cb-27"
            tag: 5
            Interface "fg-997d43cb-27"
                type: internal
        Port "qvoba76f8a7-50"
            tag: 2
            Interface "qvoba76f8a7-50"
        Port "qvod01a61c2-59"
            tag: 2
            Interface "qvod01a61c2-59"
        Port "enp50s0f0_60"
            tag: 2
            Interface "enp50s0f0_60"
        Port "tap15879e40-d9"
            tag: 2
            Interface "tap15879e40-d9"
        Port "int-br-physnet1"
            Interface "int-br-physnet1"
                type: patch
                options: {peer="phy-br-physnet1"}
        Port "qg-3f2f0aa4-fa"
            tag: 5
            Interface "qg-3f2f0aa4-fa"
                type: internal
        Port "ha-a481b72f-6a"
            tag: 4
            Interface "ha-a481b72f-6a"
                type: internal
        Port "enp50s0f0_62"
            tag: 2
            Interface "enp50s0f0_62"
        Port "sg-5e3c2880-79"
            tag: 3
            Interface "sg-5e3c2880-79"
                type: internal
        Port "tap62452ada-90"
            tag: 3
            Interface "tap62452ada-90"
        Port br-int
            Interface br-int
                type: internal
        Port "qvo75f1a5f8-e6"
            tag: 2
            Interface "qvo75f1a5f8-e6"
        Port "tapbc9a8481-c2"
            tag: 2
            Interface "tapbc9a8481-c2"
        Port "tapd7155195-25"
            tag: 1
            Interface "tapd7155195-25"
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qvo1e23c34e-5f"
            tag: 2
            Interface "qvo1e23c34e-5f"
        Port "enp50s0f0_61"
            tag: 2
            Interface "enp50s0f0_61"
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
        Port "vxlan-c0a80a0b"
            Interface "vxlan-c0a80a0b"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="192.168.11.11", out_key=flow, remote_ip="192.168.10.11"}
    ovs_version: "2.10.0"

Features for all interfaces :
$ for i in br-bond0 bond0 enp50s0f0 enp113s0f0; do sudo ethtool -k $i; echo "======"; done
Features for br-bond0:
rx-checksumming: off [fixed]
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: on
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: on
        tx-tcp-mangleid-segmentation: on
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: on [fixed]
tx-gso-robust: off [requested on]
tx-fcoe-segmentation: off [requested on]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: on
tx-esp-segmentation: on
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
======
Features for bond0:
rx-checksumming: off [fixed]
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [requested on]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: on
        tx-tcp-mangleid-segmentation: on
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: on [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
======
Features for enp50s0f0:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: on [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: on
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
======
Features for enp113s0f0:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: on [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: on
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
======

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: openvswitch-switch 2.10.0-0ubuntu2~cloud0 [origin: Canonical]
ProcVersionSignature: Ubuntu 4.15.0-47.50-generic 4.15.18
Uname: Linux 4.15.0-47-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.6
Architecture: amd64
CrashDB:
 {
                "impl": "launchpad",
                "project": "cloud-archive",
                "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
             }
Date: Mon Apr 29 09:28:24 2019
ProcEnviron:
 TERM=screen-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
SourcePackage: openvswitch
UpgradeStatus: No upgrade log present (probably fresh install)

[James Page (james-page)]

@axino

Just to confirm whether you tested - if you take OVS out of the picture and just test with bonded network interfaces, do you see offloading still being disabled? Slightly tricky for the VXLAN test but wanted to see if you had tested that combination.

I'm wondering whether this is an OVS issue or whether there is an issue with hardware offloading with linux bonds more generally.

[Junien F (axino)]

I don't think I tested with OVS out of the picture. However, none of the physical interface was enslaved to an OVS bridge, so I'd say this is not an ovs bug and can likely be repro'ed outside of OVS. I don't have time to do this in the short term though.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Junien F (axino)]

Correction : I tested today and this bug cannot be reproduced without OVS, at least on 4.15.0-55-generic. Offloading works properly even when the interface is in a linuxbridge or in a bond.

[James Page (james-page)]

Revisiting this bug with new information.

I can confirm that when a bridge is involved in the data path, hardware offloading of packets gets disabled as the network flow bumps to and from the kernel and the NIC.

However with bonding and appropriate switchdev configuration, offloading works fine and this is a more typical configuration. Sticking to OVS bridges rather than using Linux bridges as well will avoid this situation.

AFAICT this is working as intended so marking this bug as a wont fix.