mlx5_core ovs/ovn hardware offload - asymmetric network performance

[James Page]

Ubuntu: Focal
OpenStack: Ussuri
OVN: 20.03.1
OVS: 2.12.1

Mellanox ConnectX 5 Ex cards connected to 100G TOR Cumulus switches in MLAG configuration.

Leaf and Spine network topology between racks.

8 core instances
Ubuntu: Focal

iperf -P4 -c XXXX -e

performance between to instances on different hypervisors is consistent by asymmetric - ~90Gbps in one direction, and ~30Gbps in the other.

some dmesg entries that might be pertinent (similar messages on both hypervisors):

[92583.318283] mlx5_core 0000:81:00.1 enp129s0f1: Matching on GENEVE options: can't match on 0 data field
[92583.318286] mlx5_core 0000:81:00.0 enp129s0f0: Matching on GENEVE options: can't match on 0 data field
[92693.121984] mlx5_core 0000:81:00.1: cmd_work_handler:887:(pid 1137976): failed to allocate command entry
[92693.131803] vport 31 error -11 reading stats

VF's for hardware offloaded ports are allocated on enp129s0f0

Monitoring the representator port on the hypervisor the initial TCP setup is observed but not the main iperf test (so offloading appears to be working).

[James Page (james-page)]

[94427.800683] mlx5_core 0000:81:00.0 enp129s0f0: Matching on GENEVE options: can't match on 0 data field

may be a clue here as I only see that on one side of the link.