mlx5_core ovs/ovn hardware offload - broken network connectivity between instances

[James Page]

Ubuntu: Focal
OpenStack: Ussuri
OVN: 20.03.1
OVS: 2.12.1

Mellanox ConnectX 5 Ex cards connected to 100G TOR Cumulus switches in MLAG configuration.

Leaf and Spine network topology between racks.

8 core instances
Ubuntu: Focal

For some instances, networking works just fine - for others they are not able to connect to other instances on the same project network running on other hypervisors.

A ping will work for the first packet (not offloaded) but subsequent packets are lost for example indicating that the issue lies somewhere in the offloaded flows on the eswitch.

[Frode Nordahl (fnordahl)]

Observation:
With everything Groovy and ct offload enabled the problem is not there. If I disable port security the problem IS there, so perhaps incorrect flows are created or the card is interpreting them incorrectly, the offloaded flows do indeed look quite different when pivoting port security on/off.

The non-offloaded instances still work dandy in either case.

[James Page (james-page)]

Example:

$ ping 10.5.2.128
PING 10.5.2.128 (10.5.2.128) 56(84) bytes of data.
64 bytes from 10.5.2.128: icmp_seq=1 ttl=64 time=12.9 ms
^C
--- 10.5.2.128 ping statistics ---
2 packets transmitted, 1 received, 50% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.938/12.938/12.938/0.000 ms

[James Page (james-page)]

Digging a bit more - I see that the instances with broken network is on the same hypervisor as an instance on the same project network:

ps5-ra4-n4.maas

all other instances ended up on different hypervisors so maybe its a conflicting offloaded flow.

[James Page (james-page)]

If I delete one of the machines on the hypervisor then rebuild the other (forcing a rebuild of some sort) connectivity is restored:

$ ssh 10.5.0.247 "ping -c 2 10.5.2.128"
Warning: Permanently added '10.5.0.247' (ECDSA) to the list of known hosts.
PING 10.5.2.128 (10.5.2.128) 56(84) bytes of data.
64 bytes from 10.5.2.128: icmp_seq=1 ttl=64 time=0.149 ms
64 bytes from 10.5.2.128: icmp_seq=2 ttl=64 time=0.197 ms

[Frode Nordahl (fnordahl)]

As mentioned in #1 everything appear to work with port security enabled and a 5.8 kernel which has support for offload of conntrack flows. But as we know that is a bit too bleeding edge for production use at this point in time.

To try an alternative approach I took the OVN VXLAN support patch [0] and backported it to the 20.03 package in OVN.

Testing so far suggests that solves the issue, so that might indicate that the root of the issue is in NIC firmware and/or driver for the Geneve offload support. So perhaps we need to go with VXLAN for now and move back to Geneve once the 5.8 kernel becomes stable enough to be consumable in production?

0: https://github.com/ovn-org/ovn/commit/b07f1bc3d068e231e40840b0d977bd57158987fd