mlx5_core ovs/ovn hardware offload - broken network connectivity between instances

Bug #1907451 reported by James Page
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
charm-ovn-chassis
New
Undecided
Unassigned

Bug Description

Ubuntu: Focal
OpenStack: Ussuri
OVN: 20.03.1
OVS: 2.12.1

Mellanox ConnectX 5 Ex cards connected to 100G TOR Cumulus switches in MLAG configuration.

Leaf and Spine network topology between racks.

8 core instances
Ubuntu: Focal

For some instances, networking works just fine - for others they are not able to connect to other instances on the same project network running on other hypervisors.

A ping will work for the first packet (not offloaded) but subsequent packets are lost for example indicating that the issue lies somewhere in the offloaded flows on the eswitch.

Tags: ps5
James Page (james-page)
tags: added: ps5
Revision history for this message
Frode Nordahl (fnordahl) wrote :

Observation:
With everything Groovy and ct offload enabled the problem is not there. If I disable port security the problem IS there, so perhaps incorrect flows are created or the card is interpreting them incorrectly, the offloaded flows do indeed look quite different when pivoting port security on/off.

The non-offloaded instances still work dandy in either case.

Revision history for this message
James Page (james-page) wrote :

Example:

$ ping 10.5.2.128
PING 10.5.2.128 (10.5.2.128) 56(84) bytes of data.
64 bytes from 10.5.2.128: icmp_seq=1 ttl=64 time=12.9 ms
^C
--- 10.5.2.128 ping statistics ---
2 packets transmitted, 1 received, 50% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.938/12.938/12.938/0.000 ms

Revision history for this message
James Page (james-page) wrote :

Digging a bit more - I see that the instances with broken network is on the same hypervisor as an instance on the same project network:

ps5-ra4-n4.maas

all other instances ended up on different hypervisors so maybe its a conflicting offloaded flow.

Revision history for this message
James Page (james-page) wrote :

If I delete one of the machines on the hypervisor then rebuild the other (forcing a rebuild of some sort) connectivity is restored:

$ ssh 10.5.0.247 "ping -c 2 10.5.2.128"
Warning: Permanently added '10.5.0.247' (ECDSA) to the list of known hosts.
PING 10.5.2.128 (10.5.2.128) 56(84) bytes of data.
64 bytes from 10.5.2.128: icmp_seq=1 ttl=64 time=0.149 ms
64 bytes from 10.5.2.128: icmp_seq=2 ttl=64 time=0.197 ms

Revision history for this message
Frode Nordahl (fnordahl) wrote :

As mentioned in #1 everything appear to work with port security enabled and a 5.8 kernel which has support for offload of conntrack flows. But as we know that is a bit too bleeding edge for production use at this point in time.

To try an alternative approach I took the OVN VXLAN support patch [0] and backported it to the 20.03 package in OVN.

Testing so far suggests that solves the issue, so that might indicate that the root of the issue is in NIC firmware and/or driver for the Geneve offload support. So perhaps we need to go with VXLAN for now and move back to Geneve once the 5.8 kernel becomes stable enough to be consumable in production?

0: https://github.com/ovn-org/ovn/commit/b07f1bc3d068e231e40840b0d977bd57158987fd

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.