mlx5_core ovs/ovn hardware offload - Offload stops working at scale

[Frode Nordahl]

Ubuntu: Focal
OpenStack: Ussuri
OVN: 20.03.1
OVS: 2.13.1

During testing hardware offload at scale we got the following error message logged:
[339311.693051] mlx5_core 0000:81:00.0: mlx5_cmd_check:769:(pid 2730780): CREATE_FLOW_TABLE(0x930) op_mod(0x0) failed, status limits exceeded(0x8), syndrome (0x4bf0e8)
[339311.710421] mlx5_core 0000:81:00.0: E-Switch: Failed to create FDB Table err -12 (table prio: 5, level: 0, size: 4194304)

At Linux 5.4 the number of large groups in the FDB flow table is hard coded into the driver.

Newer kernel versions allow tuning this through a devlink parameter as Mellanox mentions in their documentation [0].

0: https://docs.mellanox.com/pages/viewpage.action?pageId=39264792