Comment 4 for bug 1849856

Managed to figure out why this was happening - I had the NUM_VFS in the card firmware configured to 127 (the maximum value) - reducing this to a lower number allowed me to successfully switch the cards into switchdev mode at which point the ip tools all worked again.

The clue that pointed to this appeared when using the proposed 5.3 hwe edge kernel:

  [ 694.027106] infiniband (null): mlx5_ib_alloc_counters:5452:(pid 47479): couldn't allocate queue counter for port 128, err -12

(128 being the pertinent value)