ip commands error with mellanox devices in switchdev mode

[James Page]

Kernel: using 5.0.0-23-generic from hwe-edge

Configuring a mellanox connectx device with configured VF's into switchdev (rather than legacy) mode result in the ip cli tool erroring when trying to query the interface state:

$ sudo ip link
Error: Buffer too small for object.
Dump terminated

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: iproute2 4.15.0-2ubuntu1
ProcVersionSignature: Ubuntu 5.0.0-23.24~18.04.1-generic 5.0.15
Uname: Linux 5.0.0-23-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.7
Architecture: amd64
Date: Fri Oct 25 14:55:36 2019
ProcEnviron:
 TERM=screen-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
SourcePackage: iproute2
UpgradeStatus: No upgrade log present (probably fresh install)

[James Page (james-page)]

Also tried with the iproute2 in bionic-backports - same result

[James Page (james-page)]

strace from failing command

[James Page (james-page)]

Managed to figure out why this was happening - I had the NUM_VFS in the card firmware configured to 127 (the maximum value) - reducing this to a lower number allowed me to successfully switch the cards into switchdev mode at which point the ip tools all worked again.

The clue that pointed to this appeared when using the proposed 5.3 hwe edge kernel:

  [ 694.027106] infiniband (null): mlx5_ib_alloc_counters:5452:(pid 47479): couldn't allocate queue counter for port 128, err -12

(128 being the pertinent value)

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1849856

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Terry Rudd (terrykrudd)]

Is there further work to do on this given the apparent resolution of the problem?

[James Page (james-page)]

@terrykrudd - I think there is a bug here somewhere as the card should support 127 VF's (and 1 PF) per port - setting the firmware to this limit results in this error.

I'm not sure whether this problem is in the kernel driver code or in the underlying firmware.

[James Page (james-page)]

FWIW same is seem with the 5.3.0 kernel that's just appeared in HWE edge.