libvirt display error after restart driver

[Chen Ben Enosh]

After restarting driver the syslog display message:
Jan 2 12:53:30 qa-h-vrt-044 libvirtd[35130]: nl_recv returned with error: No buffer space available.

However, all the system seems to work normally

*****************System-Information*****************

Host-Name : qa-h-vrt-044
Manufacturer : Dell Inc.
Product Name : PowerEdge R720
Cores : 24
Architecture : x86_64
Bios-Version : 2.5.2 Release Date: 01/28/2015
Linux Distro : Ubuntu 16.04.1 LTS

Kernel-Version : 4.8.0-34-generic
------------------OS Informationn-------------------
MFT-Version : 4.6.0-33
lldpad-Version : ethtool-Version: 4.5
----------------- HCA's--------------------
HCA 1: CX354A - ConnectX-3 Pro QSFP
FW Version : 2.40.5000

HCA 2: FW Version :
HCA 3: CX456A - ConnectX-4 QSFP
FW Version : 12.17.2020

HCA 4: CB192A - Connect-IB QSFP
FW Version : 10.16.1020

HCA 5: CX4121A - ConnectX-4 LX SFP28
FW Version : 14.17.2020

*****************System-Information*****************

root@qa-h-vrt-045:~# dpkg --list | grep libvirt
ii gir1.2-libvirt-glib-1.0:amd64 0.2.2-0.1ubuntu1 amd64 libvirt glib mainloop integration
ii libvirt-bin 1.3.1-1ubuntu10.6 amd64 programs for the libvirt library
ii libvirt-glib-1.0-0:amd64 0.2.2-0.1ubuntu1 amd64 libvirt glib mainloop integration
ii libvirt0:amd64 1.3.1-1ubuntu10.6 amd64 library for interfacing with different virtualization systems
ii python-libvirt 1.3.1-1ubuntu1 amd64 libvirt Python bindings

root@qa-h-vrt-045:~# uname -a
Linux qa-h-vrt-045 4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:55:08 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[Joshua Powers (powersj)]

Thanks for reporting this issue.

nl_recv is from libnl3. Can you provide the full libvirt and syslog to see if there is any additional data that might help? Also do you have identically configured systems that are also working as expected?

This message can happen when you have a number of I/O cards, especially newer ones with SRIOV capabilities that require more buffer space, but not sure if that is the case here. In Xenial the page size is set by:

 682 if (page_size == 0)
 683 page_size = getpagesize() * 4;

[Chen Ben Enosh (chenb)]

syslog after restart driver:
Jan 5 09:29:24 qa-h-vrt-044 systemd[1]: Reached target Host and Network Name Lookups.
Jan 5 09:29:24 qa-h-vrt-044 systemd[1]: Starting NFS status monitor for NFSv2/3 locking....
Jan 5 09:29:24 qa-h-vrt-044 systemd[1]: Starting Notify NFS peers of a restart...
Jan 5 09:29:24 qa-h-vrt-044 rpc.statd[7847]: Version 1.2.8 starting
Jan 5 09:29:24 qa-h-vrt-044 rpc.statd[7847]: Flags: TI-RPC
Jan 5 09:29:24 qa-h-vrt-044 sm-notify[7846]: Version 1.2.8 starting
Jan 5 09:29:24 qa-h-vrt-044 systemd[1]: Started Notify NFS peers of a restart.
Jan 5 09:29:24 qa-h-vrt-044 systemd[1]: Started NFS status monitor for NFSv2/3 locking..
Jan 5 09:29:24 qa-h-vrt-044 kernel: [57029.219054] FS-Cache: Loaded
Jan 5 09:29:24 qa-h-vrt-044 kernel: [57029.236040] FS-Cache: Netfs 'nfs' registered for caching
Jan 5 09:29:28 qa-h-vrt-044 kernel: [57033.623710] mlx4_en: enp65s0: Close port called
Jan 5 09:29:28 qa-h-vrt-044 kernel: [57033.884115] mlx4_en 0000:41:00.0: removed PHC
Jan 5 09:29:28 qa-h-vrt-044 kernel: [57033.885654] mlx4_en: enp65s0d1: Close port called
Jan 5 09:29:28 qa-h-vrt-044 systemd[1]: Stopping ifup for enp65s0...
Jan 5 09:29:28 qa-h-vrt-044 ifdown[7896]: Cannot find device "enp65s0"
Jan 5 09:29:29 qa-h-vrt-044 ifdown[7896]: Cannot find device "enp65s0"
Jan 5 09:29:29 qa-h-vrt-044 systemd[1]: Stopped if...

[Christian Ehrhardt  (paelzer)]

I'd quote something from that thread that I agree to for us I think:
"Up to now our position has been that this problem should be fixed in
libnl, so we have preferred to not patch libvirt for it, but instead get
libnl fixed."

Therefore adding libnl task.

Also in the past this was very driver specific to ixgbe.
From the log snippet I only see mlx4 warnings.

Could you elaborate on the cards/drivers available and used in your system?

[Chen Ben Enosh (chenb)]

which details do you need from my cards and drivers?

[Joshua Powers (powersj)]

@paelzer per the description and syslog this looks to be with Mellanox cards:

HCA 1: CX354A - ConnectX-3 Pro QSFP
HCA 3: CX456A - ConnectX-4 QSFP
HCA 4: CB192A - Connect-IB QSFP
HCA 5: CX4121A - ConnectX-4 LX SFP28

Which will apparently use the mlx4 and mlx5 drivers.

[Christian Ehrhardt  (paelzer)]

Clearing out old unresolved issues, this is really actually a problem within libnl more than anything else. The default should be ok (as Josh posted). I marked this a dup as there was another bug with better link references already.

But the TL;DR stays as I said in comment 3 - it is non fatal and should be addressed in libnl.