[ Impact ]
Memory leak causing growing memory footprints in long running libvirt processes. In a fairly busy openstack env, this showed a steady linear growth up to ~15GB after a couple of months.
This would impact many openstack deployments and anyone else using libvirt with particular PCI devices (VPD capable), forcing them to restart libvirt regularly to reset the memory consumption. This memory leak has only been observed so far in a hardware (metal) environment with mellanox devices, but ostensibly occurs wherever a VPD capable device exists.
[ Test Plan ]
It is only possible to reproduce this on certain hardware, seemingly hosts that have PCI cards that present VPD (Vital Product Data). For example, this was noticed on a host where libvirt was obtaining data from a mellanox card that presented vpd data. You can tell if a PCI device presents vpd data by looking at the sysfs entry /sys/bus/pci/devices/{address}/vpd, or from `lswh` if you see 'vpd' in the list of capabilities, for example:
*-network:0
description: Ethernet interface
product: MT2892 Family [ConnectX-6 Dx]
vendor: Mellanox Technologies
...snip...
capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation
...snip...
1. Knowing the server has a VPD capable device, and monitoring the memory consumption over time can show if the issue is present as well as when it is fixed. Before the fix there is a clear linear growth, which should flatten out after applying the patch.
2. This is also another simple test that can be done:
Run "virsh nodedev-list" for 1000 times, and check the memory occupied by the libvirtd service.
#!/bin/sh
systemctl start libvirtd
systemctl status libvirtd
i=0
while [ $i -ne 1000 ]
do
virsh nodedev-list
i=$(($i+1))
echo "$i"
done
systemctl status libvirtd
and watch the "Memory:" field grow (or not, if the fix is there).
[ Where problems could occur ]
The functions changed are only called in environments where VPD devices exist, and the patch adjusts pointers and contents of data structures related to VPD capable PCI devices found by libvirt.
Things could go wrong in environments wherever VPD capable devices are present, and may show up as garbage data about the device, null pointer where there should be data, segfaults.
[ Other Info ]
The backport is derived from an upstream fix:
https://github.com/libvirt/libvirt/commit/64d32118540aca3d42bc5ee21c8b780cafe04bfa
This patch is missing from Jammy and Kinetic, but present in Lunar+. The same issue has not been observed in a similar environment running Focal.
Running libvirt in valgrind which will stacks like the following:
==3411871== 7,559,541 (407,160 direct, 7,152,381 indirect) bytes in 16,965 blocks are definitely lost in loss record 2,846 of 2,846
==3411871== at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==3411871== by 0x4D53C50: g_malloc0 (gmem.c:161)
==3411871== by 0x49A2832: virPCIVPDParse (virpcivpd.c:672)
==3411871== by 0x4983BD8: virPCIDeviceGetVPD (virpci.c:2694)
==3411871== by 0x4A2CEB7: UnknownInlinedFun (node_device_conf.c:3032)
==3411871== by 0x4A2CEB7: virNodeDeviceGetPCIDynamicCaps (node_device_conf.c:3065)
==3411871== by 0x4A2D03D: virNodeDeviceUpdateCaps (node_device_conf.c:2636)
==3411871== by 0xFC8CD35: nodeDeviceGetXMLDesc (node_device_driver.c:370)
==3411871== by 0x4B7E9D1: virNodeDeviceGetXMLDesc (libvirt-nodedev.c:275)
==3411871== by 0x15519A: UnknownInlinedFun (remote_daemon_dispatch_stubs.h:15507)
==3411871== by 0x15519A: remoteDispatchNodeDeviceGetXMLDescHelper.lto_priv.0 (remote_daemon_dispatch_stubs.h:15484)
==3411871== by 0x4A59785: UnknownInlinedFun (virnetserverprogram.c:428)
==3411871== by 0x4A59785: virNetServerProgramDispatch (virnetserverprogram.c:302)
==3411871== by 0x4A60067: UnknownInlinedFun (virnetserver.c:140)
==3411871== by 0x4A60067: virNetServerHandleJob (virnetserver.c:160)
==3411871== by 0x499B982: virThreadPoolWorker (virthreadpool.c:164)
==3411871== by 0x499A4D8: virThreadHelper (virthread.c:241)
==3411871== by 0x514CB42: start_thread (pthread_create.c:442)
==3411871== by 0x51DDBB3: clone (clone.S:100)
Attached debdiff for jammy