There are libvirtd errors in /var/log/syslog when powering on or off virtual machine in Ubuntu 22.04 LTS

[conie chang]

Hardware Configuration:
Server MTM: Bona POR SE450
OS installed: Ubuntu 22.04 LTS (ubuntu-22.04-live-server-amd64.iso)
BMC Version: 1.40 (Build ID: USX311E)
UEFI Version: 1.03 (Build ID: CME105C)
LXPM Version: 3.19 (Build ID: CML103A)
CPU: Xeon Silver 4314 16C 135W 2.4GHz
DIMM2,4,7,9: Hynix 32GB 2Rx4 3200 MHz RDIMM HMA84GR7CJR4N-XNTG; HMA84GR7CJR4N-XNT4
DIMM1,3,8,10: Intel Optane DCPMM_128GB QVR2
IO Config:
RAID: Intel VROC (VMD NVMe RAID) Premium
SLOT5: NIC: ThinkSystem Mellanox ConnectX-6 Dx 100GbE QSFP56 2-port PCIe 4 Ethernet Adapter
SLOT7: NIC: Intel I350 1GbE RJ45 4-port Internal Lock OCP Ethernet Adapter
Storage:
DISK1: ThinkSystem 2.5” 7mm U.3 7400P 1.92TB Read Intensive NVMe PCIe 4.0 x4 Trayless SSD
DISK2: ThinkSystem 2.5” 7mm U.3 7400P 1.92TB Read Intensive NVMe PCIe 4.0 x4 Trayless SSD
PSU1: CFFv4 AC 1100W (Delta) 110V
PSU2: CFFv4 AC 1100W (Delta) 110V
M/B build version: PASS5 MB

Steps:
1. Power on SUT and install Ubuntu 22.04 LTS on VROC NVMe RAID 0 disk.
2. Install KVM features and desktop GUI.
3. Run "apt upgrade" to update packages.
4. Create a namespace on BPS DIMM App Direct Not Interleaved mode.
5. Run "virt-manager" then install Windows Server 2022 on the namespace.
6. Reboot Guest OS and Host OS to check if there is any unexpected error in system logs. (=> failed, there are libvirtd errors in /var/log/syslog when booting into OS or powering on/off the virtual machine:
libvirtd: internal error: A field data length violates the resource length boundary.
libvirtd: message repeated 3 times: [internal error: A field data length violates the resource length boundary.] )

Expected behaviors:
There are no any unexpected errors in system logs.

Current behaviors:
There are libvirtd errors in /var/log/syslog when booting into OS or powering on/off the virtual machine:
libvirtd: internal error: A field data length violates the resource length boundary.
libvirtd: message repeated 3 times: [internal error: A field data length violates the resource length boundary.]
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu82.1
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: pass
DistroRelease: Ubuntu 22.04
InstallationDate: Installed on 2022-08-18 (0 days ago)
InstallationMedia: Ubuntu-Server 22.04.1 LTS "Jammy Jellyfish" - Release amd64 (20220816)
Lsusb:
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 001 Device 003: ID 0624:0407 Avocent Corp. IBM 39M2894
 Bus 001 Device 004: ID 17ef:b000 Lenovo Virtual Keyboard and Mouse
 Bus 001 Device 002: ID 1d6b:0107 Linux Foundation USB Virtual Hub
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: Lenovo ThinkEdge SE450
Package: linux (not installed)
ProcCmdline: BOOT_IMAGE=/vmlinuz-5.15.0-46-generic root=/dev/mapper/ubuntu--vg--1-ubuntu--lv ro
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.15.0-46-generic root=/dev/mapper/ubuntu--vg--1-ubuntu--lv ro
ProcVersionSignature: Ubuntu 5.15.0-46.49-generic 5.15.39
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-5.15.0-46-generic N/A
 linux-backports-modules-5.15.0-46-generic N/A
 linux-firmware 20220329.git681281e4-0ubuntu3.4
RfKill:

Tags: jammy uec-images
Uname: Linux 5.15.0-46-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 08/08/2022
dmi.bios.release: 1.23
dmi.bios.vendor: Lenovo
dmi.bios.version: CME105F-1.23
dmi.board.asset.tag: none
dmi.board.name: 7D8TCTO1WW
dmi.board.vendor: Lenovo
dmi.board.version: 05
dmi.chassis.asset.tag: none
dmi.chassis.type: 23
dmi.chassis.vendor: Lenovo
dmi.chassis.version: none
dmi.ec.firmware.release: 1.40
dmi.modalias: dmi:bvnLenovo:bvrCME105F-1.23:bd08/08/2022:br1.23:efr1.40:svnLenovo:pnThinkEdgeSE450:pvr05:rvnLenovo:rn7D8TCTO1WW:rvr05:cvnLenovo:ct23:cvrnone:sku7D8TCTO1WW:
dmi.product.family: ThinkSystem
dmi.product.name: ThinkEdge SE450
dmi.product.sku: 7D8TCTO1WW
dmi.product.version: 05
dmi.sys.vendor: Lenovo

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1986520

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[conie chang (conie)]

There are libvirtd errors in /var/log/syslog when powering on or off virtual machine in Ubuntu 22.04 LTS

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1986520

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Lucas Kanashiro (lucaskanashiro)]

Thanks for taking the time to report this bug an trying to make Ubuntu better.

Is this the only relevant message you found in the syslog? Is this error causing some unexpected behavior apart from this message in the syslog? You mentioned the error in the log but did not mention what is the impact of that, is this not allowing you to reboot the VM?

I am setting the libvirt task to Incomplete until you provide more information, when you do that please set it back to New and we will take a look again.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1986520

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Christian Ehrhardt  (paelzer)]

You can usually replicate that behavior by starting a few guests and
instead of shutdown stopping the libvirt-guests service.
If that works in your case as well it will easen having a look, so let
me ask does that work for you?

[conie chang (conie)]

apport information

[conie chang (conie)]

apport information

[conie chang (conie)]

apport information

[conie chang (conie)]

apport information

[conie chang (conie)]

apport information

[conie chang (conie)]

apport information

[conie chang (conie)]

apport information

[conie chang (conie)]

apport information

[conie chang (conie)]

apport information

[conie chang (conie)]

apport information

[conie chang (conie)]

apport information

[conie chang (conie)]

apport information

[conie chang (conie)]

apport information

[conie chang (conie)]

apport information

[conie chang (conie)]

apport information

[conie chang (conie)]

apport information

[conie chang (conie)]

Even I does not create one VM, the messages still show below errors.
I will try Christian's suggest as comment#7 if it works.

===error message===
Aug 19 03:39:32 conie libvirtd[7772]: internal error: A field data length violates the resource length boundary.
Aug 19 03:39:32 conie libvirtd[7772]: message repeated 3 times: [ internal error: A field data length violates the resource length boundary.]

Thank you.

[conie chang (conie)]

Reply Christian's suggest #7

I tried not to create one VM and check the syslog after run "virt-manager", and then it also shows the these error message.

I am trying to narrow down the issue, and only execue the "systemctl restart libvirtd" instead of open "virt-manager". And the error message is also shown in syslog.

"internal error: A field data length violates the resource length boundary."

[conie chang (conie)]

After removing i350 OCP, the errors disappears in the syslog.
I also found there is this message "Unknown small resource type 06, will not decode more." when using "lspci -vvv" as comment 12# "Lspci.txt"

It looks like intel i350 OCP related issue. Any idea?

[Christian Ehrhardt  (paelzer)]

Hey,
the error message inside libvirt is from parsing PCI VPD Data.
If you say removing that card removes the message it seems to indicate that the VPD of that device is either
a) broken as coming from the device and needs a FW Update (or report to Intel to create one)
or
b) valid VPD data but uncommon and breaking the parser in libvirt

#1. libvirt
You could gather some info and then decide if it is more (a) or (b) to then report it directly to intel or to upstream libvirt [1] to have a look.

I think to check what libvirt can (or can not read) you could run
$ virsh nodedev-list
# now select the i350 OCP
$ virsh nodedev-dumpxml <the i350 device ID from above cmd>

That should cause libvirt to try to read and report you the VPD data of the device.
I'd expect this triggers the issue, maybe you find anything odd in there already ...

1b. debug
If 1 recerates the issue you might want to run the same with livbirt debug info enabled [3]
and report that here and/or upstream depending what you see.

#2. kernel
This is rather kernel dependent, it might be worth trying a few older/newer mainline kernels [2] to see if any might already behave different

#3. data
It might be great to report the actual VPD data exposed.
That would be
$ sudo cat /sys/devices/pci.../<ID>/<ID>/vpd

Until we (or intel or upstreams) have that data there is not much we can do, hence setting the bug to incomplete for now.

[1]: https://gitlab.com/libvirt/libvirt/-/issues
[2]: https://kernel.ubuntu.com/~kernel-ppa/mainline/
[3]: https://libvirt.org/docs/libvirt-appdev-guide-python/en-US/html/libvirt_application_development_guide_using_python-Debug.html

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for libvirt (Ubuntu) because there has been no activity for 60 days.]