Not able to passthrough > 32 PCIe devices to a KVM Guest
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Invalid
|
Undecided
|
Unassigned | ||
qemu (Ubuntu) |
Invalid
|
Undecided
|
Unassigned |
Bug Description
Using an Ubuntu Server 16.04-based host with KVM hypervisor installed, we are unable to launch a vanilla Ubuntu Server 16.04.4 guest with >= 32 PCIe devices. It is 100% reproducible. Using fewer PCIe devices works fine. We are using the vanilla kvm and qemu packages from the Canonical repos.
The ultimate goal is to create a KVM Guest wherein I can passthrough 44 PCI devices.
When a KVM Guest launches, it also has some internal PCIe devices including host bridge, USB, IDE (for virtual disk), and virtual nic etc.
Script used to launch all devices looks like this:
#!/bin/bash
NAME=16gpuvm
sudo qemu-img create -f qcow2 /home/lab/
sudo virt-install \
--name ${NAME} \
--ram 716800 \
--vcpus 88 \
--disk path=/home/
--network bridge=virbr0 \
--graphics none \
--host-device 34:00.0 \
--host-device 36:00.0 \
--host-device 39:00.0 \
--host-device 3b:00.0 \
--host-device 57:00.0 \
--host-device 59:00.0 \
--host-device 5c:00.0 \
--host-device 5e:00.0 \
--host-device 61:00.0 \
--host-device 62:00.0 \
--host-device 63:00.0 \
--host-device 65:00.0 \
--host-device 66:00.0 \
--host-device 67:00.0 \
--host-device 35:00.0 \
--host-device 3a:00.0 \
--host-device 58:00.0 \
--host-device 5d:00.0 \
--host-device 2e:00.0 \
--host-device 2f:00.0 \
--host-device 51:00.0 \
--host-device 52:00.0 \
--host-device b7:00.0 \
--host-device b9:00.0 \
--host-device bc:00.0 \
--host-device be:00.0 \
--host-device e0:00.0 \
--host-device e2:00.0 \
--host-device e5:00.0 \
--host-device e7:00.0 \
--host-device c1:00.0 \
--host-device c2:00.0 \
--host-device c3:00.0 \
--host-device c5:00.0 \
--host-device c6:00.0 \
--host-device c7:00.0 \
--host-device b8:00.0 \
--host-device bd:00.0 \
--host-device e1:00.0 \
--host-device e6:00.0 \
--host-device b1:00.0 \
--host-device b2:00.0 \
--host-device da:00.0 \
--host-device db:00.0 \
--console pty,target_
--location http://
--initrd-
--extra-args="
console=
locale=en_US
console-
console-
console-
keyboard-
keyboard-
hostname=${NAME}
file=file:
"
Passing > 32 device causes this issue: 32nd device hits a DPC error and the host/HV crashes:
Apr 25 22:34:35 xpl-evt-16 kernel: [18125.977496] dpc 0000:5b:
Apr 25 22:34:35 xpl-evt-16 kernel: [18125.977500] dpc 0000:5b:
Apr 25 22:34:35 xpl-evt-16 kernel: [18125.994326] vfio-pci 0000:5e:00.0: Refused to change power state, currently in D3
Apr 25 22:34:35 xpl-evt-16 kernel: [18125.994427] iommu: Removing device 0000:5e:00.0 from group 92
From syslog (attached)
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.194358] dpc 0000:bb:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.194387] dpc 0000:bb:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.194413] dpc 0000:d9:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.194439] dpc 0000:d9:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.194472] dpc 0000:d9:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.194499] dpc 0000:d9:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.194526] dpc 0000:d9:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.194553] dpc 0000:d9:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.194583] dpc 0000:df:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.194619] dpc 0000:df:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.194649] dpc 0000:df:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.194679] dpc 0000:e4:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.194709] dpc 0000:e4:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.194742] dpc 0000:e4:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.194763] pciehp 0000:00:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.195036] pciehp 0000:60:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.195278] pciehp 0000:60:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.195513] pciehp 0000:c0:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.195753] pciehp 0000:c0:
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.196196] efifb: probing for efifb
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.196242] efifb: framebuffer at 0x9c000000, using 1920k, total 1920k
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.196247] efifb: mode is 800x600x32, linelength=3200, pages=1
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.196250] efifb: scrolling: redraw
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.196254] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.206652] Console: switching to colour frame buffer device 100x37
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.217034] fb0: EFI VGA frame buffer device
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.217173] intel_idle: MWAIT substates: 0x2020
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.217174] intel_idle: v0.4.1 model 0x55
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.220874] intel_idle: lapic_timer_
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.221219] input: Power Button as /devices/
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.221590] ACPI: Power Button [PWRF]
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.231089] ERST: Error Record Serialization Table (ERST) support is initialized.
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.231312] pstore: using zlib compression
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.231444] pstore: Registered erst as persistent store backend
Apr 25 22:37:13 xpl-evt-16 kernel: [ 2.232503] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
All PCI devices go offline include NVMe.
OS Drives go away, RAID-1 is remounted as RO, and eventually system crashes.
Apr 25 22:37:13 xpl-evt-16 rsyslogd-2007: action 'action 9' suspended, next retry is Wed Apr 25 22:37:43 2018 [v8.16.0 try http://
Apr 25 22:37:13 xpl-evt-16 systemd-
Apr 25 22:37:13 xpl-evt-16 systemd-
Apr 25 22:37:13 xpl-evt-16 systemd[1]: Starting Apply Kernel Variables...
Apr 25 22:37:13 xpl-evt-16 systemd[1]: Mounted Configuration File System.
Apr 25 22:37:13 xpl-evt-16 systemd[1]: Mounted FUSE Control File System.
Apr 25 22:37:13 xpl-evt-16 systemd[1]: Started Apply Kernel Variables.
Apr 25 22:37:13 xpl-evt-16 systemd[1]: Found device /dev/disk/
Apr 25 22:37:13 xpl-evt-16 systemd[1]: Starting File System Check on /dev/disk/
Apr 25 22:37:13 xpl-evt-16 systemd[1]: Started File System Check Daemon to report status.
Apr 25 22:37:13 xpl-evt-16 systemd-fsck[1576]: fsck.fat 3.0.28 (2015-05-16)
Apr 25 22:37:13 xpl-evt-16 systemd-fsck[1576]: /dev/nvme0n1p1: 10 files, 1168/130812 clusters
Apr 25 22:37:13 xpl-evt-16 systemd[1]: Started File System Check on /dev/disk/
Apr 25 22:37:13 xpl-evt-16 systemd[1]: Mounting /boot/efi...
Apr 25 22:37:13 xpl-evt-16 systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.
Apr 25 22:37:13 xpl-evt-16 systemd[1]: Mounted /boot/efi.
Apr 25 22:37:13 xpl-evt-16 systemd[1]: Reached target Local File Systems.
Apr 25 22:37:13 xpl-evt-16 systemd[1]: Starting Preprocess NFS configuration...
Apr 25 22:37:13 xpl-evt-16 systemd[1]: Starting Create Volatile Files and Directories...
Apr 25 22:37:13 xpl-evt-16 systemd-
Apr 25 22:37:13 xpl-evt-16 systemd[1]: Starting openibd - configure Mellanox devices...
Apr 25 22:37:13 xpl-evt-16 kernel: [ 0.000000] random: get_random_bytes called from start_kernel+
Changed in qemu (Ubuntu): | |
importance: | Undecided → Critical |
Changed in qemu (Ubuntu): | |
importance: | Critical → High |
Changed in qemu (Ubuntu): | |
importance: | High → Undecided |
Changed in linux (Ubuntu): | |
status: | Incomplete → Confirmed |
If the host kernel crashes, this is certainly rather a KVM bug than a QEMU bug, so you should report this to the KVM / kernel mailing list instead of opening an (upstream) QEMU bug ticket.