PCI-E passthrough of Nvidia GTX GFX card to Win 10 guest fails with "kvm_set_phys_mem: error registering slot: Invalid argument"

[Joe Clifford]

Problem:
Passthrough of a PCI-E Nvidia GTX 970 GFX card to a Windows 10 guest from a Debian Stretch host fails after recent changes to kvm in QEMU master/trunk. Before this recent commit, everything worked as expected.

QEMU Version:
Master/trunk pulled from github 4/10/17 ( git reflog: d147f7e815 HEAD@{0} )

Host:
Debian Stretch kernel SMP Debian 4.9.30-2+deb9u5 (2017-09-19) x86_64 GNU/Linux

Guest:
Windows 10 Professional

Issue is with this commit:
https://github.com/qemu/qemu/commit/f357f564be0bd45245b3ccfbbe20ace08fe83ca8

Subsequent commit does not help:
https://github.com/qemu/qemu/commit/3110cdbd8a4845c5b5fb861b0a664c56d993dd3c#diff-7b7a17f6e8ba4195198dd685073f43cb

Error output from qemu:
(qemu) kvm_set_phys_mem: error registering slot: Invalid argument

QEMU commandline used:

./sources/qemu/x86_64-softmmu/qemu-system-x86_64 -machine q35,accel=kvm -serial none -parallel none -name Windows \
-enable-kvm -cpu host,kvm=off,hv_vendor_id=sugoidesu,-hypervisor -smp 6,sockets=1,cores=3,threads=2 \
-m 8G -mem-path /dev/hugepages -mem-prealloc -balloon none \
-drive if=pflash,format=raw,readonly,file=vms/ovmf-x64/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=vms/ovmf-x64/ovmf-x64/OVMF_VARS-pure-efi.fd \
-rtc clock=host,base=localtime \
-readconfig ./vms/q35-virtio-graphical.cfg \
-object iothread,id=iothread0 -object iothread,id=iothread1 -object iothread,id=iothread2 -object iothread,id=iothread3 \
-device virtio-scsi-pci,iothread=iothread0,id=scsi0 -device virtio-scsi-pci,iothread=iothread1,id=scsi1 -device virtio-scsi-pci,iothread=iothread2,id=scsi2 -device virtio-scsi-pci,iothread=iothread3,id=scsi3 \
-device scsi-hd,bus=scsi0.0,drive=drive0,bootindex=1 -device scsi-hd,bus=scsi1.0,drive=drive1 -device scsi-hd,bus=scsi2.0,drive=drive2 -device scsi-hd,bus=scsi3.0,drive=drive3 -device scsi-hd,bus=scsi1.0,drive=drive4 -device scsi-hd,bus=scsi2.0,drive=drive5 -device scsi-hd,bus=scsi3.0,drive=drive6 -device scsi-hd,bus=scsi1.0,drive=drive7 -device scsi-hd,bus=scsi2.0,drive=drive8 -device scsi-hd,bus=scsi3.0,drive=drive9 \
-drive if=none,id=drive0,file=vms/w10p64.qcow2,format=qcow2,cache=none,discard=unmap \
-drive if=none,id=drive1,file=vms/w10p64-2.qcow2,format=qcow2,cache=none,discard=unmap \
-drive if=none,id=drive2,file=/dev/mapper/w10p64-3,format=raw,cache=none \
-drive if=none,id=drive3,file=vms/w10p64-4.qcow2,format=qcow2,cache=none \
-drive if=none,id=drive4,file=vms/w10p64-5.qcow2,format=qcow2,cache=none \
-drive if=none,id=drive5,file=vms/w10p64-6.qcow2,format=qcow2,cache=none,discard=unmap \
-drive if=none,id=drive6,file=/dev/mapper/w10p64-7,format=raw,cache=none \
-drive if=none,id=drive7,file=vms/w10p64-8.qcow2,format=qcow2,cache=none,discard=unmap \
-device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,multifunction=on \
-netdev type=tap,id=net1,ifname=tap1,script=no,downscript=no,vhost=on \
-device virtio-net-pci,netdev=net1,mac=52:54:00:18:32:c9,bus=pcie.2,addr=00.0,ioeventfd=on \
-device usb-host,bus=usb.0,hostbus=3,hostport=2.1 \
-device usb-host,hostbus=3,hostport=2.2 \
-device usb-host,bus=ich9-ehci-1.0,hostbus=3,hostport=2.4 \
-object input-linux,id=kbd1,evdev=/dev/input/event0,grab_all=yes,repeat=on \
-drive if=none,id=drive8,file=vms/w10p64.qcow2-9,format=qcow2,discard=unmap \
-drive if=none,id=drive9,file=vms/w10p64-10.qcow2,format=qcow2,cache=none,discard=unmap \
-device usb-host,bus=usb.0,hostbus=3,hostport=9 \
-monitor stdio

[Joe Clifford (joeclifford)]

lspci -tv
-[0000:00]-+-00.0 Intel Corporation 4th Gen Core Processor DRAM Controller
           +-01.0-[01]--+-00.0 NVIDIA Corporation GM204 [GeForce GTX 970]
           | \-00.1 NVIDIA Corporation GM204 High Definition Audio Controller
           +-02.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller
           +-03.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller
           +-14.0 Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI
           +-16.0 Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1
           +-19.0 Intel Corporation Ethernet Connection I217-V
           +-1a.0 Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2
           +-1b.0 Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller
           +-1c.0-[02]--
           +-1c.3-[03]----00.0 Broadcom Limited BCM4352 802.11ac Wireless Network Adapter
           +-1d.0 Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1
           +-1f.0 Intel Corporation Z87 Express LPC Controller
           +-1f.2 Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode]
           \-1f.3 Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller

[Joe Clifford (joeclifford)]

I should probably add that I am using a recent OVMF firmware from this repo: https://www.kraxel.org/repos/jenkins/edk2/ namely edk2.git-ovmf-x64-0-20170914.b2974.g7f2f96f1a8.noarch.rpm dated 18/9/17

[Laszlo Ersek (Red Hat) (lersek)]

There's another bug report / discussion thread on qemu-devel about the same commit:

http://<email address hidden>

I'll add a note to that thread about this LP report too.

[Joe Clifford (joeclifford)]

Apologies, the title of this bug might be very misleading. I've made a huge assumption that PCI-E GFX card pass-through was triggering the bug, purely because a Debian Stretch guest VM on the same host, using the same build of QEMU, which uses virtio-vga for graphics and the same version of OVMF firmware, boots up fine.

[Joe Clifford (joeclifford)]

More noise... OK, so I've tested the Windows 10 guest VM using the same criteria but eliminating PCI-E pass-through and using virtio-vga graphics instead, and the the guest boots up fine so perhaps my assumption is correct.

Let me know if you need to see the memory I/O regions or dmesg of my host etc.

[David Hildenbrand (davidhildenbrand)]

Fix will end up in QEMU master soon.