Can't pass "0000:01:00:1 NVIDIA Corporation TU102 High Definition Audio Controller" from qemu + xen + libvirt + virt-manager VM to windows 10 guest os

[Marietto]

Hello to everyone.

I'm trying to pass-through my nvidia RTX 2080 ti with qemu + xen + libvirt + virt-manager. I've realized that I can't pass every device located inside the IOMMU group. Basically,if I try to pass the "0000:01:00:1 NVIDIA Corporation TU102 High Definition Audio Controller",windows 10 guest os is not able to boot properly. It freezes forever on the boot. Anyway,if I decide to don't pass through that device to the vm,I'm not able to make work the nvidia driver and it reports the error 43,I'm sure because it needs also the missing component to work properly. I don't have this problem if I don't use libvirt. what to do ? I'm using ubuntu 20.04,xen version :

root@ziomario-z390aoruspro:/etc/xen# xl info
host : ziomario-z390aoruspro
release : 5.4.0-42-generic
version : #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020
machine : x86_64
nr_cpus : 16
max_cpu_id : 15
nr_nodes : 1
cores_per_socket : 8
threads_per_core : 2
cpu_mhz : 3600.603
hw_caps : bfebfbff:77faf3ff:2c100800:00000121:0000000f:009c6fbf:00000000:00000100
virt_caps : hvm hvm_directio
total_memory : 32627
free_memory : 14117
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 11
xen_extra : .4-pre
xen_version : 4.11.4-pre
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset :
xen_commandline : placeholder
cc_compiler : gcc (Ubuntu 9.2.1-31ubuntu3) 9.2.1 20200306
cc_compile_by : ubuntu-devel-di
cc_compile_domain : lists.ubuntu.com
cc_compile_date : Tue Mar 10 09:04:06 UTC 2020
build_id : 70edf50fce444a706eb5c69735c35c1838e4eaee
xend_config_format : 4

 and this xen / libvirt cfg file :

<domain type='xen'>
  <name>win10-sdb</name>
  <uuid>bd40e4ee-9ee5-4d63-9cec-ecd2424cb283</uuid>
  <memory unit='KiB'>2048000</memory>
  <currentMemory unit='KiB'>2048000</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='xenfv'>hvm</type>
    <loader type='rom'>/usr/lib/xen-4.11/boot/hvmloader</loader>
    <boot dev='hd'/>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='host-model'/>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/lib/xen-4.11/bin/qemu-system-i386</emulator>
    <disk type='block' device='disk'>
      <driver name='phy' type='raw'/>
      <source dev='/dev/sdb'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver type='raw'/>
      <target dev='sdb' bus='scsi'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='xenbus' index='0'/>
    <controller type='scsi' index='0'/>
    <controller type='usb' index='0'/>
    <interface type='bridge'>
      <mac address='00:16:3e:29:2d:36'/>
      <source bridge='virbr0'/>
      <model type='e1000'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' keymap='lt'>
      <listen type='address'/>
    </graphics>
    <video>
      <model type='vga' vram='16384' heads='1' primary='yes'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='no'>
      <driver name='xen'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='no'>
      <driver name='xen'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x2'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='no'>
      <driver name='xen'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x3'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x045e'/>
        <product id='0x02c4'/>
      </source>
    </hostdev>
    <memballoon model='xen'/>
  </devices>
</domain>

let me know.

[Marietto (marietto2008)]

lspci -nnk | grep nvidia

root@ziomario-z390aoruspro:/etc/xen# lspci -nnk | grep nvidia

Kernel modules: nvidiafb, nouveau, nvidia\_drm, nvidia

Kernel modules: i2c\_nvidia\_gpu

this is how to I assign the 4 devices of the NVIDIA iommu group to xen :

root@ziomario-z390aoruspro:/etc/xen# ./pciback-0000:01:00.0.sh 0000:01:00.0

root@ziomario-z390aoruspro:/etc/xen# ./pciback-0000:01:00.0.sh 0000:01:00.1

root@ziomario-z390aoruspro:/etc/xen# ./pciback-0000:01:00.0.sh 0000:01:00.2

root@ziomario-z390aoruspro:/etc/xen# ./pciback-0000:01:00.0.sh 0000:01:00.3

this is the script pciback.sh :

#!/bin/bash

if [ $# -eq 0 ]; then

echo "Require PCI devices in format: <domain>:<bus>:<slot>.<function>"

echo "Eg: $(basename $0) 0000:00:1b.0"

exit 1

fi

modprobe xen-pciback

for pcidev in $@; do

if [ -h /sys/bus/pci/devices/"$pcidev"/driver ]; then

echo "Unbinding $pcidev from" $(basename $(readlink /sys/bus/pci/devices/"$pcidev>

echo -n "$pcidev" > /sys/bus/pci/devices/"$pcidev"/driver/unbind

fi

echo "Binding $pcidev to pciback"

echo -n "$pcidev" > /sys/bus/pci/drivers/pciback/new_slot

echo -n "$pcidev" > /sys/bus/pci/drivers/pciback/bind

done

as I said,it works if I don't use libvirt / virt-manager. It means that I can pass also the audio device.

[Marietto (marietto2008)]

BEFORE to assign the devices to xen :

root@ziomario-z390aoruspro:/home/ziomario# lspci -nnk | grep nvidia

Kernel modules: nvidiafb, nouveau

Kernel driver in use: nvidia-gpu

Kernel modules: i2c\_nvidia\_gpu

ASSIGNING the devices to xen :

root@ziomario-z390aoruspro:/etc/xen# ./pciback-0000:01:00.0.sh 0000:01:00.0

Binding 0000:01:00.0 to pciback

root@ziomario-z390aoruspro:/etc/xen# ./pciback-0000:01:00.0.sh 0000:01:00.1

Unbinding 0000:01:00.1 from snd_hda_intel

Binding 0000:01:00.1 to pciback

root@ziomario-z390aoruspro:/etc/xen# ./pciback-0000:01:00.0.sh 0000:01:00.2

Unbinding 0000:01:00.2 from xhci_hcd

Binding 0000:01:00.2 to pciback

root@ziomario-z390aoruspro:/etc/xen# ./pciback-0000:01:00.0.sh 0000:01:00.3

Unbinding 0000:01:00.3 from nvidia-gpu

Binding 0000:01:00.3 to pciback

AFTER having assigned the devices to xen :

root@ziomario-z390aoruspro:/etc/xen# lspci -nnk | grep nvidia

Kernel modules: nvidiafb, nouveau

Kernel modules: i2c\_nvidia\_gpu

is this good ? because windows is not able to boot if I give this command :

root@ziomario-z390aoruspro:/etc/xen# ./pciback-0000:01:00.0.sh 0000:01:00.1

I see that the CPU is working hard,but windows does not boot after a long time.

I'm also using audio device : AC97

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping make Ubuntu better.

I believe this is too obscure a configuration combination for Ubuntu package maintainers to justify prioritising it, so I'm setting Importance to Low accordingly. I don't expect it to receive expert attention soon. Feel free to continue to use this bug to coordinate with others, and if you do find a specific solution then we can help with getting it landed.

In case it's not a bug, you can find community support options here: http://www.ubuntu.com/support/community