pci device not reset (passthrough)

[Gerben]

A restarted kvm client with a passthough pci device does not work anymore (properly) with that device. The device does not seem to be re-set properly.

If the KVM host is re-started the client one time starts properly with a correctly functioning pci device, when the client is re-started (without re-starting the kvm host) the pci device is not working properly.

Seen on device:
   USB device with linux client (USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller)
  The device is implemented with MSI-X, as a test a keyboard is connected.

First boot client (on fresh start of KVM host):
The client is presented the pci USB device (usb 2.0 and usb 3.0) and the keyboard is presented as well. The keyboard properly works.

Reboot:
The client is presented the pci USB device (usb 2.0 and usb 3.0), but no keyboard is presented (it is plugged in). Disconnecting and re-connecting the usb keyboard is not helping. Unregistering and re-registering with it's driver (xhci_hcd) did not help. Trying to reset through "echo 1 > reset" within /sys/devices/ did not help. Trying to use pci-config -s 03:00.0 -S and pci-config -s 03:00.0 -W did not help.

I've been looking for a solution to get this working properly, but failed so far to find a solution.

package used to start the client: libvirt (virsh and virt-manager)

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: kvm 1:84+dfsg-0ubuntu16+1.0+noroms+0ubuntu14.2
ProcVersionSignature: Ubuntu 3.2.0-31.50-generic 3.2.28
Uname: Linux 3.2.0-31-generic x86_64
NonfreeKernelModules: fglrx
ApportVersion: 2.0.1-0ubuntu13
Architecture: amd64
Date: Thu Oct 11 20:03:16 2012
InstallationMedia: Ubuntu 12.04 LTS "Precise Pangolin" - Release amd64 (20120425)
MachineType: To be filled by O.E.M. To be filled by O.E.M.
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-31-generic root=UUID=2583ca5d-0a8a-46bc-b0c3-d8468dc2a421 ro quiet splash iommu=pt vt.handoff=7
SourcePackage: qemu-kvm
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 03/12/2012
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1102
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: SABERTOOTH 990FX
dmi.board.vendor: ASUSTeK COMPUTER INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: To Be Filled By O.E.M.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1102:bd03/12/2012:svnTobefilledbyO.E.M.:pnTobefilledbyO.E.M.:pvrTobefilledbyO.E.M.:rvnASUSTeKCOMPUTERINC.:rnSABERTOOTH990FX:rvrRev1.xx:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.name: To be filled by O.E.M.
dmi.product.version: To be filled by O.E.M.
dmi.sys.vendor: To be filled by O.E.M.

[Gerben (gerbgeus)]

$ lsb_release -rd
Description: Ubuntu 12.04.1 LTS
Release: 12.04

$ apt-cache policy qemu-kvm
qemu-kvm:
  Installed: 1.0+noroms-0ubuntu14.2
  Candidate: 1.0+noroms-0ubuntu14.3
  Version table:
     1.0+noroms-0ubuntu14.3 0
        500 http://nl.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
 *** 1.0+noroms-0ubuntu14.2 0
        500 http://security.ubuntu.com/ubuntu/ precise-security/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0+noroms-0ubuntu13 0
        500 http://nl.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages

$ apt-cache policy libvirt-bin
libvirt-bin:
  Installed: 0.9.8-2ubuntu17.4
  Candidate: 0.9.8-2ubuntu17.4
  Version table:
 *** 0.9.8-2ubuntu17.4 0
        500 http://nl.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     0.9.8-2ubuntu17 0
        500 http://nl.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages

[Gerben (gerbgeus)]

Fresh start:
@pci-02:~$ lsusb
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 002: ID 045e:0750 Microsoft Corp. Wired Keyboard 600
(On the KVM host the KVM process is nice flat, e.g. 1% cpu)

Power off and again starting using virsh start...
@pci-02:~$ lsusb
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
(On the KVM host the KVM process uses 67% cpu continuously, but no cpu used shown in the kvm client, nothing started there either.)

KVM Host top:
  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 4679 libvirt- 20 0 4002m 1.0g 7284 S 54 6.6 2:58.40 kvm
 4713 root -51 0 0 0 0 S 19 0.0 0:49.76 irq/73-kvm:0000
 1401 root 20 0 835m 7500 4356 S 19 0.0 0:51.85 libvirtd
 4689 root 20 0 721m 92m 17m S 19 0.6 0:53.77 python

[Serge Hallyn (serge-hallyn)]

Thanks for submitting this bug. Is there any way you could test this with either the quantal package, or with upstream qemu version 1.2?

[Gerben (gerbgeus)]

Hi,

I've tested this in quantal, and everything works OK.

The pci device is present with every restart of the client, including the connected usb device. All devices are working properly, including removing and adding the usb divice.

Thanks for pointing me to quantal, I will test some different pci device (amd hd5440), but this issue for me can be closed.

Details for package used:

 apt-cache policy qemu-kvm
qemu-kvm:
  Installed: 1.2.0+noroms-0ubuntu2
  Candidate: 1.2.0+noroms-0ubuntu2
  Version table:
 *** 1.2.0+noroms-0ubuntu2 0
        500 http://nl.archive.ubuntu.com/ubuntu/ quantal/main amd64 Packages
        100 /var/lib/dpkg/status

[Serge Hallyn (serge-hallyn)]

Thanks, Gerben. That means we may be able to find a cherry-pickable patchset to fix this in precise.

[Steve Langasek (vorlon)]

The Precise Pangolin has reached end of life, so this bug will not be fixed for that release