Comment 4 for bug 1781891

Trying to recreate this, I have a xhci Host controller (no extra card, but as part of the chipset).
  00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05)

00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05) (prog-if 30 [XHCI])
        Subsystem: Hewlett-Packard Company C610/X99 series chipset USB xHCI Host Controller
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin D routed to IRQ 19
        NUMA node: 0
        Region 0: Memory at 39ffff00000 (64-bit, non-prefetchable) [size=64K]
        Capabilities: <access denied>
        Kernel driver in use: xhci_hcd

First check if it can be unbound:
echo 0000:00:14.0 | sudo tee /sys/bus/pci/devices/0000:00:14.0/driver/unbind

Works and I see:
[2071597.213764] xhci_hcd 0000:00:14.0: remove, state 4
[2071597.213778] usb usb5: USB disconnect, device number 1
[2071597.215019] xhci_hcd 0000:00:14.0: USB bus 5 deregistered
[2071597.215036] xhci_hcd 0000:00:14.0: remove, state 4
[2071597.215046] usb usb4: USB disconnect, device number 1
[2071597.215049] usb 4-3: USB disconnect, device number 2
[2071597.218160] xhci_hcd 0000:00:14.0: USB bus 4 deregistered

FYI: Libvirt should do the unbind/bind for you at runtime if you configured it as managed hostdev

Check ID
$ lspci -n -s 00:14.0
00:14.0 0c03: 8086:8d31 (rev 05)

Tell vfio-pci to handle that
$ echo 8086 8d31 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
I get a fail to bind it like:
  vfio-pci: probe of 0000:00:14.0 failed with error

But then on this system I always failed to get vfio working due to FW issues (not a Linux issue).

You might try the above but for your device to initially rule out all of the modprobe/boot timing that might affect it.
After boot just try to:
1. unbind your device from xhci
2. make the ID known to vfio-pci
   (that should autoload it then)

Report back the kernel you have and the success or fail when doing so, along a dmesg log of the try.
That should clarify if we look at vfio-pci no more being able to load at all (above test fails) or just at how to prep cour config correctly so that it works again.