iommu dmar pci passthrough regression

[daemon dog]

I'd like to reduce clutter and get the most use out of my existing infrastructure so I wanted to stop using an external VOIP gateway for my PSTN lines and use a dedicated card with Asterisk instead via VT-d and the iommu under Virtualbox 4.1.34. The hardware I have is capable and the relevant BIOS and kernel options are enabled. I have set intel_iommu=on and I have also tried using the additional kernel option iommu=pt.

After testing I can confirm that this works correctly using linux 3.2.0-70-generic but not with 3.5.0-54-generic, 3.13.0-37-generic or 3.16.6-031606-generic (from mainline kernel-ppa). I noted that there may be a possible solution in https://bugzilla.kernel.org/show_bug.cgi?id=44881#c49 but that this would not be backported.

Release: Ubuntu 12.04.5 LTS
Linux 3.13.0-37-generic #64~precise1-Ubuntu SMP Wed Sep 24 21:37:11 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Motherboard: Gigabyte Technology Co., Ltd. Z87X-D3H/Z87X-D3H-CF, BIOS F9 08/25/2014

The card I'd like to pass to the guest...

03:00.0 Ethernet controller: Digium, Inc. Wildcard TDM410 4-port analog card (rev 11)

I have setup pci-stub with the above and set vboxpci to softdep on it. Not sure if that's necessary or not as there are no host drivers available.
The error I'm seeing when loading the kernel module in the guest (except when using kernel 3.2 which works properly). Modprobe hangs in the guest and eventually causes the guest to freeze.

Oct 24 09:05:08 kernel: [ 81.496003] dmar: DRHD: handling fault status reg 3
Oct 24 09:05:08 kernel: [ 81.496032] dmar: DMAR:[DMA Read] Request device [03:00.0] fault addr 7349c000
Oct 24 09:05:08 kernel: [ 81.496032] DMAR:[fault reason 06] PTE Read access is not set

Another error which may be related...

WARNING: CPU: 4 PID: 1 at /build/buildd/linux-lts-trusty-3.13.0/drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x66/0x80()

If there is any possibility of trying the patch mentioned in bugzilla with the current kernel that would be great. I'm going to look and see if there is a later kernel available to try as I'm not sure if 3.16.6 has it or not.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.13.0-37-generic 3.13.0-37.64~precise1
ProcVersionSignature: Ubuntu 3.13.0-37.64~precise1-generic 3.13.11.7
Uname: Linux 3.13.0-37-generic x86_64
InstallationMedia: Ubuntu-Server 12.04 LTS "Precise Pangolin" - Release amd64 (20120424.1)
MachineType: Gigabyte Technology Co., Ltd. Z87X-D3H
MarkForUpload: True
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-37-generic root=UUID=c3169056-4938-4faf-b6eb-d70b4268b135 ro intel_iommu=on iommu=pt
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-37-generic N/A
 linux-backports-modules-3.13.0-37-generic N/A
 linux-firmware 1.79.18
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux-lts-trusty
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 08/25/2014
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: F9
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: Z87X-D3H-CF
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: Gigabyte Technology Co., Ltd.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrF9:bd08/25/2014:svnGigabyteTechnologyCo.,Ltd.:pnZ87X-D3H:pvrTobefilledbyO.E.M.:rvnGigabyteTechnologyCo.,Ltd.:rnZ87X-D3H-CF:rvrx.x:cvnGigabyteTechnologyCo.,Ltd.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.name: Z87X-D3H
dmi.product.version: To be filled by O.E.M.
dmi.sys.vendor: Gigabyte Technology Co., Ltd.

[daemon dog (nodenet-1)]

Tried Linux version 3.17.1-031701-generic intel_iommu=on also with and without iommu=pt

Does appear to resolve the pci_find_upstream_pcie_bridge error message but PCI passthrough is still not working....

Oct 26 06:35:39 kernel: [ 227.795652] dmar: DRHD: handling fault status reg 3
Oct 26 06:35:39 kernel: [ 227.795678] dmar: DMAR:[DMA Read] Request device [03:00.0] fault addr 73ac6000
Oct 26 06:35:39 kernel: [ 227.795678] DMAR:[fault reason 06] PTE Read access is not set

Also tried Linux version 3.18.0-031800rc1-generic

pci_find_upstream_pcie_bridge error message resolved but there were more error messages...

Oct 26 08:09:11 kernel: [ 5.591085] vboxdrv: Successfully loaded version 4.1.34 (interface 0x00190000).
Oct 26 08:09:11 kernel: [ 5.809189] dmar: DRHD: handling fault status reg 2
Oct 26 08:09:11 kernel: [ 5.809615] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 7300000000
Oct 26 08:09:11 kernel: [ 5.809615] DMAR:[fault reason 05] PTE Write access is not set

Oct 26 08:09:13 kernel: [ 7.688709] vboxpci: iommu_attach_device() success
Oct 26 08:09:13 kernel: [ 7.804686] dmar: DRHD: handling fault status reg 2
Oct 26 08:09:13 kernel: [ 7.805118] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 7300000000
Oct 26 08:09:13 kernel: [ 7.805118] DMAR:[fault reason 05] PTE Write access is not set

Oct 26 08:10:56 kernel: [ 111.141785] dmar: DRHD: handling fault status reg 3
Oct 26 08:10:56 kernel: [ 111.141811] dmar: DMAR:[DMA Read] Request device [03:00.0] fault addr 728be000
Oct 26 08:10:56 kernel: [ 111.141811] DMAR:[fault reason 06] PTE Read access is not set
Oct 26 08:10:56 kernel: [ 111.141847] dmar: DRHD: handling fault status reg 3
Oct 26 08:10:56 kernel: [ 111.141863] dmar: DMAR:[DMA Read] Request device [03:00.0] fault addr 0
Oct 26 08:10:56 kernel: [ 111.141863] DMAR:[fault reason 06] PTE Read access is not set

[Bjorn Helgaas (bjorn-helgaas)]

Reported upstream as https://bugzilla.kernel.org/show_bug.cgi?id=86951

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[penalvch (penalvch)]

daemon dog, thank you for reporting this and helping make Ubuntu better.

At your earliest convenience, could you please test the latest upstream kernel available from the very top line at the top of the page from http://kernel.ubuntu.com/~kernel-ppa/mainline/?C=N;O=D (the release names are irrelevant for testing, and please do not test the daily folder)? Install instructions are available at https://wiki.ubuntu.com/Kernel/MainlineBuilds . This will allow additional upstream developers to examine the issue.

If testing on your main install would be inconvenient, one may:
1) Install Ubuntu to a different partition and then test this there.
2) Backup, or clone the primary install.

If the latest kernel did not allow you to test to the issue (ex. you couldn't boot into the OS) please make a comment in your report about this, and continue to test the next most recent kernel version until you can test to the issue. Once you've tested the upstream kernel, please comment on which kernel version specifically you tested. If this issue is fixed in the mainline kernel, please add the following tags by clicking on the yellow circle with a black pencil icon, next to the word Tags, located at the bottom of the report description:
kernel-fixed-upstream
kernel-fixed-upstream-X.Y-rcZ

Where X, and Y are the first two numbers of the kernel version, and Z is the release candidate number if it exists.

If the mainline kernel does not fix the issue, please add the following tags:
kernel-bug-exists-upstream
kernel-bug-exists-upstream-X.Y-rcZ

Please note, an error to install the kernel does not fit the criteria of kernel-bug-exists-upstream.

Once testing of the latest upstream kernel is complete, please mark this report's Status as Confirmed. Please let us know your results.

Thank you for your understanding.