Ubuntu Raring , IOMMU, fails to boot

[mdalacu]

My system, Ubuntu 13.04, at boot, it drops to busybox if i have AMD IOMMU enabled in BIOS.
If i have the option enabled then the system only boots with a kernel version <= 3.6.
I have tried all the kernels from Ubuntu Kernel Mainline, 3.8.4 included.

The error that it throws during boot, it is:
   AMD-VI: Event logged.. INVALID DEVICE REQUEST
Thank you.

ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: linux-image-3.8.0-13-generic 3.8.0-13.23
ProcVersionSignature: Ubuntu 3.8.0-13.23-generic 3.8.3
Uname: Linux 3.8.0-13-generic x86_64
NonfreeKernelModules: nvidia
ApportVersion: 2.9.2-0ubuntu2
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: mdalacu 2411 F.... pulseaudio
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Date: Sat Mar 23 21:39:49 2013
InstallationDate: Installed on 2010-10-18 (886 days ago)
InstallationMedia: Ubuntu 10.10 "Maverick Meerkat" - Release amd64 (20101007)
IwConfig:
 eth0 no wireless extensions.

 lo no wireless extensions.
MachineType: To Be Filled By O.E.M. To Be Filled By O.E.M.
MarkForUpload: True
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.8.0-13-generic root=UUID=d17fe8b8-a39b-48b6-991a-ea7e7c7a230d ro quiet splash
RelatedPackageVersions:
 linux-restricted-modules-3.8.0-13-generic N/A
 linux-backports-modules-3.8.0-13-generic N/A
 linux-firmware 1.104
RfKill:

SourcePackage: linux
UpgradeStatus: Upgraded to raring on 2013-03-13 (9 days ago)
dmi.bios.date: 01/09/2012
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: P1.70
dmi.board.name: 890FX Deluxe4
dmi.board.vendor: ASRock
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: To Be Filled By O.E.M.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrP1.70:bd01/09/2012:svnToBeFilledByO.E.M.:pnToBeFilledByO.E.M.:pvrToBeFilledByO.E.M.:rvnASRock:rn890FXDeluxe4:rvr:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.name: To Be Filled By O.E.M.
dmi.product.version: To Be Filled By O.E.M.
dmi.sys.vendor: To Be Filled By O.E.M.

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.9 kernel[0] (Not a kernel in the daily directory) and install both the linux-image and linux-image-extra .deb packages.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.9-rc4-raring/

[mdalacu (mdalacu)]

I have tested the kernel that you indicate, by installing these two files:
linux-image-3.9.0-030900rc4-generic_3.9.0-030900rc4.201303232035_amd64.deb
linux-image-extra-3.9.0-030900rc4-generic_3.9.0-030900rc4.201303232035_amd64.deb

The result is the same, if i enable the IOMMU option in BIOS, the system is unable to boot, it drops to Busybox, with this error:
 AMD-VI: Event logged.. INVALID DEVICE REQUEST

I have added two tags 'kernel-bug-exists-upstream' (because it is not fixed) and 'kernel-unable-to-test-upstream' for not being able to boot. If i only have to add one tag which what it is ?
Thank you.

[Joseph Salisbury (jsalisbury)]

This issue appears to be an upstream bug, since you tested the latest upstream kernel. Would it be possible for you to open an upstream bug report[0]? That will allow the upstream Developers to examine the issue, and may provide a quicker resolution to the bug.

Please follow the instructions on the wiki page[0]. The first step is to email the appropriate mailing list. If no response is received, then a bug may be opened on bugzilla.kernel.org.

[0] https://wiki.ubuntu.com/Bugs/Upstream/kernel

[Joseph Salisbury (jsalisbury)]

I'd like to perform a bisect to figure out what commit caused this regression. It would be very helpful to know the earliest kernel where the issue started happening as well as the latest kernel that did not have this issue.

Can you test the following kernels and report back? We are looking for the first kernel version that exhibits this bug:

v3.7-rc4: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.7-rc4-raring/

If v3.7-rc4 does not exhibit the bug then test v3.7-rc6:
v3.7-rc6: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.7-rc6-raring/

If v3.7-rc4 does exhibit the bug then test v3.7-rc2:
v3.7-rc2: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.7-rc2-raring/

You don't have to test every kernel, just up until the kernel that first has this bug.

One thing to note, you will need to install both the linux-image and linux-image-extra .deb packages.

Thanks in advance!

[mdalacu (mdalacu)]

I have tested all of the indicated kernels and, none of them are booting my system, same error INVALID DEVICE REQUEST.
What's next?
Thank you.

[mdalacu (mdalacu)]

Read this:
http://forums.fedoraforum.org/showthread.php?t=261804
do you think that i have a buggy BIOS. Note that Quantal kernels are booting just fine.

[Joseph Salisbury (jsalisbury)]

Can you perform on additional test. Can you test v3.7-rc1:
 http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.7-rc1-raring/

If v3.7-rc1 has the bug, I will bisect between v3.6 final and v3.7-rc1. If it does not have the bug, I'll bisect between v3.7-rc1 and v3.7-rc2.

[mdalacu (mdalacu)]

Hi, i have tested this kernel (3.7.0-030700rc1) and still it does not boot, same error.
Waiting for the next link...

[Joseph Salisbury (jsalisbury)]

I started a kernel bisect between v3.6 final and v3.7-rc1. The kernel bisect will require testing of about 7-10 test kernels.

I built the first test kernel, up to the following commit:
24d7b40a60cf19008334bcbcbd98da374d4d9c64

The test kernel can be downloaded from:
http://people.canonical.com/~jsalisbury/lp1159219

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

One thing to note, you will need to install both the linux-image and linux-image-extra .deb packages.

Thanks in advance

[mdalacu (mdalacu)]

So, i have tested the indicated kernel (3.6.0-030600.201304031435_amd64) and it boots perfectly!
I will wait for your next kernel build.
Thank you.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
21c8715f0a1f4df8bfa2bd6f3915e5e33c1c2e6e

The test kernel can be downloaded from:
http://people.canonical.com/~jsalisbury/lp1159219

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[mdalacu (mdalacu)]

The latest build boots just fine!
What's next?

Thank you.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
23d5385f382a7c7d8b6bf19b0c2cfb3acbb12d31

The test kernel can be downloaded from:
http://people.canonical.com/~jsalisbury/lp1159219

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[mdalacu (mdalacu)]

So, the last kernel build does not boot, same old error.
I will wait for your next build.
Thanks.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
50e0d10232db05c6776afcf6098459bff47e8b15

The test kernel can be downloaded from:
http://people.canonical.com/~jsalisbury/lp1159219

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[mdalacu (mdalacu)]

Same as the last time, it doesn't boot.
I will wait for your next build.
Thanks.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
0fd0ba5f9e8ebae66afded580f5f34936f740ac7

The test kernel can be downloaded from:
http://people.canonical.com/~jsalisbury/lp1159219

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[mdalacu (mdalacu)]

Hey, this one boots! :-)
I will wait for your next build if it is still needed.
Thanks.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
7035cdf36d5c4d913f68ff97e1c2e5603500d946

The test kernel can be downloaded from:
http://people.canonical.com/~jsalisbury/lp1159219

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[mdalacu (mdalacu)]

Hi, it doesn't boot with this version.
I will wait for the next one.
Thanks.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
6432f2128414edbea5fd4f6c4fa4c28d0e1c6151

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1159219

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[mdalacu (mdalacu)]

Sorry for the delayed answer, i was on a vacation.
So, this new build still does not boot, same errors.
I will wait for the next one.
Thanks.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
ba39ebb61401cfe0ccd58dd0cd4da88465528c0a

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1159219

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[mdalacu (mdalacu)]

I have tired the one from 6432f2128414edbea5fd4f6c4fa4c28d0e1c6151 folder, the other one was empty.
The result was the same, it still does not boot.
I will wait for the next one.
Thanks.

[Joseph Salisbury (jsalisbury)]

Sorry about that. The kernels to test are now at:
http://kernel.ubuntu.com/~jsalisbury/lp1159219

[mdalacu (mdalacu)]

No problem, but, there is no extra deb file. Should i test only with the generic one?

[Joseph Salisbury (jsalisbury)]

Not every kernel version has the extra deb file. It was introduced in for a short time, and is not built anymore. Some kernels will have it and some wont. All kernels for Saucy and going forward will not have it.

[mdalacu (mdalacu)]

So, this last build boots perfectly!
What's next?
Thanks.

[Joseph Salisbury (jsalisbury)]

There will be about 5 more kernels to test until the bisect identifies the first bad commit.

I built the next test kernel, up to the following commit:
c0703c12ef6744b6d2565ec67a15aaf25d534abd

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1159219

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[mdalacu (mdalacu)]

The last one does not boot.
So, 5 to go.
Waiting for the next one.
Thanks.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
c18d2388170132ce015475f1fdab272b62c1e2da

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1159219

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[mdalacu (mdalacu)]

Same result. It does not boot.
Waiting for your next one.
Thanks.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
7ef2798deb695f112f25e348b199dca79eb1ea68

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1159219

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[mdalacu (mdalacu)]

Still no go. It does not boot
Waiting for your next one.
Thanks.

[Joseph Salisbury (jsalisbury)]

I think this may be the commit that introduced this regression:
commit 0ea2c422bc8da99d14baa46d4789861a4f8d4ec0
Author: Joerg Roedel <email address hidden>
Date: Fri Jun 15 18:05:20 2012 +0200

    iommu/amd: Allocate data structures to keep track of irq remapping tables

I'll build a kernel with this commit reverted and post a link shortly.

[mdalacu (mdalacu)]

Great! I will wait for your build.

[Joseph Salisbury (jsalisbury)]

I attempted to revert commit 0ea2c422. However, there were a few other commits that this commit were dependent on, so it's probably best to finish the bisect.

I built the next test kernel, up to the following commit:
0ea2c422bc8da99d14baa46d4789861a4f8d4ec0

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1159219

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[mdalacu (mdalacu)]

Yes, this one boots.
I will wait for your next one.
Thanks.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
33f28c59e18d83fd2aeef258d211be66b9b80eb3

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1159219

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[mdalacu (mdalacu)]

This one does not boot, same error.
Waiting for your next one.
Thanks.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
eb1eb7ae65a9d32f6c16a90419caf01221f94734

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1159219

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[mdalacu (mdalacu)]

Yes, this one boots.
I will wait for your next one.
Thanks.

[Joseph Salisbury (jsalisbury)]

33f28c59 was identified as the first bad commit:

33f28c59e18d83fd2aeef258d211be66b9b80eb3 is the first bad commit
commit 33f28c59e18d83fd2aeef258d211be66b9b80eb3
Author: Joerg Roedel <email address hidden>
Date: Fri Jun 15 18:03:31 2012 +0200

    iommu/amd: Split device table initialization into irq and dma part

I'll build a Raring test kernel with this commit reverted.

[Joseph Salisbury (jsalisbury)]

I built a raring test kernel with commit 33f28c59 reverted. The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1159219/

Can you test this kernel and confirm that it fixes this bug? Note, you will need to install both the linux-image and linux-image-extra .deb packages.

Thanks in advance!

[mdalacu (mdalacu)]

Sorry to say but it still does not boot. I have installed both debs and choose the .27 version from grub. The same drop to busybox happened.
Do you need me to run some tests?
What's next if there is any?
Thank you for your hard work it's much appreciated.

[Joseph Salisbury (jsalisbury)]

Sorry for the delay, I was away for a few days. Thanks for testing that kernel. I'll have to investigate the set of iommu/amd patches further.

The v3.11-rc4 kernel is now out. Can you also test this kernel, just to see if the bug is already fixed in mainline?
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.11-rc4-saucy/

Thanks again for all your help!

[mdalacu (mdalacu)]

Also sorry for the delay. :-) I was in vacation.
I have tested it and still no go, same error.
So it is not fixed yet. What's next.
Thanks.

[Joseph Salisbury (jsalisbury)]

Can you test the v3.12-rc3 kernel:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.12-rc3-saucy/

[mdalacu (mdalacu)]

Hi, i have done the testing with the same errors. Does not boot. drops to Busybox.
Thanks.

[Joseph Salisbury (jsalisbury)]

The 3.12 final kernel is now out. Can you give this kernel a try to see if the bug still exists in mainline:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.12-saucy/

[mdalacu (mdalacu)]

I have tested it an got the same results. Does not boot.

[Joseph Salisbury (jsalisbury)]

Thanks for testing. I'll investigate the IOMMU patches further.

Was there ever an upstream bug report opened, per the request in comment #5?

[mdalacu (mdalacu)]

I don't think so.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu Raring) because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu Saucy) because there has been no activity for 60 days.]