AMD-Vi: Completion-Wait loop timed out

[MasterCATZ]

AMD-Vi: Completion-Wait loop timed out

I have an F2A85X-UP4 with 32 Gig Ram and amd-a10-5800k

I have disabled turbo and any thing that might change in cpu freq
I also disabled cpu virtualization

I have a feeling this might be related to using ZFS as when ever I do something that requires a lot of data from my ZFS pool of 16x HDD's this seems to happen

ppa:zfs-native/daily

how ever I am sure it has happened before with out the pool imported

and happened more often with c6 state enabled

any idea's where to start looking

this happened with both 12.04 LTS and Ubuntu 12.10

how ever sticking with Ubuntu 12.10 as its video drivers seem to be better and I can get a days use out of it with out this issue rearing its head

with 12.04 LTS it did not take long for mouse to become unresponsive

# uname -a
Linux aio 3.5.0-18-generic #29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

sorry using my windows box to post this as Ubuntu is doing that thing atm and I am waiting for it to finish something before doing another reboot
---
ApportVersion: 2.6.1-0ubuntu9
Architecture: amd64
DistroRelease: Ubuntu 12.10
MarkForUpload: True
NonfreeKernelModules: zfs zcommon znvpair zavl zunicode
Package: linux (not installed)
ProcEnviron:
 LANGUAGE=en_AU:en
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_AU.UTF-8
 SHELL=/bin/bash
Tags: quantal
Uname: Linux 3.6.8-030608-generic x86_64
UnreportableReason: The running kernel is not an Ubuntu kernel
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1073384/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Fabio Marconi (fabiomarconi)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. This bug did not have a package associated with it, which is important for ensuring that it gets looked at by the proper developers. You can learn more about finding the right package at https://wiki.ubuntu.com/Bugs/FindRightPackage . I have classified this bug as a bug in linux.

When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://help.ubuntu.com/community/ReportingBugs.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1073384

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[MasterCATZ (mastercatz)]

It has been running rock solid with IOMMU off

and I just rebooted now to turn IOMMU back on to see how it goes ( last time it lasted a few days instead of minutes )

# uname -a
Linux aio 3.6.8-030608-generic #201211271040 SMP Tue Nov 27 15:41:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

this is the highest kernel I have been able to run whilst having zfs

its not really package related issue , I am unsure what actually happened with the discussions about it beeing the bios / IOMMU or where my log files got to

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.8 kernel[0] (Not a kernel in the daily directory) and install both the linux-image and linux-image-extra .deb packages.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8-rc1-raring/

[MasterCATZ (mastercatz)]

v3.8-rc1-raring/22-Dec-2012 02:58- v3.8-rc2-raring/
http://kernel.ubuntu.com/~kernel-ppa/mainline/
which one ?

Master CATZ

> Date: Thu, 3 Jan 2013 16:45:31 +0000
> From: <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1073384] Re: AMD-Vi: Completion-Wait loop timed out
>
> Would it be possible for you to test the latest upstream kernel? Refer
> to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest
> v3.8 kernel[0] (Not a kernel in the daily directory) and install both
> the linux-image and linux-image-extra .deb packages.
>
> If this bug is fixed in the mainline kernel, please add the following
> tag 'kernel-fixed-upstream'.
>
> If the mainline kernel does not fix this bug, please add the tag:
> 'kernel-bug-exists-upstream'.
>
> If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
> Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".
>
>
> Thanks in advance.
>
> [0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8-rc1-raring/
>
>
> ** Changed in: linux (Ubuntu)
> Importance: Undecided => Medium
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1073384
>
> Title:
> AMD-Vi: Completion-Wait loop timed out
>
> Status in “linux” package in Ubuntu:
> Incomplete
>
> Bug description:
> AMD-Vi: Completion-Wait loop timed out
>
>
> I have an F2A85X-UP4 with 32 Gig Ram and amd-a10-5800k
>
> I have disabled turbo and any thing that might change in cpu freq
> I also disabled cpu virtualization
>
>
> I have a feeling this might be related to using ZFS as when ever I do something that requires a lot of data from my ZFS pool of 16x HDD's this seems to happen
>
> ppa:zfs-native/daily
>
> how ever I am sure it has happened before with out the pool imported
>
> and happened more often with c6 state enabled
>
>
> any idea's where to start looking
>
> this happened with both 12.04 LTS and Ubuntu 12.10
>
> how ever sticking with Ubuntu 12.10 as its video drivers seem to be
> better and I can get a days use out of it with out this issue rearing
> its head
>
> with 12.04 LTS it did not take long for mouse to become unresponsive
>
>
> # uname -a
> Linux aio 3.5.0-18-generic #29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
>
>
> sorry using my windows box to post this as Ubuntu is doing that thing atm and I am waiting for it to finish something before doing another reboot
> ---
> ApportVersion: 2.6.1-0ubuntu9
> Architecture: amd64
> DistroRelease: Ubuntu 12.10
> MarkForUpload: True
> NonfreeKernelModules: zfs zcommon znvpair zavl zunicode
> Package: linux (not installed)
> ProcEnviron:
> LANGUAGE=en_AU:en
> TERM=xterm
> PATH=(custom, no user)
> LANG=en_AU.UTF-8
> SHELL=/bin/bash
> Tags: quantal
> Uname: Linux 3.6.8-030608-generic x86_64
> UnreportableReason: The running kernel is not an Ubuntu kernel
> UpgradeStatus: No upgrade log present (probably fresh install)
> UserGrou...

[Joseph Salisbury (jsalisbury)]

v3.8-rc2

On 01/03/2013 11:34 PM, Master CATZ wrote:
> v3.8-rc1-raring/
> <http://kernel.ubuntu.com/%7Ekernel-ppa/mainline/v3.8-rc1-raring/>
> 22-Dec-2012 02:58 -
>
> v3.8-rc2-raring/
> <http://kernel.ubuntu.com/%7Ekernel-ppa/mainline/v3.8-rc2-raring/>
>
>
> http://kernel.ubuntu.com/~kernel-ppa/mainline/
> <http://kernel.ubuntu.com/%7Ekernel-ppa/mainline/>
>
> which one ?
>
> Master CATZ
>
> > Date: Thu, 3 Jan 2013 16:45:31 +0000
> > From: <email address hidden>
> > To: <email address hidden>
> > Subject: [Bug 1073384] Re: AMD-Vi: Completion-Wait loop timed out
> >
> > Would it be possible for you to test the latest upstream kernel? Refer
> > to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest
> > v3.8 kernel[0] (Not a kernel in the daily directory) and install both
> > the linux-image and linux-image-extra .deb packages.
> >
> > If this bug is fixed in the mainline kernel, please add the following
> > tag 'kernel-fixed-upstream'.
> >
> > If the mainline kernel does not fix this bug, please add the tag:
> > 'kernel-bug-exists-upstream'.
> >
> > If you are unable to test the mainline kernel, for example it will
> not boot, please add the tag: 'kernel-unable-to-test-upstream'.
> > Once testing of the upstream kernel is complete, please mark this
> bug as "Confirmed".
> >
> >
> > Thanks in advance.
> >
> > [0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8-rc1-raring/
> >
> >
> > ** Changed in: linux (Ubuntu)
> > Importance: Undecided => Medium
> >
> > --
> > You received this bug notification because you are subscribed to the bug
> > report.
> > https://bugs.launchpad.net/bugs/1073384
> >
> > Title:
> > AMD-Vi: Completion-Wait loop timed out
> >
> > Status in “linux” package in Ubuntu:
> > Incomplete
> >
> > Bug description:
> > AMD-Vi: Completion-Wait loop timed out
> >
> >
> > I have an F2A85X-UP4 with 32 Gig Ram and amd-a10-5800k
> >
> > I have disabled turbo and any thing that might change in cpu freq
> > I also disabled cpu virtualization
> >
> >
> > I have a feeling this might be related to using ZFS as when ever I
> do something that requires a lot of data from my ZFS pool of 16x HDD's
> this seems to happen
> >
> > ppa:zfs-native/daily
> >
> > how ever I am sure it has happened before with out the pool imported
> >
> > and happened more often with c6 state enabled
> >
> >
> > any idea's where to start looking
> >
> > this happened with both 12.04 LTS and Ubuntu 12.10
> >
> > how ever sticking with Ubuntu 12.10 as its video drivers seem to be
> > better and I can get a days use out of it with out this issue rearing
> > its head
> >
> > with 12.04 LTS it did not take long for mouse to become unresponsive
> >
> >
> > # uname -a
> > Linux aio 3.5.0-18-generic #29-Ubuntu SMP Fri Oct 19 10:26:51 UTC
> 2012 x86_64 x86_64 x86_64 GNU/Linux
> >
> >
> > sorry using my windows box to post this as Ubuntu is doing that
> thing atm and I am waiting for it to finish something before doing
> another reboot
> > ---
> > ApportVersion: 2.6.1-0ubuntu9
> > Architecture: amd64
> > DistroRelease: Ubuntu 12.10
> > MarkForUpload: True
> > NonfreeKernelModules: zfs zcommon znvpair zavl zunicode
> > Package: linux (n...

[MasterCATZ (mastercatz)]

ok will try rc2 I did not use it because it was the same date as the daily
i did try rc1 and it failed in under 1 hr

> Date: Fri, 4 Jan 2013 17:18:46 +0000
> From: <email address hidden>
> To: <email address hidden>
> Subject: Re: [Bug 1073384] Re: AMD-Vi: Completion-Wait loop timed out
>
> v3.8-rc2
>
> On 01/03/2013 11:34 PM, Master CATZ wrote:
> > v3.8-rc1-raring/
> > <http://kernel.ubuntu.com/%7Ekernel-ppa/mainline/v3.8-rc1-raring/>
> > 22-Dec-2012 02:58 -
> >
> > v3.8-rc2-raring/
> > <http://kernel.ubuntu.com/%7Ekernel-ppa/mainline/v3.8-rc2-raring/>
> >
> >
> > http://kernel.ubuntu.com/~kernel-ppa/mainline/
> > <http://kernel.ubuntu.com/%7Ekernel-ppa/mainline/>
> >
> > which one ?
> >
> > Master CATZ
> >
> > > Date: Thu, 3 Jan 2013 16:45:31 +0000
> > > From: <email address hidden>
> > > To: <email address hidden>
> > > Subject: [Bug 1073384] Re: AMD-Vi: Completion-Wait loop timed out
> > >
> > > Would it be possible for you to test the latest upstream kernel? Refer
> > > to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest
> > > v3.8 kernel[0] (Not a kernel in the daily directory) and install both
> > > the linux-image and linux-image-extra .deb packages.
> > >
> > > If this bug is fixed in the mainline kernel, please add the following
> > > tag 'kernel-fixed-upstream'.
> > >
> > > If the mainline kernel does not fix this bug, please add the tag:
> > > 'kernel-bug-exists-upstream'.
> > >
> > > If you are unable to test the mainline kernel, for example it will
> > not boot, please add the tag: 'kernel-unable-to-test-upstream'.
> > > Once testing of the upstream kernel is complete, please mark this
> > bug as "Confirmed".
> > >
> > >
> > > Thanks in advance.
> > >
> > > [0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8-rc1-raring/
> > >
> > >
> > > ** Changed in: linux (Ubuntu)
> > > Importance: Undecided => Medium
> > >
> > > --
> > > You received this bug notification because you are subscribed to the bug
> > > report.
> > > https://bugs.launchpad.net/bugs/1073384
> > >
> > > Title:
> > > AMD-Vi: Completion-Wait loop timed out
> > >
> > > Status in “linux” package in Ubuntu:
> > > Incomplete
> > >
> > > Bug description:
> > > AMD-Vi: Completion-Wait loop timed out
> > >
> > >
> > > I have an F2A85X-UP4 with 32 Gig Ram and amd-a10-5800k
> > >
> > > I have disabled turbo and any thing that might change in cpu freq
> > > I also disabled cpu virtualization
> > >
> > >
> > > I have a feeling this might be related to using ZFS as when ever I
> > do something that requires a lot of data from my ZFS pool of 16x HDD's
> > this seems to happen
> > >
> > > ppa:zfs-native/daily
> > >
> > > how ever I am sure it has happened before with out the pool imported
> > >
> > > and happened more often with c6 state enabled
> > >
> > >
> > > any idea's where to start looking
> > >
> > > this happened with both 12.04 LTS and Ubuntu 12.10
> > >
> > > how ever sticking with Ubuntu 12.10 as its video drivers seem to be
> > > better and I can get a days use out of it with out this issue rearing
> > > its head
> > >
> > > with 12.04 LTS it did not take long for mouse to become unresponsive
> > >
> > >
> > ...

[MasterCATZ (mastercatz)]

attached var log files created during 3.8 kernel tries

if their is anything else you would need to find this problem let me know

I really need iommu back on pc is so slow without it

[Suravee Suthikulpanit (suravee-suthikulpanit)]

I have recently submitted a patch (http://marc.info/?l=linux-kernel&m=135905510523514&w=2) to implement workaround for an erratum which might be related to IOMMU stop processing commands as discussed here (http://marc.info/?l=linux-kernel&m=135867802432660&w=2). It should be now included upstream (v3.8-rc5+). Could you please try it and see if the issue has resolved.

[MasterCATZ (mastercatz)]

it seems to have stoped the AMD-vi issue
but its acting like immou was never turned back on .. going slow in 86h31m and i will reboot to make sure I indeed did save by bios before I left normally with immou turned on it takes just over a day to zfs scrub 16tb of data just a lot of reboots from the AMD-vi issue

> Date: Fri, 1 Feb 2013 21:54:13 +0000
> From: <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1073384] Re: AMD-Vi: Completion-Wait loop timed out
>
> I have recently submitted a patch (http://marc.info/?l=linux-
> kernel&m=135905510523514&w=2) to implement workaround for an erratum
> which might be related to IOMMU stop processing commands as discussed
> here (http://marc.info/?l=linux-kernel&m=135867802432660&w=2). It
> should be now included upstream (v3.8-rc5+). Could you please try it
> and see if the issue has resolved.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1073384
>
> Title:
> AMD-Vi: Completion-Wait loop timed out
>
> Status in “linux” package in Ubuntu:
> Incomplete
>
> Bug description:
> AMD-Vi: Completion-Wait loop timed out
>
>
> I have an F2A85X-UP4 with 32 Gig Ram and amd-a10-5800k
>
> I have disabled turbo and any thing that might change in cpu freq
> I also disabled cpu virtualization
>
>
> I have a feeling this might be related to using ZFS as when ever I do something that requires a lot of data from my ZFS pool of 16x HDD's this seems to happen
>
> ppa:zfs-native/daily
>
> how ever I am sure it has happened before with out the pool imported
>
> and happened more often with c6 state enabled
>
>
> any idea's where to start looking
>
> this happened with both 12.04 LTS and Ubuntu 12.10
>
> how ever sticking with Ubuntu 12.10 as its video drivers seem to be
> better and I can get a days use out of it with out this issue rearing
> its head
>
> with 12.04 LTS it did not take long for mouse to become unresponsive
>
>
> # uname -a
> Linux aio 3.5.0-18-generic #29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
>
>
> sorry using my windows box to post this as Ubuntu is doing that thing atm and I am waiting for it to finish something before doing another reboot
> ---
> ApportVersion: 2.6.1-0ubuntu9
> Architecture: amd64
> DistroRelease: Ubuntu 12.10
> MarkForUpload: True
> NonfreeKernelModules: zfs zcommon znvpair zavl zunicode
> Package: linux (not installed)
> ProcEnviron:
> LANGUAGE=en_AU:en
> TERM=xterm
> PATH=(custom, no user)
> LANG=en_AU.UTF-8
> SHELL=/bin/bash
> Tags: quantal
> Uname: Linux 3.6.8-030608-generic x86_64
> UnreportableReason: The running kernel is not an Ubuntu kernel
> UpgradeStatus: No upgrade log present (probably fresh install)
> UserGroups:
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1073384/+subscriptions

[MasterCATZ (mastercatz)]

verified IMMOU was indeed turned on

system is stable and no AMD-vi issue in the last few days but the system no longer has the performance gains that enabling IMMOU had ( which was normally 10x the speed when accessing data )

[Suravee Suthikulpanit (suravee-suthikulpanit)]

Actually, the patch should not have affected the IOMMU performance. I'm not quite familiar with ZFS. If you give me some pointers of how you do your testing, I can try to experiment. Could you recheck the dmesg to see if there are other issues?

[MasterCATZ (mastercatz)]

well the good news is system still has no AMD-Vi: Completion-Wait loop timed out

also unsure how to test this to find out why its running with out the performance gains IOMMU gave me before
I do have some spare HDD's I could use to try bench marking with with out using ZFS to see if i can crash the system again with an older kernel and try and replicate the IOMMU performance gains / loss's

pretty much the way ZFS worked it normally ramps up its speed
with out IOMMU it normally started at 14 mbs then ramped up to around 100mbs ( takes a couple of days with 16tb)
with IOMMU ( and the constant crashes ) it started around 100 mbs and ramped up to 900 mbs doing scrubs ( took around a day and a few reboots with 16tb )or I would bootup to FreeBSD ( which also had crashes so it looks like they have similar issue )

normally just to crash it quickly I just had to play a bluray disk whilst doing a srcub

> Date: Wed, 20 Feb 2013 15:24:32 +0000
> From: <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1073384] Re: AMD-Vi: Completion-Wait loop timed out
>
> Actually, the patch should not have affected the IOMMU performance. I'm
> not quite familiar with ZFS. If you give me some pointers of how you do
> your testing, I can try to experiment. Could you recheck the dmesg to
> see if there are other issues?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1073384
>
> Title:
> AMD-Vi: Completion-Wait loop timed out
>
> Status in “linux” package in Ubuntu:
> Incomplete
>
> Bug description:
> AMD-Vi: Completion-Wait loop timed out
>
>
> I have an F2A85X-UP4 with 32 Gig Ram and amd-a10-5800k
>
> I have disabled turbo and any thing that might change in cpu freq
> I also disabled cpu virtualization
>
>
> I have a feeling this might be related to using ZFS as when ever I do something that requires a lot of data from my ZFS pool of 16x HDD's this seems to happen
>
> ppa:zfs-native/daily
>
> how ever I am sure it has happened before with out the pool imported
>
> and happened more often with c6 state enabled
>
>
> any idea's where to start looking
>
> this happened with both 12.04 LTS and Ubuntu 12.10
>
> how ever sticking with Ubuntu 12.10 as its video drivers seem to be
> better and I can get a days use out of it with out this issue rearing
> its head
>
> with 12.04 LTS it did not take long for mouse to become unresponsive
>
>
> # uname -a
> Linux aio 3.5.0-18-generic #29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
>
>
> sorry using my windows box to post this as Ubuntu is doing that thing atm and I am waiting for it to finish something before doing another reboot
> ---
> ApportVersion: 2.6.1-0ubuntu9
> Architecture: amd64
> DistroRelease: Ubuntu 12.10
> MarkForUpload: True
> NonfreeKernelModules: zfs zcommon znvpair zavl zunicode
> Package: linux (not installed)
> ProcEnviron:
> LANGUAGE=en_AU:en
> TERM=xterm
> PATH=(custom, no user)
> LANG=en_AU.UTF-8
> SHELL=/bin/bash
> Tags: quantal
> Uname: Linux 3.6.8-030608-...

[Suravee Suthikulpanit (suravee-suthikulpanit)]

I have a couple questions here:

1) Are you running in some sort of virtual machine? Looking at the old log files you sent along in the "var log.zip", it seems like you are running virtual box.

2) How are you setting up your system?

3) How are you setting up your zfs? Are you using the zfs-fuse package from Ubuntu?

4) Could you please send the new set of log files?

5) Could you also send the out put of "lsmod"?

6) I also seeing "radeon" driver in the dmesg. Is this the standard radeon driver from the kernel?

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]