[ivb] hang on pageflip (IPEHR: 0x0a000001 or 0x0a080001 depending on pipe)

[Sidhanti Sudheendra]

False GPU lockup IPEHR: 0x0a000001

ProblemType: Crash
DistroRelease: Ubuntu 14.04
Package: xserver-xorg-video-intel 2:2.99.907-0ubuntu1
ProcVersionSignature: Ubuntu 3.13.0-5.20-generic 3.13.0
Uname: Linux 3.13.0-5-generic x86_64
.tmp.unity.support.test.0:

ApportVersion: 2.13.2-0ubuntu2
Architecture: amd64
CompizPlugins: No value set for `/apps/compiz-1/general/screen0/options/active_plugins'
CompositorRunning: compiz
CompositorUnredirectDriverBlacklist: '(nouveau|Intel).*Mesa 8.0'
CompositorUnredirectFSW: true
Date: Wed Jan 29 23:32:19 2014
DistUpgraded: 2013-12-25 22:45:46,524 DEBUG enabling apt cron job
DistroCodename: trusty
DistroVariant: ubuntu
DuplicateSignature: GPU lockup IPEHR: 0x0a000001 Ubuntu 14.04
ExecutablePath: /usr/share/apport/apport-gpu-error-intel.py
ExtraDebuggingInterest: Yes
GraphicsCard:
 Intel Corporation 3rd Gen Core processor Graphics Controller [8086:0166] (rev 09) (prog-if 00 [VGA controller])
   Subsystem: Lenovo Device [17aa:21f4]
 NVIDIA Corporation GF108M [NVS 5400M] [10de:0def] (rev a1) (prog-if 00 [VGA controller])
   Subsystem: Lenovo Device [17aa:21f4]
InstallationDate: Installed on 2013-05-30 (245 days ago)
InstallationMedia: Ubuntu 13.04 "Raring Ringtail" - Release amd64 (20130424)
InterpreterPath: /usr/bin/python3.3
MachineType: LENOVO 2342CTO
ProcCmdline: /usr/bin/python3 /usr/share/apport/apport-gpu-error-intel.py
ProcEnviron:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-5-generic root=UUID=db82f5ed-cf1b-4bea-bcd8-8ec4fb76a850 ro acpi_backlight=vendor acpi_osi=Linux quiet splash i915.i915_enable_rc6=1 pcie_aspm=force vt.handoff=7
RelatedPackageVersions:
 xserver-xorg 1:7.7+1ubuntu7
 libdrm2 2.4.52-1
 xserver-xorg-video-intel 2:2.99.907-0ubuntu1
SourcePackage: xserver-xorg-video-intel
Title: False GPU lockup IPEHR: 0x0a000001
UpgradeStatus: Upgraded to trusty on 2013-12-26 (35 days ago)
UserGroups:

dmi.bios.date: 10/19/2012
dmi.bios.vendor: LENOVO
dmi.bios.version: G1ET73WW (2.09 )
dmi.board.asset.tag: Not Available
dmi.board.name: 2342CTO
dmi.board.vendor: LENOVO
dmi.board.version: Win8 STD DPK TPG
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvrG1ET73WW(2.09):bd10/19/2012:svnLENOVO:pn2342CTO:pvrThinkPadT430:rvnLENOVO:rn2342CTO:rvrWin8STDDPKTPG:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 2342CTO
dmi.product.version: ThinkPad T430
dmi.sys.vendor: LENOVO
version.compiz: compiz 1:0.9.10+13.10.20131011-0ubuntu1
version.ia32-libs: ia32-libs N/A
version.libdrm2: libdrm2 2.4.52-1
version.libgl1-mesa-dri: libgl1-mesa-dri 10.0.1-1ubuntu2
version.libgl1-mesa-dri-experimental: libgl1-mesa-dri-experimental N/A
version.libgl1-mesa-glx: libgl1-mesa-glx 10.0.1-1ubuntu2
version.xserver-xorg-core: xserver-xorg-core 2:1.14.5-1ubuntu2
version.xserver-xorg-input-evdev: xserver-xorg-input-evdev 1:2.8.2-1ubuntu1
version.xserver-xorg-video-ati: xserver-xorg-video-ati 1:7.3.0-1ubuntu1
version.xserver-xorg-video-intel: xserver-xorg-video-intel 2:2.99.907-0ubuntu1
version.xserver-xorg-video-nouveau: xserver-xorg-video-nouveau 1:1.0.10-1ubuntu1
xserver.bootTime: Thu Jan 30 20:19:46 2014
xserver.configfile: default
xserver.errors:
 Failed to load module "nvidia" (module does not exist, 0)
 Failed to load module "nvidia" (module does not exist, 0)
 NOUVEAU(G0): [XvMC] Failed to initialize extension.
xserver.logfile: /var/log/Xorg.0.log
xserver.outputs:
 product id 8510
 vendor AUO
xserver.version: 2:1.14.5-1ubuntu2

[Lkml-m (lkml-m)]

Created attachment 92776
GPU hang log

Hi all,

Playing movies on my HTPC causes hanging in random intervals. Checking dmesg I see the request to create a post here along with the GPU hang file.

Running i3-3240 on Ubuntu 13.10 x64 , play back in XBMC alpha 10 with SOFTWARE playback.

Regards,
Bjoern

[Chris Wilson (ickle)]

It appears to have hung trying to execute a pageflip.

0x00007eb0: 0x0a000001: MI_DISPLAY_BUFFER_INFO
0x00007eb4: 0x00001e01: dword 1
0x00007eb8: 0x06446000: dword 2
0x00007ebc: 0x00000000: MI_NOOP

which looks consistent with

  fence[14] = 6c3d03b06446001
    valid, x-tiled, pitch: 7680, start: 0x06446000, size: 8355840

and

Pinned [33]:
...
  06446000 8355840 41 00 0 0 P X dirty uncached (name: 49) (fence: 14)

Can you please attach the full dmesg leading to the hang? Does this happen frequently or was this a one-off event?

[Lkml-m (lkml-m)]

Created attachment 92780
dmesg log

[Lkml-m (lkml-m)]

Find attached the dmesg log file.

I can reproduce this quite consistently. I noticed it yester when I watched a movie, a "freeze" came every few minutes (just the video, HDMI audio was fine). It still happens today, even after reboot.

[Chris Wilson (ickle)]

Theories for why the GPU may be upset:

1. Multiple render response messages
2. Page flip with flip outstanding
3. Forcewake is required
4. The hardware hates us

[Lkml-m (lkml-m)]

I need to do a check here on my end. After going to 3.13.0 I see some errors in dmesg (e.g. "factorial" or "conftest") poping up, even running a 3.11.10 kernel doesn't change this. Not sure, maybe something is wrong with Linux now or my hardware. It worked perfectly till I went to 3.13.0...

I'll run a memtest etc. and will update here asap.

[Chris Wilson (ickle)]

One wicked theory I have is that the intoduction of the working SRM is breaking the flips...

Can you please test:

diff --git a/drivers/gpu/drm/i915/intel_display.c b/drivers/gpu/drm/i915/intel_display.c
index 5b7ce3f09681..de70260e50f3 100644
--- a/drivers/gpu/drm/i915/intel_display.c
+++ b/drivers/gpu/drm/i915/intel_display.c
@@ -8593,7 +8593,7 @@ static int intel_gen7_queue_flip(struct drm_device *dev,

        len = 4;
        if (ring->id == RCS)
- len += 6;
+ len += 4;

        ret = intel_ring_begin(ring, len);
        if (ret)
@@ -8614,10 +8614,7 @@ static int intel_gen7_queue_flip(struct drm_device *dev,
                intel_ring_emit(ring, ~(DERRMR_PIPEA_PRI_FLIP_DONE |
                                        DERRMR_PIPEB_PRI_FLIP_DONE |
                                        DERRMR_PIPEC_PRI_FLIP_DONE));
- intel_ring_emit(ring, MI_STORE_REGISTER_MEM(1) |
- MI_SRM_LRM_GLOBAL_GTT);
- intel_ring_emit(ring, DERRMR);
- intel_ring_emit(ring, ring->scratch.gtt_offset + 256);
+ intel_ring_emit(ring, MI_NOOP);
        }

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Chris Wilson (ickle)]

Can you please 'cat /sys/kernel/debug/dri/0/i915_fbc_status'

[Ville-syrjala-e (ville-syrjala-e)]

Created attachment 93135
One potential idea

I tried to look through our gen7 page flip code. Looks like everything's according to spec, except we allow the MI_DISPLAY_FLIP to straddle two cachelines. This patch fixes that. Worth a shot I suppose even if the hanging flip in the error state didn't hit this. There were flips in the ring that would have hit this though.

[Ramesh Jude Fernando (rameshjfernando)]

Here you go
ramesh@ramesh-Aspire-V3-771:~$ cat /sys/kernel/debug/dri/0/i915_fbc_status
cat: /sys/kernel/debug/dri/0/i915_fbc_status: Permission denied
ramesh@ramesh-Aspire-V3-771:~$ sudo cat
/sys/kernel/debug/dri/0/i915_fbc_status
[sudo] password for ramesh:
FBC disabled: disabled per chip default
ramesh@ramesh-Aspire-V3-771:~$

Regards,
Ramesh

On Fri, Jan 31, 2014 at 7:32 AM, Chris Wilson <email address hidden>wrote:

> Can you please 'cat /sys/kernel/debug/dri/0/i915_fbc_status'
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1274746).
> https://bugs.launchpad.net/bugs/1274779
>
> Title:
> [ivb] hang on pageflip (IPEHR: 0x0a000001)
>
> Status in X.org xf86-video-intel:
> Incomplete
> Status in "xserver-xorg-video-intel" package in Ubuntu:
> Confirmed
>
> Bug description:
> False GPU lockup IPEHR: 0x0a000001
>
> ProblemType: Crash
> DistroRelease: Ubuntu 14.04
> Package: xserver-xorg-video-intel 2:2.99.907-0ubuntu1
> ProcVersionSignature: Ubuntu 3.13.0-5.20-generic 3.13.0
> Uname: Linux 3.13.0-5-generic x86_64
> .tmp.unity.support.test.0:
>
> ApportVersion: 2.13.2-0ubuntu2
> Architecture: amd64
> CompizPlugins: No value set for
> `/apps/compiz-1/general/screen0/options/active_plugins'
> CompositorRunning: compiz
> CompositorUnredirectDriverBlacklist: '(nouveau|Intel).*Mesa 8.0'
> CompositorUnredirectFSW: true
> Date: Wed Jan 29 23:32:19 2014
> DistUpgraded: 2013-12-25 22:45:46,524 DEBUG enabling apt cron job
> DistroCodename: trusty
> DistroVariant: ubuntu
> DuplicateSignature: GPU lockup IPEHR: 0x0a000001 Ubuntu 14.04
> ExecutablePath: /usr/share/apport/apport-gpu-error-intel.py
> ExtraDebuggingInterest: Yes
> GraphicsCard:
> Intel Corporation 3rd Gen Core processor Graphics Controller
> [8086:0166] (rev 09) (prog-if 00 [VGA controller])
> Subsystem: Lenovo Device [17aa:21f4]
> NVIDIA Corporation GF108M [NVS 5400M] [10de:0def] (rev a1) (prog-if 00
> [VGA controller])
> Subsystem: Lenovo Device [17aa:21f4]
> InstallationDate: Installed on 2013-05-30 (245 days ago)
> InstallationMedia: Ubuntu 13.04 "Raring Ringtail" - Release amd64
> (20130424)
> InterpreterPath: /usr/bin/python3.3
> MachineType: LENOVO 2342CTO
> ProcCmdline: /usr/bin/python3 /usr/share/apport/apport-gpu-error-intel.py
> ProcEnviron:
>
> ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-5-generic
> root=UUID=db82f5ed-cf1b-4bea-bcd8-8ec4fb76a850 ro acpi_backlight=vendor
> acpi_osi=Linux quiet splash i915.i915_enable_rc6=1 pcie_aspm=force
> vt.handoff=7
> RelatedPackageVersions:
> xserver-xorg 1:7.7+1ubuntu7
> libdrm2 2.4.52-1
> xserver-xorg-video-intel 2:2.99.907-0ubuntu1
> SourcePackage: xserver-xorg-video-intel
> Title: False GPU lockup IPEHR: 0x0a000001
> UpgradeStatus: Upgraded to trusty on 2013-12-26 (35 days ago)
> UserGroups:
>
> dmi.bios.date: 10/19/2012
> dmi.bios.vendor: LENOVO
> dmi.bios.version: G1ET73WW (2.09 )
> dmi.board.asset.tag: Not Available
> dmi.board.name: 2342CTO
> dmi.board.vendor: LENOVO
> dmi.board.version: Win8 STD DPK TPG
> dmi.chassis.asset.tag: No...

[Bruce Pieterse (octoquad)]

Just got the same error a couple of minutes ago. I have this in my dmesg log:

[ 5069.629929] [drm] stuck on render ring
[ 5069.629932] [drm] GPU crash dump saved to /sys/class/drm/card0/error
[ 5069.629933] [drm] GPU hangs can indicate a bug anywhere in the entire gfx stack, including userspace.
[ 5069.629934] [drm] Please file a _new_ bug report on bugs.freedesktop.org against DRI -> DRM/Intel
[ 5069.629935] [drm] drm/i915 developers can then reassign to the right component if it's not a kernel issue.
[ 5069.629936] [drm] The gpu crash dump is required to analyze gpu hangs, so please always attach it.
[ 5069.978334] atl1c 0000:0e:00.0: vpd r/w failed. This is likely a firmware bug on this device. Contact the card vendor for a firmware update.
[ 5070.070446] atl1c 0000:0e:00.0: vpd r/w failed. This is likely a firmware bug on this device. Contact the card vendor for a firmware update.

dmesg and crash dump from /sys/class/drm/card0/error attached.

Out of curioristy would the constant flickering of my cursor (sometimes hidden for few seconds before returning) be a result of this bug or caused by something else?

[Chris Wilson (ickle)]

It is unlikely to explain the cursor flicker - the hang here is quite rare. But a regular GPU hang could produce the symptoms you describe if the hang coincided with the cursor being hidden itself by software (the command to show the cursor will then need to wait for the hangcheck to be detected in some extreme case). Given that it requires a combination of unlikely situations, I doubt it actually explains the cursor flickering (and if it was a constant hang, the kernel will declare the hardware as wedged).

[Bruce Pieterse (octoquad)]

This is the first time I've had this hang, and it might have been introduced by the 3.13.0-6-generic kernel which I upgraded to today. When the gpu hang occurred my mouse cursor was hidden for about five seconds.

[Lkml-m (lkml-m)]

Chris, Ville: Neither patched helped. Around once a minute 20-30 frames are dropped in one go. I'm on git 3.13.0 and just apply your changes to that one - compared to my code I'm around 300 lines off from where you guys are doing the changes... I'll do a new git pull and then try again.

"FBC unsupported on this chipset" is what I get for i915_fbc_status.

[Bruce Pieterse (octoquad)]

I had another lockup today, but this time using an HDMI connection instead of VGA. This occurred while watching a Vimeo video which is slightly different from my first gpu hang.

[Lkml-m (lkml-m)]

I did more testing with a clean install of Ubuntu x64 13.10 and getting the usual updates and an XBMC nightly. Conclusion: XBMC runs fine without any issues as long as I don't use "bitstream" audio. Once bitstreaming is enabled and using ALSA directly the skipped frames occur.

So what I will do now is I'll take Ville's patch suggestion first and see where that brings me. If I don't get this error again then I'll test again without it. Then I'll take up Chris suggestion if the issue is not resolved. Once I know something in this regard I'll update you.

[Steve Langasek (vorlon)]

FWIW, I've just been redirected to this bug. While this is listed as a 'false GPU lockup', in my case the GPU was locked up for about 30 seconds before it spontaneously recovered.

[Chris Wilson (ickle)]

Starting to see multiple sightings in Ubuntu. We'll have to queue up a revert of RCS flips unless we can find the answer.

[Chris Wilson (ickle)]

*** Bug 74569 has been marked as a duplicate of this bug. ***

[Chris Wilson (ickle)]

Created attachment 93513
Frob FORCEWAKE around RCS flips

When in doubt, tell the GPU not to go to sleep.

[Enrico-tagliavini (enrico-tagliavini)]

Created attachment 93514
yet another crash dump

I'm affected as well. Running gentoo, happened with kernel 3.11, 3.12, 3.13 for sure, video-intel 2.21.15, 2.99.907, 2.99.909 (all with SNA enabled), mesa 9.2.5

dmesg:

[ 763.125484] [drm] stuck on render ring
[ 763.125485] [drm] GPU crash dump saved to /sys/class/drm/card0/error
[ 763.125485] [drm] GPU hangs can indicate a bug anywhere in the entire gfx stack, including userspace.
[ 763.125486] [drm] Please file a _new_ bug report on bugs.freedesktop.org against DRI -> DRM/Intel
[ 763.125486] [drm] drm/i915 developers can then reassign to the right component if it's not a kernel issue.
[ 763.125486] [drm] The gpu crash dump is required to analyze gpu hangs, so please always attach it.

~ # cat /sys/kernel/debug/dri/0/i915_fbc_status
FBC disabled: multiple pipes are enabled

[Chris Wilson (ickle)]

One question for everybody: Do you only see this in multi-monitor setups?

[Alexandru-damian-t (alexandru-damian-t)]

It happens for me on a multi-monitor setup, yes.

[Bruce Pieterse (octoquad)]

Mine also occurs on multi-monitor setup with VGA to VGA , HDMI to DVI and HDMI to HDMI connection.

[Chris Wilson (ickle)]

The multiple simultaneous render response messages theory is flawed; the original crash dump hung with only a single active pipe.

[Enrico-tagliavini (enrico-tagliavini)]

Created attachment 93534
crash dump when using patch from comment #13

Tried patch from comment #13 . Apparently it didn't solved the issue, but there is some notable change: usually I was able to spot the hang because everything freeze, only the mouse moves. Eventually and often, for some very weird reason, firefox fonts are corrupted, some latin letter is replaced with non latin one or anyway some other symbol, the only fix is to restart firefox. This time just the latter happened. Luckly I checked dmesg and I saw the "stuck at render ring" notification.

[Chris Wilson (ickle)]

(In reply to comment #18)
> Created attachment 93534 [details]
> crash dump when using patch from comment #13
>
> Tried patch from comment #13 . Apparently it didn't solved the issue,

That's actually reassuring. The crash dump shows that the patch is working and the GPU is awake, so forcewake is definitely not an issue here.

> but
> there is some notable change: usually I was able to spot the hang because
> everything freeze, only the mouse moves. Eventually and often, for some very
> weird reason, firefox fonts are corrupted, some latin letter is replaced
> with non latin one or anyway some other symbol, the only fix is to restart
> firefox. This time just the latter happened. Luckly I checked dmesg and I
> saw the "stuck at render ring" notification.

That just sounds like the usual dangers with a hung gpu and discarding work before resetting. (We have plans to fix it.)

[Chris Wilson (ickle)]

Enrico, if you have the opportunity can you try Ville's patch from comment 8?

[Enrico-tagliavini (enrico-tagliavini)]

(In reply to comment #20)
> Enrico, if you have the opportunity can you try Ville's patch from comment 8?

Hi Chris, I compiled and run the 3.13.1 kernel with named patch applied. For now everything is ok, but I still don't yell to victory. This bug is fairly hard to reproduce in my case. Usually it never happen more then once or twice a day so a single day without a crash can be a simple statistical fluctuation. I'll report back on Monday. As I said it happens mostly (only?) when I use ny dual monitor setup, and during the weekend I don't use it.

Cross your fingers!

[Enrico-tagliavini (enrico-tagliavini)]

Well no hangs for now! I downloaded the 3.13.2 kernel, applied the patch again and compiled it. From now on I'll start using this instead of the 3.13.1.

The difference I might experience compared to when running without the patch, but I'm not sure at all, if there is a difference it is quite small: during firefox rendering (or something else inside the app) freeze for a fraction of second (like half second or so). During such short and temporary freezes it was, rarely, happening the hang. As I said now there is no GPU hang, dmesg is always 100% clear for drm and i915 stuff, but those micro freezes might be a little more frequent with the patch. Again I want to stress this is very hard to quantify and so feel free to simply ignore this. It might just be part of the fact I'm keeping a lot more attention than usual on rendering times to spot an hang.

[Daniel-ffwll (daniel-ffwll)]

Assigning to Ville so he can submit the patch. To avoid bikeshedding this to death: I prefer if we add a new intel_ring_begin_cacheline_safe or so which encapsulates the logic. And obviously puts a WARN_ON if the requested length is bigger than 1 cachline ;-)

[Chris Wilson (ickle)]

commit f66fab8e1cd6b3127ba4c5c0d11539fbe1de1e36
Author: Ville Syrjälä <email address hidden>
Date: Tue Feb 11 19:52:06 2014 +0200

    drm/i915: Prevent MI_DISPLAY_FLIP straddling two cachelines on IVB

    According to BSpec the entire MI_DISPLAY_FLIP packet must be contained
    in a single cacheline. Make sure that happens.

    v2: Use intel_ring_begin_cacheline_safe()
    v3: Use intel_ring_cacheline_align() (Chris)

    Cc: Bjoern C <email address hidden>
    Cc: Alexandru DAMIAN <email address hidden>
    Cc: Enrico Tagliavini <email address hidden>
    Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=74053
    Signed-off-by: Ville Syrjälä <email address hidden>
    Cc: <email address hidden>
    Signed-off-by: Daniel Vetter <email address hidden>

[Enrico-tagliavini (enrico-tagliavini)]

Thank you very much, much appreciated the help. Can't wait for the next release!

Best regards
Enrico

[Chris Wilson (ickle)]

*** Bug 73437 has been marked as a duplicate of this bug. ***

[Enrico-tagliavini (enrico-tagliavini)]

Created attachment 95815
Crash dump with kernel 3.13.6 (including the final patch)

Hi There. Unfortunately this doesn't look solved. Had 2 hangs this week. This is better than before, but ultimately the issue doesn't look solved :(

Attached you can find my last crash dump.

Kind regards
Enrico

[Chris Wilson (ickle)]

*** Bug 76229 has been marked as a duplicate of this bug. ***

[Gordon Jin (gordon-jin)]

so looks like this bug should be reopened?

[Chris Wilson (ickle)]

Why? We are tracking the continuing saga in #77104.

[Chris Wilson (ickle)]

Fixed queued for 3.17.

[Anmar Oueja (anmar)]

Great that the bug is queued for 3.17. However, does that mean it will make the 14.10 and 14.04 kernels as a backport?

[Jean-Paul Calderone (exarkun)]

It would be great to know if there are any plans to backport this, and if so to which releases. Thanks.

[Timo Aaltonen (tjaalton)]

closing, the fix is in distro kernels 3.19 and up, which are also backported to 14.04