[i965gm] GPU lockup render.IPEHR: 0x02000004

[Ben Gamari]

[Impact]
Lockup several times a day preceded by corruption, started immediately after upgrading to precise on 1st April. Does not occur with Unity 2D. Seems to be more frequently triggered when using the Unity 3D application switcher.

Bug is believed to be specific to gen4 hardware (e.g. i965).

[Development Fix]
Fixed upstream, and backported to the 8.0 stable branch.
This fix was cherrypicked and is uploaded to Quantal.

The patch essentially moves a chunk of code to be executed only after the start of drawing. This chunk of code creates a single-level miptree for rendering to as a workaround for a gen4-specific limitation. The problem is that this was causing some code to use the renderbuffer's contents before it was actually ready to be used (thus the glitches).

[Stable Fix]
Since the same version of mesa is (currently) in Precise as in Quantal, the same patch can be used.

[Test Case]
1. On affected i965 systems,
2. boot into Unity 3d
3. Repeatedly use application switching, open and close windows, etc.
4. Should see the described bug in the course of 1 day of such testing

[Regression Potential]
The patch is on the large side, which can be a bit worrying, but the changes are essentially just shifting code from one area to another. On the plus side, the patch has been carried in the upstream tree and the upstream stable branch for a number of months now.

The chunk of code appears to be limited to execution only on certain hardware. It tests for certain missing features which I gather only occur on certain i965 hardware. So, if there were a regression, the scope of the regression would be limited to just that class of hardware.

ProblemType: Crash
DistroRelease: Ubuntu 12.04
Package: xserver-xorg-video-intel 2:2.17.0-1ubuntu4
ProcVersionSignature: Ubuntu 3.2.0-23.36-generic 3.2.14
Uname: Linux 3.2.0-23-generic x86_64
.tmp.unity.support.test.0:

ApportVersion: 2.0.1-0ubuntu2
Architecture: amd64
Chipset: i965gm
CompizPlugins: No value set for `/apps/compiz-1/general/screen0/options/active_plugins'
CompositorRunning: compiz
Date: Thu Apr 12 11:54:19 2012
DistUpgraded: 2012-04-01 17:31:17,679 DEBUG enabling apt cron job
DistroCodename: precise
DistroVariant: ubuntu
DuplicateSignature: [i965gm] GPU lockup render.IPEHR: 0x02000004 Ubuntu 12.04
ExecutablePath: /usr/share/apport/apport-gpu-error-intel.py
ExtraDebuggingInterest: Yes, whatever it takes to get this fixed in Ubuntu
GpuHangFrequency: Several times a day
GpuHangReproducibility: Seems to happen randomly
GpuHangStarted: Immediately after installing this version of Ubuntu
GraphicsCard:
 Intel Corporation Mobile GM965/GL960 Integrated Graphics Controller (primary) [8086:2a02] (rev 0c) (prog-if 00 [VGA controller])
   Subsystem: Dell Device [1028:01fe]
   Subsystem: Dell Device [1028:01fe]
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Release amd64 (20111012)
InterpreterPath: /usr/bin/python2.7
MachineType: Dell Inc. Latitude D830
PccardctlIdent:
 Socket 0:
   no product info available
PccardctlStatus:
 Socket 0:
   no card
ProcCmdline: /usr/bin/python /usr/share/apport/apport-gpu-error-intel.py
ProcEnviron:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-23-generic root=UUID=27880cc8-df42-4098-8e07-3c4fb9dba0a5 ro quiet splash vt.handoff=7
RelatedPackageVersions:
 xserver-xorg 1:7.6+12ubuntu1
 libdrm2 2.4.32-1ubuntu1
 xserver-xorg-video-intel 2:2.17.0-1ubuntu4
SourcePackage: xserver-xorg-video-intel
Title: [i965gm] GPU lockup render.IPEHR: 0x02000004
UpgradeStatus: Upgraded to precise on 2012-04-01 (10 days ago)
UserGroups:

dmi.bios.date: 01/04/2010
dmi.bios.vendor: Dell Inc.
dmi.bios.version: A15
dmi.board.vendor: Dell Inc.
dmi.chassis.type: 8
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvrA15:bd01/04/2010:svnDellInc.:pnLatitudeD830:pvr:rvnDellInc.:rn:rvr:cvnDellInc.:ct8:cvr:
dmi.product.name: Latitude D830
dmi.sys.vendor: Dell Inc.
version.compiz: compiz 1:0.9.7.6-0ubuntu1~ppa3
version.ia32-libs: ia32-libs N/A
version.libdrm2: libdrm2 2.4.32-1ubuntu1
version.libgl1-mesa-dri: libgl1-mesa-dri 8.0.2-0ubuntu3
version.libgl1-mesa-dri-experimental: libgl1-mesa-dri-experimental N/A
version.libgl1-mesa-glx: libgl1-mesa-glx 8.0.2-0ubuntu3
version.xserver-xorg-core: xserver-xorg-core 2:1.11.4-0ubuntu10
version.xserver-xorg-input-evdev: xserver-xorg-input-evdev 1:2.7.0-0ubuntu1
version.xserver-xorg-video-ati: xserver-xorg-video-ati 1:6.14.99~git20111219.aacbd629-0ubuntu2
version.xserver-xorg-video-intel: xserver-xorg-video-intel 2:2.17.0-1ubuntu4
version.xserver-xorg-video-nouveau: xserver-xorg-video-nouveau 1:0.0.16+git20111201+b5534a1-1build2

[Ben Gamari (bgamari)]

Lockup is generally preceded by obvious display corruption. Eventually GPU locks up, resulting in a blank screen and unresponsive machine, even to sysrq.

[Ben Gamari (bgamari)]

Here is an example of the sort of corruption exhibited. Note the color and grey boxes of the menu bar at the top of the screen.

[Ben Gamari (bgamari)]

It seems likely that this is related to (probably even a duplicate of) #768184

[Ben Gamari (bgamari)]

It actually seems that the system will respond to sysrq if issued not too long after the screen turns blank. After a few seconds however, it will not respond.

The application switcher seems to be very good at reproducing the crash, which occurs quite often, usually within five minutes of logging in. The machine is completely stable under Unity 2D.

[Ben Gamari (bgamari)]

It seems this is likely the same issue as #978836.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Bryce Harrington (bryce)]

Forwarding this bug from Ubuntu reporter Ben Gamari:
http://bugs.launchpad.net/ubuntu/+source/xserver-xorg-video-intel/+bug/980017

[Problem]
Lockup several times a day preceded by corruption, started immediately after upgrading to precise on 1st April. Does not occur with Unity 2D. Seems to be more frequently triggered when using the Unity 3D application switcher.

We saw quite a few 0x02000004 bugs last cycle in Ubuntu Oneiric, but this cycle just a couple. (LP #978836 being the other report).

[Original Description]
Machine sporadically locks up while running compiz.

Lockup is generally preceded by obvious display corruption. Eventually GPU locks up, resulting in a blank screen and unresponsive machine, even to sysrq.

Attached is an example of the sort of corruption exhibited. Note the color and grey boxes of the menu bar at the top of the screen.

It actually seems that the system will respond to sysrq if issued not too long after the screen turns blank. After a few seconds however, it will not respond.

The application switcher seems to be very good at reproducing the crash, which occurs quite often, usually within five minutes of logging in. The machine is completely stable under Unity 2D.

ProblemType: Crash
DistroRelease: Ubuntu 12.04
Package: xserver-xorg-video-intel 2:2.17.0-1ubuntu4
ProcVersionSignature: Ubuntu 3.2.0-23.36-generic 3.2.14
Uname: Linux 3.2.0-23-generic x86_64
.tmp.unity.support.test.0:

ApportVersion: 2.0.1-0ubuntu2
Architecture: amd64
Chipset: i965gm
CompizPlugins: No value set for `/apps/compiz-1/general/screen0/options/active_plugins'
CompositorRunning: compiz
Date: Thu Apr 12 11:54:19 2012
DistUpgraded: 2012-04-01 17:31:17,679 DEBUG enabling apt cron job
DistroCodename: precise
DistroVariant: ubuntu
DuplicateSignature: [i965gm] GPU lockup render.IPEHR: 0x02000004 Ubuntu 12.04
ExecutablePath: /usr/share/apport/apport-gpu-error-intel.py
ExtraDebuggingInterest: Yes, whatever it takes to get this fixed in Ubuntu
GpuHangFrequency: Several times a day
GpuHangReproducibility: Seems to happen randomly
GpuHangStarted: Immediately after installing this version of Ubuntu
GraphicsCard:
 Intel Corporation Mobile GM965/GL960 Integrated Graphics Controller (primary) [8086:2a02] (rev 0c) (prog-if 00 [VGA controller])
   Subsystem: Dell Device [1028:01fe]
   Subsystem: Dell Device [1028:01fe]
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Release amd64 (20111012)
InterpreterPath: /usr/bin/python2.7
MachineType: Dell Inc. Latitude D830
PccardctlIdent:
 Socket 0:
   no product info available
PccardctlStatus:
 Socket 0:
   no card
ProcCmdline: /usr/bin/python /usr/share/apport/apport-gpu-error-intel.py
ProcEnviron:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-23-generic root=UUID=27880cc8-df42-4098-8e07-3c4fb9dba0a5 ro quiet splash vt.handoff=7
RelatedPackageVersions:
 xserver-xorg 1:7.6+12ubuntu1
 libdrm2 2.4.32-1ubuntu1
 xserver-xorg-video-intel 2:2.17.0-1ubuntu4
SourcePackage: xserver-xorg-video-intel
Title: [i965gm] GPU lockup render.IPEHR: 0x02000004
UpgradeStatus: Upgraded to precise on 2012-04-01 (10 days ago)
UserGroups:

dmi.bios.date: 01/04/2010
dmi.bios.vendor: Dell Inc.
d...

[Bryce Harrington (bryce)]

Created attachment 60189
BootDmesg.txt

[Bryce Harrington (bryce)]

Created attachment 60190
CurrentDmesg.txt

[Bryce Harrington (bryce)]

Created attachment 60191
i915_error_state.txt

[Bryce Harrington (bryce)]

Created attachment 60192
XorgLog.txt

[Bryce Harrington (bryce)]

Ben Gamari - I've forwarded this bug upstream to https://bugs.freedesktop.org/show_bug.cgi?id=48829 - please subscribe yourself to this bug, in case they need further information or wish you to test something. Thanks ahead of time!

[Bryce Harrington (bryce)]

Created attachment 60193
Screenshot from 2012-04-12 12:05:52.png

[Chris Wilson (ickle)]

The error-state looks ordinary and more importantly self-consistent. There are not the tell-tales of recent bugs, so I currently have no explanation for the hang. Can you please attach a few more error-states to see if a pattern forms?

[Ben Gamari (bgamari)]

Created attachment 60321
Another crash dump

The panel was inactive although the machine was responsive over SSH.

[Chris Wilson (ickle)]

In the second crash dump, mesa overwrote our batch performing a depth-clear.

One more...

[Ben Gamari (bgamari)]

Created attachment 60322
Yet another dump

[Chris Wilson (ickle)]

(In reply to comment #9)
> Created attachment 60322 [details]
> Yet another dump

This wasn't a hang, so I'm not going to use its vote as to whether there is an underlying UXA bug here...

[Ben Gamari (bgamari)]

Created attachment 60323
i915_error_state from odd hang

This time I couldn't get the rest of the dump as cat BUGs in i915_batchbuffer_info. Nevertheless, here is i915_error_state.

[Ben Gamari (bgamari)]

Created attachment 60324
crash-20120419-1230

[Chris Wilson (ickle)]

That time, the stray depth clear hit a Mesa batch buffer. With 3 clear errors, let's presume this is the first and foremost the stray clear that's causing the hangs.

[Ben Gamari (bgamari)]

Created attachment 60325
crash-20120419-1233

[Ben Gamari (bgamari)]

Created attachment 60327
crash-20120419-1242

[Ben Gamari (bgamari)]

I can confirm that the problem appears to be gone with mesa master (dbf48e88)

[Ben Gamari (bgamari)]

The 8.0 branch (6fe42b6) exhibits the problem.

[Ben Gamari (bgamari)]

(In reply to comment #17)
> The 8.0 branch (6fe42b6) exhibits the problem.

To clarify the 8.0 branch (currently 49ed43b6) exhibits the issue as does 6fe42b6, the point where master diverged from 8.0.

[Ben Gamari (bgamari)]

8f5c172c does not exhibit the problem

[Ben Gamari (bgamari)]

952ca07 exhibits the problem.

[Ben Gamari (bgamari)]

7335cf1c exhibits the problem.

[Ben Gamari (bgamari)]

9be0f9 exhibits the issue.

[Ben Gamari (bgamari)]

dbadd39 does not exhibit the problem.

[Ben Gamari (bgamari)]

f00c97b does not exhibit the problem.
117a0e9 exhibits the problem.
308c6be exhibits the problem.
fbe8543 does not exhibit the problem.
e2dce7f does not exhibit the problem.

[Ben Gamari (bgamari)]

Here is the first working commit:

commit e2dce7f7ee3e7da9cbb0bb33307ecd79e824426d
Author: Eric Anholt <email address hidden>
Date: Fri Feb 10 12:54:25 2012 -0800

    intel: Fix rendering from textures after RenderTexture().

    There's a serious trap for drivers: RenderTexture() does not indicate
    that the texture is currently bound to the draw buffer, despite
    FinishRenderTexture() signaling that the texture is just now being
    unbound from the draw buffer.

    We were acting as if RenderTexture() *was* the start of rendering and
    that we could make texturing incoherent with the current contents of
    the renderbuffer. This caused intel oglconform sRGB
    Mipmap.1D_textures to fail, because we got a call to TexImage() and
    thus RenderTexture() on a texture bound to a framebuffer that wasn't
    the draw buffer, so we skipped validating the new image into the
    texture object used for rendering.

    We can't (easily) make RenderTexture() indicate the start of drawing,
    because both our driver and gallium are using it as the moment to set
    up the renderbuffer wrapper used for things like MapRenderbuffer().
    Instead, postpone the setup of the workaround render target miptree
    until update_renderbuffer time, so that we no longer need to skip
    validation of miptrees used as render targets. As a bonus, this
    should make GL_NV_texture_barrier possible.

    (This also fixes a regression in the gen4 small-mipmap rendering since
    3b38b33c1648b07e75dc4d8340758171e109c598, which switched
    set_draw_offset from image->mt to irb->mt but didn't move the irb->mt
    replacement up before set_draw_offset).

    Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=44961
    NOTE: This is a candidate for the 8.0 branch.

[Ben Gamari (bgamari)]

The problem is fixed in mesa master. I bisected the fix to be e2dce7f7. This fix should be cherry-picked into the 8.0 branch shortly and a new mesa package needs to be spun before release if we want compiz to work for Intel users out of the box.

[Chris Morgan (chris.morgan)]

Upgraded to from Ubuntu 11.10 to 12.04 on my HP 6710b laptop (Intel integrated graphics, naturally) and I was immediately hit by this upon opening the application switcher with Alt-Tab (I don't use the default application switcher, ccsm reminds me that the one I use is the Static Application Switcher; by the way, I use Unity and haven't tried Unity 2D). As soon as the application switcher opened, the first time all the squares (window previews and application icons) were black and a couple of seconds later the display lost all input. The second and third times I can't remember what happened with those squares. The fourth time (just now, intentionally) they all appeared as white squares with RGB rainbow fuzz around the edges. In all cases (whether the switcher was shown briefly or Alt-Tab held down), the display panel lost input a couple of seconds later.

Also in all cases Alt+SysRq+REISUB worked for me (it has been a pleasantly long time since I have had recourse to the sequence, sadly now broken by this), but the graphics driver was completely dead (so no Ctrl+Alt+F1 terminal, etc.).

This is to me (to put it mildly) rather an important issue. I've already caused it twice more intentionally, once accidentally and *almost* done it by accident a couple of times in this present session; fortunately I have the popup window delay set to 0.05, so it doesn't immediately pop up the switcher window which makes it die. I'll try the default application switcher after submitting this comment and see what happens with it.

If you want more information from me, just ask and I'll provide it*.

(* Conditions apply. Of course.)

[Ben Gamari (bgamari)]

Last week I spent nearly an entire day bisecting for the fix so that the patch could be cherry-picked into the Ubuntu tree. Unfortunately, it looks like no one ever followed up on this work. This is mildly annoying considering the work put in to the solve this very severe issue. Sadly, it seems that the 12.04 upgrade will be a complete mess for a large portion of the user base due to this bug.

[Chris Morgan (chris.morgan)]

The Application Switcher Compiz plugin didn't work, but the default Unity switcher did, so I'm now using it. That quite certainly mitigates my problem.

[beccatoria (beccatoria)]

This is affecting me too - rendering 12.04 upgrade totally unuseable unless I'm in a desktop with no effects which is pretty annoying. Thanks to anyone who's working on fixing this - any updates on how I can work around it, if at all?

It doesn't matter what desktop switcher I use - it'll work until I try to *close* an application on a desktop other than the first, then all hell breaks loose and the display crashes followed by total nonresponsiveness as described by others above.

[Ben Gamari (bgamari)]

I am currently waiting for any sort of response from the packagers. It's been weeks since the exact fix necessary was identified and yet nothing has happened from the Ubuntu side. The patch in question was pulled onto mesa's 8.0 branch as 93e94cbb48. The release should have never gone out with such a blatant and easily fixed bug. Moreover, it's quite embarrassing that we are still struggling to get a fix out.

I am currently running mesa built directly from the 8.0 branch. Perhaps I can try building a package to push up to a PPA in lieu of any action from the packagers. Unfortunately, this probably won't happen until tomorrow.

[horaciod (horaciod)]

Same problem with notebook hp530, thanks in advance for the fix

[beccatoria (beccatoria)]

Ben Gamari - I agree with everything you've said and thank you so much for your proactive work in identifying and fixing this issue. I would be incredibly grateful for any sort of assistance in fixing this - a PPA with the build you described would be fantastic - by tomorrow, even? Sir, you are a god among, um, fellow Ubuntu users. Heck, I'd be impressed if you'd said next week. Seriously - anytime you have the time to do this given nothing seems to be being done officially would be fantastic in my book. :)

[Eric Anholt (eric-anholt)]

Pushed the cherry pick.

[beccatoria (beccatoria)]

Just checked the freedesktop.org bug report on the issue linked above (https://bugs.freedesktop.org/show_bug.cgi?id=48829) and it looks like the fix got pushed into the main mesa 8.0 branch yesterday. Hopefully this means Ubuntu will take action on putting out an update soon... *crosses fingers*

[Ben Gamari (bgamari)]

I've been having some difficulty pushing the new package to a PPA. Those affected by the issue could consider pulling the mesa packages from the xorg-edgers repository (https://launchpad.net/~xorg-edgers/+archive/ppa) in lieu of my package. I've not tested this but xorg-edgers appears to have 8.1 which should have the needed fix.

Beyond that, hopefully the mesa maintainers will acknowledge this soon.

[beccatoria (beccatoria)]

Awesome - thank you VERY much, I'm happy to report that adding the xorg-edgers repository has largely solved my issues.

For information, I am still getting the error message a few minutes after startup most of the time (a few times it has not happened). And if I try to report it it seems to freak out and spawn infinite errors. But if I cancel reporting it and ignore it, then there's no further issue, I receive no further error notifications, and Unity + Compiz seem to be working fine together. When changing some settings in the Compiz Manager and on a few other occasions, it seems as though Compiz has rebooted itself (I briefly get a blank desktop with nothing but my wallpaper before everything reappears), but this has only happened to me on once outside of actively fiddling with the Compiz settings and every time it's recovered in 2 - 3 seconds rather than crashing and dying, or providing a corrupted display as it was previously.

It's very useable now as long as I don't try to report that initial error. *gigantic thumbs up*

Thanks again for being so helpful, Ben!

[Gabriele Vivinetto (gabriele.vivinetto)]

As of today, with the latest updates, the bug is still affecting me.

[Ben Gamari (bgamari)]

Can someone involved in the packaging of mesa please let us know what the status of this is? Will a new release be cut from the 8.0 branch soon?

[beccatoria (beccatoria)]

As a further update to my previous comment, while the xorg-edgers ppa has resolved my problems to the point my system is useable when running compiz, time has revealed slightly more severe problems than previously reported. Largely it's as I said - as long as I ignore the error report, everything works pretty much fine. But I have gotten a few system-wide crashes since I last commented - most commonly right after I log in. If everything goes fine in the few minutes after login, I generally have a stable system after that, as long as I cancel the option to report any errors.

I hope that's helpful information to anyone tracking the issue.

[beccatoria (beccatoria)]

...and in a spectacular demonstration of why I should update my system before reporting stuff like this, about an hour and a half ago, my update manager popped up with a lot of new stuff from xorg-edgers, and out of curiosity, I rebooted my computer to see if it had helped with the situation. Unfortunately, rather than helping it appears to have made things worse - I'm now back in the same position I was before. Doing almost anything in unity or classic desktop with compiz enabled causes graphic degradation, crashes and freezes requiring a hard reboot. So I'm back to the 2d desktop.

I'm completely back to where I was before and not sure what to do about it. Well...it was a nice four days that I had compiz back and running? *sigh*

[Launchpad Janitor (janitor)]

This bug was fixed in the package mesa - 8.0.2-0ubuntu4

---------------
mesa (8.0.2-0ubuntu4) quantal; urgency=low

  * Add 118_fix_rendering_from_textures.patch: Postpone the setup of the
    workaround render target miptree until update_renderbuffer time.
    Fixes a GPU hang preceded by corruption which can be triggered using
    the Unity 3D application switcher. Cherrypick from the upstream
    8.0 (stable) tree.
    (LP: #980017)
 -- Bryce Harrington <email address hidden> Mon, 14 May 2012 10:06:36 -0700

[Chris Halse Rogers (raof)]

Hello Ben, or anyone else affected,

Accepted mesa into precise-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Timo Aaltonen (tjaalton)]

I've tested it with an Ivybridge desktop for over a week now, and I can confirm that it hangs only maybe once a day with this fix, instead of right after login or within the first hour or so. The other hang is due to the kernel, 3.4 is solid.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mesa - 8.0.2-0ubuntu3.1

---------------
mesa (8.0.2-0ubuntu3.1) precise-proposed; urgency=low

  * Add 118_fix_rendering_from_textures.patch: Postpone the setup of the
    workaround render target miptree until update_renderbuffer time.
    Fixes a GPU hang preceded by corruption which can be triggered using
    the Unity 3D application switcher. Cherrypick from the upstream
    8.0 (stable) tree.
    (LP: #980017)
 -- Bryce Harrington <email address hidden> Mon, 14 May 2012 10:06:36 -0700