[radeon] Compiz memory leak and blank screen unable to login when using radeon driver

[Ronald Duncan]

The issue appears to be that radeon's implementation of glXWaitVideoSyncSGI (from libgl1-mesa-glx) hangs and leaks X events while the screen is locked and asleep. The leak is around 200 bytes (one XEvent) per frame or about 12.5KB per second.

It is most noticeable if some application is continuously redrawing during this time. But only if the application itself does not sync to vblank. For example: KVM, QEMU or:
    env vblank_mode=0 glxgears

WORKAROUNDS:

1. Before leaving/locking the PC run ccsm, and in the OpenGL section disable "Sync To VBlank".

2. Before leaving/locking the PC, minimize all active windows.

3. Install fglrx driver from AMD/ATI. But then you'll have bug 969860.

ORIGINAL DESCRIPTION:
There are two issues.

Compiz has a memory leak so it expands to take up all memory plus swap (12 + 4 gig)

Unity shows a blank screen and does not come up with a login prompt, only the mouse pointer is displayed.

ctl+alt F1 works so you can get into a console and create the attached bug report. This was after killing vviewer and before restarting lightdm

I was running KVM and vviewer as well and killed vviewer but still did not get get login screen
Killed compiz which freed the memory still did not get login screen
had to restart lightdm to get back into Unity and restart everything.

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: unity 4.24.0-0ubuntu2b1
ProcVersionSignature: Ubuntu 3.1.0-2.6-generic 3.1.0
Uname: Linux 3.1.0-2-generic x86_64
ApportVersion: 1.23-0ubuntu4
Architecture: amd64
CompizPlugins: [core,bailer,detection,composite,opengl,decor,mousepoll,vpswitch,regex,animation,snap,expo,move,compiztoolbox,place,grid,imgpng,gnomecompat,wall,ezoom,workarounds,resize,fade,unitymtgrabhandles,scale,session,unityshell]
CurrentDmesg: Error: command ['sh', '-c', 'dmesg | comm -13 --nocheck-order /var/log/dmesg -'] failed with exit code 1: comm: /var/log/dmesg: Permission denied
Date: Wed Jan 11 13:16:58 2012
DistUpgraded: Log time: 2011-10-17 17:02:13.788852
DistroCodename: oneiric
DistroVariant: ubuntu
ExecutablePath: /usr/bin/compiz
GraphicsCard:
 ATI Technologies Inc Cedar PRO [Radeon HD 5450] [1002:68f9] (prog-if 00 [VGA controller])
   Subsystem: Micro-Star International Co., Ltd. Device [1462:2341]
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Release amd64 (20110427.1)
MachineType: Gigabyte Technology Co., Ltd. H67M-UD2H-B3
ProcEnviron:
 LANGUAGE=en_GB:en
 PATH=(custom, no user)
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.1.0-2-generic root=UUID=75185a0a-ffd1-4a37-9bda-52f17a050b24 ro quiet splash vt.handoff=7
SourcePackage: unity
UpgradeStatus: Upgraded to oneiric on 2011-10-17 (85 days ago)
dmi.bios.date: 02/22/2011
dmi.bios.vendor: Award Software International, Inc.
dmi.bios.version: F2
dmi.board.name: H67M-UD2H-B3
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.type: 3
dmi.chassis.vendor: Gigabyte Technology Co., Ltd.
dmi.modalias: dmi:bvnAwardSoftwareInternational,Inc.:bvrF2:bd02/22/2011:svnGigabyteTechnologyCo.,Ltd.:pnH67M-UD2H-B3:pvr:rvnGigabyteTechnologyCo.,Ltd.:rnH67M-UD2H-B3:rvrx.x:cvnGigabyteTechnologyCo.,Ltd.:ct3:cvr:
dmi.product.name: H67M-UD2H-B3
dmi.sys.vendor: Gigabyte Technology Co., Ltd.
version.compiz: compiz 1:0.9.6+bzr20110929-0ubuntu6
version.ia32-libs: ia32-libs 20090808ubuntu26
version.libdrm2: libdrm2 2.4.26-1ubuntu1
version.libgl1-mesa-dri: libgl1-mesa-dri 7.11-0ubuntu3
version.libgl1-mesa-dri-experimental: libgl1-mesa-dri-experimental N/A
version.libgl1-mesa-glx: libgl1-mesa-glx 7.11-0ubuntu3
version.xserver-xorg: xserver-xorg 1:7.6+7ubuntu7
version.xserver-xorg-input-evdev: xserver-xorg-input-evdev 1:2.6.0-1ubuntu13
version.xserver-xorg-video-ati: xserver-xorg-video-ati 1:6.14.99~git20110811.g93fc084-0ubuntu1
version.xserver-xorg-video-intel: xserver-xorg-video-intel 2:2.15.901-1ubuntu2.1
version.xserver-xorg-video-nouveau: xserver-xorg-video-nouveau 1:0.0.16+git20110411+8378443-1

[Ronald Duncan (ronaldduncan)]

Compiz used up the memory whilst I was away from the computer for a couple of days after restarting lightdm it went back down to 626meg, from about 8gig prior to creating the bug report.

[Ronald Duncan (ronaldduncan)]

Added in symbol packages and ran compiz for a couple of minutes under valgrind.

==14236== LEAK SUMMARY:
==14236== definitely lost: 57,142 bytes in 707 blocks
==14236== indirectly lost: 121,649 bytes in 3,809 blocks
==14236== possibly lost: 505,138 bytes in 2,110 blocks
==14236== still reachable: 15,955,329 bytes in 74,116 blocks
==14236== suppressed: 40,536 bytes in 402 blocks
==14236== Reachable blocks (those to which a pointer was found) are not shown.
==14236== To see them, rerun with: --leak-check=full --show-reachable=yes

Going to leave it running whilst I have a break to recreate the conditions when it goes to screen saver and does not come back.

[Ronald Duncan (ronaldduncan)]

Attached is valgrind.log

==15020== LEAK SUMMARY:
==15020== definitely lost: 71,205 bytes in 1,101 blocks
==15020== indirectly lost: 303,006 bytes in 4,119 blocks
==15020== possibly lost: 39,590 bytes in 470 blocks
==15020== still reachable: 6,922,482 bytes in 41,776 blocks
==15020== suppressed: 40,336 bytes in 400 blocks

After going away and leaving the machine idle for an hour.

[Ronald Duncan (ronaldduncan)]

I have had this problem a number of times in the past, it normally occurs if my machine is left running for a few days whilst I am away, so an hours log probably does not have this issue, but there are certainly a few memory leaks in the current version (or its libraries).

[Bilal Akhtar (bilalakhtar)]

I've also had memory leaks in the past, but none as massive as this one.

That could well be because I never left my computer on for that much time.

[Damien Churchill (damoxc)]

When I arrived into work this morning I couldn't unlock the computer, I was just getting blank screens. A switch to VT1 and running top (both of which were awfully slow) revealed that compiz was using 12G of memory, so there is definitely a rather bad leak somewhere.

[Ronald Duncan (ronaldduncan)]

I got the leak again. I was only away from the machine for 8 hours.

I was able to switch to VT1 and kill firefox and skype, which freed enough memory that the login prompt would come up, and I was able to login.

Compiz has been running since 22 Jan and is now up at 3.5 gig (I have stopped by kvm session that was using the rest of the memory.) So I could reopen firefox.

ronald@dev5:~$ top

top - 11:39:18 up 20 days, 2:00, 5 users, load average: 0.14, 0.15, 0.29
Tasks: 204 total, 2 running, 201 sleeping, 0 stopped, 1 zombie
Cpu(s): 2.8%us, 1.8%sy, 0.0%ni, 95.0%id, 0.4%wa, 0.1%hi, 0.0%si, 0.0%st
Mem: 16419052k total, 7087412k used, 9331640k free, 190120k buffers
Swap: 7999484k total, 1105540k used, 6893944k free, 1113952k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28891 ronald 20 0 4911m 3.5g 8840 R 6 22.5 510:06.85 compiz

ps results
ronald@dev5:~$ ps -eF|grep compiz
ronald 24915 27784 0 3641 896 1 11:39 pts/3 00:00:00 grep --color=auto compiz
ronald 28891 28814 1 1243782 3702344 2 Jan22 ? 08:30:09 compiz
ronald 28965 28891 0 1066 112 2 Jan22 ? 00:00:00 /bin/sh -c /usr/bin/compiz-decorator

The problem is that if running valgrind things are very very slow, so I need to remember to do it the next time I will be away for a few days.

[Omer Akram (om26er)]

Are you using indicator-multiload? There was a fix made to nux that may fix this issue or related one. I have backported the fix for oneiric and waiting for it to get sponsored. so the update may soon reach oneiric users as well, see bug 888039

[James Grace (jamesgrace)]

I have tested, and this bug is isolated to x86-64 architecture. Another words, this bug is non existant in 32 bit systems. Anyone know why?

When I tested on x86-64, I was using indicator-multiload = lets hope the fix gets sponsored soon.

[Tim Penhey (thumper)]

I ran the 64 bit one for a significant time without any of these errors.

My guess is that this is a leak in indicator-multiload as that is the only difference we had.

[Michael Hofmann (mh21)]

As I understand the bug report, the memory leak is in compiz?

Closing as invalid for indicator-multiload.

[kayandus (bierfiltertje)]

I'm also affected by this bug. If I'm not using my PC for I think around 10 hours, there's like a 50% chance compiz has used up almost all memory. Only thing you can do is go to the console and SIGKILL it (won't listen to SIGTERM). Also on 64-bit and NOT running indicator-multiload.

[kayandus (bierfiltertje)]

Oh and not using unity either, just plain compiz.

[kayandus (bierfiltertje)]

Attached the full valgrind log, while running compiz for about 8 hours, with options --leak-check=full --show-reachable=yes enabled. Most interesting part is probably the last few lines of the log:

==17463== 314,222,480 bytes in 1,510,685 blocks are still reachable in loss record 7,754 of 7,755
==17463== at 0x4C28F9F: malloc (vg_replace_malloc.c:236)
==17463== by 0x4E7223E: _XEnq (in /usr/lib/x86_64-linux-gnu/libX11.so.6.3.0)
==17463== by 0x4E6EE92: ??? (in /usr/lib/x86_64-linux-gnu/libX11.so.6.3.0)
==17463== by 0x4E6FC6F: _XReply (in /usr/lib/x86_64-linux-gnu/libX11.so.6.3.0)
==17463== by 0x11730A5A: ??? (in /usr/lib/x86_64-linux-gnu/mesa/libGL.so.1.2)
==17463== by 0x1172EE30: ??? (in /usr/lib/x86_64-linux-gnu/mesa/libGL.so.1.2)
==17463== by 0x11706F76: ??? (in /usr/lib/x86_64-linux-gnu/mesa/libGL.so.1.2)
==17463== by 0x114DB1A3: PrivateGLScreen::waitForVideoSync() (in /usr/lib/compiz/libopengl.so)
==17463== by 0x114DB368: PrivateGLScreen::paintOutputs(std::list<CompOutput*, std::allocator<CompOutput*> >&, unsigned int, CompRegion const&) (in /usr/lib/compiz/libopengl.so)
==17463== by 0x112A5F08: CompositeScreen::paint(std::list<CompOutput*, std::allocator<CompOutput*> >&, unsigned int) (in /usr/lib/compiz/libcomposite.so)
==17463== by 0x112A777E: CompositeScreen::handlePaintTimeout() (in /usr/lib/compiz/libcomposite.so)
==17463== by 0x424DBA: CompTimer::triggerCallback() (in /usr/bin/compiz)
==17463==
==17463== 315,954,288 bytes in 1,519,011 blocks are still reachable in loss record 7,755 of 7,755
==17463== at 0x4C28B35: operator new(unsigned long) (vg_replace_malloc.c:261)
==17463== by 0x436FF9: std::list<_XEvent, std::allocator<_XEvent> >::operator=(std::list<_XEvent, std::allocator<_XEvent> > const&) (in /usr/bin/compiz)
==17463== by 0x431BF3: PrivateScreen::processEvents() (in /usr/bin/compiz)
==17463== by 0x45EAD0: CompEventSource::callback() (in /usr/bin/compiz)
==17463== by 0x623A9DE: Glib::Source::dispatch_vfunc(_GSource*, int (*)(void*), void*) (in /usr/lib/libglibmm-2.4.so.1.3.0)
==17463== by 0x66A6A5C: g_main_context_dispatch (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3000.0)
==17463== by 0x66A7257: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3000.0)
==17463== by 0x66A7791: g_main_loop_run (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3000.0)
==17463== by 0x42E8B5: CompScreen::eventLoop() (in /usr/bin/compiz)
==17463== by 0x422279: main (in /usr/bin/compiz)
==17463==
==17463== LEAK SUMMARY:
==17463== definitely lost: 420 bytes in 5 blocks
==17463== indirectly lost: 240 bytes in 10 blocks
==17463== possibly lost: 356,292 bytes in 3,767 blocks
==17463== still reachable: 638,139,513 bytes in 3,076,735 blocks
==17463== suppressed: 0 bytes in 0 blocks
==17463==
==17463== For counts of detected and suppressed errors, rerun with: -v
==17463== Use --track-origins=yes to see where uninitialised values come from
==17463== ERROR SUMMARY: 9591 errors from 1260 contexts (suppressed: 23 from 8)

Hope the bug will be fixed soon.

[Daniel van Vugt (vanvugt)]

Thanks kayandus, that valgrind output is intriguing.

It would appear the two leaks are the result of a blockage in compiz event loop. I am guessing that after many hours, maybe your monitor has gone to sleep. And that is causing PrivateGLScreen::waitForVideoSync to hang inside either glXGetVideoSyncSGI or glXWaitVideoSyncSGI. I've never heard of an inactive monitor to cause such a hang, but it does sound plausible.

So while the monitor is inactive, unhandled events queue up in the compiz process. And it would appear even after the monitor wakes, the backlog of 1.5 million events is too large. So compiz appears to hang, or never catches up.

That's my theory, which I think is reasonable. If I'm right, then it's kind of a driver bug and kind of an architectural limitation (lack of threads) in compiz.

Could everyone affected by this kind of leak, particularly kayandus, please provide your graphics driver details? Are you using intel, radeon, nouveau, fglrx or nvidia?

[Daniel van Vugt (vanvugt)]

A workaround to avoid this problem should be:
Run ccsm, and in the OpenGL section disable "Sync To VBlank".

If you do this just before leaving your computer for the night (and assuming my theory is right), then that should avoid the bug.

[Daniel van Vugt (vanvugt)]

In fact, the number 1,500,000 makes sense. That's probably the number of video frames (monitor refresh rate) that your system would normally render in 8 hours.

However, compiz won't try to render a new frame unless some part of the screen is changing. So maybe another simple workaround is to minimize any windows that change lots before leaving your PC for many hours.

[kayandus (bierfiltertje)]

It's an onboard Radeon HD3200 using the radeon driver (with KMS) on a 20" Dell 2007WFP monitor.

Attached dmesg.log, Xorg.0.log and my current ccsm config.

[Daniel van Vugt (vanvugt)]

Thanks. Please try the two workarounds I suggested:

1. Disable "Sync To VBlank"; or
2. Minimize windows that are redrawing lots (like a browser or VM).

You only need to do either before locking/leaving your PC for several hours.

[kayandus (bierfiltertje)]

" So while the monitor is inactive, unhandled events queue up in the compiz process. And it would appear even after the monitor wakes, the backlog of 1.5 million events is too large. So compiz appears to hang, or never catches up. "

When I become active again after a long period of inactivity, and the monitor comes out of its sleep, I always see an old frame, with the clock showing the time of a few minutes after I have left (so probably when the the monitor goes into sleep).

Then either compiz hangs and I need to switch to a vc to force KILL it, or I'll get new frames after a few seconds. Then still a lot of memory is in use, but a compiz --replace solves the job. That doesn't work if compiz hangs, then I won't get new frames and a KILL is required. I think the hanging happens after a longer period of inactivity.

If the problem is indeed the graphic driver putting the monitor into sleep, another workaround should also be to disable sleep in the power management settings and turn off the monitor manually. I will test this next time.

[Daniel van Vugt (vanvugt)]

A third workaround that might work would be to try AMD's fglrx ATI driver. You will lose KMS, but it's probably worth testing for a little while, just to confirm this bug.

[Daniel van Vugt (vanvugt)]

I don't think turning off the monitor will necessarily be a viable workaround. It may in fact trigger this bug also.

[Daniel van Vugt (vanvugt)]

Actually, the primary leak in _XEnq is going to happen to any OpenGL app, not just compiz. And the secondary leak in PrivateScreen::processEvents() is caused completely by the primary leak.

So this is not a compiz-specific issue. It's just that compiz is usually the only OpenGL program that most people are running for long periods.

[Daniel van Vugt (vanvugt)]

It's still not clear which Mesa/DRM/radeon packages this bug should be assigned to. But it's some combination thereof.

[kayandus (bierfiltertje)]

Memory usage of compiz was up to 5% (100MB) again with VSync disabled and not using pc for 3 hours, didn't restart compiz though, don't know if that is necessary when changing that option.

[Daniel van Vugt (vanvugt)]

Sounds like better news if the PC was still usable, but the 100MB is still not great. It is however a much lower rate of leakage per hour than before.

I would be interested to see what valgrind (or massif) reports as the source of the leaks (or bloat) when vsync is disabled.

Please also try the workarounds overnight (for 8 hours or more) so we have a good comparison with the original problem.

[kayandus (bierfiltertje)]

Hmm, oops, 5% is 200MB, or even 400MB if I should include swap, don't know what went wrong there.

This weekend I left my pc idle for more than 24 hours, when I checked via ssh after 20 hours it was up to the usual 12% (the percentage I always see after 8+% hours of being idle. But when I tried to use my pc again, compiz was hanging and using 60% memory. So the 'Sync to VBlank option' really didn't help. This is what I notice all the time, the memory usage is between 1-12%, or the memory usage is 50+% and compiz is hanging, I've never see something in between. (And this is probably why it were the last two lines in the valgrind log with the tremendous memory usage?).

On another note, last night I disabled the 'turn off monitor' setting in gnome AND disconnected the DVI cable from my monitor. Now the memory usage of compiz went from 0.9% (the value compiz starts) to 0.8%, so it actually decreased. And it's 0.6% at the moment! Today I will leave the DVI cable in the monitor and put the monitor into standby manually, let's see what happens.

[Daniel van Vugt (vanvugt)]

The previous valgrind output showed the primary leak originated from waitForVideoSync. If it's still leaking with "Sync To VBlank" disabled then you must have a leak originating somewhere else. Please provide valgrind leak details from with "Sync To VBlank" disabled.

Also, please try the fglrx driver.

[kayandus (bierfiltertje)]

I double checked and turning "Sync to VBlank" off does solve the leak. I will check again tonight with "Sync to VBlank" enabled and every program that redraws minimised or shut down.

[kayandus (bierfiltertje)]

Using the fglrx driver solves the bug.

But I'm still not sure about "Sync to VBlank" solving it, because it seems memory usage does go up again. How can I check if compiz really is not syncing to VBlank?

[Daniel van Vugt (vanvugt)]

I confirmed the bug with radeon today. It leaked some gigabytes fairly quickly. Couldn't reproduce any such leak with intel graphics under the same conditions.

Though, the leak only happened with radeon if I left some program running that continuously redraws while the screen is locked and in power savings mode. Furthermore, the continuous redraws had to come from an app which itself does not sync to vblank, such as KVM or glxgears with vblank_mode=0:
    env vblank_mode=0 glxgears

I don't think you can confirm whether compiz is syncing to vblank, other than to drag a window around quickly and look for tearing. If there is tearing then it's not syncing. If no tearing then it is syncing.

Next step, if possible will be to run it under valgrind's "massif" plugin:
    valgrind --tool=massif --massif-out-file=massif.out compiz ...
and then later:
    ms_print massif.out > massif.txt
I haven't had time to do this but if someone else could then it should help.

[Omer Akram (om26er)]

a new bug report maybe?

[Daniel van Vugt (vanvugt)]

No need for new bug reports. There's only one topic here.

[Otus (jan-varho)]

Just a note that

01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI RV516 [Radeon X1300/X1550 Series]

also triggers this so it is not specific to R600+ drivers.

For these old GPUs FGLRX is not available so there is no workaround...

[Daniel van Vugt (vanvugt)]

This appears to have been fixed in Mesa some time by 12.10. Last time I could reproduce it was in oneiric (Mesa 7.11) but can't reproduce it now in quantal (Mesa 9.0). Given the comments on this bug kind of stopped around April, it could be that the fix was released in Ubuntu 12.04 with Mesa 8.0...

With quantal right now I can confirm that radeon's glXWaitVideoSyncSGI no longer hangs when the screen is asleep. I see it returning regularly. So even if no leak was explicitly fixed, the fact that it doesn't hang at all any more could be working around any potential leak.

Next step is to find out if the bug is fixed in precise too (Mesa 8.0.4).

[Daniel van Vugt (vanvugt)]

Duplicate bug 998708 suggests this was still a problem on precise in Mesa 8.0.2 at least.

[Mr.Bananas (crapmail715)]

Hi folks,

I'm still regularly stricken by the memory leak in the compiz process. I have to kill compiz every couple of days after it bloats to several GB in size. From reading this bug, it's not clear to me what has been fixed, where any possible fix is destined to, how I could pick it up, or whether I'm even following the right bug. Any insight would be greatly appreciated.

I'm running ubuntu 12.04 (religiously apply all official updates) and below are the details about my video card, compiz version, and mesa version:

0> lsmod | grep radeon
radeon 804518 5
ttm 76949 1 radeon
drm_kms_helper 46978 1 radeon
drm 241971 7 radeon,ttm,drm_kms_helper
i2c_algo_bit 13423 1 radeon

0> lspci | grep adeon
01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cedar PRO [Radeon HD 5450]
01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cedar HDMI Audio [Radeon HD 5400/6300 Series]

0> compiz --version
Compiz 0.9.7.12

0> dpkg -l | grep -i mesa
ii libgl1-mesa-dri 8.0.4-0ubuntu0.3 free implementation of the OpenGL API -- DRI modules
ii libgl1-mesa-dri:i386 8.0.4-0ubuntu0.3 free implementation of the OpenGL API -- DRI modules
ii libgl1-mesa-glx 8.0.4-0ubuntu0.3 free implementation of the OpenGL API -- GLX runtime
ii libgl1-mesa-glx:i386 8.0.4-0ubuntu0.3 free implementation of the OpenGL API -- GLX runtime
ii libglapi-mesa 8.0.4-0ubuntu0.3 free implementation of the GL API -- shared library
ii libglapi-mesa:i386 8.0.4-0ubuntu0.3 free implementation of the GL API -- shared library
ii libglu1-mesa 8.0.4-0ubuntu0.3 Mesa OpenGL utility library (GLU)
ii libglu1-mesa:i386 8.0.4-0ubuntu0.3 Mesa OpenGL utility library (GLU)
ii mesa-utils 8.0.1+git20110129+d8f7d6b-0ubuntu2 Miscellaneous Mesa GL utilities

thanks in advance...

[Daniel van Vugt (vanvugt)]

Nominated precise. Though I know it's unlikely to get a fix any time soon, unless someone can identify the change between Mesa 8.0 and 9.0 that (apparently) fixed the bug.