Compiz gets into a state where going fullscreen in apps causes an X11 BadAlloc error

Bug #194638 reported by Tristan Schmelcher on 2008-02-23
This bug affects 2 people
Affects Status Importance Assigned to Milestone
compiz (Ubuntu)
Nominated for Intrepid by adso58

Bug Description

Binary package hint: compiz

I have a up-to-date i686 gutsy install on a Dell XPS M1710 laptop with an NVIDIA GeForce Go 7900 GS (except with some packages installed from hardy, including the nvidia driver).

I am using compiz and I am seeing a situation where it sometimes gets into a state where apps are unable to go into fullscreen mode. Attempting to do so results in an X11 BadAlloc error with the explanation of "insufficient resources for operation". Disabling compiz fixes the problem, after which it can be re-enabled and fullscreen will work with it again. I have 2GB of system RAM, 4GB of swap, and 256MB of video RAM, so I don't think I am actually running out of resources.

I have seen this with Totem, MPlayer, and SMPlayer. In the latter two, it occurs with all output modules that I have tried (which are xv, gl, and gl2). Note that in SMPlayer the X11 error simply causes video to not show up in fullscreen, whereas in Totem and MPlayer it causes the program to exit. I will test other apps and other settings (particularly turning off/on the legacy fullscreen option in compiz) when this bug happens next.

I am attaching .xsession-errors, Xorg.0.log, and the output of lspci -vvnn, since I have noticed that people often ask for that. For the first two, they are taken from after I encountered the problem, turned off compiz, and then turned it back on.

$ cat /etc/lsb-release

$ apt-show-versions | grep compiz
compiz-gnome/gutsy uptodate 1:0.6.2+git20071119-0ubuntu1~gutsy1
python-compizconfig/gutsy uptodate 0.5.2+git20070912-0ubuntu1
libcompizconfig-backend-gconf/gutsy uptodate 0.5.2+git20071010-0ubuntu1
compizconfig-settings-manager/gutsy uptodate 0.5.2+git20070912-0ubuntu1
compiz-plugins/gutsy uptodate 1:0.6.2+git20071119-0ubuntu1~gutsy1
compiz-fusion-plugins-extra/gutsy uptodate 0.6.0+git20071121-0ubuntu1~gutsy1
libcompizconfig0/gutsy uptodate 0.5.2+git20070919-0ubuntu3
compiz-fusion-plugins-main/gutsy uptodate 0.5.2+git20070928-0ubuntu2
compiz-core/gutsy uptodate 1:0.6.2+git20071119-0ubuntu1~gutsy1
compiz/gutsy uptodate 1:0.6.2+git20071119-0ubuntu1~gutsy1

Oh, I should probably also mentions that I am using TwinView.

I've now tried changing the "Legacy Fullscreen Support" and "Unredirect Fullscreen Windows" options while experiencing this problem, but that had no effect. I also checked my memory usage when this happened and it was only at around 50%. Not sure what else to check.

Basilio Kublik (sourcercito) wrote :

Hi Tristan
you report this issue some time ago and hasn't been any activity in a while, i was wondering if this is still an issue for you with the latest upgrades in Hardy Heron.

Thanks in advance

Changed in compiz:
assignee: nobody → sourcercito
importance: Undecided → Low
status: New → Incomplete

Yes, this still occurs today, even with the Compiz/X/NVIDIA packages from Hardy Heron. Attaching logs taken with the Hardy packages and changing back to New.

These logs were taken with me using SMPlayer and the 'xv' output module.

Changed in compiz:
status: Incomplete → New

Anything I can do to help debug this? This is the last bug preventing me from using Compiz. I'm a computer engineer so you can feel free to ask me to use gdb and stuff.

I haven't seen this in ages. Changing to Invalid.

Changed in compiz:
status: New → Invalid

I take it back, this happened again today. Definitely less common than it used to be, though. Note that I am using pure up-to-date Hardy now.

Changed in compiz:
status: Invalid → New
Dara Adib (daradib) wrote :

Perhaps related to Bug #216322?

Seems unlikely to me. The description is very different.

Rui Boon (ruiboon) wrote :

Please create a backtrace and attach it here as a non-compressed text file. Instructions are available at and

Relevant debug package may include xserver-xorg-core-dbgsym, compiz-core-dbgsym. libx11-6-dbgsym, libx11-6-dbg, libc6-dbg

Changed in compiz:
status: New → Incomplete

This bug is not a crash, it is just an error that makes the program exit. How can you get a backtrace for that?

Changed in compiz:
status: Incomplete → New
Rui Boon (ruiboon) wrote :

ahh.. my fault for forgetting to mention setting breakpoint. Could you try to break at exit. Interrupt gdb by ctrl-c and type 'break exit'

Gotcha. I will do this the next time this bug happens. (Could be a while though.)

Happened again today, except this time the bug occurred whenever the video WASN'T in fullscreen mode. I got backtraces for both Totem and MPlayer; they are attached. They don't really say much though, basically just that the program gave a frame to the X server and got back this error.

At the time the bug happened, I was only using 75% RAM and 5% of SWAP. After getting the backtraces, I noticed that I had about a dozen instances of SMPlayer open (probably due to a stuck shortcut key that opened too many). After closing them, the bug went away, even though my RAM usage had barely changed (75% -> 72%). Then I tried opening a dozen instances of SMPlayer again (-> 78% RAM usage) and the bug came back.

Perhaps the problem is that X and/or Compiz reserves video memory for each app that might need it, even when they haven't asked for it? But that the memory for fullscreen mode is shared? Though that wouldn't explain the original situation where the bug happens in fullscreen mode ...

adso58 (imr581) wrote :

I have the same problem. My card is a nVidia 660GT with 128MB. I'm using Ubuntu Hardy and I tried with the Nvidia restricted drivers 169.12, 173.14.12 (envyng) and the latest 177.80 (from the nvidia web).

I can reproduce the problem whenever I want, I think it's related with the amount of texture memory used. I'm using a resolution of 1920x1200 and when I have a few windows opened I can't watch videos in fullscreen, Totem, Mplayer and VLC just die with a BadAlloc error.

If I have a lot of windows opened I can't even watch the video at all, the players close with the same error before the video is shown. But if I minimize all the windows, I can play the video in fullscreen, and when the video is playing I can re-maximize the previously minimized windows and the video will keep playing (sometimes it fails too).

I'm only having this problem with compiz enabled.

Yay, finally some confirmation. :)

Changed in compiz:
status: New → Confirmed
adso58 (imr581) wrote :

I'm having the same problem using Sabayon 32/64 bit.

FYI, I hadn't seen this for a while, but then today it happened a lot when I was trying to play a 1080p video. So it's probably more likely with larger video resolutions. Which of course would make sense.

Daniel Hobe (hobe) wrote :

I've seen this as well. It seems to be related to the number of windows I have open at the time. If I close a few of them the problem goes away.

This is on Intrepid on a 8Gig 64bit Intel box with a GeForce 7600 GT (256MB Memory) and 2 1920x1200 monitors running in twinview.

Is there a good way to see what Compiz is allocating for textures etc?

Still occurs in Jaunty Jackalope.

dap (dap-rock) wrote :

I'm using Fedora 11, this is my problem too on my GeForce 7600 GS (256Mb of RAM). It's not Ubuntu specific.

Current versions:
Nvidia driver 185.18.14 / 180.51
Compiz 0.7.8

My previous Fedora 9 _working well_ on the same hardware, there's:
Xorg 1.5.2
Nvidia driver 177.82
Compiz 0.7.6

I have to open many windows to trigger the bug (~10-15 firefox). If I switch to another VT and back / restart compiz with --replace / kill some windows, then I can play videos with XV for a while (5-10 minutes). ALT+TAB in compiz is much slower when the XV doesn't work.

Daniel, I cannot found a method to check used texture memory, googled for hours for nothing.

It also happens to me. Jaunty with a Nividia 8400GS video card 512MB, 4 gig of system ram and a 1670X1050 resolution. Only does it when I run emerald.

Hmm, I'm also using Emerald. Maybe this only occurs when using Compiz _and_ Emerald?

Has anyone seen this when not using Emerald?

dap (dap-rock) wrote :

I'm using gtk-window-decorator..

NVnews topic:

Nikola Kotur (kotnik) wrote :

It happens to me as well, Ubuntu 9.10, nvidia drivers 185.18.36-0ubuntu9.

Mplayer issue can be easily resolved with setting video output to x11, but tvtime application can't change it's output and it doesn't work. Also, screensavers are extremely slow with 100% CPU usage.

It makes compiz unusable, so I had to turn it off.

David Monro (davidm-ub) wrote :

I see this on my AGP 7600GT (256M); as well as stopping video going full screen, it makes mythtv frontend with the opengl drawing engine unbelievably slow and cpu hungry (presumably it falls back to software 3d rendering or something). Would this be caused by texture memory fragmentation or something? (Does video and/or whatever mythtv does need a large _contiguous_ chunk of it? Or does that not make sense - does texture memory get virtualised by an mmu (gart?) or something?).

Sadly I've had to stop using compiz as a result.

Nikola Kotur (kotnik) wrote :

I found solution that works for me.

I deleted whole ~/.config/compiz and ~/.gconf/apps/compiz directories, and compiz is behaving properly afterwards - no more xv issues.


Thank you for taking the time to report this bug and helping to make Ubuntu better. We are sorry that we do not always have the capacity to look at all reported bugs in a timely manner. There have been many changes in Ubuntu since that time you reported the bug and your problem may have been fixed with some of the updates. It would help us a lot if you could test it on a currently supported Ubuntu version. When you test it and it is still an issue, kindly upload the updated logs by running apport-collect <bug #> and any other logs that are relevant for this particular issue.

Changed in compiz (Ubuntu):
status: Confirmed → Incomplete
assignee: Basilio Kublik (sourcercito) → nobody

I no longer have the computer that experienced this issue, and my new one has tons more video RAM so I can't determine whether this issue still exists or not. I'll leave it as Incomplete in case someone else is still experiencing it and can provide the info.

Launchpad Janitor (janitor) wrote :

[Expired for compiz (Ubuntu) because there has been no activity for 60 days.]

Changed in compiz (Ubuntu):
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers