I'm also having the same issue (with compiz enabled) when maximizing the video player.
I'm on 9.10, NVIDIA Driver 190.32, Quadro NVS 110M (256 MB with TurboCache)

My xorg.conf is:

Section "Monitor"
	Identifier	"Configured Monitor"
EndSection

Section "Screen"
	Identifier	"Default Screen"
	Monitor		"Configured Monitor"
	Device		"Configured Video Device"
	DefaultDepth	24
EndSection

Section "Module"
	Load	"glx"
EndSection

Section "Device"
	Identifier	"Configured Video Device"
	Driver	"nvidia"
	Option	"NoLogo"	"True"
EndSection

Note that I have
COMPIZ_OPTIONS="$COMPIZ_OPTIONS --indirect-rendering "
in /usr/bin/compiz to avoid the full-screen black windows problem

for example:
attila@blackbird:~/Desktop$ vlc urka.avi 
VLC media player 1.0.2 Goldeneye
[0x16a3888] main libvlc: Esecuzione di vlc con l'interfaccia predefinita. Usa 'cvlc' per utilizzare vlc senza interfaccia.
QPainter::begin: Paint device returned engine == 0, type: 1
QPainter::begin: Paint device returned engine == 0, type: 1
[0x359a588] pulse audio output: No. of Audio Channels: 2
[????????] x11 video output error: X11 request 133.19 failed with error code 11:
 BadAlloc (insufficient resources for operation)
X Error of failed request:  BadAlloc (insufficient resources for operation)
  Major opcode of failed request:  133 (XVideo)
  Minor opcode of failed request:  19 ()
  Serial number of failed request:  102
  Current serial number in output stream:  103

attila@blackbird:~/Desktop$ totem urka.avi 
The program 'totem' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadAlloc (insufficient resources for operation)'.
  (Details: serial 353 error_code 11 request_code 133 minor_code 19)
  (Note to programmers: normally, X errors are reported asynchronously;
   that is, you will receive the error a while after causing it.
   To debug your program, run it with the --sync command line
   option to change this behavior. You can then get a meaningful
   backtrace from your debugger if you break on the gdk_x_error() function.)

attila@blackbird:~/Desktop$ mplayer urka.avi 
MPlayer UNKNOWN-4.4.1 (C) 2000-2009 MPlayer Team
mplayer: could not connect to socket
mplayer: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing urka.avi.
AVI file format detected.
[aviheader] Video stream found, -vid 0
[aviheader] Audio stream found, -aid 1
VIDEO:  [XVID]  720x576  12bpp  25.000 fps  274.0 kbps (33.5 kbyte/s)
Clip info:
 Software: MEncoder UNKNOWN-4.4.1
open: No such file or directory
[MGA] Couldn't open: /dev/mga_vid
open: No such file or directory
[MGA] Couldn't open: /dev/mga_vid
[VO_TDFXFB] Can't open /dev/fb0: No such file or directory.
[VO_3DFX] Unable to open /dev/3dfx.
==========================================================================
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
Selected video codec: [ffodivx] vfm: ffmpeg (FFmpeg MPEG-4)
==========================================================================
==========================================================================
Opening audio decoder: [mp3lib] MPEG layer-2, layer-3
AUDIO: 44100 Hz, 2 ch, s16le, 32.0 kbit/2.27% (ratio: 4000->176400)
Selected audio codec: [mp3] afm: mp3lib (mp3lib MPEG layer-2, layer-3)
==========================================================================
[pulse] working around probably broken pause functionality,
        see http://www.pulseaudio.org/ticket/440
AO: [pulse] 44100Hz 2ch s16le (2 bytes per sample)
Starting playback...
VDec: vo config request - 720 x 576 (preferred colorspace: Planar YV12)
VDec: using Planar YV12 as output csp (no 0)
Movie-Aspect is 1.78:1 - prescaling to correct movie aspect.
VO: [xv] 720x576 => 1024x576 Planar YV12 
X11 error: BadAlloc (insufficient resources for operation)0.5% 0 0 
X11 error: BadAlloc (insufficient resources for operation)0.5% 0 0 
X11 error: BadAlloc (insufficient resources for operation)0.6% 1 0 
X11 error: BadAlloc (insufficient resources for operation)0.5% 1 0 
X11 error: BadAlloc (insufficient resources for operation)
X11 error: BadAlloc (insufficient resources for operation)
X11 error: BadAlloc (insufficient resources for operation)
X11 error: BadAlloc (insufficient resources for operation)0.5% 1 0 
X11 error: BadAlloc (insufficient resources for operation)0.5% 1 0 
X11 error: BadAlloc (insufficient resources for operation)0.5% 1 0 
X11 error: BadAlloc (insufficient resources for operation)0.5% 1 0