Opening vlc with non-existing file causes Xorg to use massive amounts of memory

[pablomme]

Steps to reproduce (this is on up-to-date oneiric):
- Install http://apt.ubuntu.com/p/vlc
- Open gnome-terminal
- Type "vlc i-do-not-exist.avi" (where "i-do-not-exist.avi" should indeed not exist)

The system will become completely unresponsive in seconds. Switching to a VT (which will take a while with all the swapping) and running "top" (and press '>' to sort by memory usage) will show that Xorg is using (in my case) 3+ GiB of memory. Issuing "killall vlc" and switching back to VT 7 will make the system responsive again after a little while, and Xorg will return to more normal memory usage levels.

I think that Xorg's ability to stop using the extra memory implies that this is not a memory leak in Xorg, but that vlc is indeed requesting Xorg to allocate all that memory.

In case it helps, the output from vlc in gnome-terminal is:

$ vlc i-do-not-exist.avi
VLC media player 1.1.11 The Luggage (revision exported)
Blocked: call to unsetenv("DBUS_ACTIVATION_ADDRESS")
Blocked: call to unsetenv("DBUS_ACTIVATION_BUS_TYPE")
[0x788120] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
Blocked: call to setlocale(6, "")
Blocked: call to putenv("LIBOVERLAY_SCROLLBAR=0")
Blocked: call to setlocale(6, "")

(process:2013): Gtk-WARNING **: Locale not supported by C library.
 Using the fallback 'C' locale.
Blocked: call to setlocale(6, "")
libdvdnav: Using dvdnav version 4.1.3
libdvdread: Encrypted DVD support unavailable.
************************************************
** **
** No css library available. See **
** /usr/share/doc/libdvdread4/README.Debian **
** for more information. **
** **
************************************************
libdvdread: Can't stat /home/pablo/i-do-not-exist.avi
No such file or directory
libdvdnav: vm: failed to open/read the DVD
[0xcbcac0] filesystem access error: cannot open file /home/pablo/i-do-not-exist.avi (No such file or directory)
[0xcad4e0] main input error: open of `file:///home/pablo/i-do-not-exist.avi' failed: (null)

(vlc:2013): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion `GTK_IS_WIDGET (widget)' failed

(vlc:2013): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion `GTK_IS_WIDGET (widget)' failed

(vlc:2013): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion `GTK_IS_WIDGET (widget)' failed

(vlc:2013): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion `GTK_IS_WIDGET (widget)' failed

(vlc:2013): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion `GTK_IS_WIDGET (widget)' failed

(vlc:2013): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion `GTK_IS_WIDGET (widget)' failed

(vlc:2013): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion `GTK_IS_WIDGET (widget)' failed

(vlc:2013): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion `GTK_IS_WIDGET (widget)' failed

(vlc:2013): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion `GTK_IS_WIDGET (widget)' failed

(vlc:2013): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion `GTK_IS_WIDGET (widget)' failed

(vlc:2013): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion `GTK_IS_WIDGET (widget)' failed

(vlc:2013): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion `GTK_IS_WIDGET (widget)' failed

(vlc:2013): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion `GTK_IS_WIDGET (widget)' failed

(vlc:2013): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion `GTK_IS_WIDGET (widget)' failed

(vlc:2013): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion `GTK_IS_WIDGET (widget)' failed

(vlc:2013): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion `GTK_IS_WIDGET (widget)' failed

(vlc:2013): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion `GTK_IS_WIDGET (widget)' failed

(vlc:2013): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion `GTK_IS_WIDGET (widget)' failed
[0x2b15e8001360] signals interface error: Caught Terminated signal, exiting...
$

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: vlc 1.1.11-2build2
ProcVersionSignature: Ubuntu 3.0.0-11.18-generic 3.0.4
Uname: Linux 3.0.0-11-generic x86_64
ApportVersion: 1.23-0ubuntu1
Architecture: amd64
Date: Sat Sep 24 21:07:20 2011
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Release amd64 (20110427.1)
ProcEnviron:
 LANGUAGE=en_GB:en
 PATH=(custom, user)
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
SourcePackage: vlc
UpgradeStatus: Upgraded to oneiric on 2011-09-24 (0 days ago)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.