Firefox crashes when a plugin calls freopen()

[penalvch]

1) lsb_release -rd
Description: Ubuntu 11.04
Release: 11.04

2) apt-cache policy mozilla-plugin-vlc
mozilla-plugin-vlc:
  Installed: 1.1.9-1ubuntu1.2
  Candidate: 1.1.9-1ubuntu1.2
  Version table:
 *** 1.1.9-1ubuntu1.2 0
        500 http://us.archive.ubuntu.com/ubuntu/ natty-updates/universe i386 Packages
        100 /var/lib/dpkg/status
     1.1.9-1ubuntu1.1 0
        500 http://security.ubuntu.com/ubuntu/ natty-security/universe i386 Packages
     1.1.9-1ubuntu1 0
        500 http://us.archive.ubuntu.com/ubuntu/ natty/universe i386 Packages

apt-cache policy firefox
firefox:
  Installed: 5.0+build1+nobinonly-0ubuntu0.11.04.2
  Candidate: 5.0+build1+nobinonly-0ubuntu0.11.04.2
  Version table:
 *** 5.0+build1+nobinonly-0ubuntu0.11.04.2 0
        500 http://us.archive.ubuntu.com/ubuntu/ natty-updates/main i386 Packages
        500 http://security.ubuntu.com/ubuntu/ natty-security/main i386 Packages
        100 /var/lib/dpkg/status
     4.0+nobinonly-0ubuntu3 0
        500 http://us.archive.ubuntu.com/ubuntu/ natty/main i386 Packages

3) What is expected to happen in Firefox is when one visits the website:

http://media.thrashermagazine.com/FiringLine_PatPasquale.mp4

mozilla-plugin-vlc plays the video with no problem.

4) What happens instead is Firefox crashes, with a prompt to report this to Mozilla instead of an apport to Launchpad. The about:crashes link may be found at:
https://crash-stats.mozilla.com/report/index/c6f47bc4-e936-469d-bcf2-b13352110713

WORKAROUND: Open in VLC via the Terminal:

wget http://media.thrashermagazine.com/FiringLine_PatPasquale.mp4 && vlc FiringLine_PatPasquale.mp4

the video plays correctly.

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: mozilla-plugin-vlc 1.1.9-1ubuntu1.2
ProcVersionSignature: Ubuntu 2.6.38-10.46-generic 2.6.38.7
Uname: Linux 2.6.38-10-generic i686
NonfreeKernelModules: fglrx
Architecture: i386
Date: Wed Jul 13 19:35:14 2011
EcryptfsInUse: Yes
InstallationMedia: Ubuntu 10.10 "Maverick Meerkat" - Release i386 (20101007)
ProcEnviron:
 LANGUAGE=en_US:en
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: vlc
UpgradeStatus: Upgraded to natty on 2011-05-14 (60 days ago)

[Rémi Denis-Courmont (rdenis)]

It's crashing while trying to close a FILE pointer from within glibc freopen() from within the lua5.1 file loader. This smells quite fishy. Can you post a valgrind trace?

[Rémi Denis-Courmont (rdenis)]

Uh? Did you actually reproduce the problem under valgrind? I can't even see VLC messages...

[penalvch (penalvch)]

Rémi Denis-Courmont:

> Uh? Did you actually reproduce the problem under valgrind? I can't even see VLC messages...

Yes. I followed the recommendations from https://wiki.ubuntu.com/Valgrind and performed the following to produce the log at the Terminal:

G_SLICE=always-malloc G_DEBUG=gc-friendly valgrind -v --tool=memcheck --leak-check=full --num-callers=40 --log-file=valgrind.log firefox

then pasted the above mentioned link into the address bar, hit enter, crash, click Quit Firefox button in report to Mozilla window, then saved valgrind log to this bug.

 apt-cache policy valgrind
valgrind:
  Installed: 1:3.6.1-0ubuntu1
  Candidate: 1:3.6.1-0ubuntu1
  Version table:
 *** 1:3.6.1-0ubuntu1 0
        500 http://us.archive.ubuntu.com/ubuntu/ natty/main i386 Packages
        100 /var/lib/dpkg/status

[Rémi Denis-Courmont (rdenis)]

Detailled valgrind trace is available on the Debian bug report. It did not help much though :-(

[Rémi Denis-Courmont (rdenis)]

So I reproduced the bug (on Debian/Iceweasel anyway) and stepped through the Lua code. I cannot see anything wrong: Lua calls fopen(filename, "r") which returns a valid FILE pointer, then it calls getc(fp) and then freopen(filename, "rb", fp). freopen() crashes while trying to close the fp. This crashes under Firefox, but not under VLC self.

Interestingly, in VLC, the FILE pointer has more data, and is directed to a different close function inside glibc:

_IO_new_file_close_it (fp=0x814e4b8) at fileops.c:165
165 fileops.c: Aucun fichier ou dossier de ce type.
        in fileops.c
(gdb) p *fp
$2 = {_flags = -72539000, _IO_read_ptr = 0xb7fcc001 "LuaQ",
  _IO_read_end = 0xb7fcc9b2 "", _IO_read_base = 0xb7fcc000 "\033LuaQ",
  _IO_write_base = 0xb7fcc000 "\033LuaQ",
  _IO_write_ptr = 0xb7fcc000 "\033LuaQ",
  _IO_write_end = 0xb7fcc000 "\033LuaQ", _IO_buf_base = 0xb7fcc000 "\033LuaQ",
  _IO_buf_end = 0xb7fcd000 "\177ELF\001\001\001", _IO_save_base = 0x0,
  _IO_backup_base = 0x0, _IO_save_end = 0x0, _markers = 0x0,
  _chain = 0xb7e8e560, _fileno = 10, _flags2 = 0, _old_offset = 0,
  _cur_column = 0, _vtable_offset = 0 '\000', _shortbuf = "",
  _lock = 0x814e550, _offset = -1, _codecvt = 0x0, _wide_data = 0x814e55c,
  _freeres_list = 0x0, _freeres_buf = 0x0, _freeres_size = 0, _mode = -1,
  _unused2 = '\000' <repeats 39 times>}
(gdb) bt
#0 _IO_new_file_close_it (fp=0x814e4b8) at fileops.c:165
#1 0xb7daba1d in freopen (
    filename=0x8150830 "/usr/lib/vlc/lua/playlist/anevia_streams.luac",
    mode=0xb6b599a8 "rb", fp=0x814e4b8) at freopen.c:69
#2 0xb6b4e115 in luaL_loadfile (L=0x81487e8,
    filename=0x8150830 "/usr/lib/vlc/lua/playlist/anevia_streams.luac")
    at lauxlib.c:574
#3 0xb6b6ccbd in probe_luascript ()
   from /usr/lib/vlc/plugins/misc/liblua_plugin.so
#4 0xb6b6f4b7 in vlclua_scripts_batch_execute ()
   from /usr/lib/vlc/plugins/misc/liblua_plugin.so
#5 0xb6b6d314 in Import_LuaPlaylist ()
   from /usr/lib/vlc/plugins/misc/liblua_plugin.so
#6 0xb7f4c260 in module_need () from /usr/lib/libvlccore.so.4
#7 0xb7ef6ce3 in __demux_New () from /usr/lib/libvlccore.so.4
#8 0xb7f05a6e in InputSourceInit () from /usr/lib/libvlccore.so.4
#9 0xb7f06df9 in Init () from /usr/lib/libvlccore.so.4
#10 0xb7f0b20c in Run () from /usr/lib/libvlccore.so.4
#11 0xb7f51a19 in thread_entry () from /usr/lib/libvlccore.so.4
#12 0xb7e9b944 in start_thread (arg=0xb7828b70) at pthread_create.c:304
#13 0xb7e18d9e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Backtrace stopped: Not enough registers or memory available to unwind further
(gdb) p _IO_stdin_used
$3 = 131073

but Firefox (actually Iceweasel):

Program received signal SIGSEGV, Segmentation fault.
0xb76c0a14 in _IO_old_file_close_it (fp=0xa92c7040) at oldfileops.c:158
158 oldfileops.c: Aucun fichier ou dossier de ce type.
        in oldfileops.c
(gdb) p *fp
$1 = {_flags = -72539000, _IO_read_ptr = 0xb2d05001 "LuaQ",
  _IO_read_end = 0xb2d059b2 "", _IO_read_base = 0xb2d05000 "\033LuaQ",
  _IO_write_base = 0xb2d05000 "\033LuaQ",
  _IO_write_ptr = 0xb2d05000 "\033LuaQ",
  _IO_write_end = 0xb2d05000 "\033LuaQ", _IO_buf_base = 0xb2d05000 "\033LuaQ",
  _IO_buf_end = 0...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mh+mozilla (mh+mozilla)]

I got the craziest crash happening on sparc linux, when running xpcshell with jemalloc enabled, it crashes in "setbuf(stdout, 0);" (which is a libc function).

It turns out this is due to the fact that the _IO_stdin_used symbol is not exported, because of the version script used to hide most symbols from programs. No idea why this only crashes on sparc, though.

[Mh+mozilla (mh+mozilla)]

Created attachment 571903
Export the _IO_stdin_used symbol

[Mh+mozilla (mh+mozilla)]

http://hg.mozilla.org/integration/mozilla-inbound/rev/109b2416d243

[Bmo-edmorley (bmo-edmorley)]

https://hg.mozilla.org/mozilla-central/rev/109b2416d243

[Reuben Morais (reuben-morais)]

This also affected x86 (https://crash-stats.mozilla.com/report/index/bp-696816ab-e67c-4beb-b60e-defec2111214 / https://launchpad.net/bugs/810214), should we ship this earlier than Firefox 11?

[penalvch (penalvch)]

This is fixed in Precise.

lsb_release -rd
Description: Ubuntu precise (development branch)
Release: 12.04

apt-cache policy firefox
firefox:
  Installed: 11.0+build1-0ubuntu1
  Candidate: 11.0+build1-0ubuntu1
  Version table:
 *** 11.0+build1-0ubuntu1 0
        500 http://us.archive.ubuntu.com/ubuntu/ precise/main i386 Packages
        100 /var/lib/dpkg/status

apt-cache policy mozilla-plugin-vlc
mozilla-plugin-vlc:
  Installed: 2.0.0-1
  Candidate: 2.0.0-1
  Version table:
 *** 2.0.0-1 0
        500 http://us.archive.ubuntu.com/ubuntu/ precise/universe i386 Packages
        100 /var/lib/dpkg/status