Comment 8 for bug 810214

So I reproduced the bug (on Debian/Iceweasel anyway) and stepped through the Lua code. I cannot see anything wrong: Lua calls fopen(filename, "r") which returns a valid FILE pointer, then it calls getc(fp) and then freopen(filename, "rb", fp). freopen() crashes while trying to close the fp. This crashes under Firefox, but not under VLC self.

Interestingly, in VLC, the FILE pointer has more data, and is directed to a different close function inside glibc:

_IO_new_file_close_it (fp=0x814e4b8) at fileops.c:165
165 fileops.c: Aucun fichier ou dossier de ce type.
        in fileops.c
(gdb) p *fp
$2 = {_flags = -72539000, _IO_read_ptr = 0xb7fcc001 "LuaQ",
  _IO_read_end = 0xb7fcc9b2 "", _IO_read_base = 0xb7fcc000 "\033LuaQ",
  _IO_write_base = 0xb7fcc000 "\033LuaQ",
  _IO_write_ptr = 0xb7fcc000 "\033LuaQ",
  _IO_write_end = 0xb7fcc000 "\033LuaQ", _IO_buf_base = 0xb7fcc000 "\033LuaQ",
  _IO_buf_end = 0xb7fcd000 "\177ELF\001\001\001", _IO_save_base = 0x0,
  _IO_backup_base = 0x0, _IO_save_end = 0x0, _markers = 0x0,
  _chain = 0xb7e8e560, _fileno = 10, _flags2 = 0, _old_offset = 0,
  _cur_column = 0, _vtable_offset = 0 '\000', _shortbuf = "",
  _lock = 0x814e550, _offset = -1, _codecvt = 0x0, _wide_data = 0x814e55c,
  _freeres_list = 0x0, _freeres_buf = 0x0, _freeres_size = 0, _mode = -1,
  _unused2 = '\000' <repeats 39 times>}
(gdb) bt
#0 _IO_new_file_close_it (fp=0x814e4b8) at fileops.c:165
#1 0xb7daba1d in freopen (
    filename=0x8150830 "/usr/lib/vlc/lua/playlist/anevia_streams.luac",
    mode=0xb6b599a8 "rb", fp=0x814e4b8) at freopen.c:69
#2 0xb6b4e115 in luaL_loadfile (L=0x81487e8,
    filename=0x8150830 "/usr/lib/vlc/lua/playlist/anevia_streams.luac")
    at lauxlib.c:574
#3 0xb6b6ccbd in probe_luascript ()
   from /usr/lib/vlc/plugins/misc/liblua_plugin.so
#4 0xb6b6f4b7 in vlclua_scripts_batch_execute ()
   from /usr/lib/vlc/plugins/misc/liblua_plugin.so
#5 0xb6b6d314 in Import_LuaPlaylist ()
   from /usr/lib/vlc/plugins/misc/liblua_plugin.so
#6 0xb7f4c260 in module_need () from /usr/lib/libvlccore.so.4
#7 0xb7ef6ce3 in __demux_New () from /usr/lib/libvlccore.so.4
#8 0xb7f05a6e in InputSourceInit () from /usr/lib/libvlccore.so.4
#9 0xb7f06df9 in Init () from /usr/lib/libvlccore.so.4
#10 0xb7f0b20c in Run () from /usr/lib/libvlccore.so.4
#11 0xb7f51a19 in thread_entry () from /usr/lib/libvlccore.so.4
#12 0xb7e9b944 in start_thread (arg=0xb7828b70) at pthread_create.c:304
#13 0xb7e18d9e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Backtrace stopped: Not enough registers or memory available to unwind further
(gdb) p _IO_stdin_used
$3 = 131073

but Firefox (actually Iceweasel):

Program received signal SIGSEGV, Segmentation fault.
0xb76c0a14 in _IO_old_file_close_it (fp=0xa92c7040) at oldfileops.c:158
158 oldfileops.c: Aucun fichier ou dossier de ce type.
        in oldfileops.c
(gdb) p *fp
$1 = {_flags = -72539000, _IO_read_ptr = 0xb2d05001 "LuaQ",
  _IO_read_end = 0xb2d059b2 "", _IO_read_base = 0xb2d05000 "\033LuaQ",
  _IO_write_base = 0xb2d05000 "\033LuaQ",
  _IO_write_ptr = 0xb2d05000 "\033LuaQ",
  _IO_write_end = 0xb2d05000 "\033LuaQ", _IO_buf_base = 0xb2d05000 "\033LuaQ",
  _IO_buf_end = 0xb2d06000 "\336\022\004\225", _IO_save_base = 0x0,
  _IO_backup_base = 0x0, _IO_save_end = 0x0, _markers = 0x0,
  _chain = 0xb76fa920, _fileno = 56, _flags2 = 0, _old_offset = -1492835812,
  _cur_column = 0, _vtable_offset = 0 '\000', _shortbuf = "\247",
  _lock = 0xa92c70d8}
(gdb) bt
#0 0xb76c0a14 in _IO_old_file_close_it (fp=0xa92c7040) at oldfileops.c:158
#1 0xb7617894 in freopen (
    filename=0xa97489a0 "/usr/lib/vlc/lua/playlist/anevia_streams.luac",
    mode=0xa6e939a8 "rb", fp=0xa92c7040) at freopen.c:62
#2 0xa6e88115 in luaL_loadfile (L=0xac6de060,
    filename=0xa97489a0 "/usr/lib/vlc/lua/playlist/anevia_streams.luac")
    at lauxlib.c:574
#3 0xa6ebccbd in probe_luascript ()
   from /usr/lib/vlc/plugins/misc/liblua_plugin.so
#4 0xa6ebf4b7 in vlclua_scripts_batch_execute ()
   from /usr/lib/vlc/plugins/misc/liblua_plugin.so
#5 0xa6ebd314 in Import_LuaPlaylist ()
   from /usr/lib/vlc/plugins/misc/liblua_plugin.so
#6 0xa899c260 in module_need () from /usr/lib/libvlccore.so.4
#7 0xa8946ce3 in __demux_New () from /usr/lib/libvlccore.so.4
#8 0xa8955a6e in InputSourceInit () from /usr/lib/libvlccore.so.4
#9 0xa8956df9 in Init () from /usr/lib/libvlccore.so.4
#10 0xa895b20c in Run () from /usr/lib/libvlccore.so.4
#11 0xa89a1a19 in thread_entry () from /usr/lib/libvlccore.so.4
#12 0xb7703944 in start_thread (arg=0xa887fb70) at pthread_create.c:304
#13 0xb7684d9e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Backtrace stopped: Not enough registers or memory available to unwind further
(gdb) p _IO_stdin_used
Address of symbol "_IO_stdin_used" is unknown.

And voilà! For whatever reasons, the Firefox binary fails to define _IO_stdin_used even though it's linked against a "recent" (e)glibc ABI, causing freopen() to wrongly enter the antique GLibc 2.1 libio support.

There are several ways to fix this. The correct one consists of fixing the Firefox linking flags or whatever in the Firefox build system causes _IO_stdin_used not to be defined in the binary. Alternatively, you could work around it by dropping support for GLibc 2.0-2.1 ABI compatibility in (e)glibc. Or you could hack all freopen()ing libraries to define the symbol (probably not a good nor practical idea). That's not my call.

Reassigning to firefox and glibc.