luaL_loadfile segfault in _IO_file_underflow

[Mark1-x]

This is a bit of a niche bug, that only happens on a particular combination of OS, CPU architecture, and LUA package, and admittedly is a pretty unusual use-case.

The original error report relates to my vlcj bindings for the vlc media player - but I have been able to distil the problem down to a very specific and simple test case.

My test case embeds the execution of a LUA script inside Java code.

I have tested this using the Oracle provided Java6 and Java7 JVMs, and the OpenJDK Java6 and Java7 JVMs. All with the same results. There is no observed problem with LUA outside of a JVM.

My test case works just fine on 64-bit Ubuntu/Kubuntu and Lubuntu.

My unchanged test case fails on 32-bit Ubuntu/Kubuntu and Lubuntu.

The nature of the failure is a fatal JVM crash caused by a segfault triggered by some lua code. This is repeatable 100% of the time.

An example JVM crash dump contains this:

# SIGSEGV (0xb) at pc=0xb767c688, pid=4407, tid=2080172864
#
# JRE version: 7.0_05-b05
# Java VM: Java HotSpot(TM) Server VM (23.1-b03 mixed mode linux-x86 )
# Problematic frame:
# C [libc.so.6+0x128688] _IO_file_underflow+0x68

The backtrace from gdb contains this:

#0 0x0028f688 in _IO_file_underflow () from /lib/i386-linux-gnu/libc.so.6
#1 0x001d83a9 in _IO_default_uflow () from /lib/i386-linux-gnu/libc.so.6
#2 0x001d81c0 in __uflow () from /lib/i386-linux-gnu/libc.so.6
#3 0x001ce6d2 in getc () from /lib/i386-linux-gnu/libc.so.6
#4 0x02c8abc4 in luaL_loadfile () from /usr/lib/i386-linux-gnu/liblua5.1.so.0

The piece of code that triggers this bug is in lauxlib.c in the luaL_loadfile method:

while ((c = getc(lf.f)) != EOF && c != LUA_SIGNATURE[0]) ; <--- Crash failure triggered here after freopen

This bug is only triggered when the LUA script being opened has been compiled by luac - if I run the same test with a source script rather than a compiled script the fatal crash is not triggered. I think the lua code essentially opens the script file, sniffs for a magic number to check if it is a binary script, and if so reopens the file. I think it is this act of reopening the file that ultimately triggers the bug.

There is a long thread on github with my entire history of investigating this issue: https://github.com/caprica/vlcj/issues/62

I am reporting this as an Ubuntu bug rather than a JVM or lua bug because if I remove the lua package provided by Ubuntu and instead build lua myself using the vanilla untouched 5.1 source code from lua.org, my test case does in fact work on 32-bit Ubuntu.

cat /etc/issue
Ubuntu 12.04.2 LTS

uname -a
Linux Ubuntu32 3.2.0-38-generic #61-Ubuntu SMP Tue Feb 19 12:20:02 UTC 2013 i686 i686 i386 GNU/Linux

lsb_release -rd
Description: Ubuntu 12.04.2 LTS
Release: 12.04

apt-cache policy liblua5.1-0-dev
liblua5.1-0-dev:
  Installed: 5.1.4-12ubuntu1
  Candidate: 5.1.4-12ubuntu1
  Version table:
 *** 5.1.4-12ubuntu1 0
        500 http://gb.archive.ubuntu.com/ubuntu/ precise/main i386 Packages
        100 /var/lib/dpkg/status

[Mark1-x (mark1-x)]

My initial report was not entirely accurate.

If I use a Java6 JVM I do *not* see the problem, even on 32-bit Ubuntu.

So there is some interplay of Java7, LUA and 32-bit Ubuntu that triggers the bug. I guess it is possible that this is a JVM bug, but the fact remains that if I use a vanilla lua build rather than the Ubuntu lua build my problem goes away, even on Java7.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Alexey Brodkin (alexey-brodkin)]

I also may confirm this issue with lua.
I faced it trying to play a video from public web-cam.
===================
$ gdb --args vlc http://83.64.164.6/mjpg/video.mjpg
GNU gdb (GDB) 7.5-ubuntu
Reading symbols from /usr/bin/vlc...Reading symbols from /usr/lib/debug/usr/bin/vlc...done.
done.
(gdb) run
Starting program: /usr/bin/vlc http://83.64.164.6/mjpg/video.mjpg
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
VLC media player 2.0.5 Twoflower (revision 2.0.5-0-g1661b7d)

Program received signal SIGSEGV, Segmentation fault.
0xb7b4425d in __GI___uflow (fp=0x80dc9c0) at genops.c:394
394 genops.c: No such file or directory.
(gdb) bt
#0 0xb7b4425d in __GI___uflow (fp=0x80dc9c0) at genops.c:394
#1 0xb7b3a772 in _IO_getc (fp=0x80dc9c0) at getc.c:41
#2 0xb4e4cd8c in luaL_loadfile (L=L@entry=0x81905a0,
    filename=filename@entry=0x81904e0 "/usr/lib/vlc/lua/meta/reader/filename.luac") at lauxlib.c:577
#3 0xb4e8c055 in run (p_this=p_this@entry=0x8188cc0,
    psz_filename=psz_filename@entry=0x81904e0 "/usr/lib/vlc/lua/meta/reader/filename.luac",
    L=L@entry=0x81905a0, luafunction=luafunction@entry=0xb4ea3f61 "read_meta") at meta.c:100
#4 0xb4e8c2a0 in read_meta (p_this=p_this@entry=0x8188cc0,
    psz_filename=0x81904e0 "/usr/lib/vlc/lua/meta/reader/filename.luac",
    user_data=user_data@entry=0x81c8dd8) at meta.c:192
#5 0xb4e8f689 in vlclua_scripts_batch_execute (p_this=p_this@entry=0x8188cc0,
    luadirname=luadirname@entry=0xb4ea3fa6 "meta/reader", func=func@entry=0xb4e8c250 <read_meta>,
    user_data=0x81c8dd8) at vlc.c:317
#6 0xb4e8c506 in ReadMeta (p_this=0x8188cc0) at meta.c:227
#7 0xb7d2c6a0 in generic_start (func=0xb4e8c4d0 <ReadMeta>, ap=0xbfffe2c8 "\350\342\377\277",
    ap@entry=0xbfffe2c4 "\300\214\030\b\350\342\377\277") at modules/modules.c:422
#8 0xb7d2cde3 in vlc_module_load (p_this=p_this@entry=0x8188cc0,
    psz_capability=psz_capability@entry=0xb7d7067f "meta reader", psz_name=<optimized out>,
    psz_name@entry=0x0, b_strict=b_strict@entry=false, probe=probe@entry=0xb7d2c690 <generic_start>)
    at modules/modules.c:347
#9 0xb7d2d302 in module_need (obj=obj@entry=0x8188cc0, cap=cap@entry=0xb7d7067f "meta reader",
    name=name@entry=0x0, strict=strict@entry=false) at modules/modules.c:437
#10 0xb7ce679b in InputSourceMeta (p_input=p_input@entry=0x80d7188, p_meta=p_meta@entry=0x818c8d0,
    p_source=<optimized out>, p_source=<optimized out>) at input/input.c:2741
#11 0xb7ce9b5d in Init (p_input=p_input@entry=0x80d7188) at input/input.c:1315
#12 0xb7cedd5b in input_Read (p_parent=p_parent@entry=0x818af40, p_item=p_item@entry=0x81c8dd8)
    at input/input.c:175
#13 0xb7cc6c05 in playlist_MLLoad (p_playlist=p_playlist@entry=0x818af40) at playlist/loadsave.c:171
#14 0xb7cc21a9 in playlist_Create (p_parent=p_parent@entry=0x804db78) at playlist/engine.c:271
#15 0xb7ca9fb5 in libvlc_InternalInit (p_libvlc=p_libvlc@entry=0x804db78, i_argc=4,
    ppsz_argv=ppsz_argv@entry=0xbfffe77c) at libvlc.c:557
#16 0xb7d9c1d8 in libvlc_new (argc=argc@entry=3, argv=argv@entry=0xbfffe7e4) at core.c:59
#17 0x08048f68 in main (i_argc=2, ppsz_argv=0xbfffea04) at vl...

[Alexey Brodkin (alexey-brodkin)]

Snap-shot of "filename.lua" from VLC's git: http://git.videolan.org/?p=vlc.git;a=blob_plain;f=share/lua/meta/reader/filename.lua;hb=HEAD

[Alexey Brodkin (alexey-brodkin)]

Confirm that only 32-bit machine is affected.
On 64-bit machine compiled script gets correctly executed.

[Alexey Brodkin (alexey-brodkin)]

As for Ubuntu-specific patches I don't agree.
Just got lua built from latest 5.2.2 sources from lua.org: http://www.lua.org/ftp/lua-5.2.2.tar.gz
and got the same behavior as with Ubuntu's binary packages:
==============
$ gdb --args ./lua luac.out
(gdb) run
Starting program: lua luac.out
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0xb7c1025d in __GI___uflow (fp=0x807bd08) at genops.c:394
394 genops.c: No such file or directory.
(gdb) bt
#0 0xb7c1025d in __GI___uflow (fp=0x807bd08) at genops.c:394
#1 0xb7c06772 in _IO_getc (fp=0x807bd08) at getc.c:41
#2 0x0805aac3 in skipBOM ()
#3 0x0805c2df in luaL_loadfilex ()
#4 0x0804bf27 in pmain ()
#5 0x0804f9f3 in luaD_precall ()
#6 0x0804fc46 in luaD_call ()
#7 0x0804c218 in f_call ()
#8 0x0804f2b4 in luaD_rawrunprotected ()
#9 0x0804fe9d in luaD_pcall ()
#10 0x0804dac5 in lua_pcallk ()
#11 0x0804b091 in main ()
==============

Once again on 64-bit machine everything works flawlessly.

[Alexey Brodkin (alexey-brodkin)]

Luckily I found out that this issue was caused by 1 special .so I had loaded automatically with LD_PRELOAD.
It was "libusb-driver.so" used by Xilinx Impact tool for communication with JTAG.
So with this library preloaded this simple test (built with "gcc test.c") crashes on the second "getc":
=========
#include <stdio.h>
char * filename = "luac.out";

int main (void)
{
 FILE* f;
 int c;

 f = fopen(filename, "r");
 if (f == NULL) {
  printf("Cannot open file %s\n", filename);
  return -1;
 }
 printf("File %s opened successfully\n", filename);
 c = getc(f);
 printf("c: %x\n", c);

 f = freopen(filename, "rb", f);
 if (f == NULL) {
  printf("Cannot re-open file %s in binary mode\n", filename);
  return -1;
 }
 printf("File %s re-opened in binary mode successfully\n", filename);
 c = getc(f);
 printf("c: %x\n", c);

 if (filename)
  fclose(f);
 return 0;
}
=========

If I disable library preload application gets executed without problems.
Moreover if I build this test statically everything will work as well.

So it is definitely not a problem in Lua.
Might be eglibc or dynamic linker.

[Mooky (simon-bernard-31)]

Hi,
  I'm a committer of Lua Development Tools (an eclipse plug-in), and this bug affect us too.
  We integrate the Lua VM and have this kind of exception at launch :

Current thread (0xb6706c00): JavaThread "main" [_thread_in_native, id=2110, stack(0xb6837000,0xb6888000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x00000037

Stack: [0xb6837000,0xb6888000], sp=0xb6882d40, free space=303k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.6+0x12ef98] _IO_file_underflow+0x68
C [liblua5.1.so+0x21c89] luaopen_string+0xef9
C [liblua5.1.so+0xa2ea] lua_resume+0x36a
C [liblua5.1.so+0xa784] lua_resume+0x804
C [liblua5.1.so+0x57b5] lua_call+0x35
C [liblua5.1.so+0x210aa] luaopen_string+0x31a
C [liblua5.1.so+0xa2ea] lua_resume+0x36a
C [liblua5.1.so+0x1545a] lua_newstate+0x3f9a
C [liblua5.1.so+0xa7d0] lua_resume+0x850
C [liblua5.1.so+0x5771] lua_pushcclosure+0x1a1
C [liblua5.1.so+0x9df3] lua_yield+0x723
C [liblua5.1.so+0x9e55] lua_yield+0x785
C [liblua5.1.so+0x5598] lua_pcall+0x68
C [libjnlua5.1.so+0xb085] Java_com_naef_jnlua_LuaState_lua_1pcall+0x205
j com.naef.jnlua.LuaState.lua_pcall(II)V+0
j com.naef.jnlua.LuaState.call(II)V+7

We can reproduce it only, on 32bit version and with a JVM 1.7. It seems this does not happened on older version of ubuntu.
I precise we do not use the lua VM packaged in ubuntu repository.

(LDT issue : https://bugs.eclipse.org/bugs/show_bug.cgi?id=413740)