TMUDF: A TMUDF missing 'language cpp' crashes sqlci with a core

[Weishiun Tsai]

As shown in the following example, 2 TMUDFs were defined: qa_tmudf1() was defined normally, while qa_tmudf2() was defined with a missing ‘language c++’ attribute. Invoking qa_tmudf2() crashes sqlci with a core at bindRowValues().

This is similar to another bug report https://bugs.launchpad.net/trafodion/+bug/1430438 ‘TMUDF: A TMUDF with nonexisting external name crashes sqlci with a core’. The stack trace of the core is similar as well. The 2 bugs can probably be addressed together. Again, the purpose of this bug report is only for this error to be handled more gracefully by returning a proper error at the definition time or at the run time, instead of crashing sqlci with a core.

This is seen on the v0305 build installed on a workstation. To reproduce it:

(1) Download the attached tar file and untar it to get the 3 files in there. Put the files in any directory <mydir>
(2) Make sure that you have run ./sqenv.sh of your Trafodion instance first as building UDF needs $MY_SQROOT for the header files.
(3) Run build.sh from <mydir> to build the UDF so file.
(4) Change the line create library qaTmudfLib file '<mydir>/qaTMUdfTest.so'; in mytest.sql and fill in <mydir>
(5) From sqlci, obey mytest.sql

-------------------------------------------------------------------------

Here is the execution output:

>>log mytest.log clear;
>>drop schema mytest cascade;

*** ERROR[1003] Schema TRAFODION.MYTEST does not exist.

--- SQL operation failed with errors.
>>create schema mytest;

--- SQL operation complete.
>>set schema mytest;

--- SQL operation complete.
>>
>>create library qaTmudfLib file '/designs/seaquest/wtsai/udf_bugs/udf_bug/qaTMUdfTest.so';

--- SQL operation complete.
>>
>>create table mytable (a int, b int);

--- SQL operation complete.
>>insert into mytable values (1,1),(2,2);

--- 2 row(s) inserted.
>>
>>create table_mapping function qa_tmudf1()
+>external name 'QA_TMUDF'
+>language cpp
+>library qaTmudfLib;

--- SQL operation complete.
>>
>>select * from UDF(qa_tmudf1(TABLE(select * from mytable)));

A B
----------- -----------

          1 1
          2 2

--- 2 row(s) selected.
>>
>>create table_mapping function qa_tmudf2()
+>external name 'QA_TMUDF'
+>-- language cpp
+>library qaTmudfLib;

--- SQL operation complete.
>>
>>select * from UDF(qa_tmudf2(TABLE(select * from mytable)));
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fffef80a975, pid=1499, tid=140737188215328
#
# JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [liboptimizer.so+0x3c6975] bindRowValues(BindWA*, ItemExpr*, ValueIdList&, RelExpr*, int)+0x475
#
# Core dump written. Default location: <my_dir>/core or core.1499
#
# An error report file with more information is saved as:
# <my_dir>/hs_err_pid1499.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Aborted (core dumped)

-------------------------------------------------------------------------

Here is the stack trace of the core file:

(gdb) bt
#0 0x00000039e28328a5 in raise () from /lib64/libc.so.6
#1 0x00000039e283400d in abort () from /lib64/libc.so.6
#2 0x00007ffff7122a55 in os::abort(bool) ()
   from /opt/home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff72a2f87 in VMError::report_and_die() ()
   from /opt/home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff712796f in JVM_handle_linux_signal ()
   from /opt/home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 <signal handler called>
#6 0x00007fffef80a975 in bindRowValues (bindWA=0x7ffffffef880,
    exprTree=<value optimized out>, vidList=..., parent=<value optimized out>,
    inTrueRoot=1) at ../optimizer/BindRelExpr.cpp:222
#7 0x00007fffef8188fa in RelRoot::bindNode (this=0x7fffd73c81d0,
    bindWA=0x7ffffffef880) at ../optimizer/BindRelExpr.cpp:5440
#8 0x00007ffff111406e in CmpMain::compile (this=0x7fffffff2330,
    input_str=0x7fffe8bab3c0 "select * from UDF(qa_tmudf2(TABLE(select * from mytable)));", charset=15, queryExpr=@0x7fffffff2268, gen_code=0x7fffd740d4a0,
    gen_code_len=0x7fffd740d498, heap=0x7fffe9741c08, phase=CmpMain::END,
    fragmentDir=0x7fffffff2488, op=3004, useQueryCache=1,
    cacheable=0x7fffffff2278, begTime=0x7fffffff2250, shouldLog=0)
    at ../sqlcomp/CmpMain.cpp:2058
#9 0x00007ffff1116e7c in CmpMain::sqlcomp (this=0x7fffffff2330,
    input_str=0x7fffe8bab3c0 "select * from UDF(qa_tmudf2(TABLE(select * from mytable)));", charset=15, queryExpr=@0x7fffffff2268, gen_code=0x7fffd740d4a0,
    gen_code_len=0x7fffd740d498, heap=0x7fffe9741c08, phase=CmpMain::END,
    fragmentDir=0x7fffffff2488, op=3004, useQueryCache=1,
    cacheable=0x7fffffff2278, begTime=0x7fffffff2250, shouldLog=0)
    at ../sqlcomp/CmpMain.cpp:1671
#10 0x00007ffff1118180 in CmpMain::sqlcomp (this=0x7fffffff2330, input=...,
    gen_code=0x7fffd740d4a0, gen_code_len=0x7fffd740d498, heap=0x7fffe9741c08,
    phase=CmpMain::END, fragmentDir=0x7fffffff2488, op=3004)
    at ../sqlcomp/CmpMain.cpp:817
#11 0x00007ffff55c7478 in CmpStatement::process (this=0x7fffd73b9d00,
    sqltext=<value optimized out>) at ../arkcmp/CmpStatement.cpp:508
#12 0x00007ffff55ba903 in CmpContext::compileDirect (this=0x7fffe8bab090,
    data=0x7fffe974e808 "\200", data_len=192, outHeap=0x7fffea1b1128,
    charset=15, op=CmpMessageObj::SQLTEXT_COMPILE, gen_code=@0x7fffffff2a10,
    gen_code_len=@0x7fffffff2a18, parserFlags=0, diagsArea=0x7fffe974e8d0)
    at ../arkcmp/CmpContext.cpp:689
#13 0x00007ffff5faf857 in CliStatement::prepare2 (this=0x7fffe975d478,
    source=0x7fffe97541e8 "select * from UDF(qa_tmudf2(TABLE(select * from mytable)));", diagsArea=..., passed_gen_code=<value optimized out>,
    passed_gen_code_len=3927642408, charset=15, unpackTdbs=1, cliFlags=129)
    at ../cli/Statement.cpp:1827
#14 0x00007ffff5fafca6 in CliStatement::prepare (this=0x7fffe975d478,
    source=0x7fffe97541e8 "select * from UDF(qa_tmudf2(TABLE(select * from mytable)));", diagsArea=..., passed_gen_code=<value optimized out>,
    passed_gen_code_len=<value optimized out>, charset=<value optimized out>,
    unpackTdbs=1, cliFlags=129) at ../cli/Statement.cpp:1420
#15 0x00007ffff5f5f984 in SQLCLI_Prepare2 (cliGlobals=0xb82c10,
    statement_id=0x1c0c400, sql_source=0x1dc1e30, gencode_ptr=0x0,
    gencode_len=0, ret_gencode_len=0x0, query_cost_info=0x7fffffff3f70,
    query_comp_stats_info=0x7fffffff2ce0, uniqueStmtId=<value optimized out>,
    uniqueStmtIdLen=0x7fffffff4028, flags=129) at ../cli/Cli.cpp:5914
#16 0x00007ffff5fbcf50 in SQL_EXEC_Prepare2 (statement_id=0x1c0c400,
    sql_source=0x1dc1e30, gencode_ptr=0x0, gencode_len=0, ret_gencode_len=0x0,
    query_cost_info=0x7fffffff3f70, comp_stats_info=0x7fffffff2ce0,
    uniqueStmtId=0x7fffffff3de0 " ", uniqueStmtIdLen=0x7fffffff4028,
    flags=129) at ../cli/CliExtern.cpp:4985
#17 0x00007ffff79cedaf in SqlCmd::do_prepare (sqlci_env=0xb819c0,
    prep_stmt=0x1dc4f40,
    sqlStmt=0x149ea90 "select * from UDF(qa_tmudf2(TABLE(select * from mytable)));", resetLastExecStmt=1, rsIndex=-49696, prepcode=0x7fffffff40fc,
    statisticsType=0x7fffffff40f8) at ../sqlci/SqlCmd.cpp:893
#18 0x00007ffff79ce1b0 in DML::process (this=0x1c0bcf0, sqlci_env=0xb819c0)
    at ../sqlci/SqlCmd.cpp:2803
#19 0x00007ffff79b4a84 in Obey::process (this=0x1d97940,
    sqlci_env=<value optimized out>) at ../sqlci/Obey.cpp:264
#20 0x00007ffff79bcf14 in SqlciEnv::run (this=0xb819c0,
    in_filename=<value optimized out>, input_string=<value optimized out>)
    at ../sqlci/SqlciEnv.cpp:726
#21 0x00000000004019c2 in main (argc=3, argv=0x7fffffff43c8)
    at ../bin/SqlciMain.cpp:326
(gdb)

[Suresh Subbiah (suresh-subbiah)]

A related instance of this bug is that if the language cpp clause is missing we may get no error at all but instead get 0 rows at runtime (though rows should have been returned). I am not sure what is correct behaviour for this instance of user error but some kind of error message would be nice.

To reproduce

create library sessionize_etl file '/opt/home/ssubbiah/trafodion/core/sql/regress/udr/libsessionizeETL.so'

[Suresh Subbiah (suresh-subbiah)]

Sorry previous comment was posted too early.

Compile the attached .cpp into an .so. and create a library on it to point to the created .so. Let the library be called sessionize_etl.

create table_mapping function sessionize_etl()
returns (vehicle_id char(25),
         begin_utc char(19),
         end_utc char(19),
  alarm_code char(2))
--language cpp
external name 'SESSIONIZEETL'
library sessionize_etl;

CREATE TABLE clicks
  (
    VEHICLE_ID CHAR(25) CHARACTER SET ISO88591 COLLATE
      DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
  , GPS_TIMESTAMP TIMESTAMP(0) NO DEFAULT NOT NULL NOT DROPPABLE
  , ALERT_CODE VARCHAR(200) CHARACTER SET ISO88591 COLLATE
      DEFAULT DEFAULT NULL)
   STORE BY (VEHICLE_ID ASC, GPS_TIMESTAMP ASC);

insert into clicks values ('12345', timestamp '2014-11-11 01:01:01', '01,02,03,04,05') ;

SELECT *
FROM UDF(sessionize_etl(TABLE(SELECT vehicle_id, gps_timestamp, alert_code
                          FROM clicks
                          PARTITION BY vehicle_id
                          ORDER BY gps_timestamp)));

-- will likely return 0 rows while it should return 5 rows

This may be because if compile time methods are missing instead of a core file we treat the tmudf language to be C, find the runtime method missing and come out with no error. An error message would be nice.

[Hans Zeller (hans-zeller)]

With my recent changes, the DLL entry points for C and C++ UDFs have the same names. So, if we get the language wrong, we call the entry point with the wrong parameters and will likely crash. We may not be able to eliminate that type of crash completely, but to minimize it, I'm planning to implement the following change:

The default language for TMUDFs will change to be C++ for C/C++ libraries and Java for Java libraries (jar files). Since a library doesn't have a language attribute, we determine its language from the file suffix. If there is no such suffix (.so, .dll, .jar), then the user will need to specify the language explicitly - this should be an unusual case.

With my fix, the code will also issue the following warning if we use the C interface for TMUDFs or if we create the now unsupported UUDFs or actions:

*** WARNING[3285] The C interface for this type of routine is deprecated, please consider migrating to a TMUDF with a C++ or Java interface.

[Hans Zeller (hans-zeller)]

Fix was delivered on March 18 as part of https://review.trafodion.org/1329

[Weishiun Tsai (wei-shiun-tsai)]

Verified on the v0330 build installed on a workstation. This problem is now fixed:

>>drop schema mytest cascade;

--- SQL operation complete.
>>create schema mytest;

--- SQL operation complete.
>>set schema mytest;

--- SQL operation complete.
>>
>>create library qaTmudfLib file '<mydir>/qaTMUdfTest.so';

--- SQL operation complete.
>>
>>create table mytable (a int, b int);

--- SQL operation complete.
>>insert into mytable values (1,1),(2,2);

--- 2 row(s) inserted.
>>
>>create table_mapping function qa_tmudf1()
+>external name 'QA_TMUDF'
+>language cpp
+>library qaTmudfLib;

--- SQL operation complete.
>>
>>select * from UDF(qa_tmudf1(TABLE(select * from mytable)));

A B
----------- -----------

          1 1
          2 2

--- 2 row(s) selected.
>>
>>create table_mapping function qa_tmudf2()
+>external name 'QA_TMUDF'
+>-- language cpp
+>library qaTmudfLib;

--- SQL operation complete.
>>
>>select * from UDF(qa_tmudf2(TABLE(select * from mytable)));

A B
----------- -----------

          1 1
          2 2

--- 2 row(s) selected.
>>
>>drop schema mytest cascade;

*** ERROR[1069] Schema TRAFODION.MYTEST could not be dropped.

--- SQL operation failed with errors.