mxosrvr cores at PCodeCfg::layoutNativeCode() during phoenix tests runs

Bug #1389347 reported by Weishiun Tsai
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Trafodion
Fix Released
High
James Capps

Bug Description

When running phoenix tests on clusters, this problem has caused mxosrvr to core and the tests to fail a couple of times. This is seen with the v1101_0830 build on a Cloudera cluster, and with the v1104_0830 build on a HortonWorks cluster as well. The problem is caused by segmentation fault and cannot be reliably reproduced. But it does show up two days in a row.

The default threads of the cores do not show any useful info. But the threads handling the client requests show that they both are in the middle of handling PCodeCfg::layoutNativeCode().

---------------------------------------------------------------------------------------------------------------------------------------------------------
Here is the core file core.1415043771.n011.29690.mxosrvr from v1101_0830 (on a Cloudera machine)

Program terminated with signal 11, Segmentation fault.
#0 0x00007fffdf2a5f34 in ?? ()

(gdb) info threads
  43 Thread 0x7fffe6ee6700 (LWP 29699) 0x00007ffff528099d in sigtimedwait ()
   from /lib64/libc.so.6
  42 Thread 0x7fffd28ac700 (LWP 16641) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  41 Thread 0x7ffff7fbb760 (LWP 29690) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  40 Thread 0x7fffe64e5700 (LWP 29700) 0x00007ffff532c253 in poll ()
   from /lib64/libc.so.6
  39 Thread 0x7fffd23a8700 (LWP 16645) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  38 Thread 0x7fffde722700 (LWP 31966) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  37 Thread 0x7fffde520700 (LWP 31968) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  36 Thread 0x7fffe50e3700 (LWP 29702) 0x00007ffff527fd85 in sigwait ()
   from /lib64/libc.so.6
  35 Thread 0x7fffde823700 (LWP 31965) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  34 Thread 0x7fffe307d700 (LWP 31960) 0x00007ffff5335f03 in epoll_wait ()
   from /lib64/libc.so.6
  33 Thread 0x7fffd9f46700 (LWP 31970) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  32 Thread 0x7fffde924700 (LWP 31964) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  31 Thread 0x7fffdec27700 (LWP 31961) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  30 Thread 0x7fffd3eb3700 (LWP 31972) 0x00007ffff4bff720 in sem_wait ()
   from /lib64/libpthread.so.0
  29 Thread 0x7fffcfc1f700 (LWP 3063) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  28 Thread 0x7fffe3c02700 (LWP 31959) 0x00007ffff52f9b8d in nanosleep ()
   from /lib64/libc.so.6
  27 Thread 0x7fffd3db2700 (LWP 31973) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  26 Thread 0x7fffe5ae4700 (LWP 29701) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  25 Thread 0x7fffd9e45700 (LWP 31971) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  24 Thread 0x7fffea20b700 (LWP 29697) 0x00007ffff53364cd in accept ()
   from /lib64/libc.so.6
  23 Thread 0x7fffe980a700 (LWP 29698) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  22 Thread 0x7fffdeb26700 (LWP 31962) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  21 Thread 0x7fffdea25700 (LWP 31963) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  20 Thread 0x7fffd0224700 (LWP 2880) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  19 Thread 0x7fffcfd20700 (LWP 16495) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  18 Thread 0x7fffd3bb0700 (LWP 31975) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  17 Thread 0x7fffd24a9700 (LWP 31984) 0x00007ffff532c253 in poll ()
   from /lib64/libc.so.6
  16 Thread 0x7fffd0a96700 (LWP 16640) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  15 Thread 0x7fffcfe21700 (LWP 16638) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  14 Thread 0x7fffd0d19700 (LWP 2855) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  13 Thread 0x7fffd22a7700 (LWP 2852) 0x00007ffff5335f03 in epoll_wait ()
   from /lib64/libc.so.6
  12 Thread 0x7fffd3aaf700 (LWP 31976) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  11 Thread 0x7fffd0b97700 (LWP 2876) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  10 Thread 0x7fffd0e1a700 (LWP 2853) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  9 Thread 0x7fffd3cb1700 (LWP 31974) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  8 Thread 0x7fffd20a5700 (LWP 2877) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  7 Thread 0x7fffd0f1b700 (LWP 2886) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  6 Thread 0x7fffda047700 (LWP 31969) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  5 Thread 0x7fffd101c700 (LWP 2890) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  4 Thread 0x7fffd21a6700 (LWP 2897) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  3 Thread 0x7fffde621700 (LWP 31967) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  2 Thread 0x7fffe4684700 (LWP 29718) 0x00007ffff2210965 in llvm::APInt::getLowBitsSet(unsigned int, unsigned int) ()
   from /opt/home/trafodion/v1101_0830/export/lib64/libtdm_sqlexp.so
* 1 Thread 0x7fffcff22700 (LWP 2902) 0x00007fffdf2a5f34 in ?? ()
(gdb) bt
#0 0x00007fffdf2a5f34 in ?? ()
#1 0x00000000c0472ce8 in ?? ()
#2 0x00000000ec50b4a8 in ?? ()
#3 0x00000000ec4e0b28 in ?? ()
#4 0x00000000ec50ab68 in ?? ()
#5 0x00000000c00ac260 in ?? ()
#6 0x00000000c035d210 in ?? ()
#7 0x00007fffcff0a740 in ?? ()
#8 0x00007fffdefae058 in ?? ()
#9 0x00007fffcff0a740 in ?? ()
#10 0x00007fffdefae233 in ?? ()
#11 0x00007fffdefae233 in ?? ()
#12 0x00000000c0472ce8 in ?? ()
#13 0x00007fffcff0a700 in ?? ()
#14 0x00000000bb6f0d72 in ?? ()
#15 0x00007fffcff0a798 in ?? ()
#16 0x00000000bb6f6c48 in ?? ()
#17 0x00000000bc78a810 in ?? ()
#18 0x00000000bb6f0eb8 in ?? ()
#19 0x00007fffcff0a6f8 in ?? ()
#20 0x00007fffcff0a790 in ?? ()
#21 0x00007fffcff0a7e0 in ?? ()
#22 0x00007fffdefae058 in ?? ()
#23 0x0000000000000000 in ?? ()
(gdb) bt
#0 0x00007ffff2210965 in llvm::APInt::getLowBitsSet(unsigned int, unsigned int) () from /opt/home/trafodion/v1101_0830/export/lib64/libtdm_sqlexp.so
#1 0x00007ffff2785b10 in llvm::ComputeMaskedBits(llvm::Value*, llvm::APInt&, llvm::APInt&, llvm::DataLayout const*, unsigned int) ()
   from /opt/home/trafodion/v1101_0830/export/lib64/libtdm_sqlexp.so
#2 0x00007ffff26ac425 in llvm::getOrEnforceKnownAlignment(llvm::Value*, unsigned int, llvm::DataLayout const*) ()
   from /opt/home/trafodion/v1101_0830/export/lib64/libtdm_sqlexp.so
#3 0x00007ffff2678b78 in llvm::InstCombiner::visitStoreInst(llvm::StoreInst&)
    () from /opt/home/trafodion/v1101_0830/export/lib64/libtdm_sqlexp.so
#4 0x00007ffff263ce7f in llvm::InstCombiner::DoOneIteration(llvm::Function&, unsigned int) ()
   from /opt/home/trafodion/v1101_0830/export/lib64/libtdm_sqlexp.so
#5 0x00007ffff263db1f in llvm::InstCombiner::runOnFunction(llvm::Function&) ()
   from /opt/home/trafodion/v1101_0830/export/lib64/libtdm_sqlexp.so
#6 0x00007ffff28ac49d in llvm::FPPassManager::runOnFunction(llvm::Function&)
    () from /opt/home/trafodion/v1101_0830/export/lib64/libtdm_sqlexp.so
#7 0x00007ffff28ac64b in llvm::FunctionPassManagerImpl::run(llvm::Function&)
    () from /opt/home/trafodion/v1101_0830/export/lib64/libtdm_sqlexp.so
#8 0x00007ffff28ac95c in llvm::FunctionPassManager::run(llvm::Function&) ()
   from /opt/home/trafodion/v1101_0830/export/lib64/libtdm_sqlexp.so
#9 0x00007ffff20ff3bf in PCodeCfg::layoutNativeCode (this=0x7fffcfffc198,
showplanSpace=0x0) at ../exp/ExpPCodeOptsNativeExpr.cpp:9079
#10 0x00007ffff20c1df1 in PCodeCfg::optimize (this=0x7fffcfffc198)
    at ../exp/ExpPCodeOptimizations.cpp:2275
#11 0x00007ffff20a1a65 in ex_expr::pCodeGenerate (this=<value optimized out>,
    space=0x7fffcfffb4a8, heap=0x7fffd095e660, f=0)
    at ../exp/ExpPCodeExpGen.cpp:975
#12 0x00007fffee24fea4 in ExpGenerator::endExprGen (this=0x7fffcffce4e8,
    expr=0x7fffe465d268, gen_last_clause=0)
    at ../generator/GenExpGenerator.cpp:4772
#13 0x00007fffee2535c3 in ExpGenerator::generateListExpr (this=0x7fffcffce4e8,
    val_id_list=..., node_type=<value optimized out>, expr=0x7fffe465d268,
    atp=<value optimized out>, atpIndex=<value optimized out>, hdrInfo=0x0)
    at ../generator/GenExpGenerator.cpp:3842
#14 0x00007fffee25b6f7 in ExpGenerator::generateContiguousMoveExpr (
    this=0x7fffcffce4e8, valIdList=..., addConvNodes=0, atp=1, atpIndex=2,
    tdataF=ExpTupleDesc::SQLARK_EXPLODED_FORMAT, tupleLength=@0x7fffe465d2ac,
    moveExpr=0x7fffe465d268, tupleDesc=0x7fffe465d250,
    tdescF=ExpTupleDesc::LONG_FORMAT, newMapTable=0x0, tgtValues=0x0,
    startOffset=0, bulkMoveSrcStartOffset=0x0, disableConstFolding=0,
    colArray=0x0, doBulkMoves=0) at ../generator/GenExpGenerator.cpp:2529
#15 0x00007fffee310c2f in HbaseAccess::codeGen (this=0x7fffe465cb10,
    generator=0x7fffe465e670) at ../generator/GenRelScan.cpp:1847
#16 0x00007fffee2f62cb in Sort::codeGen (this=0x7fffcffcb7f8,
    generator=0x7fffe465e670) at ../generator/GenRelMisc.cpp:3551
#17 0x00007fffee2fb605 in RelRoot::codeGen (this=0x7fffcffc9fd0,
    generator=0x7fffe465e670) at ../generator/GenRelMisc.cpp:1379
#18 0x00007fffee24509e in Generator::genCode (this=0x7fffe465e670,
    source=0x7fffd09456d0 "select column_name, column_number, column_class, fs_data_type, column_size, column_precision, column_scale, datetime_start_field, datetime_end_field, trim(is_upshifted), column_flags, nullable, trim(c"...,
    expr_node=0x7fffcffc9fd0) at ../generator/Generator.cpp:506
#19 0x00007ffff01c801a in CmpMain::compile (this=0x7fffe4661600,
    input_str=0x7fffd09456d0 "select column_name, column_number, column_class, fs_data_type, column_size, column_precision, column_scale, datetime_start_field, datetime_end_field, trim(is_upshifted), column_flags, nullable, trim(c"...,
    charset=15, queryExpr=@0x7fffe4661538, gen_code=0x7fffd096f388,
    gen_code_len=0x7fffd096f380, heap=0x7fffd1e2e948, phase=CmpMain::END,
    fragmentDir=0x7fffe4661758, op=3004, useQueryCache=<value optimized out>,
    cacheable=0x7fffe4661548, begTime=0x7fffe4661520, shouldLog=0)
    at ../sqlcomp/CmpMain.cpp:2269
#20 0x00007ffff01c96cc in CmpMain::sqlcomp (this=0x7fffe4661600,
    input_str=0x7fffd09456d0 "select column_name, column_number, column_class, fs_data_type, column_size, column_precision, column_scale, datetime_start_field, datetime_end_field, trim(is_upshifted), column_flags, nullable, trim(c"...,
    charset=15, queryExpr=@0x7fffe4661538, gen_code=0x7fffd096f388,
    gen_code_len=0x7fffd096f380, heap=0x7fffd1e2e948, phase=CmpMain::END,
fragmentDir=0x7fffe4661758, op=3004, useQueryCache=1,
    cacheable=0x7fffe4661548, begTime=0x7fffe4661520, shouldLog=0)
    at ../sqlcomp/CmpMain.cpp:1660
#21 0x00007ffff01ca8bc in CmpMain::sqlcomp (this=0x7fffe4661600, input=...,
    gen_code=0x7fffd096f388, gen_code_len=0x7fffd096f380, heap=0x7fffd1e2e948,
    phase=CmpMain::END, fragmentDir=0x7fffe4661758, op=3004)
    at ../sqlcomp/CmpMain.cpp:809
#22 0x00007fffedfd28d8 in CmpStatement::process (this=0x7fffd093d120,
    sqltext=<value optimized out>) at ../arkcmp/CmpStatement.cpp:488
#23 0x00007fffedfc72de in CmpContext::compileDirect (this=0x7fffd0b98090,
    data=0x7fffd0868438 "\200", data_len=592, outHeap=0x7fffe3c050f0,
    charset=15, op=CmpMessageObj::SQLTEXT_COMPILE, gen_code=@0x7fffe4661ce0,
    gen_code_len=@0x7fffe4661ce8, parserFlags=4358144,
    diagsArea=0x7fffd0868690) at ../arkcmp/CmpContext.cpp:688
#24 0x00007ffff49c7a67 in CliStatement::prepare2 (this=0x7fffd00ff1e8,
    source=0x7fffd062e910 "select column_name, column_number, column_class, fs_data_type, column_size, column_precision, column_scale, datetime_start_field, datetime_end_field, trim(is_upshifted), column_flags, nullable, trim(c"...,
    diagsArea=..., passed_gen_code=<value optimized out>,
    passed_gen_code_len=3821031664, charset=15, unpackTdbs=1, cliFlags=144)
    at ../cli/Statement.cpp:1797
#25 0x00007ffff49c7eb6 in CliStatement::prepare (this=0x7fffd00ff1e8,
source=0x7fffd062e910 "select column_name, column_number, column_class, fs_data_type, column_size, column_precision, column_scale, datetime_start_field, dat
etime_end_field, trim(is_upshifted), column_flags, nullable, trim(c"...,
    diagsArea=..., passed_gen_code=<value optimized out>,
    passed_gen_code_len=<value optimized out>, charset=<value optimized out>,
    unpackTdbs=1, cliFlags=144) at ../cli/Statement.cpp:1388
#26 0x00007ffff49928cc in SQLCLI_Prepare2 (cliGlobals=0xf45040,
    statement_id=0x1e0a110, sql_source=0x1c90760, gencode_ptr=0x0,
    gencode_len=0, ret_gencode_len=0x0, query_cost_info=0x7fffe46630b0,
    query_comp_stats_info=0x7fffe4661fb0, uniqueStmtId=<value optimized out>,
    uniqueStmtIdLen=0x0, flags=0) at ../cli/Cli.cpp:5910
#27 0x00007ffff49d2cf7 in SQL_EXEC_Prepare2 (statement_id=0x1e0a110,
    sql_source=0x1c90760, gencode_ptr=0x0, gencode_len=0, ret_gencode_len=0x0,
    query_cost_info=0x7fffe46630b0, comp_stats_info=0x7fffe4661fb0,
    uniqueStmtId=0x0, uniqueStmtIdLen=0x0, flags=0)
    at ../cli/CliExtern.cpp:4955
#28 0x00007ffff371c85b in ExeCliInterface::prepare (this=0x7fffe46658c0,
    stmtStr=<value optimized out>, module=<value optimized out>,
    stmt=0x1e0a110, sql_src=0x1c90760, input_desc=0x322c0f0,
    output_desc=0x324e910, outputBuf=0x7fffe46658f8, outputVarPtrList=0x0,
    inputBuf=0x7fffe4665930, inputVarPtrList=0x0, uniqueStmtId=0x0,
    uniqueStmtIdLen=0x0, query_cost_info=0x0, comp_stats_info=0x0,
    monitorThis=0) at ../executor/ExExeUtilCli.cpp:321
#29 0x00007ffff371e25a in ExeCliInterface::fetchRowsPrologue (
    this=0x7fffe46658c0,
    sqlStrBuf=0x7fffe4663360 "select column_name, column_number, column_class, fs_data_type, column_size, column_precision, column_scale, datetime_start_field, datetime_end_field, trim(is_upshifted), column_flags, nullable, trim(c"...,
    noExec=0, monitorThis=0, stmtName=0x0) at ../executor/ExExeUtilCli.cpp:1020
#30 0x00007ffff371f326 in ExeCliInterface::fetchAllRows (this=0x7fffe46658c0,
    infoList=@0x7fffe4664080,
    query=0x7fffe4663360 "select column_name, column_number, column_class, fs_data_type, column_size, column_precision, column_scale, datetime_start_field, datetime_end_field, trim(is_upshifted), column_flags, nullable, trim(c"...,
    inNumOutputEntries=0, varcharFormat=0, monitorThis=<value optimized out>,
    initInfoList=1) at ../executor/ExExeUtilCli.cpp:1103
#31 0x00007ffff01f8296 in CmpSeabaseDDL::getSeabaseColumnInfo (
    this=<value optimized out>, cliInterface=0x7fffe46658c0,
    objUID=300664944826988162, direction=0x7fffe4665ca0 "' '",
    isTableSalted=0x7fffe4665d24, numCols=0x7fffe4665d28,
    outColInfoArray=0x7fffe4665d00) at ../sqlcomp/CmpSeabaseDDLtable.cpp:5599
#32 0x00007ffff02103c9 in CmpSeabaseDDL::getSeabaseUserTableDesc (
    this=0x7fffe4665ea0, catName=..., schName=..., objName=...,
    objType=<value optimized out>, includeInvalidDefs=0)
    at ../sqlcomp/CmpSeabaseDDLtable.cpp:5938
#33 0x00007ffff0216629 in CmpSeabaseDDL::getSeabaseTableDesc (
    this=0x7fffe4665ea0, catName=..., schName=..., objName=...,
    objType=<value optimized out>, includeInvalidDefs=0)
    at ../sqlcomp/CmpSeabaseDDLtable.cpp:6611
#34 0x00007fffeebbb6d5 in NATableDB::get (this=0x7fffe2601918, corrName=...,
    bindWA=0x7fffe4669440, inTableDescStruct=0x0)
    at ../optimizer/NATable.cpp:7590
#35 0x00007fffee92bf8c in BindWA::getNATable (this=0x7fffe4669440,
    corrName=..., catmanCollectTableUsages=1, inTableDescStruct=0x0)
    at ../optimizer/BindRelExpr.cpp:1446
#36 0x00007fffee93acc3 in GenericUpdate::bindNode (this=0x7fffd2d47b60,
    bindWA=0x7fffe4669440) at ../optimizer/BindRelExpr.cpp:11792
#37 0x00007fffee950823 in Insert::bindNode (this=0x7fffd2d47b60,
    bindWA=0x7fffe4669440) at ../optimizer/BindRelExpr.cpp:9121
#38 0x00007fffee91bdf7 in RelExpr::bindChildren (this=0x7fffd2d48a10,
    bindWA=0x7fffe4669440) at ../optimizer/BindRelExpr.cpp:2181
#39 0x00007fffee956b7e in RelRoot::bindNode (this=0x7fffd2d48a10,
    bindWA=0x7fffe4669440) at ../optimizer/BindRelExpr.cpp:5204
#40 0x00007ffff01c6fb3 in CmpMain::compile (this=0x7fffe466b730,
    input_str=0x7fffd2d796e0 "UPSERT INTO TABLE_WITH_SALTING VALUES(?,?,?,?,?)", charset=15, queryExpr=@0x7fffe466b668, gen_code=0x7fffd04e2ca8,
    gen_code_len=0x7fffd04e2ca0, heap=0x7fffe3193bf8, phase=CmpMain::END,
    fragmentDir=0x7fffe466b888, op=3004, useQueryCache=<value optimized out>,
    cacheable=0x7fffe466b678, begTime=0x7fffe466b650, shouldLog=0)
    at ../sqlcomp/CmpMain.cpp:2003
#41 0x00007ffff01c96cc in CmpMain::sqlcomp (this=0x7fffe466b730,
    input_str=0x7fffd2d796e0 "UPSERT INTO TABLE_WITH_SALTING VALUES(?,?,?,?,?)", charset=15, queryExpr=@0x7fffe466b668, gen_code=0x7fffd04e2ca8,
    gen_code_len=0x7fffd04e2ca0, heap=0x7fffe3193bf8, phase=CmpMain::END,
    fragmentDir=0x7fffe466b888, op=3004, useQueryCache=1,
    cacheable=0x7fffe466b678, begTime=0x7fffe466b650, shouldLog=0)
    at ../sqlcomp/CmpMain.cpp:1660
#42 0x00007ffff01ca8bc in CmpMain::sqlcomp (this=0x7fffe466b730, input=...,
    gen_code=0x7fffd04e2ca8, gen_code_len=0x7fffd04e2ca0, heap=0x7fffe3193bf8,
    phase=CmpMain::END, fragmentDir=0x7fffe466b888, op=3004)
    at ../sqlcomp/CmpMain.cpp:809
#43 0x00007fffedfd28d8 in CmpStatement::process (this=0x7fffd2d23ff8,
    sqltext=<value optimized out>) at ../arkcmp/CmpStatement.cpp:488
#44 0x00007fffedfc72de in CmpContext::compileDirect (this=0x7fffe25fc090,
    data=0x7fffe31f7968 "\200", data_len=184, outHeap=0x7fffe3c050f0,
    charset=15, op=CmpMessageObj::SQLTEXT_COMPILE, gen_code=@0x7fffe466be10,
    gen_code_len=@0x7fffe466be18, parserFlags=4194304,
    diagsArea=0x7fffe31f7a28) at ../arkcmp/CmpContext.cpp:688
#45 0x00007ffff49c7a67 in CliStatement::prepare2 (this=0x7fffd0565c30,
    source=0x7fffcf63e978 "UPSERT INTO TABLE_WITH_SALTING VALUES(?,?,?,?,?)",
    diagsArea=..., passed_gen_code=<value optimized out>,
    passed_gen_code_len=3821031664, charset=15, unpackTdbs=1, cliFlags=144)
    at ../cli/Statement.cpp:1797
#46 0x00007ffff49c7eb6 in CliStatement::prepare (this=0x7fffd0565c30,
    source=0x7fffcf63e978 "UPSERT INTO TABLE_WITH_SALTING VALUES(?,?,?,?,?)",
    diagsArea=..., passed_gen_code=<value optimized out>,
    passed_gen_code_len=<value optimized out>, charset=<value optimized out>,
    unpackTdbs=1, cliFlags=144) at ../cli/Statement.cpp:1388
#47 0x00007ffff49928cc in SQLCLI_Prepare2 (cliGlobals=0xf45040,
    statement_id=0x338d7c8, sql_source=0x7fffe466c150, gencode_ptr=0x0,
    gencode_len=0, ret_gencode_len=0x0, query_cost_info=0x338d9f8,
    query_comp_stats_info=0x338da48, uniqueStmtId=<value optimized out>,
    uniqueStmtIdLen=0x338d92c, flags=0) at ../cli/Cli.cpp:5910
#48 0x00007ffff49d2cf7 in SQL_EXEC_Prepare2 (statement_id=0x338d7c8,
    sql_source=0x7fffe466c150, gencode_ptr=0x0, gencode_len=0,
    ret_gencode_len=0x0, query_cost_info=0x338d9f8, comp_stats_info=0x338da48,
    uniqueStmtId=0x338d930 "", uniqueStmtIdLen=0x338d92c, flags=0)
    at ../cli/CliExtern.cpp:4955
#49 0x00007ffff6e7a8d9 in SRVR::WSQL_EXEC_Prepare2 (statement_id=0x338d7c8,
    sql_source=<value optimized out>, gencode_ptr=<value optimized out>,
    gencode_len=<value optimized out>, ret_gencode_len=<value optimized out>,
    query_cost_info=<value optimized out>, comp_stats_info=0x338da48,
    uniqueQueryId=0x338d930 "", uniqueQueryIdLen=0x338d92c, flags=0)
    at SQLWrapper.cpp:800
#50 0x00007ffff6e6f275 in SRVR::PREPARE2 (pSrvrStmt=0x338d1b0,
    isFromExecDirect=248) at sqlinterface.cpp:5016
#51 0x00007ffff6e9f670 in odbc_SQLSvc_Prepare2_sme_ (inputRowCnt=0,
    sqlStmtType=256, stmtLabel=<value optimized out>,
    sqlString=0x1c20750 "UPSERT INTO TABLE_WITH_SALTING VALUES(?,?,?,?,?)",
    holdableCursor=0, returnCode=0x7fffe466c93c,
    sqlWarningOrErrorLength=0x7fffe466c938, sqlWarningOrError=@0x7fffe466c920,
    sqlQueryType=0x7fffe466c934, stmtHandle=0x7fffe466c918,
    estimatedCost=0x7fffe466c930, inputDescLength=0x7fffe466c92c,
    inputDesc=@0x7fffe466c910, outputDescLength=0x7fffe466c928,
    outputDesc=@0x7fffe466c908, isFromExecDirect=false) at srvrothers.cpp:926
#52 0x00000000004c5210 in odbc_SQLSrvr_Prepare_ame_ (objtag_=0x24d3800,
    call_id_=0x24d3858, dialogueId=105906255, sqlAsyncEnable=0,
    queryTimeout=0, stmtType=<value optimized out>, sqlStmtType=256,
    stmtLength=10, stmtLabel=0x1c20736 "SQL_CUR_3", stmtLabelCharset=1,
    cursorLength=0, cursorName=0x0, cursorCharset=0, moduleNameLength=0,
    moduleName=0x0, moduleCharset=0, moduleTimestamp=0, sqlStringLength=49,
    sqlString=0x1c20750 "UPSERT INTO TABLE_WITH_SALTING VALUES(?,?,?,?,?)",
    sqlStringCharset=1, setStmtOptionsLength=0, setStmtOptions=0x0,
    stmtExplainLabelLength=0, stmtExplainLabel=0x0, maxRowsetSize=0, txnID=0,
    extTransId=0x0, holdableCursor=0) at SrvrConnect.cpp:6265
#53 0x00000000004999e1 in SQLPREPARE_IOMessage (objtag_=0x24d3800,
    call_id_=0x24d3858) at Interface/odbcs_srvr.cpp:925
#54 0x00000000004a1b2c in DISPATCH_TCPIPRequest (objtag_=0x24d3800,
    call_id_=0x24d3858, operation_id=<value optimized out>)
    at Interface/odbcs_srvr.cpp:1793
#55 0x00000000004328e2 in BUILD_TCPIP_REQUEST (pnode=0x24d3800)
    at ../Common/TCPIPSystemSrvr.cpp:603
#56 0x000000000043327d in PROCESS_TCPIP_REQUEST (pnode=0x24d3800)
    at ../Common/TCPIPSystemSrvr.cpp:581
#57 0x00000000004709b6 in CNSKListenerSrvr::tcpip_listener (arg=0xde5450)
    at Interface/linux/Listener_srvr_ps.cpp:397
#58 0x00007ffff50329f0 in sb_thread_sthr_disp (pp_arg=0xf17550)
    at threadl.cpp:253
#59 0x00007ffff4bf9851 in start_thread () from /lib64/libpthread.so.0
#60 0x00007ffff533590d in clone () from /lib64/libc.so.6

---------------------------------------------------------------------------------------------------------------------------------------------------------
Here is the core file core.1415121533.n001.14030.mxosrvr from v1104_0830 (on a HortonWorks machine)

Program terminated with signal 11, Segmentation fault.
#0 0x00007fffdf1ebb48 in ?? ()

(gdb) bt
#0 0x00007fffdf1ebb48 in ?? ()
#1 0x00000000c0472d70 in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0xbc012fb0eb8ca760 in ?? ()
#4 0x00000000eb8ca760 in ?? ()
#5 0x00000000eb9430c8 in ?? ()
#6 0x00000000f8904d98 in ?? ()
#7 0x0000000000000003 in ?? ()
#8 0x00000000f8843b88 in ?? ()
#9 0x00007fffceddd7b0 in ?? ()
#10 0x00007fffdefae350 in ?? ()
#11 0x00007fffdefae350 in ?? ()
#12 0x00000000c0472d70 in ?? ()
#13 0x00007fffceddd770 in ?? ()
#14 0x00000000bc019b0f in ?? ()
#15 0x00007fffceddd7c8 in ?? ()
#16 0x00000000bc01e2f0 in ?? ()
#17 0x00000000bc6e6b08 in ?? ()
#18 0x00000000bc019bc8 in ?? ()
#19 0x00007fffceddd768 in ?? ()
#20 0x00007fffceddd7c8 in ?? ()
#21 0x00007fffceddd830 in ?? ()
#22 0x00007fffdefa84e7 in ?? ()
#23 0x0000000000000000 in ?? ()

(gdb) bt
#0 0x00007ffff527fa3e in __libc_sigaction () from /lib64/libc.so.6
#1 0x00007ffff20ffcac in PCodeCfg::layoutNativeCode (
    this=<value optimized out>, showplanSpace=0x0)
    at ../exp/ExpPCodeOptsNativeExpr.cpp:9350
#2 0x00007ffff20c1ec1 in PCodeCfg::optimize (this=0x7fffd15141c0)
    at ../exp/ExpPCodeOptimizations.cpp:2275
#3 0x00007ffff20a1b35 in ex_expr::pCodeGenerate (this=<value optimized out>,
    space=0x7fffd153e800, heap=0x7fffd063f660, f=0)
    at ../exp/ExpPCodeExpGen.cpp:975
#4 0x00007fffee24fea4 in ExpGenerator::endExprGen (this=0x7fffd1505668,
    expr=0x7fffe465dde0, gen_last_clause=0)
    at ../generator/GenExpGenerator.cpp:4772
#5 0x00007fffee251de7 in ExpGenerator::generateExpr (this=0x7fffd1505668,
    val_id=..., node_type=ex_expr::exp_SCAN_PRED, expr=0x7fffe465dde0)
    at ../generator/GenExpGenerator.cpp:2833
#6 0x00007fffee31109a in HbaseAccess::codeGen (this=0x7fffe465d680, generator=
    0x7fffe465f490) at ../generator/GenRelScan.cpp:2000
#7 0x00007fffee2b93bb in ProbeCache::codeGen (this=0x7fffd19d5310,
    generator=0x7fffe465f490) at ../generator/GenProbeCache.cpp:118
#8 0x00007fffee2e5800 in NestedJoin::codeGen (this=0x7fffd150c278,
    generator=0x7fffe465f490) at ../generator/GenRelJoin.cpp:3369
#9 0x00007fffee2fb645 in RelRoot::codeGen (this=0x7fffd19d6370,
generator=0x7fffe465f490) at ../generator/GenRelMisc.cpp:1379
#10 0x00007fffee24509e in Generator::genCode (this=0x7fffe465f490,
    source=0x7fffd0635438 "select trim(O.catalog_name || '.' || '\"' || O.schema_name || '\"' || '.' || '\"' || O.object_name || '\"' ) constr_name, trim(O2.catalog_name || '.' || '\"' || O2.schema_name || '\"' || '.' || '\"' || O2.ob"..., expr_node=0x7fffd19d6370) at ../generator/Generator.cpp:506
#11 0x00007ffff01c801a in CmpMain::compile (this=0x7fffe4662420,
    input_str=0x7fffd0635438 "select trim(O.catalog_name || '.' || '\"' || O.schema_name || '\"' || '.' || '\"' || O.object_name || '\"' ) constr_name, trim(O2.catalog_name || '.' || '\"' || O2.schema_name || '\"' || '.' || '\"' || O2.ob"..., charset=15, queryExpr=@0x7fffe4662358, gen_code=0x7fffd064fd58,
    gen_code_len=0x7fffd064fd50, heap=0x7fffe31a4828, phase=CmpMain::END,
    fragmentDir=0x7fffe4662578, op=3004, useQueryCache=<value optimized out>,
    cacheable=0x7fffe4662368, begTime=0x7fffe4662340, shouldLog=0)
    at ../sqlcomp/CmpMain.cpp:2269
#12 0x00007ffff01c96cc in CmpMain::sqlcomp (this=0x7fffe4662420,
    input_str=0x7fffd0635438 "select trim(O.catalog_name || '.' || '\"' || O.schema_name || '\"' || '.' || '\"' || O.object_name || '\"' ) constr_name, trim(O2.catalog_name || '.' || '\"' || O2.schema_name || '\"' || '.' || '\"' || O2.ob"..., charset=15, queryExpr=@0x7fffe4662358, gen_code=0x7fffd064fd58,
    gen_code_len=0x7fffd064fd50, heap=0x7fffe31a4828, phase=CmpMain::END,
    fragmentDir=0x7fffe4662578, op=3004, useQueryCache=1,
    cacheable=0x7fffe4662368, begTime=0x7fffe4662340, shouldLog=0)
at ../sqlcomp/CmpMain.cpp:1660
#13 0x00007ffff01ca8bc in CmpMain::sqlcomp (this=0x7fffe4662420, input=...,
    gen_code=0x7fffd064fd58, gen_code_len=0x7fffd064fd50, heap=0x7fffe31a4828,
    phase=CmpMain::END, fragmentDir=0x7fffe4662578, op=3004)
    at ../sqlcomp/CmpMain.cpp:809
#14 0x00007fffedfd28d8 in CmpStatement::process (this=0x7fffd061e120,
    sqltext=<value optimized out>) at ../arkcmp/CmpStatement.cpp:488
#15 0x00007fffedfc72de in CmpContext::compileDirect (this=0x7fffd0879090,
    data=0x7fffd02a0d80 "\200", data_len=696, outHeap=0x7fffe3c050f0,
    charset=15, op=CmpMessageObj::SQLTEXT_COMPILE, gen_code=@0x7fffe4662b00,
    gen_code_len=@0x7fffe4662b08, parserFlags=4358144, diagsArea=
    0x7fffd02a1040) at ../arkcmp/CmpContext.cpp:688
#16 0x00007ffff49c7a67 in CliStatement::prepare2 (this=0x7fffe31acc40,
    source=0x7fffd0414318 "select trim(O.catalog_name || '.' || '\"' || O.schema_name || '\"' || '.' || '\"' || O.object_name || '\"' ) constr_name, trim(O2.catalog_name || '.' || '\"' || O2.schema_name || '\"' || '.' || '\"' || O2.ob"..., diagsArea=..., passed_gen_code=<value optimized out>,
    passed_gen_code_len=3821031664, charset=15, unpackTdbs=1, cliFlags=144)
    at ../cli/Statement.cpp:1797
#17 0x00007ffff49c7eb6 in CliStatement::prepare (this=0x7fffe31acc40,
source=0x7fffd0414318 "select trim(O.catalog_name || '.' || '\"' || O.schema_name || '\"' || '.' || '\"' || O.object_name || '\"' ) constr_name, trim(O2.catalog_name || '.' || '\"' || O2.schema_name || '\"' || '.' || '\"' || O2.ob"..., diagsArea=..., passed_gen_code=<value optimized out>,
    passed_gen_code_len=<value optimized out>, charset=<value optimized out>,
    unpackTdbs=1, cliFlags=144) at ../cli/Statement.cpp:1388
#18 0x00007ffff49928cc in SQLCLI_Prepare2 (cliGlobals=0xf40b30,
    statement_id=0x37ae8f0, sql_source=0x37ae970, gencode_ptr=0x0,
    gencode_len=0, ret_gencode_len=0x0, query_cost_info=0x7fffe4663ed0,
    query_comp_stats_info=0x7fffe4662dd0, uniqueStmtId=<value optimized out>,
    uniqueStmtIdLen=0x0, flags=0) at ../cli/Cli.cpp:5910
#19 0x00007ffff49d2cf7 in SQL_EXEC_Prepare2 (statement_id=0x37ae8f0,
    sql_source=0x37ae970, gencode_ptr=0x0, gencode_len=0, ret_gencode_len=0x0,
    query_cost_info=0x7fffe4663ed0, comp_stats_info=0x7fffe4662dd0,
    uniqueStmtId=0x0, uniqueStmtIdLen=0x0, flags=0)
    at ../cli/CliExtern.cpp:4955
#20 0x00007ffff371c8cb in ExeCliInterface::prepare (this=0x7fffe46658c0,
    stmtStr=<value optimized out>, module=<value optimized out>,
    stmt=0x37ae8f0, sql_src=0x37ae970, input_desc=0x1046450,
    output_desc=0x1046490, outputBuf=0x7fffe46658f8, outputVarPtrList=0x0,
    inputBuf=0x7fffe4665930, inputVarPtrList=0x0, uniqueStmtId=0x0,
    uniqueStmtIdLen=0x0, query_cost_info=0x0, comp_stats_info=0x0,
    monitorThis=0) at ../executor/ExExeUtilCli.cpp:321
#21 0x00007ffff371e2ca in ExeCliInterface::fetchRowsPrologue (
    this=0x7fffe46658c0,
sqlStrBuf=0x7fffe4664260 "select trim(O.catalog_name || '.' || '\"' || O.schema_name || '\"' || '.' || '\"' || O.object_name || '\"' ) constr_name, trim(O2.
catalog_name || '.' || '\"' || O2.schema_name || '\"' || '.' || '\"' || O2.ob"..., noExec=0, monitorThis=0, stmtName=0x0) at ../executor/ExExeUtilCli.cpp:1020
#22 0x00007ffff371f396 in ExeCliInterface::fetchAllRows (this=0x7fffe46658c0,
    infoList=@0x7fffe4665cd0,
    query=0x7fffe4664260 "select trim(O.catalog_name || '.' || '\"' || O.schema_name || '\"' || '.' || '\"' || O.object_name || '\"' ) constr_name, trim(O2.catalog_name || '.' || '\"' || O2.schema_name || '\"' || '.' || '\"' || O2.ob"...,
    inNumOutputEntries=0, varcharFormat=0, monitorThis=<value optimized out>,
    initInfoList=1) at ../executor/ExExeUtilCli.cpp:1103
#23 0x00007ffff0212e18 in CmpSeabaseDDL::getSeabaseUserTableDesc (
    this=0x7fffe4665ea0, catName=..., schName=..., objName=...,
    objType=<value optimized out>, includeInvalidDefs=<value optimized out>)
    at ../sqlcomp/CmpSeabaseDDLtable.cpp:6332
#24 0x00007ffff0216629 in CmpSeabaseDDL::getSeabaseTableDesc (
    this=0x7fffe4665ea0, catName=..., schName=..., objName=...,
    objType=<value optimized out>, includeInvalidDefs=0)
    at ../sqlcomp/CmpSeabaseDDLtable.cpp:6611
#25 0x00007fffeebbb6d5 in NATableDB::get (this=0x7fffe2601918, corrName=...,
    bindWA=0x7fffe4669440, inTableDescStruct=0x0)
    at ../optimizer/NATable.cpp:7590
#26 0x00007fffee92bf8c in BindWA::getNATable (this=0x7fffe4669440,
    corrName=..., catmanCollectTableUsages=1, inTableDescStruct=0x0)
at ../optimizer/BindRelExpr.cpp:1446
#27 0x00007fffee93acc3 in GenericUpdate::bindNode (this=0x7fffd1cfbb00,
    bindWA=0x7fffe4669440) at ../optimizer/BindRelExpr.cpp:11792
#28 0x00007fffee950823 in Insert::bindNode (this=0x7fffd1cfbb00,
    bindWA=0x7fffe4669440) at ../optimizer/BindRelExpr.cpp:9121
#29 0x00007fffee91bdf7 in RelExpr::bindChildren (this=0x7fffd1cfc9b0,
    bindWA=0x7fffe4669440) at ../optimizer/BindRelExpr.cpp:2181
#30 0x00007fffee956b7e in RelRoot::bindNode (this=0x7fffd1cfc9b0,
    bindWA=0x7fffe4669440) at ../optimizer/BindRelExpr.cpp:5204
#31 0x00007ffff01c6fb3 in CmpMain::compile (this=0x7fffe466b730,
    input_str=0x7fffd1d00818 "upsert into ATABLE( ORGANIZATION_ID, ENTITY_ID, A_STRING, B_STRING, A_INTEGER, A_DATE, X_DECIMAL, X_LONG, X_INTEGER, Y_INTEGER, A_BYTE, A_SHORT, A_FLOAT,"...,
    charset=15, queryExpr=@0x7fffe466b668, gen_code=0x7fffd1d04870,
    gen_code_len=0x7fffd1d04868, heap=0x7fffe3193bf8, phase=CmpMain::END,
    fragmentDir=0x7fffe466b888, op=3004, useQueryCache=<value optimized out>,
    cacheable=0x7fffe466b678, begTime=0x7fffe466b650, shouldLog=0)
    at ../sqlcomp/CmpMain.cpp:2003
#32 0x00007ffff01c96cc in CmpMain::sqlcomp (this=0x7fffe466b730,
    input_str=0x7fffd1d00818 "upsert into ATABLE( ORGANIZATION_ID, ENTITY_ID, A_STRING, B_STRING, A_INTEGER, A_DATE, X_DECIMAL, X_LONG, X_INTEGER, Y_INTEGER, A_BYTE, A_SHORT, A_FLOAT,"...,
    charset=15, queryExpr=@0x7fffe466b668, gen_code=0x7fffd1d04870,
gen_code_len=0x7fffd1d04868, heap=0x7fffe3193bf8, phase=CmpMain::END,
    fragmentDir=0x7fffe466b888, op=3004, useQueryCache=1,
    cacheable=0x7fffe466b678, begTime=0x7fffe466b650, shouldLog=0)
    at ../sqlcomp/CmpMain.cpp:1660
#33 0x00007ffff01ca8bc in CmpMain::sqlcomp (this=0x7fffe466b730, input=...,
    gen_code=0x7fffd1d04870, gen_code_len=0x7fffd1d04868, heap=0x7fffe3193bf8,
    phase=CmpMain::END, fragmentDir=0x7fffe466b888, op=3004)
    at ../sqlcomp/CmpMain.cpp:809
#34 0x00007fffedfd28d8 in CmpStatement::process (this=0x7fffd1cadff8,
    sqltext=<value optimized out>) at ../arkcmp/CmpStatement.cpp:488
#35 0x00007fffedfc72de in CmpContext::compileDirect (this=0x7fffe25fc090,
    data=0x7fffe31a95f0 "\200", data_len=440, outHeap=0x7fffe3c050f0,
    charset=15, op=CmpMessageObj::SQLTEXT_COMPILE, gen_code=@0x7fffe466be10,
    gen_code_len=@0x7fffe466be18, parserFlags=4194304,
    diagsArea=0x7fffe31a97b0) at ../arkcmp/CmpContext.cpp:688
#36 0x00007ffff49c7a67 in CliStatement::prepare2 (this=0x7fffd0419fe8,
    source=0x7fffd044c9f0 "upsert into ATABLE( ORGANIZATION_ID, ENTITY_ID, A_STRING, B_STRING, A_INTEGER, A_DATE, X_DECIMAL, X_LONG, X_INTEGER, Y_INTEGER, A_BYTE, A_SHORT, A_FLOAT,"...,
    diagsArea=..., passed_gen_code=<value optimized out>,
    passed_gen_code_len=3821031664, charset=15, unpackTdbs=1, cliFlags=144)
    at ../cli/Statement.cpp:1797
#37 0x00007ffff49c7eb6 in CliStatement::prepare (this=0x7fffd0419fe8,
source=0x7fffd044c9f0 "upsert into ATABLE( ORGANIZATION_ID, ENTITY_ID
, A_STRING, B_STRING, A_INTEGER, A_DATE, X_DECIMAL, X_LONG, X_INTEGER, Y_INTEGER, A_BYTE, A_SHORT, A_FLOAT,"...,
    diagsArea=..., passed_gen_code=<value optimized out>,
    passed_gen_code_len=<value optimized out>, charset=<value optimized out>,
    unpackTdbs=1, cliFlags=144) at ../cli/Statement.cpp:1388
#38 0x00007ffff49928cc in SQLCLI_Prepare2 (cliGlobals=0xf40b30,
    statement_id=0x21501b8, sql_source=0x7fffe466c150, gencode_ptr=0x0,
    gencode_len=0, ret_gencode_len=0x0, query_cost_info=0x21503e8,
    query_comp_stats_info=0x2150438, uniqueStmtId=<value optimized out>,
    uniqueStmtIdLen=0x215031c, flags=0) at ../cli/Cli.cpp:5910
#39 0x00007ffff49d2cf7 in SQL_EXEC_Prepare2 (statement_id=0x21501b8,
    sql_source=0x7fffe466c150, gencode_ptr=0x0, gencode_len=0,
    ret_gencode_len=0x0, query_cost_info=0x21503e8, comp_stats_info=0x2150438,
    uniqueStmtId=0x2150320 "", uniqueStmtIdLen=0x215031c, flags=0)
    at ../cli/CliExtern.cpp:4955
#40 0x00007ffff6e7a8d9 in SRVR::WSQL_EXEC_Prepare2 (statement_id=0x21501b8,
    sql_source=<value optimized out>, gencode_ptr=<value optimized out>,
    gencode_len=<value optimized out>, ret_gencode_len=<value optimized out>,
    query_cost_info=<value optimized out>, comp_stats_info=0x2150438,
    uniqueQueryId=0x2150320 "", uniqueQueryIdLen=0x215031c, flags=0)
    at SQLWrapper.cpp:800
#41 0x00007ffff6e6f275 in SRVR::PREPARE2 (pSrvrStmt=0x214fba0,
isFromExecDirect=232) at sqlinterface.cpp:5016
#42 0x00007ffff6e9f670 in odbc_SQLSvc_Prepare2_sme_ (inputRowCnt=0,
    sqlStmtType=256, stmtLabel=<value optimized out>,
    sqlString=0x29f80e0 "upsert into ATABLE( ORGANIZATION_ID, ENTITY_ID, A_STRING, B_STRING, A_INTEGER, A_DATE, X_DECIMAL, X_LONG, X_INTEGER, Y_INTEGER, A_BYTE, A_SHORT, A_FLOAT,"...,
    holdableCursor=0, returnCode=0x7fffe466c93c,
    sqlWarningOrErrorLength=0x7fffe466c938, sqlWarningOrError=@0x7fffe466c920,
    sqlQueryType=0x7fffe466c934, stmtHandle=0x7fffe466c918,
    estimatedCost=0x7fffe466c930, inputDescLength=0x7fffe466c92c,
    inputDesc=@0x7fffe466c910, outputDescLength=0x7fffe466c928,
    outputDesc=@0x7fffe466c908, isFromExecDirect=false) at srvrothers.cpp:926
#43 0x00000000004c5210 in odbc_SQLSrvr_Prepare_ame_ (objtag_=0x37b1da0,
    call_id_=0x37b1df8, dialogueId=1320028801, sqlAsyncEnable=0,
    queryTimeout=0, stmtType=<value optimized out>, sqlStmtType=256,
    stmtLength=10, stmtLabel=0x29f80c6 "SQL_CUR_3", stmtLabelCharset=1,
    cursorLength=0, cursorName=0x0, cursorCharset=0, moduleNameLength=0,
    moduleName=0x0, moduleCharset=0, moduleTimestamp=0, sqlStringLength=312,
    sqlString=0x29f80e0 "upsert into ATABLE( ORGANIZATION_ID, ENTITY_ID, A_STRING, B_STRING, A_INTEGER, A_DATE, X_DECIMAL, X_LONG, X_INTEGER, Y_INTEGER, A_BYTE, A_SHORT, A_FLOAT,"...,
    sqlStringCharset=1, setStmtOptionsLength=0, setStmtOptions=0x0,
    stmtExplainLabelLength=0, stmtExplainLabel=0x0, maxRowsetSize=0, txnID=0,
extTransId=0x0, holdableCursor=0) at SrvrConnect.cpp:6265
#44 0x00000000004999e1 in SQLPREPARE_IOMessage (objtag_=0x37b1da0,
    call_id_=0x37b1df8) at Interface/odbcs_srvr.cpp:925
#45 0x00000000004a1b2c in DISPATCH_TCPIPRequest (objtag_=0x37b1da0,
    call_id_=0x37b1df8, operation_id=<value optimized out>)
    at Interface/odbcs_srvr.cpp:1793
#46 0x00000000004328e2 in BUILD_TCPIP_REQUEST (pnode=0x37b1da0)
    at ../Common/TCPIPSystemSrvr.cpp:603
#47 0x000000000043327d in PROCESS_TCPIP_REQUEST (pnode=0x37b1da0)
    at ../Common/TCPIPSystemSrvr.cpp:581
#48 0x00000000004709b6 in CNSKListenerSrvr::tcpip_listener (arg=0xde0f40)
    at Interface/linux/Listener_srvr_ps.cpp:397
#49 0x00007ffff50329f0 in sb_thread_sthr_disp (pp_arg=0xf13040)
    at threadl.cpp:253
#50 0x00007ffff4bf9851 in start_thread () from /lib64/libpthread.so.0
#51 0x00007ffff533590d in clone () from /lib64/libc.so.6

Tags: sql-exe
Changed in trafodion:
assignee: nobody → James Capps (james-capps)
Revision history for this message
Weishiun Tsai (wei-shiun-tsai) wrote :
Download full text (22.9 KiB)

On the v1105_0830 daily build, SQL regression (not phoenix tests) also ran into this problem with a core:

[trafodion@n001 501]$ sqid
MY_SQROOT=/opt/home/trafodion/v1105_0830
who@host=trafodion@amethyst-hn1
JAVA_HOME=/usr/java/jdk1.7.0_67
linux=2.6.32-279.el6.x86_64
redhat=6.3
Release 0.9.1 (Build release [0.9.0-71-g0235c0c_Bld59], branch 0235c0c-master, date 20141105_083000)

(gdb) info threads
  55 Thread 0x7fffcee35700 (LWP 29389) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  54 Thread 0x7fffd2341700 (LWP 29393) 0x00007ffff5335f03 in epoll_wait ()
   from /lib64/libc.so.6
  53 Thread 0x7fffd1e3c700 (LWP 29546) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  52 Thread 0x7fffd0eca700 (LWP 29721) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  51 Thread 0x7fffced34700 (LWP 29394) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  50 Thread 0x7fffd2644700 (LWP 29551) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  49 Thread 0x7fffceab3700 (LWP 29397) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  48 Thread 0x7fffd2442700 (LWP 29587) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  47 Thread 0x7fffd0641700 (LWP 29398) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  46 Thread 0x7fffd213f700 (LWP 29588) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  45 Thread 0x7fffd0742700 (LWP 29590) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  44 Thread 0x7fffcf137700 (LWP 29545) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  43 Thread 0x7fffe4684700 (LWP 615) 0x00007ffff2882e13 in llvm::Instruction::Instruction(llvm::Type*, unsigned int, llvm::Use*, unsigned int, llvm::Instruction*) () from /opt/home/trafodion/v1105_0830/export/lib64/libtdm_sqlexp.so
  42 Thread 0x7fffe980a700 (LWP 604) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  41 Thread 0x7fffce8b1700 (LWP 29972) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  40 Thread 0x7fffd3aaf700 (LWP 9386) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  39 Thread 0x7fffd203e700 (LWP 29401) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  38 Thread 0x7fffd0cc8700 (LWP 29591) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  37 Thread 0x7fffcec33700 (LWP 29387) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  36 Thread 0x7fffd3db2700 (LWP 9383) 0x00007ffff4bfd43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  35 Thread 0x7fffd17c3700 (LWP 30717) 0x00007ffff4bfd7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  34 Thread 0x7fffd19c5700 (LWP 306...

Revision history for this message
Weishiun Tsai (wei-shiun-tsai) wrote :
Download full text (14.3 KiB)

On the v1106_0830 daily build, SQL regression (not phoenix tests) also ran into this problem with a core:

#0 0x00007ffff2478f3f in llvm::MachineInstr::readsWritesVirtualRegister(unsigned int, llvm::SmallVectorImpl<unsigned int>*) const ()
   from /opt/home/trafodion/v1106_0830/export/lib64/libtdm_sqlexp.so
#1 0x00007ffff254c1da in llvm::VirtRegAuxInfo::CalculateWeightAndHint(llvm::LiveInterval&) ()
   from /opt/home/trafodion/v1106_0830/export/lib64/libtdm_sqlexp.so
#2 0x00007ffff254d091 in llvm::CalculateSpillWeights::runOnMachineFunction(llvm::MachineFunction&) ()
   from /opt/home/trafodion/v1106_0830/export/lib64/libtdm_sqlexp.so
#3 0x00007ffff28ac56d in llvm::FPPassManager::runOnFunction(llvm::Function&)
    () from /opt/home/trafodion/v1106_0830/export/lib64/libtdm_sqlexp.so
#4 0x00007ffff28ac71b in llvm::FunctionPassManagerImpl::run(llvm::Function&)
    () from /opt/home/trafodion/v1106_0830/export/lib64/libtdm_sqlexp.so
#5 0x00007ffff28aca2c in llvm::FunctionPassManager::run(llvm::Function&) ()
   from /opt/home/trafodion/v1106_0830/export/lib64/libtdm_sqlexp.so
#6 0x00007ffff23eb464 in llvm::JIT::jitTheFunction(llvm::Function*, llvm::MutexGuard const&) ()
   from /opt/home/trafodion/v1106_0830/export/lib64/libtdm_sqlexp.so
#7 0x00007ffff23eb77c in llvm::JIT::runJITOnFunctionUnlocked(llvm::Function*, llvm::MutexGuard const&) ()
   from /opt/home/trafodion/v1106_0830/export/lib64/libtdm_sqlexp.so
#8 0x00007ffff23ebb4a in llvm::JIT::runJITOnFunction(llvm::Function*, llvm::MachineCodeInfo*) ()
   from /opt/home/trafodion/v1106_0830/export/lib64/libtdm_sqlexp.so
#9 0x00007ffff20ff580 in PCodeCfg::layoutNativeCode (this=0x7fffcfc24d10,
    showplanSpace=0x0) at ../exp/ExpPCodeOptsNativeExpr.cpp:9122
#10 0x00007ffff20c1ec1 in PCodeCfg::optimize (this=0x7fffcfc24d10)
    at ../exp/ExpPCodeOptimizations.cpp:2275
#11 0x00007ffff20a1b35 in ex_expr::pCodeGenerate (this=<value optimized out>,
    space=0x7fffcfc24640, heap=0x7fffd0409658, f=0)
    at ../exp/ExpPCodeExpGen.cpp:975
#12 0x00007fffee24fea4 in ExpGenerator::endExprGen (this=0x7fffcfbf5610,
    expr=0x7fffe465ebe0, gen_last_clause=0)
    at ../generator/GenExpGenerator.cpp:4772
#13 0x00007fffee251de7 in ExpGenerator::generateExpr (this=0x7fffcfbf5610,
    val_id=..., node_type=ex_expr::exp_SCAN_PRED, expr=0x7fffe465ebe0)
    at ../generator/GenExpGenerator.cpp:2833
#14 0x00007fffee31109a in HbaseAccess::codeGen (this=0x7fffe465e480,
    generator=0x7fffe465ffe0) at ../generator/GenRelScan.cpp:2000
#15 0x00007fffee2f630b in Sort::codeGen (this=0x7fffcfbf2da0,
    generator=0x7fffe465ffe0) at ../generator/GenRelMisc.cpp:3551
#16 0x00007fffee2fb645 in RelRoot::codeGen (this=0x7fffcfbf1358,
    generator=0x7fffe465ffe0) at ../generator/GenRelMisc.cpp:1379
#17 0x00007fffee24509e in Generator::genCode (this=0x7fffe465ffe0,
    source=0x7fffd0177d80 "select column_name, column_number, column_class, fs_data_type, column_size, column_precision, column_scale, datetime_start_field, dat
etime_end_field, trim(is_upshifted), column_flags, nullable, trim(c"...,
    expr_node=0x7fffcfbf1358) at ../generator/Generator.cpp:506
#18 0x00007ffff01c801a in CmpMain::compile (this=0x7fffe4662f70,
 ...

Changed in trafodion:
status: New → In Progress
Revision history for this message
Weishiun Tsai (wei-shiun-tsai) wrote :

Development still does not have a good understanding of this problem. Lower the severity to High for now. We will continue to collect more info when we run into this problem.

Changed in trafodion:
importance: Critical → High
Revision history for this message
Weishiun Tsai (wei-shiun-tsai) wrote :
Download full text (15.2 KiB)

Ran into this core again on the v1119_0830 build again:

#0 0x00007ffff24694bd in llvm::MachineInstr::addOperand(llvm::MachineOperand const&) () from /opt/home/trafodion/v1119_0830/export/lib64/libtdm_sqlexp.so
#1 0x00007ffff238bea0 in llvm::InstrEmitter::AddRegisterOperand(llvm::MachineInstr*, llvm::SDValue, unsigned int, llvm::MCInstrDesc const*, llvm::DenseMap<llvm::SDValue, unsigned int, llvm::DenseMapInfo<llvm::SDValue> >&, bool, bool, bool) () from /opt/home/trafodion/v1119_0830/export/lib64/libtdm_sqlexp.so
#2 0x00007ffff238c70c in llvm::InstrEmitter::AddOperand(llvm::MachineInstr*, llvm::SDValue, unsigned int, llvm::MCInstrDesc const*, llvm::DenseMap<llvm::SDValue, unsigned int, llvm::DenseMapInfo<llvm::SDValue> >&, bool, bool, bool) ()
   from /opt/home/trafodion/v1119_0830/export/lib64/libtdm_sqlexp.so
#3 0x00007ffff238f9d1 in llvm::InstrEmitter::EmitMachineNode(llvm::SDNode*, bool, bool, llvm::DenseMap<llvm::SDValue, unsigned int, llvm::DenseMapInfo<llvm::SDValue> >&) () from /opt/home/trafodion/v1119_0830/export/lib64/libtdm_sqlexp.so
#4 0x00007ffff23855a3 in llvm::ScheduleDAGSDNodes::EmitSchedule(llvm::MachineBasicBlock::bundle_iterator<llvm::MachineInstr, llvm::ilist_iterator<llvm::MachineInstr> >&) () from /opt/home/trafodion/v1119_0830/export/lib64/libtdm_sqlexp.so
#5 0x00007ffff2300410 in llvm::SelectionDAGISel::CodeGenAndEmitDAG() ()
   from /opt/home/trafodion/v1119_0830/export/lib64/libtdm_sqlexp.so
#6 0x00007ffff2302869 in llvm::SelectionDAGISel::SelectBasicBlock(llvm::ilist_iterator<llvm::Instruction const>, llvm::ilist_iterator<llvm::Instruction const>, bool&) () from /opt/home/trafodion/v1119_0830/export/lib64/libtdm_sqlexp.so
#7 0x00007ffff2302fbe in llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&) ()
   from /opt/home/trafodion/v1119_0830/export/lib64/libtdm_sqlexp.so
#8 0x00007ffff2304407 in llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&) ()
   from /opt/home/trafodion/v1119_0830/export/lib64/libtdm_sqlexp.so
#9 0x00007ffff289e36d in llvm::FPPassManager::runOnFunction(llvm::Function&)
    () from /opt/home/trafodion/v1119_0830/export/lib64/libtdm_sqlexp.so
#10 0x00007ffff289e51b in llvm::FunctionPassManagerImpl::run(llvm::Function&)
    () from /opt/home/trafodion/v1119_0830/export/lib64/libtdm_sqlexp.so
#11 0x00007ffff289e82c in llvm::FunctionPassManager::run(llvm::Function&) ()
   from /opt/home/trafodion/v1119_0830/export/lib64/libtdm_sqlexp.so
#12 0x00007ffff23dd264 in llvm::JIT::jitTheFunction(llvm::Function*, llvm::MutexGuard const&) ()
   from /opt/home/trafodion/v1119_0830/export/lib64/libtdm_sqlexp.so
#13 0x00007ffff23dd57c in llvm::JIT::runJITOnFunctionUnlocked(llvm::Function*, llvm::MutexGuard const&) ()
   from /opt/home/trafodion/v1119_0830/export/lib64/libtdm_sqlexp.so
#14 0x00007ffff23dd94a in llvm::JIT::runJITOnFunction(llvm::Function*, llvm::MachineCodeInfo*) ()
   from /opt/home/trafodion/v1119_0830/export/lib64/libtdm_sqlexp.so
#15 0x00007ffff20f1380 in PCodeCfg::layoutNativeCode (this=0x7fffd1704d28,
    showplanSpace=0x0) at ../exp/ExpPCodeOptsNativeExpr.cpp:9122
#16 0x00007ffff20b3cc1 in PCodeCfg::optimize (this=0x7fffd1704d28)
    at ../exp/Exp...

Revision history for this message
Weishiun Tsai (wei-shiun-tsai) wrote :
Download full text (15.6 KiB)

Ran into this core again when running the full SQL regression tests on the v1202_0830 build.

#0 0x00007ffff52bd30c in _int_malloc () from /lib64/libc.so.6
#1 0x00007ffff52be911 in malloc () from /lib64/libc.so.6
#2 0x00007ffff2744438 in llvm::ValueSymbolTable::createValueName(llvm::StringRef, llvm::Value*) ()
   from /opt/home/trafodion/v1202_0830/export/lib64/libtdm_sqlexp.so
#3 0x00007ffff2740202 in llvm::Value::setName(llvm::Twine const&) ()
   from /opt/home/trafodion/v1202_0830/export/lib64/libtdm_sqlexp.so
#4 0x00007ffff2710b48 in llvm::BitCastInst::BitCastInst(llvm::Value*, llvm::Type*, llvm::Twine const&, llvm::Instruction*) ()
   from /opt/home/trafodion/v1202_0830/export/lib64/libtdm_sqlexp.so
#5 0x00007ffff2713331 in llvm::CastInst::Create(llvm::Instruction::CastOps, llvm::Value*, llvm::Type*, llvm::Twine const&, llvm::Instruction*) ()
   from /opt/home/trafodion/v1202_0830/export/lib64/libtdm_sqlexp.so
#6 0x00007ffff25f7bbb in llvm::SCEVExpander::ReuseOrCreateCast(llvm::Value*, llvm::Type*, llvm::Instruction::CastOps, llvm::ilist_iterator<llvm::Instruction>)
    () from /opt/home/trafodion/v1202_0830/export/lib64/libtdm_sqlexp.so
#7 0x00007ffff25f9690 in llvm::SCEVExpander::InsertNoopCastOfTo(llvm::Value*, llvm::Type*) ()
   from /opt/home/trafodion/v1202_0830/export/lib64/libtdm_sqlexp.so
#8 0x00007ffff249e582 in (anonymous namespace)::LSRInstance::Expand((anonymous namespace)::LSRFixup const&, (anonymous namespace)::Formula const&, llvm::ilist_iterator<llvm::Instruction>, llvm::SCEVExpander&, llvm::SmallVectorImpl<llvm::WeakVH>&) const ()
   from /opt/home/trafodion/v1202_0830/export/lib64/libtdm_sqlexp.so
#9 0x00007ffff249f50b in (anonymous namespace)::LSRInstance::ImplementSolution(llvm::SmallVectorImpl<(anonymous namespace)::Formula const*> const&, llvm::Pass*) () from /opt/home/trafodion/v1202_0830/export/lib64/libtdm_sqlexp.so
#10 0x00007ffff24a1337 in (anonymous namespace)::LSRInstance::LSRInstance(llvm::TargetLowering const*, llvm::Loop*, llvm::Pass*) ()
   from /opt/home/trafodion/v1202_0830/export/lib64/libtdm_sqlexp.so
#11 0x00007ffff24a169c in (anonymous namespace)::LoopStrengthReduce::runOnLoop(llvm::Loop*, llvm::LPPassManager&) ()
   from /opt/home/trafodion/v1202_0830/export/lib64/libtdm_sqlexp.so
#12 0x00007ffff25a857c in llvm::LPPassManager::runOnFunction(llvm::Function&)
    () from /opt/home/trafodion/v1202_0830/export/lib64/libtdm_sqlexp.so
#13 0x00007ffff272e59d in llvm::FPPassManager::runOnFunction(llvm::Function&)
    () from /opt/home/trafodion/v1202_0830/export/lib64/libtdm_sqlexp.so
#14 0x00007ffff272e74b in llvm::FunctionPassManagerImpl::run(llvm::Function&)
    () from /opt/home/trafodion/v1202_0830/export/lib64/libtdm_sqlexp.so
#15 0x00007ffff272ea5c in llvm::FunctionPassManager::run(llvm::Function&) ()
   from /opt/home/trafodion/v1202_0830/export/lib64/libtdm_sqlexp.so
#16 0x00007ffff226d494 in llvm::JIT::jitTheFunction(llvm::Function*, llvm::MutexGuard const&) ()
   from /opt/home/trafodion/v1202_0830/export/lib64/libtdm_sqlexp.so
#17 0x00007ffff226d7ac in llvm::JIT::runJITOnFunctionUnlocked(llvm::Function*, llvm::MutexGuard const&) ()
   from /opt/home/trafodion/v1202_0830/export/lib64/...

Changed in trafodion:
milestone: none → r1.1
Changed in trafodion:
status: In Progress → Fix Committed
Revision history for this message
James Capps (james-capps) wrote :

Changed the makefile for mxosrvr to link libjsig into mxosrvr ... early on in the linking process. The libjsig library comes as part of Java and is intended to allow other parts of your application to have signal handlers for SIGSEGV (and other signals that Java has handlers for) without interfering with Java's use of the same signals. The inclusion of libjsig into mxosrvr should resolve the intermittent problem reported in the LP bug.

Revision history for this message
Weishiun Tsai (wei-shiun-tsai) wrote :

We have been monitoring this problem all the way until the last build of R1.0 (rc3 build) and have not seen this core again. It's believed that the problem has been fixed by linking libjsig. We are closing this bug report.

Changed in trafodion:
milestone: r1.1 → r1.0
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.