mxosrvr cores at PCodeCfg::layoutNativeCode() during phoenix tests runs
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Trafodion |
Fix Released
|
High
|
James Capps |
Bug Description
When running phoenix tests on clusters, this problem has caused mxosrvr to core and the tests to fail a couple of times. This is seen with the v1101_0830 build on a Cloudera cluster, and with the v1104_0830 build on a HortonWorks cluster as well. The problem is caused by segmentation fault and cannot be reliably reproduced. But it does show up two days in a row.
The default threads of the cores do not show any useful info. But the threads handling the client requests show that they both are in the middle of handling PCodeCfg:
-------
Here is the core file core.1415043771
Program terminated with signal 11, Segmentation fault.
#0 0x00007fffdf2a5f34 in ?? ()
(gdb) info threads
43 Thread 0x7fffe6ee6700 (LWP 29699) 0x00007ffff528099d in sigtimedwait ()
from /lib64/libc.so.6
42 Thread 0x7fffd28ac700 (LWP 16641) 0x00007ffff4bfd7bb in pthread_
41 Thread 0x7ffff7fbb760 (LWP 29690) 0x00007ffff4bfd43c in pthread_
40 Thread 0x7fffe64e5700 (LWP 29700) 0x00007ffff532c253 in poll ()
from /lib64/libc.so.6
39 Thread 0x7fffd23a8700 (LWP 16645) 0x00007ffff4bfd7bb in pthread_
38 Thread 0x7fffde722700 (LWP 31966) 0x00007ffff4bfd43c in pthread_
37 Thread 0x7fffde520700 (LWP 31968) 0x00007ffff4bfd43c in pthread_
36 Thread 0x7fffe50e3700 (LWP 29702) 0x00007ffff527fd85 in sigwait ()
from /lib64/libc.so.6
35 Thread 0x7fffde823700 (LWP 31965) 0x00007ffff4bfd43c in pthread_
34 Thread 0x7fffe307d700 (LWP 31960) 0x00007ffff5335f03 in epoll_wait ()
from /lib64/libc.so.6
33 Thread 0x7fffd9f46700 (LWP 31970) 0x00007ffff4bfd43c in pthread_
32 Thread 0x7fffde924700 (LWP 31964) 0x00007ffff4bfd43c in pthread_
31 Thread 0x7fffdec27700 (LWP 31961) 0x00007ffff4bfd43c in pthread_
30 Thread 0x7fffd3eb3700 (LWP 31972) 0x00007ffff4bff720 in sem_wait ()
from /lib64/
29 Thread 0x7fffcfc1f700 (LWP 3063) 0x00007ffff4bfd7bb in pthread_
28 Thread 0x7fffe3c02700 (LWP 31959) 0x00007ffff52f9b8d in nanosleep ()
from /lib64/libc.so.6
27 Thread 0x7fffd3db2700 (LWP 31973) 0x00007ffff4bfd43c in pthread_
26 Thread 0x7fffe5ae4700 (LWP 29701) 0x00007ffff4bfd43c in pthread_
25 Thread 0x7fffd9e45700 (LWP 31971) 0x00007ffff4bfd43c in pthread_
24 Thread 0x7fffea20b700 (LWP 29697) 0x00007ffff53364cd in accept ()
from /lib64/libc.so.6
23 Thread 0x7fffe980a700 (LWP 29698) 0x00007ffff4bfd43c in pthread_
22 Thread 0x7fffdeb26700 (LWP 31962) 0x00007ffff4bfd43c in pthread_
21 Thread 0x7fffdea25700 (LWP 31963) 0x00007ffff4bfd43c in pthread_
20 Thread 0x7fffd0224700 (LWP 2880) 0x00007ffff4bfd7bb in pthread_
19 Thread 0x7fffcfd20700 (LWP 16495) 0x00007ffff4bfd7bb in pthread_
18 Thread 0x7fffd3bb0700 (LWP 31975) 0x00007ffff4bfd43c in pthread_
17 Thread 0x7fffd24a9700 (LWP 31984) 0x00007ffff532c253 in poll ()
from /lib64/libc.so.6
16 Thread 0x7fffd0a96700 (LWP 16640) 0x00007ffff4bfd7bb in pthread_
15 Thread 0x7fffcfe21700 (LWP 16638) 0x00007ffff4bfd7bb in pthread_
14 Thread 0x7fffd0d19700 (LWP 2855) 0x00007ffff4bfd7bb in pthread_
13 Thread 0x7fffd22a7700 (LWP 2852) 0x00007ffff5335f03 in epoll_wait ()
from /lib64/libc.so.6
12 Thread 0x7fffd3aaf700 (LWP 31976) 0x00007ffff4bfd7bb in pthread_
11 Thread 0x7fffd0b97700 (LWP 2876) 0x00007ffff4bfd7bb in pthread_
10 Thread 0x7fffd0e1a700 (LWP 2853) 0x00007ffff4bfd43c in pthread_
9 Thread 0x7fffd3cb1700 (LWP 31974) 0x00007ffff4bfd43c in pthread_
8 Thread 0x7fffd20a5700 (LWP 2877) 0x00007ffff4bfd7bb in pthread_
7 Thread 0x7fffd0f1b700 (LWP 2886) 0x00007ffff4bfd7bb in pthread_
6 Thread 0x7fffda047700 (LWP 31969) 0x00007ffff4bfd7bb in pthread_
5 Thread 0x7fffd101c700 (LWP 2890) 0x00007ffff4bfd7bb in pthread_
4 Thread 0x7fffd21a6700 (LWP 2897) 0x00007ffff4bfd7bb in pthread_
3 Thread 0x7fffde621700 (LWP 31967) 0x00007ffff4bfd43c in pthread_
2 Thread 0x7fffe4684700 (LWP 29718) 0x00007ffff2210965 in llvm::APInt:
from /opt/home/
* 1 Thread 0x7fffcff22700 (LWP 2902) 0x00007fffdf2a5f34 in ?? ()
(gdb) bt
#0 0x00007fffdf2a5f34 in ?? ()
#1 0x00000000c0472ce8 in ?? ()
#2 0x00000000ec50b4a8 in ?? ()
#3 0x00000000ec4e0b28 in ?? ()
#4 0x00000000ec50ab68 in ?? ()
#5 0x00000000c00ac260 in ?? ()
#6 0x00000000c035d210 in ?? ()
#7 0x00007fffcff0a740 in ?? ()
#8 0x00007fffdefae058 in ?? ()
#9 0x00007fffcff0a740 in ?? ()
#10 0x00007fffdefae233 in ?? ()
#11 0x00007fffdefae233 in ?? ()
#12 0x00000000c0472ce8 in ?? ()
#13 0x00007fffcff0a700 in ?? ()
#14 0x00000000bb6f0d72 in ?? ()
#15 0x00007fffcff0a798 in ?? ()
#16 0x00000000bb6f6c48 in ?? ()
#17 0x00000000bc78a810 in ?? ()
#18 0x00000000bb6f0eb8 in ?? ()
#19 0x00007fffcff0a6f8 in ?? ()
#20 0x00007fffcff0a790 in ?? ()
#21 0x00007fffcff0a7e0 in ?? ()
#22 0x00007fffdefae058 in ?? ()
#23 0x0000000000000000 in ?? ()
(gdb) bt
#0 0x00007ffff2210965 in llvm::APInt:
#1 0x00007ffff2785b10 in llvm::ComputeMa
from /opt/home/
#2 0x00007ffff26ac425 in llvm::getOrEnfo
from /opt/home/
#3 0x00007ffff2678b78 in llvm::InstCombi
() from /opt/home/
#4 0x00007ffff263ce7f in llvm::InstCombi
from /opt/home/
#5 0x00007ffff263db1f in llvm::InstCombi
from /opt/home/
#6 0x00007ffff28ac49d in llvm::FPPassMan
() from /opt/home/
#7 0x00007ffff28ac64b in llvm::FunctionP
() from /opt/home/
#8 0x00007ffff28ac95c in llvm::FunctionP
from /opt/home/
#9 0x00007ffff20ff3bf in PCodeCfg:
showplanSpace=0x0) at ../exp/
#10 0x00007ffff20c1df1 in PCodeCfg::optimize (this=0x7fffcff
at ../exp/
#11 0x00007ffff20a1a65 in ex_expr:
space=
at ../exp/
#12 0x00007fffee24fea4 in ExpGenerator:
expr=
at ../generator/
#13 0x00007fffee2535c3 in ExpGenerator:
val_
atp=<value optimized out>, atpIndex=<value optimized out>, hdrInfo=0x0)
at ../generator/
#14 0x00007fffee25b6f7 in ExpGenerator:
this=
tdataF=
moveExpr=
tdescF=
startOffset=0, bulkMoveSrcStar
colArray=0x0, doBulkMoves=0) at ../generator/
#15 0x00007fffee310c2f in HbaseAccess:
generator=
#16 0x00007fffee2f62cb in Sort::codeGen (this=0x7fffcff
generator=
#17 0x00007fffee2fb605 in RelRoot::codeGen (this=0x7fffcff
generator=
#18 0x00007fffee24509e in Generator::genCode (this=0x7fffe46
source=
expr_
#19 0x00007ffff01c801a in CmpMain::compile (this=0x7fffe46
input_
charset=15, queryExpr=
gen_
fragmentDir
cacheable=
at ../sqlcomp/
#20 0x00007ffff01c96cc in CmpMain::sqlcomp (this=0x7fffe46
input_
charset=15, queryExpr=
gen_
fragmentDir=
cacheable=
at ../sqlcomp/
#21 0x00007ffff01ca8bc in CmpMain::sqlcomp (this=0x7fffe46
gen_
phase=
at ../sqlcomp/
#22 0x00007fffedfd28d8 in CmpStatement:
sqltext=<value optimized out>) at ../arkcmp/
#23 0x00007fffedfc72de in CmpContext:
data=
charset=15, op=CmpMessageOb
gen_
diagsArea=
#24 0x00007ffff49c7a67 in CliStatement:
source=
diagsArea=..., passed_
passed_
at ../cli/
#25 0x00007ffff49c7eb6 in CliStatement:
source=
etime_end_field, trim(is_upshifted), column_flags, nullable, trim(c"...,
diagsArea=..., passed_
passed_
unpackTdbs=1, cliFlags=144) at ../cli/
#26 0x00007ffff49928cc in SQLCLI_Prepare2 (cliGlobals=
statement_
gencode_len=0, ret_gencode_
query_
uniqueStmtI
#27 0x00007ffff49d2cf7 in SQL_EXEC_Prepare2 (statement_
sql_
query_
uniqueStmtI
at ../cli/
#28 0x00007ffff371c85b in ExeCliInterface
stmtStr=<value optimized out>, module=<value optimized out>,
stmt=0x1e0a110, sql_src=0x1c90760, input_desc=
output_
inputBuf=
uniqueStmtI
monitorThis=0) at ../executor/
#29 0x00007ffff371e25a in ExeCliInterface
this=
sqlStrBuf=
noExec=0, monitorThis=0, stmtName=0x0) at ../executor/
#30 0x00007ffff371f326 in ExeCliInterface
infoList=
query=
inNumOutput
initInfoList=1) at ../executor/
#31 0x00007ffff01f8296 in CmpSeabaseDDL:
this=<value optimized out>, cliInterface=
objUID=
isTableSalt
outColInfoA
#32 0x00007ffff02103c9 in CmpSeabaseDDL:
this=
objType=<value optimized out>, includeInvalidD
at ../sqlcomp/
#33 0x00007ffff0216629 in CmpSeabaseDDL:
this=
objType=<value optimized out>, includeInvalidD
at ../sqlcomp/
#34 0x00007fffeebbb6d5 in NATableDB::get (this=0x7fffe26
bindWA=
at ../optimizer/
#35 0x00007fffee92bf8c in BindWA::getNATable (this=0x7fffe46
corrName=..., catmanCollectTa
at ../optimizer/
#36 0x00007fffee93acc3 in GenericUpdate:
bindWA=
#37 0x00007fffee950823 in Insert::bindNode (this=0x7fffd2d
bindWA=
#38 0x00007fffee91bdf7 in RelExpr:
bindWA=
#39 0x00007fffee956b7e in RelRoot::bindNode (this=0x7fffd2d
bindWA=
#40 0x00007ffff01c6fb3 in CmpMain::compile (this=0x7fffe46
input_
gen_
fragmentDir
cacheable=
at ../sqlcomp/
#41 0x00007ffff01c96cc in CmpMain::sqlcomp (this=0x7fffe46
input_
gen_
fragmentDir
cacheable=
at ../sqlcomp/
#42 0x00007ffff01ca8bc in CmpMain::sqlcomp (this=0x7fffe46
gen_
phase=
at ../sqlcomp/
#43 0x00007fffedfd28d8 in CmpStatement:
sqltext=<value optimized out>) at ../arkcmp/
#44 0x00007fffedfc72de in CmpContext:
data=
charset=15, op=CmpMessageOb
gen_
diagsArea=
#45 0x00007ffff49c7a67 in CliStatement:
source=
diagsArea=..., passed_
passed_
at ../cli/
#46 0x00007ffff49c7eb6 in CliStatement:
source=
diagsArea=..., passed_
passed_
unpackTdbs=1, cliFlags=144) at ../cli/
#47 0x00007ffff49928cc in SQLCLI_Prepare2 (cliGlobals=
statement_
gencode_len=0, ret_gencode_
query_
uniqueStmtI
#48 0x00007ffff49d2cf7 in SQL_EXEC_Prepare2 (statement_
sql_
ret_
uniqueStmtI
at ../cli/
#49 0x00007ffff6e7a8d9 in SRVR::WSQL_
sql_
gencode_
query_
uniqueQuery
at SQLWrapper.cpp:800
#50 0x00007ffff6e6f275 in SRVR::PREPARE2 (pSrvrStmt=
isFromExecD
#51 0x00007ffff6e9f670 in odbc_SQLSvc_
sqlStmtType
sqlString=
holdableCur
sqlWarningO
sqlQueryTyp
estimatedCo
inputDesc=
outputDesc=
#52 0x00000000004c5210 in odbc_SQLSrvr_
call_
queryTimeout=0, stmtType=<value optimized out>, sqlStmtType=256,
stmtLength=10, stmtLabel=0x1c20736 "SQL_CUR_3", stmtLabelCharset=1,
cursorLength=0, cursorName=0x0, cursorCharset=0, moduleNameLength=0,
moduleName=0x0, moduleCharset=0, moduleTimestamp=0, sqlStringLength=49,
sqlString=
sqlStringCh
stmtExplain
extTransId=0x0, holdableCursor=0) at SrvrConnect.
#53 0x00000000004999e1 in SQLPREPARE_
call_
#54 0x00000000004a1b2c in DISPATCH_
call_
at Interface/
#55 0x00000000004328e2 in BUILD_TCPIP_REQUEST (pnode=0x24d3800)
at ../Common/
#56 0x000000000043327d in PROCESS_
at ../Common/
#57 0x00000000004709b6 in CNSKListenerSrv
at Interface/
#58 0x00007ffff50329f0 in sb_thread_sthr_disp (pp_arg=0xf17550)
at threadl.cpp:253
#59 0x00007ffff4bf9851 in start_thread () from /lib64/
#60 0x00007ffff533590d in clone () from /lib64/libc.so.6
-------
Here is the core file core.1415121533
Program terminated with signal 11, Segmentation fault.
#0 0x00007fffdf1ebb48 in ?? ()
(gdb) bt
#0 0x00007fffdf1ebb48 in ?? ()
#1 0x00000000c0472d70 in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0xbc012fb0eb8ca760 in ?? ()
#4 0x00000000eb8ca760 in ?? ()
#5 0x00000000eb9430c8 in ?? ()
#6 0x00000000f8904d98 in ?? ()
#7 0x0000000000000003 in ?? ()
#8 0x00000000f8843b88 in ?? ()
#9 0x00007fffceddd7b0 in ?? ()
#10 0x00007fffdefae350 in ?? ()
#11 0x00007fffdefae350 in ?? ()
#12 0x00000000c0472d70 in ?? ()
#13 0x00007fffceddd770 in ?? ()
#14 0x00000000bc019b0f in ?? ()
#15 0x00007fffceddd7c8 in ?? ()
#16 0x00000000bc01e2f0 in ?? ()
#17 0x00000000bc6e6b08 in ?? ()
#18 0x00000000bc019bc8 in ?? ()
#19 0x00007fffceddd768 in ?? ()
#20 0x00007fffceddd7c8 in ?? ()
#21 0x00007fffceddd830 in ?? ()
#22 0x00007fffdefa84e7 in ?? ()
#23 0x0000000000000000 in ?? ()
(gdb) bt
#0 0x00007ffff527fa3e in __libc_sigaction () from /lib64/libc.so.6
#1 0x00007ffff20ffcac in PCodeCfg:
this=<value optimized out>, showplanSpace=0x0)
at ../exp/
#2 0x00007ffff20c1ec1 in PCodeCfg::optimize (this=0x7fffd15
at ../exp/
#3 0x00007ffff20a1b35 in ex_expr:
space=
at ../exp/
#4 0x00007fffee24fea4 in ExpGenerator:
expr=
at ../generator/
#5 0x00007fffee251de7 in ExpGenerator:
val_id=..., node_type=
at ../generator/
#6 0x00007fffee31109a in HbaseAccess:
0x7fffe465f490) at ../generator/
#7 0x00007fffee2b93bb in ProbeCache::codeGen (this=0x7fffd19
generator=
#8 0x00007fffee2e5800 in NestedJoin::codeGen (this=0x7fffd15
generator=
#9 0x00007fffee2fb645 in RelRoot::codeGen (this=0x7fffd19
generator=
#10 0x00007fffee24509e in Generator::genCode (this=0x7fffe46
source=
#11 0x00007ffff01c801a in CmpMain::compile (this=0x7fffe46
input_
gen_
fragmentDir
cacheable=
at ../sqlcomp/
#12 0x00007ffff01c96cc in CmpMain::sqlcomp (this=0x7fffe46
input_
gen_
fragmentDir
cacheable=
at ../sqlcomp/
#13 0x00007ffff01ca8bc in CmpMain::sqlcomp (this=0x7fffe46
gen_
phase=
at ../sqlcomp/
#14 0x00007fffedfd28d8 in CmpStatement:
sqltext=<value optimized out>) at ../arkcmp/
#15 0x00007fffedfc72de in CmpContext:
data=
charset=15, op=CmpMessageOb
gen_
0x7fffd02a1040) at ../arkcmp/
#16 0x00007ffff49c7a67 in CliStatement:
source=
passed_
at ../cli/
#17 0x00007ffff49c7eb6 in CliStatement:
source=
passed_
unpackTdbs=1, cliFlags=144) at ../cli/
#18 0x00007ffff49928cc in SQLCLI_Prepare2 (cliGlobals=
statement_
gencode_len=0, ret_gencode_
query_
uniqueStmtI
#19 0x00007ffff49d2cf7 in SQL_EXEC_Prepare2 (statement_
sql_
query_
uniqueStmtI
at ../cli/
#20 0x00007ffff371c8cb in ExeCliInterface
stmtStr=<value optimized out>, module=<value optimized out>,
stmt=0x37ae8f0, sql_src=0x37ae970, input_desc=
output_
inputBuf=
uniqueStmtI
monitorThis=0) at ../executor/
#21 0x00007ffff371e2ca in ExeCliInterface
this=
sqlStrBuf=
catalog_name || '.' || '\"' || O2.schema_name || '\"' || '.' || '\"' || O2.ob"..., noExec=0, monitorThis=0, stmtName=0x0) at ../executor/
#22 0x00007ffff371f396 in ExeCliInterface
infoList=
query=
inNumOutput
initInfoList=1) at ../executor/
#23 0x00007ffff0212e18 in CmpSeabaseDDL:
this=
objType=<value optimized out>, includeInvalidD
at ../sqlcomp/
#24 0x00007ffff0216629 in CmpSeabaseDDL:
this=
objType=<value optimized out>, includeInvalidD
at ../sqlcomp/
#25 0x00007fffeebbb6d5 in NATableDB::get (this=0x7fffe26
bindWA=
at ../optimizer/
#26 0x00007fffee92bf8c in BindWA::getNATable (this=0x7fffe46
corrName=..., catmanCollectTa
at ../optimizer/
#27 0x00007fffee93acc3 in GenericUpdate:
bindWA=
#28 0x00007fffee950823 in Insert::bindNode (this=0x7fffd1c
bindWA=
#29 0x00007fffee91bdf7 in RelExpr:
bindWA=
#30 0x00007fffee956b7e in RelRoot::bindNode (this=0x7fffd1c
bindWA=
#31 0x00007ffff01c6fb3 in CmpMain::compile (this=0x7fffe46
input_
charset=15, queryExpr=
gen_
fragmentDir
cacheable=
at ../sqlcomp/
#32 0x00007ffff01c96cc in CmpMain::sqlcomp (this=0x7fffe46
input_
charset=15, queryExpr=
gen_code_
fragmentDir
cacheable=
at ../sqlcomp/
#33 0x00007ffff01ca8bc in CmpMain::sqlcomp (this=0x7fffe46
gen_
phase=
at ../sqlcomp/
#34 0x00007fffedfd28d8 in CmpStatement:
sqltext=<value optimized out>) at ../arkcmp/
#35 0x00007fffedfc72de in CmpContext:
data=
charset=15, op=CmpMessageOb
gen_
diagsArea=
#36 0x00007ffff49c7a67 in CliStatement:
source=
diagsArea=..., passed_
passed_
at ../cli/
#37 0x00007ffff49c7eb6 in CliStatement:
source=
, A_STRING, B_STRING, A_INTEGER, A_DATE, X_DECIMAL, X_LONG, X_INTEGER, Y_INTEGER, A_BYTE, A_SHORT, A_FLOAT,"...,
diagsArea=..., passed_
passed_
unpackTdbs=1, cliFlags=144) at ../cli/
#38 0x00007ffff49928cc in SQLCLI_Prepare2 (cliGlobals=
statement_
gencode_len=0, ret_gencode_
query_
uniqueStmtI
#39 0x00007ffff49d2cf7 in SQL_EXEC_Prepare2 (statement_
sql_
ret_
uniqueStmtI
at ../cli/
#40 0x00007ffff6e7a8d9 in SRVR::WSQL_
sql_
gencode_
query_
uniqueQuery
at SQLWrapper.cpp:800
#41 0x00007ffff6e6f275 in SRVR::PREPARE2 (pSrvrStmt=
isFromExecDirec
#42 0x00007ffff6e9f670 in odbc_SQLSvc_
sqlStmtType
sqlString=
holdableCur
sqlWarningO
sqlQueryTyp
estimatedCo
inputDesc=
outputDesc=
#43 0x00000000004c5210 in odbc_SQLSrvr_
call_
queryTimeout=0, stmtType=<value optimized out>, sqlStmtType=256,
stmtLength=10, stmtLabel=0x29f80c6 "SQL_CUR_3", stmtLabelCharset=1,
cursorLength=0, cursorName=0x0, cursorCharset=0, moduleNameLength=0,
moduleName=0x0, moduleCharset=0, moduleTimestamp=0, sqlStringLength
sqlString=
sqlStringCh
stmtExplain
extTransId=0x0, holdableCursor=0) at SrvrConnect.
#44 0x00000000004999e1 in SQLPREPARE_
call_
#45 0x00000000004a1b2c in DISPATCH_
call_
at Interface/
#46 0x00000000004328e2 in BUILD_TCPIP_REQUEST (pnode=0x37b1da0)
at ../Common/
#47 0x000000000043327d in PROCESS_
at ../Common/
#48 0x00000000004709b6 in CNSKListenerSrv
at Interface/
#49 0x00007ffff50329f0 in sb_thread_sthr_disp (pp_arg=0xf13040)
at threadl.cpp:253
#50 0x00007ffff4bf9851 in start_thread () from /lib64/
#51 0x00007ffff533590d in clone () from /lib64/libc.so.6
Changed in trafodion: | |
assignee: | nobody → James Capps (james-capps) |
Changed in trafodion: | |
status: | New → In Progress |
Changed in trafodion: | |
milestone: | none → r1.1 |
Changed in trafodion: | |
status: | In Progress → Fix Committed |
On the v1105_0830 daily build, SQL regression (not phoenix tests) also ran into this problem with a core:
[trafodion@n001 501]$ sqid /opt/home/ trafodion/ v1105_0830 trafodion@ amethyst- hn1 /usr/java/ jdk1.7. 0_67 6.32-279. el6.x86_ 64 71-g0235c0c_ Bld59], branch 0235c0c-master, date 20141105_083000)
MY_SQROOT=
who@host=
JAVA_HOME=
linux=2.
redhat=6.3
Release 0.9.1 (Build release [0.9.0-
(gdb) info threads cond_timedwait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0 cond_timedwait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0 cond_timedwait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0 cond_wait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0 cond_timedwait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0 cond_timedwait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0 cond_timedwait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0 cond_timedwait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0 cond_timedwait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0 cond_timedwait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0 cond_timedwait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0 on::Instruction (llvm:: Type*, unsigned int, llvm::Use*, unsigned int, llvm::Instruction*) () from /opt/home/ trafodion/ v1105_0830/ export/ lib64/libtdm_ sqlexp. so cond_wait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0 cond_timedwait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0 cond_timedwait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0 cond_timedwait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0 cond_timedwait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0 cond_timedwait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0 cond_wait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0 cond_timedwait@ @GLIBC_ 2.3.2 () from /lib64/ libpthread. so.0
55 Thread 0x7fffcee35700 (LWP 29389) 0x00007ffff4bfd7bb in pthread_
54 Thread 0x7fffd2341700 (LWP 29393) 0x00007ffff5335f03 in epoll_wait ()
from /lib64/libc.so.6
53 Thread 0x7fffd1e3c700 (LWP 29546) 0x00007ffff4bfd7bb in pthread_
52 Thread 0x7fffd0eca700 (LWP 29721) 0x00007ffff4bfd7bb in pthread_
51 Thread 0x7fffced34700 (LWP 29394) 0x00007ffff4bfd43c in pthread_
50 Thread 0x7fffd2644700 (LWP 29551) 0x00007ffff4bfd7bb in pthread_
49 Thread 0x7fffceab3700 (LWP 29397) 0x00007ffff4bfd7bb in pthread_
48 Thread 0x7fffd2442700 (LWP 29587) 0x00007ffff4bfd7bb in pthread_
47 Thread 0x7fffd0641700 (LWP 29398) 0x00007ffff4bfd7bb in pthread_
46 Thread 0x7fffd213f700 (LWP 29588) 0x00007ffff4bfd7bb in pthread_
45 Thread 0x7fffd0742700 (LWP 29590) 0x00007ffff4bfd7bb in pthread_
44 Thread 0x7fffcf137700 (LWP 29545) 0x00007ffff4bfd7bb in pthread_
43 Thread 0x7fffe4684700 (LWP 615) 0x00007ffff2882e13 in llvm::Instructi
42 Thread 0x7fffe980a700 (LWP 604) 0x00007ffff4bfd43c in pthread_
41 Thread 0x7fffce8b1700 (LWP 29972) 0x00007ffff4bfd7bb in pthread_
40 Thread 0x7fffd3aaf700 (LWP 9386) 0x00007ffff4bfd7bb in pthread_
39 Thread 0x7fffd203e700 (LWP 29401) 0x00007ffff4bfd7bb in pthread_
38 Thread 0x7fffd0cc8700 (LWP 29591) 0x00007ffff4bfd7bb in pthread_
37 Thread 0x7fffcec33700 (LWP 29387) 0x00007ffff4bfd7bb in pthread_
36 Thread 0x7fffd3db2700 (LWP 9383) 0x00007ffff4bfd43c in pthread_
35 Thread 0x7fffd17c3700 (LWP 30717) 0x00007ffff4bfd7bb in pthread_
34 Thread 0x7fffd19c5700 (LWP 306...