Explain queries cause mxosrvr to hang at ex_root_tcb::fetchMultiple()

[Weishiun Tsai]

Certain explain queries have caused mxosrvrs to hang at ex_root_tcb::fetchMultiple(). When this happens, killing the client trafci connection does not release the mxosrvr either. The mxosrvr continues to stay in the same looping state and can’t be reused.

This is seen on the v0320 build installed on a 6-node cluster. The initial analysis is that it has something to do with the CQD explain_space_opt, which is on by default. An experiment to turn it off does seem to get the same test that was stuck previously going again.

This is a test stopper for QA as a large number of QA SQL regression tests use explain queries to verify the plan quality. For now, we will try to set this CQD in the defaults table to turn it off manually. But the suggestion is to turn it off from the code in the next build until the underlying issue is figured out.

One of the tests that encountered this problem and got mxosrvr to hang for 20+ hours was the test in compiler/explain: tcase.test152. The explain query is shown here as 'explainit'. The actually query is a query that uses QA's tpch2x tables. This problem shows up only as a combination of the actual query plan and the explainit query, as the same set of tests have executed explainit earlier with other queries and they did not see this problem:

>>set schema g_tpch2x;

--- SQL operation complete.
>>showddl lineitem;

CREATE TABLE TRAFODION.G_TPCH2X.LINEITEM
  (
    L_ORDERKEY INT NO DEFAULT NOT NULL NOT DROPPABLE
  , L_PARTKEY INT NO DEFAULT NOT NULL NOT DROPPABLE
  , L_SUPPKEY INT NO DEFAULT NOT NULL NOT DROPPABLE
  , L_LINENUMBER INT NO DEFAULT NOT NULL NOT DROPPABLE
  , L_QUANTITY NUMERIC(12, 2) NO DEFAULT NOT NULL NOT
      DROPPABLE
  , L_EXTENDEDPRICE NUMERIC(12, 2) NO DEFAULT NOT NULL NOT
      DROPPABLE
  , L_DISCOUNT NUMERIC(12, 2) NO DEFAULT NOT NULL NOT
      DROPPABLE
  , L_TAX NUMERIC(12, 2) NO DEFAULT NOT NULL NOT
      DROPPABLE
  , L_RETURNFLAG CHAR(1) CHARACTER SET ISO88591 COLLATE
      DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
  , L_LINESTATUS CHAR(1) CHARACTER SET ISO88591 COLLATE
      DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
  , L_SHIPDATE DATE NO DEFAULT NOT NULL NOT DROPPABLE
  , L_COMMITDATE DATE NO DEFAULT NOT NULL NOT DROPPABLE
  , L_RECEIPTDATE DATE NO DEFAULT NOT NULL NOT DROPPABLE
  , L_SHIPINSTRUCT CHAR(25) CHARACTER SET ISO88591 COLLATE
      DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
  , L_SHIPMODE CHAR(10) CHARACTER SET ISO88591 COLLATE
      DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
  , L_COMMENT VARCHAR(44) CHARACTER SET ISO88591 COLLATE
      DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
  , PRIMARY KEY (L_SHIPDATE ASC, L_ORDERKEY ASC, L_LINENUMBER ASC)
  )
  SALT USING 8 PARTITIONS
;

--- SQL operation complete.
>>select count(*) from lineitem;

(EXPR)
--------------------

            11997996

--- 1 row(s) selected.

prepare explainIt from
select substring(cast(SEQ_NUM+100 as char(3)),2,2) s,
cast(LEFT_CHILD_SEQ_NUM as char(2)) lc,
cast(RIGHT_CHILD_SEQ_NUM as char(2)) rc,
substring(operator,1,16) operator,
substring
(substring(tname from (1+locate('.',tname))),
(1+locate('.',substring(tname from (1+locate('.',tname))))),
10
) tab_name,
cast(cardinality as char(11)) cardinal,
cast(operator_cost as char(11)) op_cost,
cast(total_cost as char(11)) tot_cost,
cast(detail_cost as char(256)) detail_cost -- This is the one that shows all the individual cost components in detail.
from table (explain(NULL,'XX'))
order by 1 desc;

prepare xx from
select [last 0]
l_returnflag,
l_linestatus,
sum(l_quantity) over (order by l_quantity
rows between unbounded preceding and 10 preceding ) as sum_qty ,
sum(l_extendedprice) over (order by l_quantity
rows between unbounded preceding and current row) as sum_base_price ,
sum(l_extendedprice * (1 - l_discount)) over (order by l_quantity
rows between unbounded preceding and unbounded following ) as sum_disc_price,
sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) over (order by l_quantity
rows between current row and current row ) as sum_charge,
avg(l_quantity) over (order by l_quantity
rows between current row and unbounded following ) as avg_qty,
avg(l_extendedprice) over (order by l_quantity
rows between current row and 5 following)as avg_price,
avg(l_discount) over (order by l_quantity
rows between 6 preceding and 4 preceding ) as avg_disc,
count(*) as count_order
from
lineitem
where
l_shipdate <= date '1998-12-01' - interval '72' day(2)
group by
l_returnflag,
l_linestatus,
l_quantity,
l_discount,
l_extendedprice,
l_tax
order by
l_returnflag,
l_linestatus;

execute explainit;

---------------------------------------------

Here is the pstack of the mxosrvr that got stuck. Thread 34 is the one handling the request:

Thread 40 (Thread 0x7fffead93700 (LWP 12490)):
#0 0x00007ffff4af64bd in accept () from /lib64/libc.so.6
#1 0x00007ffff73639b5 in SB_Trans::Sock_Listener::accept (this=0xeaa950) at soc
k.cpp:618
#2 0x00007ffff736a506 in SB_Trans::Sock_Stream_Accept_Thread::run (this=0xeab8f
0) at sockstream.cpp:2085
#3 0x00007ffff736a233 in sock_stream_accept_thread_fun (pp_arg=0xeab8f0) at soc
kstream.cpp:2019
#4 0x00007ffff47efb9f in SB_Thread::Thread::disp (this=0xeab8f0, pp_arg=0xeab8f
0) at thread.cpp:211
#5 0x00007ffff47efff7 in thread_fun (pp_arg=0xeab8f0) at thread.cpp:307
#6 0x00007ffff47f3290 in sb_thread_sthr_disp (pp_arg=0xeaba20) at threadl.cpp:2
53
#7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 39 (Thread 0x7fffea392700 (LWP 12491)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff47f2656 in SB_Thread::CV::wait (this=0xeac288) at /home/jenkins/w
orkspace/traf-pub-release-ahw2.2/trafodion/core/sqf/export/include/seabed/int/th
read.inl:552
#2 0x00007ffff47f2732 in SB_Thread::CV::wait (this=0xeac288, pv_lock=false) at
/home/jenkins/workspace/traf-pub-release-ahw2.2/trafodion/core/sqf/export/includ
e/seabed/int/thread.inl:591
#3 0x00007ffff735cc67 in SB_Sig_Queue::remove (this=0xeac230) at queue.inl:473
#4 0x00007ffff736a997 in SB_Trans::Sock_Stream_Helper_Thread::run (this=0xeabce
0) at sockstream.cpp:2179
#5 0x00007ffff736a25a in sock_helper_thread_fun (pp_arg=0xeabce0) at sockstream
.cpp:2029
#6 0x00007ffff47efb9f in SB_Thread::Thread::disp (this=0xeabce0, pp_arg=0xeabce
0) at thread.cpp:211
#7 0x00007ffff47efff7 in thread_fun (pp_arg=0xeabce0) at thread.cpp:307
#8 0x00007ffff47f3290 in sb_thread_sthr_disp (pp_arg=0xeac3f0) at threadl.cpp:2
53
#9 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 38 (Thread 0x7fffe7a6e700 (LWP 12494)):
#0 0x00007ffff4a4071d in sigtimedwait () from /lib64/libc.so.6
#1 0x00007ffff73285e4 in local_monitor_reader (pp_arg=0x18c1) at ../../../monit
or/linux/clio.cxx:130
#2 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#3 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 37 (Thread 0x7fffe706d700 (LWP 12495)):
#0 0x00007ffff4aec0d3 in poll () from /lib64/libc.so.6
#1 0x00007ffff63c4482 in do_io (v=0xeaeec0) at src/mt_adaptor.c:387
#2 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#3 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 36 (Thread 0x7fffe666c700 (LWP 12496)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff63c426b in do_completion (v=0xeaeec0) at src/mt_adaptor.c:463
#2 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#3 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7fffe5c6b700 (LWP 12513)):
#0 0x00007ffff4a3fb05 in sigwait () from /lib64/libc.so.6
#1 0x00007ffff7371abf in SB_Timer_Thread::run (this=0xeb6f30) at timer.cpp:270
#2 0x00007ffff73718c7 in sb_timer_thread_fun (pp_arg=0xeb6f30) at timer.cpp:233
#3 0x00007ffff47efb9f in SB_Thread::Thread::disp (this=0xeb6f30, pp_arg=0xeb6f3
0) at thread.cpp:211
#4 0x00007ffff47efff7 in thread_fun (pp_arg=0xeb6f30) at thread.cpp:307
#5 0x00007ffff47f3290 in sb_thread_sthr_disp (pp_arg=0xeb0380) at threadl.cpp:2
53
#6 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#7 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7fffe520c700 (LWP 12514)):
#0 0x00007ffff45c398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/lib
pthread.so.0
#1 0x00007ffff47f27ec in SB_Thread::CV::wait (this=0xed6688, pv_sec=60, pv_us=2
00000) at /home/jenkins/workspace/traf-pub-release-ahw2.2/trafodion/core/sqf/exp
ort/include/seabed/int/thread.inl:649
#2 0x00007ffff47f28b1 in SB_Thread::CV::wait (this=0xed6688, pv_lock=true, pv_s
ec=60, pv_us=200000) at /home/jenkins/workspace/traf-pub-release-ahw2.2/trafodio
n/core/sqf/export/include/seabed/int/thread.inl:701
#3 0x00007ffff733a8eb in SB_Ms_Event_Mgr::wait (this=0xed6590, pv_us=60200000)
at mseventmgr.inl:351
#4 0x00007ffff735b18e in XWAIT_com (pv_mask=1024, pv_time=6020, pv_residual=fal
se) at pctl.cpp:979
#5 0x00007ffff735af8f in XWAITNO0 (pv_mask=1024, pv_time=6020) at pctl.cpp:902
#6 0x00007ffff3f74daa in IpcSetOfConnections::waitOnSet (this=0x7fffe3c07268, t
imeout=-1, calledByESP=0, timedout=0x7fffe51f3ff8) at ../common/Ipc.cpp:1593
#7 0x00007ffff2d5af24 in waitOnAll (waitTime=<optimized out>, timedout=0x7fffe5
1f3ff8, calledByESP=0, timeout=-1, this=0x7fffe3c07200) at ../common/Ipc.h:3089
#8 ex_root_tcb::fetch (this=0x7fffe3d379f8, cliGlobals=<optimized out>, glob=0x
7fffe3d1dfd8, output_desc=0x7fffe3d4f808, diagsArea=@0x7fffe51f4180: 0x0, timeLi
mit=-1, newOperation=1, closeCursorOnError=@0x7fffe51f418c: 1) at ../executor/ex
_root.cpp:1934
#9 0x00007ffff2d5c46a in ex_root_tcb::fetchMultiple (this=0x7fffe3d379f8, cliGl
obals=0xee53d0, glob=0x7fffe3d1dfd8, output_desc=0x7fffe3d4f808, diagsArea=@0x7f
ffe51f4180: 0x0, timeLimit=-1, newOperation=1, closeCursorOnError=@0x7fffe51f418
c: 1, eodSeen=@0x7fffe51f4188: 0) at ../executor/ex_root.cpp:2010
#10 0x00007ffff437709a in CliStatement::fetch (this=0x7fffe3d33258, cliGlobals=0
xee53d0, output_desc=0x7fffe3d4f808, diagsArea=..., newOperation=1) at ../cli/St
atement.cpp:4768
#11 0x00007ffff431ddff in SQLCLI_PerformTasks(CliGlobals *, ULng32, SQLSTMT_ID *
, SQLDESC_ID *, SQLDESC_ID *, Lng32, Lng32, typedef __va_list_tag __va_list_tag
*, SQLCLI_PTR_PAIRS *, SQLCLI_PTR_PAIRS *) (cliGlobals=0xee53d0, tasks=5924, sta
tement_id=0x6a3c8a8, input_descriptor=0x0, output_descriptor=0x6a3c908, num_inpu
t_ptr_pairs=-472582136, num_output_ptr_pairs=0, ap=0x7fffe51f4320, input_ptr_pai
rs=0x0, output_ptr_pairs=0x0) at ../cli/Cli.cpp:3383
#12 0x00007ffff431e81d in SQLCLI_Fetch(CliGlobals *, SQLSTMT_ID *, SQLDESC_ID *,
Lng32, typedef __va_list_tag __va_list_tag *, SQLCLI_PTR_PAIRS *) (cliGlobals=<
optimized out>, statement_id=<optimized out>, output_descriptor=<optimized out>,
num_ptr_pairs=<optimized out>, ap=<optimized out>, ptr_pairs=<optimized out>) a
t ../cli/Cli.cpp:3821
#13 0x00007ffff43833cb in SQL_EXEC_Fetch (statement_id=0x6a3c8a8, output_descrip
tor=0x6a3c908, num_ptr_pairs=0) at ../cli/CliExtern.cpp:2737
#14 0x00007ffff6870e01 in SRVR::WSQL_EXEC_Fetch (statement_id=0x6a3c8a8, output_
descriptor=0x6a3c908, num_ptr_pairs=0) at SQLWrapper.cpp:514
#15 0x00007ffff685f64a in SRVR::FETCH2bulk (pSrvrStmt=0x6a3c290) at sqlinterface
.cpp:6154
#16 0x00007ffff688dbea in odbc_SQLSrvr_FetchPerf_sme_ (objtag_=<optimized out>,
call_id_=<optimized out>, returnCode=0x7fffe51f467c, dialogueId=<optimized out>,
stmtLabel=0x6585cc4 "EXPLAINIT", maxRowCnt=100, maxRowLen=0, sqlAsyncEnable=0,
queryTimeout=0, rowsAffected=0x7fffe51f4678, outValuesFormat=0x7fffe51f4674, out
putDataValue=0x7fffe51f4650, sqlWarningOrErrorLength=0x7fffe51f4670, sqlWarningO
rError=@0x7fffe51f4668: 0x0) at srvrothers.cpp:5815
#17 0x00000000004b6e07 in odbc_SQLSrvr_Fetch_ame_ (objtag_=0x69701c0, call_id_=0
x6970218, dialogueId=468739187, operation_id=<optimized out>, sqlAsyncEnable=0,
queryTimeout=0, stmtHandle=0, stmtLabel=0x6585cc4 "EXPLAINIT", maxRowCnt=100, ma
xRowLen=0) at SrvrConnect.cpp:7498
#18 0x000000000048bd52 in SQLFETCH_IOMessage (objtag_=0x69701c0, call_id_=0x6970
218, operation_id=3009) at Interface/odbcs_srvr.cpp:1021
#19 0x000000000049427e in DISPATCH_TCPIPRequest (objtag_=0x69701c0, call_id_=0x6
970218, operation_id=<optimized out>) at Interface/odbcs_srvr.cpp:1803
#20 0x0000000000433872 in BUILD_TCPIP_REQUEST (pnode=0x69701c0) at ../Common/TCP
IPSystemSrvr.cpp:603
#21 0x000000000043420d in PROCESS_TCPIP_REQUEST (pnode=0x69701c0) at ../Common/T
CPIPSystemSrvr.cpp:581
#22 0x00000000004623f6 in CNSKListenerSrvr::tcpip_listener (arg=0xda0e50) at Int
erface/linux/Listener_srvr_ps.cpp:400
#23 0x00007ffff47f3290 in sb_thread_sthr_disp (pp_arg=0xed1b80) at threadl.cpp:2
53
#24 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#25 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7fffe478a700 (LWP 23365)):
#0 0x00007ffff4ab9a3d in nanosleep () from /lib64/libc.so.6
#1 0x00007ffff6ee32d9 in Sleep (milliSecs=<optimized out>) at traf_misc.cpp:136
#2 0x00007ffff435bd37 in memMonitorUpdateThread (param=0x7fffe478caf0) at ../cl
i/memorymonitor.cpp:73
#3 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7fffe3c05700 (LWP 23366)):
#0 0x00007ffff4af5ef3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007ffff7362cf5 in SB_Trans::Sock_Controller::epoll_wait (this=0x7ffff75b
3ca0 <gv_sock_ctlr>, pp_where=0x7ffff7385c08 "Sock_Comp_Thread::run", pv_timeout
=-1) at sock.cpp:336
#2 0x00007ffff7362213 in SB_Trans::Sock_Comp_Thread::run (this=0xee7ad0) at soc
k.cpp:104
#3 0x00007ffff736204d in sock_comp_thread_fun (pp_arg=0xee7ad0) at sock.cpp:74
#4 0x00007ffff47efb9f in SB_Thread::Thread::disp (this=0xee7ad0, pp_arg=0xee7ad
0) at thread.cpp:211
#5 0x00007ffff47efff7 in thread_fun (pp_arg=0xee7ad0) at thread.cpp:307
#6 0x00007ffff47f3290 in sb_thread_sthr_disp (pp_arg=0xee7bf0) at threadl.cpp:2
53
#7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7fffdf7af700 (LWP 23367)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff5b45423 in os::PlatformEvent::park() () from /usr/java/jdk1.7.0_7
5/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from /usr/jav
a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff5882328 in GCTaskThread::run() () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#6 0x00007ffff5b4bb88 in java_start(Thread*) () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7fffdf6ae700 (LWP 23368)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff5b45423 in os::PlatformEvent::park() () from /usr/java/jdk1.7.0_7
5/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from /usr/jav
a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff5882328 in GCTaskThread::run() () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#6 0x00007ffff5b4bb88 in java_start(Thread*) () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7fffdf5ad700 (LWP 23369)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff5b45423 in os::PlatformEvent::park() () from /usr/java/jdk1.7.0_7
5/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from /usr/jav
a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff5882328 in GCTaskThread::run() () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#6 0x00007ffff5b4bb88 in java_start(Thread*) () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7fffdf4ac700 (LWP 23370)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff5b45423 in os::PlatformEvent::park() () from /usr/java/jdk1.7.0_7
5/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from /usr/jav
a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff5882328 in GCTaskThread::run() () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#6 0x00007ffff5b4bb88 in java_start(Thread*) () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fffdf3ab700 (LWP 23371)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff5b45423 in os::PlatformEvent::park() () from /usr/java/jdk1.7.0_7
5/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from /usr/jav
a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff5882328 in GCTaskThread::run() () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#6 0x00007ffff5b4bb88 in java_start(Thread*) () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fffdf2aa700 (LWP 23372)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff5b45423 in os::PlatformEvent::park() () from /usr/java/jdk1.7.0_7
5/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from /usr/jav
a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff5882328 in GCTaskThread::run() () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#6 0x00007ffff5b4bb88 in java_start(Thread*) () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fffdf1a9700 (LWP 23373)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff5b45423 in os::PlatformEvent::park() () from /usr/java/jdk1.7.0_7
5/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from /usr/jav
a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff5882328 in GCTaskThread::run() () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#6 0x00007ffff5b4bb88 in java_start(Thread*) () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fffdf0a8700 (LWP 23374)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff5b45423 in os::PlatformEvent::park() () from /usr/java/jdk1.7.0_7
5/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from /usr/jav
a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff5882328 in GCTaskThread::run() () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#6 0x00007ffff5b4bb88 in java_start(Thread*) () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fffdefa7700 (LWP 23375)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff5b45423 in os::PlatformEvent::park() () from /usr/java/jdk1.7.0_7
5/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from /usr/jav
a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff5882328 in GCTaskThread::run() () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#6 0x00007ffff5b4bb88 in java_start(Thread*) () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fffdeea6700 (LWP 23376)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff5b45423 in os::PlatformEvent::park() () from /usr/java/jdk1.7.0_7
5/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from /usr/jav
a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff5882328 in GCTaskThread::run() () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#6 0x00007ffff5b4bb88 in java_start(Thread*) () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fffda347700 (LWP 23377)):
#0 0x00007ffff45c398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/lib
pthread.so.0
#1 0x00007ffff5b4aaef in os::PlatformEvent::park(long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b0c9ee in Monitor::IWait(Thread*, long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5cd3eb9 in VMThread::loop() () from /usr/java/jdk1.7.0_75/jre/lib
/amd64/server/libjvm.so
#5 0x00007ffff5cd41c0 in VMThread::run() () from /usr/java/jdk1.7.0_75/jre/lib/
amd64/server/libjvm.so
#6 0x00007ffff5b4bb88 in java_start(Thread*) () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fffda246700 (LWP 23378)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff5b45423 in os::PlatformEvent::park() () from /usr/java/jdk1.7.0_7
5/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b39cbd in ObjectMonitor::wait(long, bool, Thread*) () from /usr/
java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff59a8538 in JVM_MonitorWait () from /usr/java/jdk1.7.0_75/jre/lib/
amd64/server/libjvm.so
#4 0x00007fffdfd39208 in ?? ()
#5 0x00000000c0034eb8 in ?? ()
#6 0x00007fffda22e2d8 in ?? ()
#7 0x00000000bb7d0170 in ?? ()
#8 0x0000000000000000 in ?? ()
Thread 19 (Thread 0x7fffda145700 (LWP 23379)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff5b45423 in os::PlatformEvent::park() () from /usr/java/jdk1.7.0_7
5/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b39cbd in ObjectMonitor::wait(long, bool, Thread*) () from /usr/
java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff59a8538 in JVM_MonitorWait () from /usr/java/jdk1.7.0_75/jre/lib/
amd64/server/libjvm.so
#4 0x00007fffdfd39208 in ?? ()
#5 0x00000000c0035720 in ?? ()
#6 0x00007ffff592433f in InterpreterRuntime::monitorenter(JavaThread*, BasicObj
ectLock*) () from /usr/java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#7 0x00007fffdfb36058 in ?? ()
#8 0x00007fffdfb36058 in ?? ()
#9 0x0000000000000000 in ?? ()
Thread 18 (Thread 0x7fffd41b3700 (LWP 23380)):
#0 0x00007ffff45c5930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007ffff5b4a7ca in check_pending_signals(bool) () from /usr/java/jdk1.7.0
_75/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b43a15 in signal_thread_entry(JavaThread*, Thread*) () from /usr
/java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5c8571f in JavaThread::thread_main_inner() () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5c85825 in JavaThread::run() () from /usr/java/jdk1.7.0_75/jre/li
b/amd64/server/libjvm.so
#5 0x00007ffff5b4bb88 in java_start(Thread*) () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#6 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#7 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fffd40b2700 (LWP 23381)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff5b45423 in os::PlatformEvent::park() () from /usr/java/jdk1.7.0_7
5/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5b0cf46 in Monitor::wait(bool, long, bool) () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5775d18 in CompileQueue::get() () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#5 0x00007ffff5779e1a in CompileBroker::compiler_thread_loop() () from /usr/jav
a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff5c8571f in JavaThread::thread_main_inner() () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#7 0x00007ffff5c85825 in JavaThread::run() () from /usr/java/jdk1.7.0_75/jre/li
b/amd64/server/libjvm.so
#8 0x00007ffff5b4bb88 in java_start(Thread*) () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#9 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fffd3fb1700 (LWP 23382)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff5b45423 in os::PlatformEvent::park() () from /usr/java/jdk1.7.0_7
5/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5b0cf46 in Monitor::wait(bool, long, bool) () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5775d18 in CompileQueue::get() () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#5 0x00007ffff5779e1a in CompileBroker::compiler_thread_loop() () from /usr/jav
a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff5c8571f in JavaThread::thread_main_inner() () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#7 0x00007ffff5c85825 in JavaThread::run() () from /usr/java/jdk1.7.0_75/jre/li
b/amd64/server/libjvm.so
#8 0x00007ffff5b4bb88 in java_start(Thread*) () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#9 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fffd3eb0700 (LWP 23383)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff5b45423 in os::PlatformEvent::park() () from /usr/java/jdk1.7.0_7
5/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5bdbd18 in ServiceThread::service_thread_entry(JavaThread*, Threa
d*) () from /usr/java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff5c8571f in JavaThread::thread_main_inner() () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff5c85825 in JavaThread::run() () from /usr/java/jdk1.7.0_75/jre/li
b/amd64/server/libjvm.so
#7 0x00007ffff5b4bb88 in java_start(Thread*) () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#8 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#9 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fffd3daf700 (LWP 23384)):
#0 0x00007ffff45c398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/lib
pthread.so.0
#1 0x00007ffff5b4aaef in os::PlatformEvent::park(long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b0c9ee in Monitor::IWait(Thread*, long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from /usr/java/jdk1
.7.0_75/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5c81741 in WatcherThread::sleep() const () from /usr/java/jdk1.7.
0_75/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff5c81c5e in WatcherThread::run() () from /usr/java/jdk1.7.0_75/jre
/lib/amd64/server/libjvm.so
#6 0x00007ffff5b4bb88 in java_start(Thread*) () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fffd24da700 (LWP 23397)):
#0 0x00007ffff4aec0d3 in poll () from /lib64/libc.so.6
#1 0x00007fffd24ef5ea in Java_org_apache_hadoop_net_unix_DomainSocketWatcher_do
Poll0 () from /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
#2 0x00007fffdfb427f8 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 12 (Thread 0x7fffd2afd700 (LWP 23410)):
#0 0x00007ffff4af5ef3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fffd30f1707 in Java_sun_nio_ch_EPollArrayWrapper_epollWait () from /u
sr/java/jdk1.7.0_75/jre/lib/amd64/libnio.so
#2 0x00007fffdfcdfc52 in ?? ()
#3 0x00000000c005a850 in ?? ()
#4 0x00007fffd2ae5058 in ?? ()
#5 0x00000000bb6afd24 in ?? ()
#6 0x00007fffd2ae50a8 in ?? ()
#7 0x00000000bb6b5e28 in ?? ()
#8 0x00000000bcc04aa0 in ?? ()
#9 0x00000000bb6afd40 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 11 (Thread 0x7fffd21d7700 (LWP 23411)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff5b4a428 in Parker::park(bool, long) () from /usr/java/jdk1.7.0_75
/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5ca93f5 in Unsafe_Park () from /usr/java/jdk1.7.0_75/jre/lib/amd6
4/server/libjvm.so
#3 0x00007fffdfb427f8 in ?? ()
#4 0x00007fffdfb36058 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 10 (Thread 0x7fffd18cc700 (LWP 23427)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00000000004cc162 in get_task (this=0x738b80 <repos_queue>) at SrvrConnect.
cpp:340
#2 SessionWatchDog (arg=<optimized out>) at SrvrConnect.cpp:499
#3 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fffd0ecb700 (LWP 23428)):
#0 0x00007ffff45c6264 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007ffff45c1523 in _L_lock_892 () from /lib64/libpthread.so.0
#2 0x00007ffff45c1407 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000004cbb67 in SessionWatchDog (arg=<optimized out>) at SrvrConnect.c
pp:366
#4 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fffd1bd1700 (LWP 23542)):
#0 0x00007ffff45c398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/lib
pthread.so.0
#1 0x00007ffff5b4aaef in os::PlatformEvent::park(long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b3955b in ObjectMonitor::wait(long, bool, Thread*) () from /usr/
java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff59a8538 in JVM_MonitorWait () from /usr/java/jdk1.7.0_75/jre/lib/
amd64/server/libjvm.so
#4 0x00007fffdfd39208 in ?? ()
#5 0x00000000c005e6a8 in ?? ()
#6 0x0000000000000000 in ?? ()
Thread 7 (Thread 0x7fffce92c700 (LWP 23543)):
#0 0x00007ffff45c398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/lib
pthread.so.0
#1 0x00007ffff5b4a314 in Parker::park(bool, long) () from /usr/java/jdk1.7.0_75
/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5ca93f5 in Unsafe_Park () from /usr/java/jdk1.7.0_75/jre/lib/amd6
4/server/libjvm.so
#3 0x00007fffdfdb3bca in ?? ()
#4 0x00000000c0040520 in ?? ()
#5 0x00007ffff348ce46 in clock_gettime () from /lib64/librt.so.1
#6 0x00007ffff5b452af in os::javaTimeNanos() () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#7 0x00007fffdfedfa38 in ?? ()
#8 0x00000000c005a650 in ?? ()
#9 0x00007fffce9142f0 in ?? ()
#10 0x00000000baf73958 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 6 (Thread 0x7fffd1fd5700 (LWP 23547)):
#0 0x00007ffff4af5ef3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fffd30f1707 in Java_sun_nio_ch_EPollArrayWrapper_epollWait () from /u
sr/java/jdk1.7.0_75/jre/lib/amd64/libnio.so
#2 0x00007fffdfcdfc52 in ?? ()
#3 0x00000000c004faf8 in ?? ()
#4 0x00007fffd1fbd158 in ?? ()
#5 0x00000000bb6afd24 in ?? ()
#6 0x00007fffd1fbd1a8 in ?? ()
#7 0x00000000bb6b5e28 in ?? ()
#8 0x00000000bcc04aa0 in ?? ()
#9 0x00000000bb6afd40 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 5 (Thread 0x7fffce82b700 (LWP 23548)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff5b4a428 in Parker::park(bool, long) () from /usr/java/jdk1.7.0_75
/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5ca93f5 in Unsafe_Park () from /usr/java/jdk1.7.0_75/jre/lib/amd6
4/server/libjvm.so
#3 0x00007fffdfb427f8 in ?? ()
#4 0x00007fffdfb36058 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 4 (Thread 0x7fffcdf37700 (LWP 5594)):
#0 0x00007ffff45c398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/lib
pthread.so.0
#1 0x00007ffff5b4aaef in os::PlatformEvent::park(long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b4b5d4 in os::sleep(Thread*, long, bool) () from /usr/java/jdk1.
7.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff59b10f2 in JVM_Sleep () from /usr/java/jdk1.7.0_75/jre/lib/amd64/
server/libjvm.so
#4 0x00007fffdfefaf91 in ?? ()
#5 0x00007fffcdf1f3b0 in ?? ()
#6 0x00007ffff348ce46 in clock_gettime () from /lib64/librt.so.1
#7 0x00007ffff5b452af in os::javaTimeNanos() () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#8 0x00007fffdfefbcf8 in ?? ()
#9 0x00000000c00405c0 in ?? ()
#10 0x00000000c00405c0 in ?? ()
#11 0x0000000000000007 in ?? ()
#12 0x0000000000000bb8 in ?? ()
#13 0x00000000ebe588c0 in ?? ()
#14 0x00000000ebe588d8 in ?? ()
#15 0x00000000ebe588e8 in ?? ()
#16 0x00007fffdfb36058 in ?? ()
#17 0x0000000f0e245731 in ?? ()
#18 0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7fffcf3b3700 (LWP 19892)):
#0 0x00007ffff45c398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/lib
pthread.so.0
#1 0x00007ffff5b4a314 in Parker::park(bool, long) () from /usr/java/jdk1.7.0_75
/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5ca93f5 in Unsafe_Park () from /usr/java/jdk1.7.0_75/jre/lib/amd6
4/server/libjvm.so
#3 0x00007fffdfdb3bca in ?? ()
#4 0x00000000c0040520 in ?? ()
#5 0x00007ffff348ce46 in clock_gettime () from /lib64/librt.so.1
#6 0x00007ffff5b452af in os::javaTimeNanos() () from /usr/java/jdk1.7.0_75/jre/
lib/amd64/server/libjvm.so
#7 0x00007fffdfee04dc in ?? ()
#8 0x000593c5d144b2ef in ?? ()
#9 0x00000000c0040530 in ?? ()
#10 0x0000000000000001 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7fffd1cd2700 (LWP 22157)):
#0 0x00007ffff45c398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/lib
pthread.so.0
#1 0x00007ffff5b4aaef in os::PlatformEvent::park(long) () from /usr/java/jdk1.7
.0_75/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff5b3955b in ObjectMonitor::wait(long, bool, Thread*) () from /usr/
java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff59a8538 in JVM_MonitorWait () from /usr/java/jdk1.7.0_75/jre/lib/
amd64/server/libjvm.so
#4 0x00007fffdfd39208 in ?? ()
#5 0x00000000edaabb50 in ?? ()
#6 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7ffff7fbcae0 (LWP 12285)):
#0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthre
ad.so.0
#1 0x00007ffff47f2656 in SB_Thread::CV::wait (this=0xeaa7a8) at /home/jenkins/w
orkspace/traf-pub-release-ahw2.2/trafodion/core/sqf/export/include/seabed/int/th
read.inl:552
#2 0x00007ffff47f26d7 in SB_Thread::CV::wait (this=0xeaa7a8, pv_lock=true) at /
home/jenkins/workspace/traf-pub-release-ahw2.2/trafodion/core/sqf/export/include
/seabed/int/thread.inl:587
#3 0x00007ffff733a7e6 in SB_Ms_Event_Mgr::wait (this=0xeaa6b0, pv_us=-1) at mse
ventmgr.inl:343
#4 0x00007ffff735b18e in XWAIT_com (pv_mask=256, pv_time=-1, pv_residual=true)
at pctl.cpp:979
#5 0x00007ffff735af3f in XWAIT (pv_mask=256, pv_time=-1) at pctl.cpp:875
#6 0x00007ffff75c733a in fs_int_fs_file_awaitiox (pp_filenum=0x7fffffff00ee, pp
p_buf=0x7ffffffeffe8, pp_xfercount=0x7ffffffeffd4, pp_tag=0x7ffffffeffd8, pv_tim
eout=-1, pp_segid=0x7ffffffeffe6, pv_int=false, pv_ts=false) at fsi.cpp:1421
#7 0x00007ffff75c0a19 in BAWAITIOX (pp_filenum=0x7fffffff00ee, ppp_buf=0x0, pp_
xfercount=0x7fffffff0068, pp_tag=0x7fffffff00d8, pv_timeout=-1, pp_segid=0x0) at
fs.cpp:558
#8 0x00007ffff75c3aad in XAWAITIOX (pp_filenum=0x7fffffff00ee, ppp_buf=0x0, pp_
xfercount=0x7fffffff00ea, pp_tag=0x7fffffff00d8, pv_timeout=-1, pp_segid=0x0) at
fs.cpp:1601
#9 0x0000000000462e0c in CNSKListenerSrvr::runProgram (this=0xda0e50, TcpProces
sName=<optimized out>, port=<optimized out>, TransportTrace=<optimized out>) at
Interface/linux/Listener_srvr_ps.cpp:474
#10 0x00000000004de8eb in main (argc=31, argv=0x7fffffff32f8, envp=<optimized ou
t>) at SrvrMain.cpp:863

[Anoop Sharma (anoop-sharma)]

fix has been delivered.

[Weishiun Tsai (wei-shiun-tsai)]

Ran the entire set of the SQL regression tests on the v0327 build installed on a 6-node cluster. Did not run into this problem again. This problem has been fixed.