Trafodion

Multiple tdm_arkcmp child processes started after receipt of HBase error

Bug #1438466 reported by Joanie Cooper on 2015-03-31

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Trafodion	New	High	Unassigned	Trafodion r2.0

Bug Description

During a fresh test of running the compGeneral regression suite
while artificially interjecting an error return from the TrxRegionEndpoint
coprocessor, numerous tdm_arkcmp child processes were started.

Before the error hit, we seemed to have a normal number of compilers

[$Z0005MG] 000,00003170 001 GEN ES--A-- $Z0002KK $Z0002IV tdm_arkcmp
[$Z0005MG] 000,00003292 001 GEN ES--A-- $Z0002P2 $Z0002KK tdm_arkcmp
[$Z0005MG] 000,00003816 001 GEN ES--A-- $Z000341 $Z0002P2 tdm_arkcmp
[$Z0005MG] 000,00003886 001 GEN ES--A-- $Z000361 $Z000341 tdm_arkcmp

After forcing the error, it looks like we have new compilers being generated,
all ultimately part of the original tdm_arkcmp parent off of the sqlci session.

This is a result of a drop statement. From the sqlci window, the
statement appears hung, as it never returns. But, it appears the
compilers keep generating new children and the query ultimately never returns.

When I killed the query, it had 174 compilers running.

I tried a pstack for one of the compilers, I’ve attached it below.

g4t3037{joaniec}3: sqps
Processing cluster.conf on local host g4t3037. [$Z000AF9] NID,PID(os) PRI TYPE STATES NAME [$Z000AF9] ------------ --- ---- ------- [$Z000AF9] 000,00018562 000 WDG ES--A-- $WDG000 NONE [$Z000AF9] 000,00018563 000 PSD ES--A-- $PSD000 NONE [$Z000AF9] 000,00018592 001 DTM ES--A-- $TM0 [$Z000AF9] 000,00019243 001 GEN ES--A-- $ZSC000 NONE [$Z000AF9] 000,00019274 001 SSMP ES--A-- $ZSM000 NONE [$Z000AF9] 000,00020982 001 GEN ES--A-- $ZLOBSRV0 NONE [$Z000AF9] 000,00007356 001 GEN ES--A-- $Z000606 NONE [$Z000AF9] 000,00007416 001 GEN ES--A-- $Z00061W [$Z000AF9] 000,00007960 001 GEN ES--A-- $Z0006HF [$Z000AF9] 000,00008021 001 GEN ES--A-- $Z0006J6 [$Z000AF9] 000,00008079 001 GEN ES--A-- $Z0006KU [$Z000AF9] 000,00008137 001 GEN ES--A-- $Z0006MH [$Z000AF9] 000,00008194 001 GEN ES--A-- $Z0006P4 [$Z000AF9] 000,00008252 001 GEN ES--A-- $Z0006QS [$Z000AF9] 000,00008312 001 GEN ES--A-- $Z0006SH [$Z000AF9] 000,00008369 001 GEN ES--A-- $Z0006U4 [$Z000AF9] 000,00008427 001 GEN ES--A-- $Z0006VS [$Z000AF9] 000,00008491 001 GEN ES--A-- $Z0006XL [$Z000AF9] 000,00009023 001 GEN ES--A-- $Z0007CT [$Z000AF9] 000,00009081 001 GEN ES--A-- $Z0007EG [$Z000AF9] 000,00009141 001 GEN ES--A-- $Z0007G6 [$Z000AF9] 000,00009202 001 GEN ES--A-- $Z0007HX [$Z000AF9] 000,00009262 001 GEN ES--A-- $Z0007JM [$Z000AF9] 000,00009320 001 GEN ES--A-- $Z0007LA [$Z000AF9] 000,00009489 001 GEN ES--A-- $Z0007R4 [$Z000AF9] 000,00009547 001 GEN ES--A-- $Z0007SS [$Z000AF9] 000,00009604 001 GEN ES--A-- $Z0007UE [$Z000AF9] 000,00009661 001 GEN ES--A-- $Z0007W1 [$Z000AF9] 000,00009728 001 GEN ES--A-- $Z0007XY [$Z000AF9] 000,00010268 001 GEN ES--A-- $Z0008DD [$Z000AF9] 000,00010364 001 GEN ES--A-- $Z0008G4 [$Z000AF9] 000,00010421 001 GEN ES--A-- $Z0008HR [$Z000AF9] 000,00010479 001 GEN ES--A-- $Z0008JE [$Z000AF9] 000,00010537 001 GEN ES--A-- $Z0008L2 [$Z000AF9] 000,00010620 001 GEN ES--A-- $Z0008NF [$Z000AF9] 000,00010677 001 GEN ES--A-- $Z0008Q2 [$Z000AF9] 000,00010735 001 GEN ES--A-- $Z0008RQ [$Z000AF9] 000,00010796 001 GEN ES--A-- $Z0008TG [$Z000AF9] 000,00010857 001 GEN ES--A-- $Z0008V7 [$Z000AF9] 000,00011367 001 GEN ES--A-- $Z00099S [$Z000AF9] 000,00011437 001 GEN ES--A-- $Z0009BS [$Z000AF9] 000,00011496 001 GEN ES--A-- $Z0009DG [$Z000AF9] 000,00011558 001 GEN ES--A-- $Z0009F8 [$Z000AF9] 000,00011618 001 GEN ES--A-- $Z0009GY [$Z000AF9] 000,00011680 001 GEN ES--A-- $Z0009IQ [$Z000AF9] 000,00011741 001 GEN ES--A-- $Z0009KG [$Z000AF9] 000,00011801 001 GEN ES--A-- $Z0009M6 [$Z000AF9] 000,00011862 001 GEN ES--A-- $Z0009NX [$Z000AF9] 000,00011921 001 GEN ES--A-- $Z0009QL [$Z000AF9] 000,00012410 001 GEN ES--A-- $Z000A4K [$Z000AF9] 000,00012509 001 GEN ES--A-- $Z000A7E [$Z000AF9] 000,00012582 001 GEN ES--A-- $Z000A9H [$Z000AF9] 000,00012646 001 GEN ES--A-- $Z000ABB [$Z000AF9] 000,00012703 001 GEN ES--A-- $Z000ACY [$Z000AF9] 000,00012760 001 GEN ES--A-- $Z000AEK [$Z000AF9] 000,00012784 001 GEN ES--A-- $Z000AF9 NONE [$Z000AF9] 001,00018560 000 WDG ES--A-- $WDG001 NONE [$Z000AF9] 001,00018561 000 PSD ES--A-- $PSD001 NONE [$Z000AF9] 001,00018640 001 DTM ES--A-- $TM1 [$Z000AF9] 001,00019258 001 GEN ES--A-- $ZSC001 NONE [$Z000AF9] 001,00019289 001 SSMP ES--A-- $ZSM001 NONE [$Z000AF9] 001,00020989 001 GEN ES--A-- $ZLOBSRV1 NONE />houston.hp.com
1.1.0 (Build release [joaniec], date 26Mar15)
PARENT PROGRAM
----------- ----------- ---------------
sqwatchdog
pstartd
NONE tm
mxsscp
mxssmp
mxlobsrvr
sqlci
$Z000606 tdm_arkcmp
$Z00061W tdm_arkcmp
$Z0006HF tdm_arkcmp
$Z0006J6 tdm_arkcmp
$Z0006KU tdm_arkcmp
$Z0006MH tdm_arkcmp
$Z0006P4 tdm_arkcmp
$Z0006QS tdm_arkcmp
$Z0006SH tdm_arkcmp
$Z0006U4 tdm_arkcmp
$Z0006VS tdm_arkcmp
$Z0006XL tdm_arkcmp
$Z0007CT tdm_arkcmp
$Z0007EG tdm_arkcmp
$Z0007G6 tdm_arkcmp
$Z0007HX tdm_arkcmp
$Z0007JM tdm_arkcmp
$Z0007LA tdm_arkcmp
$Z0007R4 tdm_arkcmp
$Z0007SS tdm_arkcmp
$Z0007UE tdm_arkcmp
$Z0007W1 tdm_arkcmp
$Z0007XY tdm_arkcmp
$Z0008DD tdm_arkcmp
$Z0008G4 tdm_arkcmp
$Z0008HR tdm_arkcmp
$Z0008JE tdm_arkcmp
$Z0008L2 tdm_arkcmp
$Z0008NF tdm_arkcmp
$Z0008Q2 tdm_arkcmp
$Z0008RQ tdm_arkcmp
$Z0008TG tdm_arkcmp
$Z0008V7 tdm_arkcmp
$Z00099S tdm_arkcmp
$Z0009BS tdm_arkcmp
$Z0009DG tdm_arkcmp
$Z0009F8 tdm_arkcmp
$Z0009GY tdm_arkcmp
$Z0009IQ tdm_arkcmp
$Z0009KG tdm_arkcmp
$Z0009M6 tdm_arkcmp
$Z0009NX tdm_arkcmp
$Z0009QL tdm_arkcmp
$Z000A4K tdm_arkcmp
$Z000A7E tdm_arkcmp
$Z000A9H tdm_arkcmp
$Z000ABB tdm_arkcmp
$Z000ACY tdm_arkcmp
shell
sqwatchdog
pstartd
NONE tm
mxsscp
mxssmp
mxlobsrvr

[$Z000GHJ] 000,00020170 001 GEN ES--A-- $Z000GGA $Z000GEK tdm_arkcmp
[$Z000GHJ] 000,00020214 001 GEN ES--A-- $Z000GHJ NONE shell
[$Z000GHJ] 001,00018560 000 WDG ES--A-- $WDG001 NONE sqwatchdog
[$Z000GHJ] 001,00018561 000 PSD ES--A-- $PSD001 NONE pstartd
[$Z000GHJ] 001,00018640 001 DTM ES--A-- $TM1 NONE tm
[$Z000GHJ] 001,00019258 001 GEN ES--A-- $ZSC001 NONE mxsscp
[$Z000GHJ] 001,00019289 001 SSMP ES--A-- $ZSM001 NONE mxssmp
[$Z000GHJ] 001,00020989 001 GEN ES--A-- $ZLOBSRV1 NONE mxlobsrvr

g4t3037{joaniec}120: pstack $Z000GGA
Usage: pstack <process-id>
g4t3037{joaniec}121: pstack 20110
Thread 30 (Thread 0x7fffee0a7700 (LWP 20111)):
#0 0x0000003e3c00e84d in accept () from /lib64/libpthread.so.0
#1 0x00007fffefec1995 in SB_Trans::Sock_Listener::accept() () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#2 0x00007fffefec84e6 in SB_Trans::Sock_Stream_Accept_Thread::run() () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#3 0x00007fffefec8213 in sock_stream_accept_thread_fun(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#4 0x00007fffee540b4f in SB_Thread::Thread::disp(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#5 0x00007fffee540fa7 in thread_fun(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#6 0x00007fffee54421c in sb_thread_sthr_disp(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7fffed6a6700 (LWP 20112)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fffee5435e2 in SB_Thread::CV::wait() () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#2 0x00007fffee5436be in SB_Thread::CV::wait(bool) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#3 0x00007fffefebac47 in SB_Sig_Queue::remove() () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#4 0x00007fffefec8977 in SB_Trans::Sock_Stream_Helper_Thread::run() () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#5 0x00007fffefec823a in sock_helper_thread_fun(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#6 0x00007fffee540b4f in SB_Thread::Thread::disp(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#7 0x00007fffee540fa7 in thread_fun(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#8 0x00007fffee54421c in sb_thread_sthr_disp(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#9 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#10 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7fffead82700 (LWP 20113)):
#0 0x0000003e3bc3399d in sigtimedwait () from /lib64/libc.so.6
#1 0x00007fffefe865c4 in local_monitor_reader(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#2 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#3 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fffea301700 (LWP 20115)):
#0 0x0000003e3c00ed2d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fffef6032b9 in Sleep(unsigned int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libwin.so
#2 0x00007ffff7747d27 in memMonitorUpdateThread(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
#3 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fffe977e700 (LWP 20116)):
#0 0x0000003e3bce8f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fffefec0cd5 in SB_Trans::Sock_Controller::epoll_wait(char const*, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#2 0x00007fffefec01f3 in SB_Trans::Sock_Comp_Thread::run() () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#3 0x00007fffefec002d in sock_comp_thread_fun(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#4 0x00007fffee540b4f in SB_Thread::Thread::disp(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#5 0x00007fffee540fa7 in thread_fun(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#6 0x00007fffee54421c in sb_thread_sthr_disp(void*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fffe523e700 (LWP 20117)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fffe513d700 (LWP 20118)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fffe503c700 (LWP 20119)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fffe4f3b700 (LWP 20120)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fffe4e3a700 (LWP 20121)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fffe4d39700 (LWP 20122)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fffe4c38700 (LWP 20123)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fffe4b37700 (LWP 20124)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fffe4a36700 (LWP 20125)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fffe4935700 (LWP 20126)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff15210a8 in GCTaskThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fffdfdec700 (LWP 20127)):
#0 0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e88ef in os::PlatformEvent::park(long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aad5e in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff1970169 in VMThread::loop() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff1970470 in VMThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fffdfceb700 (LWP 20128)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17d7add in ObjectMonitor::wait(long, bool, Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff1647378 in JVM_MonitorWait () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007fffe56ba7f8 in ?? ()
#5 0x00007fffdfcd3390 in ?? ()
#6 0x0000000000c33ca0 in ?? ()
#7 0x00007fffdfcd31c0 in ?? ()
#8 0x00007fffdfcd3158 in ?? ()
#9 0x0000000000000000 in ?? ()
Thread 13 (Thread 0x7fffdfbea700 (LWP 20129)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17d7add in ObjectMonitor::wait(long, bool, Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff1647378 in JVM_MonitorWait () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007fffe56ba7f8 in ?? ()
#5 0x00007fffdfbd22e0 in ?? ()
#6 0x0000000000c3e000 in ?? ()
#7 0x0000000000c38210 in ?? ()
#8 0x00007fffdfbd2268 in ?? ()
#9 0x0000000000000000 in ?? ()
Thread 12 (Thread 0x7fffdfae9700 (LWP 20130)):
#0 0x0000003e3c00d720 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007ffff17e85ca in check_pending_signals(bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17e1835 in signal_thread_entry(JavaThread*, Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff1922d9f in JavaThread::thread_main_inner() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff1922ea5 in JavaThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fffdf9e8700 (LWP 20131)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab2b6 in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff1414ad8 in CompileQueue::get() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff1418bda in CompileBroker::compiler_thread_loop() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff1922d9f in JavaThread::thread_main_inner() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x00007ffff1922ea5 in JavaThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#8 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#9 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#10 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fffdf8e7700 (LWP 20132)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab2b6 in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff1414ad8 in CompileQueue::get() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff1418bda in CompileBroker::compiler_thread_loop() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff1922d9f in JavaThread::thread_main_inner() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x00007ffff1922ea5 in JavaThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#8 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#9 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#10 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fffdf7e6700 (LWP 20133)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e3243 in os::PlatformEvent::park() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff1879388 in ServiceThread::service_thread_entry(JavaThread*, Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff1922d9f in JavaThread::thread_main_inner() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff1922ea5 in JavaThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#8 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fffdf6e5700 (LWP 20134)):
#0 0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e88ef in os::PlatformEvent::park(long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff17aad5e in Monitor::IWait(Thread*, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff191edc1 in WatcherThread::sleep() const () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff191f2de in WatcherThread::run() () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff17e9988 in java_start(Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fffde317700 (LWP 20137)):
#0 0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e8114 in Parker::park(bool, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff1946a05 in Unsafe_Park () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007fffe56ba7f8 in ?? ()
#4 0x0000000000bb4968 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 6 (Thread 0x7fffddedb700 (LWP 20141)):
#0 0x0000003e3bcdf253 in poll () from /lib64/libc.so.6
#1 0x00007fffde32a2ca in Java_org_apache_hadoop_net_unix_DomainSocketWatcher_doPoll0 () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/sql/local_hadoop/hadoop-2.5.0-cdh5.3.0/lib/native/libhadoop.so.1.0.0
#2 0x00007fffe56ba7f8 in ?? ()
#3 0x00000000020a4800 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 5 (Thread 0x7fffde2de700 (LWP 20153)):
#0 0x0000003e3bce8f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fffde96b707 in Java_sun_nio_ch_EPollArrayWrapper_epollWait () from /opt/home/tools/jdk1.7.0_67/jre/lib/amd64/libnio.so
#2 0x00007fffe56ba7f8 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 4 (Thread 0x7fffddbd8700 (LWP 20154)):
#0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e8228 in Parker::park(bool, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff1946a05 in Unsafe_Park () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007fffe56ba7f8 in ?? ()
#4 0x00007fffe56ae058 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7fffddfdc700 (LWP 20156)):
#0 0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e8114 in Parker::park(bool, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff1946a05 in Unsafe_Park () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007fffe56ba7f8 in ?? ()
#4 0x00007fffe56ae058 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7fffdddda700 (LWP 20236)):
#0 0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff17e8114 in Parker::park(bool, long) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff1946a05 in Unsafe_Park () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007fffe56ba7f8 in ?? ()
#4 0x00007fffe56ae058 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7fffee315c00 (LWP 20110)):
#0 0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fffee543778 in SB_Thread::CV::wait(int, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#2 0x00007fffee54383d in SB_Thread::CV::wait(bool, int, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
#3 0x00007fffefe988cb in SB_Ms_Event_Mgr::wait(long) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#4 0x00007fffefeb916e in XWAIT_com(short, int, bool) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#5 0x00007fffefeb8f1f in XWAIT(short, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
#6 0x00007fffefc3731a in fs_int_fs_file_awaitiox(short*, void**, int*, long*, int, short*, bool, bool) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbfs.so
#7 0x00007fffefc30e15 in BAWAITIOXTS(short*, void**, int*, long*, int, short*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbfs.so
#8 0x00007ffff70be1f7 in GuaConnectionToServer::wait(int, unsigned int*, IpcAwaitiox*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#9 0x00007ffff70b51cb in IpcSetOfConnections::waitOnSet(int, int, int*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#10 0x00007ffff70b633d in IpcMessageStream::waitOnMsgStream(int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#11 0x00007ffff70b6692 in IpcMessageStream::receive(int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#12 0x00007ffff70bdf42 in GuaConnectionToServer::tryToStartNewIO() () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#13 0x00007ffff70beb38 in GuaConnectionToServer::send(IpcMessageBuffer*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#14 0x00007ffff70b72e4 in IpcMessageStream::send(int, long) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#15 0x00007ffff773fad5 in ExSqlComp::sendRequest(CmpMessageObj*, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
#16 0x00007ffff773e620 in ExSqlComp::sendRequest(CmpMessageObj::MessageTypeEnum, char const*, unsigned int, int, long*, int, int, char const*, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
#17 0x00007ffff775fdd9 in CliStatement::prepare2(char*, ComDiagsArea&, char*, unsigned int, int, int, unsigned int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
#18 0x00007ffff7760846 in CliStatement::prepare(char*, ComDiagsArea&, char*, unsigned int, int, int, unsigned int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
#19 0x00007ffff77130d4 in SQLCLI_Prepare2 () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
#20 0x00007ffff776dc6e in SQL_EXEC_Prepare2 () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
#21 0x00007ffff3c819d2 in ExeCliInterface::prepare(char const*, SQLMODULE_ID_t*, SQLCLI_OBJ_ID*, SQLCLI_OBJ_ID*, SQLCLI_OBJ_ID*, SQLCLI_OBJ_ID*, char**, Queue*, char**, Queue*, char*, int*, SQL_QUERY_COST_INFO*, SQL_QUERY_COMPILER_STATS_INFO*, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libexecutor.so
#22 0x00007ffff3c83460 in ExeCliInterface::fetchRowsPrologue(char const*, int, int, char*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libexecutor.so
#23 0x00007ffff7a627fb in CmpSeabaseDDL::getObjectUIDandOwners(ExeCliInterface*, char const*, char const*, char const*, ComObjectType, int&, int&, bool, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
#24 0x00007ffff7ae763d in CmpSeabaseDDL::getSeabaseUserTableDesc(NAString const&, NAString const&, NAString const&, ComObjectType, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
#25 0x00007ffff7aea687 in CmpSeabaseDDL::getSeabaseTableDesc(NAString const&, NAString const&, NAString const&, ComObjectType, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
#26 0x00007ffff4d5d304 in NATableDB::get(CorrName&, BindWA*, desc_struct*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
#27 0x00007ffff4ab2efc in BindWA::getNATable(CorrName&, int, desc_struct*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
#28 0x00007ffff4ad361e in Scan::bindNode(BindWA*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
#29 0x00007ffff4a9fe07 in RelExpr::bindChildren(BindWA*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
#30 0x00007ffff4adffde in RelRoot::bindNode(BindWA*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
#31 0x00007ffff7a3f50e in CmpMain::compile(char const*, int, RelExpr*&, char**, unsigned int*, NAMemory*, CmpMain::CompilerPhase, FragmentDir**, int, int, int*, timeval*, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
#32 0x00007ffff7a4232c in CmpMain::sqlcomp(char const*, int, RelExpr*&, char**, unsigned int*, NAMemory*, CmpMain::CompilerPhase, FragmentDir**, int, int, int*, timeval*, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
#33 0x00007ffff7a43630 in CmpMain::sqlcomp(QueryText&, int, char**, unsigned int*, NAMemory*, CmpMain::CompilerPhase, FragmentDir**, int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
#34 0x00007ffff40dfc08 in CmpStatement::process(CmpMessageSQLText const&) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libarkcmplib.so
#35 0x00007ffff40ca93d in ExCmpMessage::actOnReceive(IpcConnection*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libarkcmplib.so
#36 0x00007ffff70a58d6 in IpcMessageStream::internalActOnReceive(IpcMessageBuffer*, IpcConnection*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#37 0x00007ffff70c02fb in GuaConnectionToClient::acceptBuffer(IpcMessageBuffer*, unsigned int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#38 0x00007ffff70c100f in GuaReceiveControlConnection::wait(int, unsigned int*, IpcAwaitiox*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#39 0x00007ffff70c23b2 in GuaConnectionToClient::wait(int, unsigned int*, IpcAwaitiox*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#40 0x00007ffff70b54aa in IpcSetOfConnections::waitOnSet(int, int, int*) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#41 0x00007ffff70b633d in IpcMessageStream::waitOnMsgStream(int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#42 0x00007ffff70b6692 in IpcMessageStream::receive(int) () from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
#43 0x00000000004064ae in main ()

One more quick note, even after the sqlci session ended,
only the initial tdm_arkcmp process was cleaned up.

The others remained:

Processing cluster.conf on local host g4t3037.houston.hp.com
[$Z000IRF] Shell/shell Version 1.0.1 Release 1.1.0 (Build release [joaniec], dat
e 26Mar15)
[$Z000IRF] %ps
[$Z000IRF] NID,PID(os) PRI TYPE STATES NAME PARENT PROGRAM
[$Z000IRF] ------------ --- ---- ------- ----------- ----------- ---------------
[$Z000IRF] 000,00018562 000 WDG ES--A-- $WDG000 NONE sqwatchdog
[$Z000IRF] 000,00018563 000 PSD ES--A-- $PSD000 NONE pstartd
[$Z000IRF] 000,00018592 001 DTM ES--A-- $TM0 NONE tm
[$Z000IRF] 000,00019243 001 GEN ES--A-- $ZSC000 NONE mxsscp
[$Z000IRF] 000,00019274 001 SSMP ES--A-- $ZSM000 NONE mxssmp
[$Z000IRF] 000,00020982 001 GEN ES--A-- $ZLOBSRV0 NONE mxlobsrvr
[$Z000IRF] 000,00007416 001 GEN ES--A-- $Z00061W NONE tdm_arkcmp
[$Z000IRF] 000,00007960 001 GEN ES--A-- $Z0006HF $Z00061W tdm_arkcmp
[$Z000IRF] 000,00008021 001 GEN ES--A-- $Z0006J6 $Z0006HF tdm_arkcmp
[$Z000IRF] 000,00008079 001 GEN ES--A-- $Z0006KU $Z0006J6 tdm_arkcmp
[$Z000IRF] 000,00008137 001 GEN ES--A-- $Z0006MH $Z0006KU tdm_arkcmp
[$Z000IRF] 000,00008194 001 GEN ES--A-- $Z0006P4 $Z0006MH tdm_arkcmp
[$Z000IRF] 000,00008252 001 GEN ES--A-- $Z0006QS $Z0006P4 tdm_arkcmp
[$Z000IRF] 000,00008312 001 GEN ES--A-- $Z0006SH $Z0006QS tdm_arkcmp
[$Z000IRF] 000,00008369 001 GEN ES--A-- $Z0006U4 $Z0006SH tdm_arkcmp
[$Z000IRF] 000,00008427 001 GEN ES--A-- $Z0006VS $Z0006U4 tdm_arkcmp
…

The new memoryUsage threshold and throttle capabilities were used to initially
see the problem.

This was on the CDH 5.3 workstation using the install_local_hadoop support
on Trafidion R1.1.0 build version git032415 or later build.

Tags:

Joanie Cooper (joanie-cooper) on 2015-03-31

Changed in trafodion:
importance:	Undecided → High

Atanu Mishra (atanu-mishra) on 2015-04-28

tags:	added: sql-cmp
Changed in trafodion:
milestone:	none → r2.0

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.