T2 Phoenix in Jenkins (Cloudera) creating cores in QueryExec tests

[Aruna Sadashiva]

On CDH and CM distros, Phoenix T2 tests failing during QueryExec* tests and creating cores with the stack trace below.

Thread 24 (Thread 0x7fffecbfb700 (LWP 14543)):
#0 0x00000030aba0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff75a918b in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff7576e78 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff75770ff in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff7635394 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff76dcca8 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff71b74c0 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#7 0x00007ffff75c118c in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#8 0x00007ffff72195ae in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#9 0x00007ffff721bd3f in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#10 0x00007ffff718d00c in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#11 0x00007ffff722145b in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#12 0x00007ffff7222ac1 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#13 0x00007ffff76e0ba5 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#14 0x00007ffff76e0e30 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#15 0x00007ffff75af9a2 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#16 0x00000030aba079d1 in start_thread () from /lib64/libpthread.so.0
#17 0x00000030ab6e88fd in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7fffc9e2c700 (LWP 14966)):
#0 0x00000030ab63371d in sigtimedwait () from /lib64/libc.so.6
#1 0x00007fffe707a5e4 in local_monitor_reader (pp_arg=0x2803) at ../../../monitor/linux/clio.cxx:130
#2 0x00000030aba079d1 in start_thread () from /lib64/libpthread.so.0
#3 0x00000030ab6e88fd in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7fffec1c8700 (LWP 14554)):
#0 0x00000030aba0b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff75ae457 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff76feba0 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007fffed0127f8 in ?? ()
#4 0x00007fffed006058 in ?? ()
#5 0x0000000000000000 in ?? ()

Thread 21 (Thread 0x7fffecdfd700 (LWP 14541)):
#0 0x00000030aba0d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007ffff75aeaec in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff75a79ba in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff76e0ba5 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff76e0e30 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff75af9a2 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#6 0x00000030aba079d1 in start_thread () from /lib64/libpthread.so.0
#7 0x00000030ab6e88fd in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7ffff5a93700 (LWP 14535)):
#0 0x00000030aba0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff75a918b in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff75774c3 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff7577a9e in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff731f0d3 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff73206f4 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff75af9a2 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#7 0x00000030aba079d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00000030ab6e88fd in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7fffecefe700 (LWP 14539)):
#0 0x00000030aba0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff75a918b in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff759ecdc in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff76a2eb8 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff7419760 in JVM_MonitorWait () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#5 0x00007fffed0127f8 in ?? ()
#6 0x0000000000000000 in ?? ()

Thread 18 (Thread 0x7ffff7bdc720 (LWP 14517)):
#0 0x00000030aba0822d in pthread_join () from /lib64/libpthread.so.0
#1 0x000000344d208ab5 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/bin/../lib/amd64/jli/libjli.so
#2 0x000000344d202574 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/bin/../lib/amd64/jli/libjli.so
#3 0x000000344d2050a8 in JLI_Launch () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/bin/../lib/amd64/jli/libjli.so
#4 0x0000000000400766 in ?? ()
#5 0x00000030ab61ed5d in __libc_start_main () from /lib64/libc.so.6
#6 0x0000000000400629 in ?? ()
#7 0x00007fffffffe3d8 in ?? ()
#8 0x000000000000001c in ?? ()
#9 0x0000000000000008 in ?? ()
#10 0x00007fffffffe65e in ?? ()
#11 0x00007fffffffe697 in ?? ()
#12 0x00007fffffffe6a2 in ?? ()
#13 0x00007fffffffe6da in ?? ()
#14 0x00007fffffffe711 in ?? ()
#15 0x00007fffffffe736 in ?? ()
#16 0x00007fffffffe768 in ?? ()
#17 0x00007fffffffe76d in ?? ()
#18 0x0000000000000000 in ?? ()

Thread 17 (Thread 0x7ffff6dca700 (LWP 14532)):
#0 0x00000030aba0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff75a918b in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff75774c3 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff7577b36 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff772989c in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff75cc2c1 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff76f3bae in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#7 0x00007ffff7633cde in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#8 0x00007fffed05f5c7 in ?? ()
#9 0x00000007870a6818 in ?? ()
#10 0x00007fffed1462c0 in ?? ()
#11 0x00007ffff6dc2800 in ?? ()
#12 0x00000007d99fc108 in ?? ()
#13 0x0000003a00000051 in ?? ()
#14 0x00000007870a6838 in ?? ()
#15 0x00000007dbddf7e0 in ?? ()
#16 0x00000007000000ae in ?? ()
#17 0x00000007870a6818 in ?? ()
#18 0x0000000000000001 in ?? ()
#19 0x0000000785000508 in ?? ()
#20 0x0000000000000000 in ?? ()

Thread 16 (Thread 0x7ffff5992700 (LWP 14536)):
#0 0x00000030aba0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff75a918b in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff75774c3 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff7577a9e in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff731f0d3 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff73206f4 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff75af9a2 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#7 0x00000030aba079d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00000030ab6e88fd in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7ffff5b94700 (LWP 14534)):
#0 0x00000030aba0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff75a918b in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff75774c3 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff7577a9e in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff731f0d3 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff73206f4 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff75af9a2 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#7 0x00000030aba079d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00000030ab6e88fd in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fffd4f18700 (LWP 14965)):
#0 0x00000030aba0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fffe4ec8656 in SB_Thread::CV::wait (this=0x7ffff0ff87b8)
    at /home/jenkins/workspace/phoenix_part1_T2-cm5.1/trafodion/core/sqf/export/include/seabed/int/thread.inl:552
#2 0x00007fffe4ec8732 in SB_Thread::CV::wait (this=0x7ffff0ff87b8, pv_lock=false)
    at /home/jenkins/workspace/phoenix_part1_T2-cm5.1/trafodion/core/sqf/export/include/seabed/int/thread.inl:591
#3 0x00007fffe70aec67 in SB_Sig_Queue::remove (this=0x7ffff0ff8760) at queue.inl:473
#4 0x00007fffe70bc997 in SB_Trans::Sock_Stream_Helper_Thread::run (this=0x7ffff0ff8210) at sockstream.cpp:2179
#5 0x00007fffe70bc25a in sock_helper_thread_fun (pp_arg=0x7ffff0ff8210) at sockstream.cpp:2029
#6 0x00007fffe4ec5b9f in SB_Thread::Thread::disp (this=0x7ffff0ff8210, pp_arg=0x7ffff0ff8210) at thread.cpp:211
#7 0x00007fffe4ec5ff7 in thread_fun (pp_arg=0x7ffff0ff8210) at thread.cpp:307
#8 0x00007fffe4ec9290 in sb_thread_sthr_disp (pp_arg=0x7ffff1002d50) at threadl.cpp:253
#9 0x00000030aba079d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00000030ab6e88fd in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7fffb616e700 (LWP 14968)):
#0 0x00000030ab6e8ef3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fffe70b4cf5 in SB_Trans::Sock_Controller::epoll_wait (this=0x7fffe7305ca0, pp_where=0x7fffe70d7c08 "Sock_Comp_Thread::run", pv_timeout=-1)
    at sock.cpp:336
#2 0x00007fffe70b4213 in SB_Trans::Sock_Comp_Thread::run (this=0x7ffff0ffbc10) at sock.cpp:104
#3 0x00007fffe70b404d in sock_comp_thread_fun (pp_arg=0x7ffff0ffbc10) at sock.cpp:74
#4 0x00007fffe4ec5b9f in SB_Thread::Thread::disp (this=0x7ffff0ffbc10, pp_arg=0x7ffff0ffbc10) at thread.cpp:211
#5 0x00007fffe4ec5ff7 in thread_fun (pp_arg=0x7ffff0ffbc10) at thread.cpp:307
#6 0x00007fffe4ec9290 in sb_thread_sthr_disp (pp_arg=0x7ffff0ffdf30) at threadl.cpp:253
#7 0x00000030aba079d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00000030ab6e88fd in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7ffff5c95700 (LWP 14533)):
#0 0x00000030aba0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff75a918b in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff75774c3 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff7577a9e in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff731f0d3 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff73206f4 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff75af9a2 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#7 0x00000030aba079d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00000030ab6e88fd in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fffd5919700 (LWP 14964)):
#0 0x00000030aba0ea5d in accept () from /lib64/libpthread.so.0
#1 0x00007fffe70b59b5 in SB_Trans::Sock_Listener::accept (this=0x7ffff1003280) at sock.cpp:618
#2 0x00007fffe70bc506 in SB_Trans::Sock_Stream_Accept_Thread::run (this=0x7ffff10041f0) at sockstream.cpp:2085
#3 0x00007fffe70bc233 in sock_stream_accept_thread_fun (pp_arg=0x7ffff10041f0) at sockstream.cpp:2019
#4 0x00007fffe4ec5b9f in SB_Thread::Thread::disp (this=0x7ffff10041f0, pp_arg=0x7ffff10041f0) at thread.cpp:211
#5 0x00007fffe4ec5ff7 in thread_fun (pp_arg=0x7ffff10041f0) at thread.cpp:307
#6 0x00007fffe4ec9290 in sb_thread_sthr_disp (pp_arg=0x7ffff1004320) at threadl.cpp:253
#7 0x00000030aba079d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00000030ab6e88fd in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fffc942b700 (LWP 14967)):
#0 0x00000030aba0ef3d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007fffe6c344cb in Sleep (milliSecs=6000) at traf_misc.cpp:136
#2 0x00007fffe51a1298 in memMonitorUpdateThread (param=0x7fffdc071af8) at ../cli/memorymonitor.cpp:73
#3 0x00000030aba079d1 in start_thread () from /lib64/libpthread.so.0
#4 0x00000030ab6e88fd in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fffe7bf7700 (LWP 14975)):
#0 0x00000030aba0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff75ae59a in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff76feba0 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007fffed0127f8 in ?? ()
#4 0x00007fffed006058 in ?? ()
#5 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7fffe7cf8700 (LWP 14902)):
#0 0x00000030aba0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff75ae59a in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff76feba0 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007fffed0127f8 in ?? ()
#4 0x00007fffed006058 in ?? ()
#5 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7fffe7af6700 (LWP 14974)):
#0 0x00000030ab6e8ef3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fffec6f1b3b in Java_sun_nio_ch_EPollArrayWrapper_epollWait () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/libnio.so
#2 0x00007fffed0127f8 in ?? ()
#3 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7fffec18f700 (LWP 14901)):
#0 0x00000030aba0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff75ae59a in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff76feba0 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007fffed0127f8 in ?? ()
#4 0x00007fffed006058 in ?? ()
#5 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7fffec9f9700 (LWP 14545)):
#0 0x00000030aba0b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff75ae8af in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff7577758 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff7577a9e in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff76da5e3 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff76dab4f in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff75af9a2 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#7 0x00000030aba079d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00000030ab6e88fd in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fffecafa700 (LWP 14544)):
#0 0x00000030aba0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff75a918b in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff75774c3 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff7577a9e in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff7638589 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff76e0ba5 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff76e0e30 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#7 0x00007ffff75af9a2 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#8 0x00000030aba079d1 in start_thread () from /lib64/libpthread.so.0
#9 0x00000030ab6e88fd in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fffeccfc700 (LWP 14542)):
#0 0x00000030aba0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff75a918b in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff7576e78 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff75770ff in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff7635394 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#5 0x00007ffff76dcca8 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#6 0x00007ffff71a9473 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#7 0x00007ffff71c53e3 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#8 0x00007ffff75e1865 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#9 0x00007ffff75df3b4 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#10 0x00007ffff75d2442 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#11 0x00007ffff75d34e2 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#12 0x00007ffff75d607f in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#13 0x00007ffff718dfc8 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#14 0x00007ffff721bbf4 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#15 0x00007ffff718d00c in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#16 0x00007ffff722145b in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#17 0x00007ffff7222ac1 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#18 0x00007ffff76e0ba5 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#19 0x00007ffff76e0e30 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#20 0x00007ffff75af9a2 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#21 0x00000030aba079d1 in start_thread () from /lib64/libpthread.so.0
#22 0x00000030ab6e88fd in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fffecfff700 (LWP 14538)):
#0 0x00000030aba0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007ffff75a918b in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#2 0x00007ffff759ecdc in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff76a2eb8 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff7419760 in JVM_MonitorWait () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#5 0x00007fffed0127f8 in ?? ()
#6 0x00007fffecffea50 in ?? ()
#7 0x00007ffff0075ff0 in ?? ()
#8 0x00007fffecffe880 in ?? ()
#9 0x00007fffecffe818 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7ffff4195700 (LWP 14537)):
#0 0x00000030ab632625 in raise () from /lib64/libc.so.6
#1 0x00000030ab633e05 in abort () from /lib64/libc.so.6
#2 0x00007ffff75ae9e5 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff772281f in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff75b3e12 in JVM_handle_linux_signal () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#5 <signal handler called>
#6 0x00007ffff7611f52 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#7 0x00007ffff769f0c6 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#8 0x00007ffff761125b in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#9 0x00007ffff7611d8b in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#10 0x00007ffff75cbd7d in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#11 0x00007ffff7723e77 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#12 0x00007ffff772a3f2 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#13 0x00007ffff7728865 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#14 0x00007ffff7728d23 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#15 0x00007ffff77291f2 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#16 0x00007ffff75af9a2 in ?? () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64/jre/lib/amd64/server/libjvm.so
#17 0x00000030aba079d1 in start_thread () from /lib64/libpthread.so.0
#18 0x00000030ab6e88fd in clone () from /lib64/libc.so.6

[Arvind Narain (arvind-narain)]

Made the following changes to test:

1. Commented out nproc entry in /etc/security/limits.d/90-nproc.conf ( to enable values set in /etc/security/limits.conf to be picked up ) - didn't help
2. Used the default java ( /usr/java/jdk1.7.0_67-cloudera ) rather than what is being used in the test - /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64 - didn't help though the stack gave more information ( attached)

stack quite similar to what is reported here - https://github.com/ochafik/nativelibs4java/issues/420
and here
https://bugs.openjdk.java.net/browse/JDK-8025834?page=com.atlassian.streams.streams-jira-plugin:activity-stream-issue-tab

#0 0x00000033ff632625 in raise () from /lib64/libc.so.6
#1 0x00000033ff633e05 in abort () from /lib64/libc.so.6
#2 0x00007ffff736fa55 in os::abort(bool) ()
   from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff74eff87 in VMError::report_and_die() ()
   from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff737496f in JVM_handle_linux_signal ()
   from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#5 <signal handler called>
#6 0x00007ffff73d9c5c in PSRootsClosure<false>::do_oop(oopDesc**) ()
   from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#7 0x00007ffff703d6dd in Dictionary::oops_do(OopClosure*) ()
   from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#8 0x00007ffff746fe78 in SystemDictionary::oops_do(OopClosure*) ()
   from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#9 0x00007ffff73da321 in ScavengeRootsTask::do_it(GCTaskManager*, unsigned int) () from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#10 0x00007ffff70a804f in GCTaskThread::run() ()
   from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#11 0x00007ffff7370988 in java_start(Thread*) ()
   from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#12 0x00000033ffa079d1 in start_thread () from /lib64/libpthread.so.0
#13 0x00000033ff6e88fd in clone () from /lib64/libc.so.6
(gdb)

3. Suspecting GC - modified the python script to download the repository for maven before running the tests - didn't help - output attached dependency tree might give a clue.

# clean the target
output = shell_call(gvars.my_EXPORT_CMD + ';mvn clean')
stdout_write(output + '\n')

# download the dependencies first
output = shell_call(gvars.my_EXPORT_CMD + ';mvn dependency:tree -Dverbose')
stdout_write(output + '\n')

# download the dependencies first
output = shell_call(gvars.my_EXPORT_CMD + ';mvn compile')
stdout_write(output + '\n')

# download the dependencies first
output = shell_call('echo $CLASSPATH')
stdout_write(output + '\n')

# do the whole build, including running tests
output = shell_call(gvars.my_EXPORT_CMD + ';mvn test -Dtest=' + ArgList._tests)
====

next step

- try by using the release version rather than debug version
- check if -XX:+CheckUnhandledOops is setup anywhere

[Weiqing Xu (wei-qing-xu)]

This core was thought to be related to CLASSPATH in the maven. I will check it after this particular checkin is merged. https://review.trafodion.org/#/c/1489/