T2 multithread test cores in SB_Trans::Trans_Stream::finish_reply_static with > 10 threads

[Aruna Sadashiva]

With mainline build from 20150106, during T2 multithread testing, this core is generated with > 10 threads.

According to DaveG, the trace suggests that the completion-thread is trying to signal a ‘dead’ thread.

Core was generated by `java -Dprop=t2prop MT 12'.
Program terminated with signal 6, Aborted.
#0 0x00007ffff70408a5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-13.el6.x86_64 glibc-2.12-1.107.el6.x86_64 hadoop-2.3.0+cdh5.1.3+824-1.cdh5.1.3.p0.13.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.9-33.el6.x86_64 libcom_err-1.41.12-12.el6.x86_64 libgcc-4.4.6-4.el6.x86_64 libselinux-2.0.94-5.3.el6.x86_64 libstdc++-4.4.6-4.el6.x86_64 nspr-4.9.2-1.el6.x86_64 nss-3.14.0.0-12.el6.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64 nss-util-3.14.0.0-2.el6.x86_64 openldap-2.4.23-26.el6.x86_64 openssl-1.0.0-20.el6_2.5.x86_64 zlib-1.2.3-27.el6.x86_64
(gdb) bt
#0 0x00007ffff70408a5 in raise () from /lib64/libc.so.6
#1 0x00007ffff7042085 in abort () from /lib64/libc.so.6
#2 0x00007fffaeea19a6 in sb_util_assert_fun_com (pv_assert=ASSERT_INTCMP,
    pp_exp=0x7fffaeea6a70 "lv_status == 0", pv_lhs=22,
    pp_op=0x7fffaeea80ae "==", pv_rhs=0,
    pp_file=0x7fffaeea6a00 "/home/jenkins/workspace/traf-pub-release/trafodion/core/sqf/export/include/seabed/int/thread.inl", pv_line=488,
    pp_fun=0x7fffaeea7440 "int SB_Thread::CV::signal(bool)") at util.cpp:271
#3 0x00007fffaeea1b7c in SB_util_assert_fun_ieq (
    pp_exp=0x7fffaeea6a70 "lv_status == 0", pv_lhs=22, pv_rhs=0,
    pp_file=0x7fffaeea6a00 "/home/jenkins/workspace/traf-pub-release/trafodion/core/sqf/export/include/seabed/int/thread.inl", pv_line=488,
    pp_fun=0x7fffaeea7440 "int SB_Thread::CV::signal(bool)") at util.cpp:364
#4 0x00007fffaee9c55f in SB_Thread::CV::signal (this=0x1500b48, pv_lock=true)
    at /home/jenkins/workspace/traf-pub-release/trafodion/core/sqf/export/include/seabed/int/thread.inl:488
#5 0x00007fffb0d26128 in SB_Ms_Event_Mgr::set_event (this=0x1500a50,
    pv_event=1024, pp_done=0xd54eba) at mseventmgr.inl:212
#6 0x00007fffb0d5a37a in SB_Trans::Trans_Stream::finish_reply_static (
    pp_md=0xd54e90, pv_fserr=0, pv_harderr=false, pv_generation=0, pp_req_map=
    0x17b81f0, pv_req_map_lock=true, pv_self=false, pv_ms_comp_callback=
    0x7fffb0d1d904 <ms_ldone_cbt(MS_Md_Type*)>) at stream.cpp:1295
#7 0x00007fffb0d5a0d0 in SB_Trans::Trans_Stream::finish_reply (
    this=0x17b7c80, pp_md=0xd54e90, pv_fserr=0, pv_harderr=false,
    pv_req_map_lock=true) at stream.cpp:1227
#8 0x00007fffb0d59111 in SB_Trans::Trans_Stream::finish_recv (this=0x17b7c80,
    pp_rd=0x17b82c8, pp_hdr=0x17b82e8, pv_client=true) at stream.cpp:842
#9 0x00007fffb0d5229e in SB_Trans::Sock_Stream::process_events (
    this=0x17b7c80, pv_events=1) at sockstream.cpp:491
---Type <return> to continue, or q <return> to quit---
#10 0x00007fffb0d56752 in SB_Trans::Sock_Stream_EH::process_events (
    this=0x1cde8a0, pv_events=1) at sockstream.cpp:2133
#11 0x00007fffb0d4ee59 in SB_Trans::Sock_Controller::epoll_wait (
    this=0x7fffb0f9fc80, pp_where=0x7fffb0d71be8 "Sock_Comp_Thread::run",
    pv_timeout=-1) at sock.cpp:363
#12 0x00007fffb0d4e213 in SB_Trans::Sock_Comp_Thread::run (this=0x13077e0)
    at sock.cpp:104
#13 0x00007fffb0d4e04d in sock_comp_thread_fun (pp_arg=0x13077e0)
    at sock.cpp:74
#14 0x00007fffaee99b9f in SB_Thread::Thread::disp (this=0x13077e0,
    pp_arg=0x13077e0) at thread.cpp:211
#15 0x00007fffaee99ff7 in thread_fun (pp_arg=0x13077e0) at thread.cpp:307
#16 0x00007fffaee9d290 in sb_thread_sthr_disp (pp_arg=0x1307900)
    at threadl.cpp:253
#17 0x00007ffff77c3851 in start_thread () from /lib64/libpthread.so.0
#18 0x00007ffff70f690d in clone () from /lib64/libc.so.6
(gdb)

[Aruna Sadashiva (aruna-sadashiva)]

With 20150322 build, it worked ok upto 48 threads, did not see this core.