Comment 3 for bug 1366227

Change 451 = https://review.trafodion.org/#/c/451/ encountered a similar stack trace in shutdown of core-regress-seabase-cdh4.4.

https://jenkins01.trafodion.org/job/core-regress-seabase-cdh4.4/345/console has

Shutting down (normal) the SQ environment!
Wed Sep 24 22:47:59 UTC 2014
Processing cluster.conf on local host slave07
[$Z0001ZW] Shell/shell Version 1.0.1 Release 0.9.0 (Build release [0.8.3rc1-279-g398c34fmaster_Bld345], date 20140924_213511)
ps
[$Z0001ZW] %ps
[$Z0001ZW] NID,PID(os) PRI TYPE STATES NAME PARENT PROGRAM
[$Z0001ZW] ------------ --- ---- ------- ----------- ----------- ---------------
[$Z0001ZW] 000,00009116 000 WDG ES--A-- $WDT000 NONE sqwatchdog
[$Z0001ZW] 000,00009117 000 PSD ES--A-- $PSD000 NONE pstartd
[$Z0001ZW] 000,00009158 001 DTM ES--A-- $TM0 NONE tm
[$Z0001ZW] 000,00002446 001 GEN ES--A-- $Z0001ZW NONE shell
[$Z0001ZW] 001,00009111 000 WDG ES--A-- $WDT001 NONE sqwatchdog
[$Z0001ZW] 001,00009112 000 PSD ES--A-- $PSD001 NONE pstartd
[$Z0001ZW] 001,00009238 001 DTM ES--A-- $TM1 NONE tm
shutdown
[$Z0001ZW] %shutdown
[$Z0001ZW] Shutdown notice, level=0 received
/home/jenkins/workspace/core-regress-seabase-cdh4.4/trafodion/core/sqf/sql/scripts/sqshell: line 7: 2446 Aborted (core dumped) shell $1 $2 $3 $4 $5 $6 $7 $8 $9
Issued a 'shutdown normal' request

Shutdown in progress
. . .
========= seabase
09/24/14 22:10:46 (RELEASE build)
09/24/14 22:14:01 TEST010 ### PASS ###
09/24/14 22:16:07 TEST011 ### PASS ###
09/24/14 22:18:38 TEST012 ### PASS ###
09/24/14 22:20:39 TEST013 ### PASS ###
09/24/14 22:20:57 TEST014 ### PASS ###
09/24/14 22:24:10 TEST016 ### PASS ###
09/24/14 22:24:41 TEST018 ### PASS ###
09/24/14 22:45:44 TEST020 ### PASS ###
09/24/14 22:46:03 TEST022 ### PASS ###
09/24/14 22:47:59 TEST024 ### PASS ###
09/24/14 22:10:46 - 22:47:59 (RELEASE build)

WARNING: Core files found in /home/jenkins/workspace/core-regress-seabase-cdh4.4/trafodion/core :
-rw-------. 1 jenkins jenkins 44548096 Sep 24 22:47 sql/regress/core.slave07.2446.shell
core file : -rw-------. 1 jenkins jenkins 44548096 Sep 24 22:47 ./sqf/sql/regress/core.slave07.2446.shell
executable : -rwxr-xr-x. 1 jenkins jenkins 2023501 Sep 24 21:40 /home/jenkins/workspace/core-regress-seabase-cdh4.4/trafodion/core/sqf/export/bin64/shell

gdb command:
  gdb /home/jenkins/workspace/core-regress-seabase-cdh4.4/trafodion/core/sqf/export/bin64/shell ./sqf/sql/regress/core.slave07.2446.shell --batch -n -x /tmp/tmp.YACcmrJ2m2 2>&1
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/1e/0a7d58f454926e2afb4797865d85801ed65ece
[New Thread 2447]
[New Thread 2446]
[Thread debugging using libthread_db enabled]
Core was generated by `shell -a'.
Program terminated with signal 6, Aborted.
#0 0x000000374b832635 in raise () from /lib64/libc.so.6

Thread 2 (Thread 0x7ffff7cdeb40 (LWP 2446)):
#0 0x000000374c00b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000410965 in Local_IO_To_Monitor::wait_on_cv (this=0x7c6e80) at clio.cxx:1978
#2 0x0000000000411e21 in Local_IO_To_Monitor::send_recv (this=0x7c6e80, pp_msg=0x7ffff6660ecc, pv_nw=false) at clio.cxx:1452
#3 0x000000000040a426 in exit_process () at shell.cxx:1874
#4 0x000000000040a75c in shutdown (level=<value optimized out>) at shell.cxx:4183
#5 0x000000000040f7f5 in process_command (token=<value optimized out>, cmd_tail=0x7cb649 "", delimiter=32 ' ') at shell.cxx:5607
#6 0x000000000041005f in main (argc=2, argv=0x7fffffff6fc8) at shell.cxx:6191

Thread 1 (Thread 0x7ffff5baa700 (LWP 2447)):
#0 0x000000374b832635 in raise () from /lib64/libc.so.6
#1 0x000000374b833e15 in abort () from /lib64/libc.so.6
#2 0x0000000000411982 in LIOTM_assert_fun (pp_exp=0x4d4f80 "0", pp_file=0x4d179e "clio.cxx", pv_line=1022, pp_fun=0x4d2da0 "int Local_IO_To_Monitor::process_notice(message_def*)") at clio.cxx:99
#3 0x0000000000413b26 in Local_IO_To_Monitor::process_notice (this=0x7c6e80, pp_msg=<value optimized out>) at clio.cxx:1022
#4 0x0000000000413e03 in Local_IO_To_Monitor::get_io (this=0x7c6e80, pv_sig=<value optimized out>, pp_siginfo=<value optimized out>) at clio.cxx:637
#5 0x0000000000414075 in local_monitor_reader (pp_arg=0x2387) at clio.cxx:154
#6 0x000000374c0079d1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000374b8e886d in clone () from /lib64/libc.so.6

To debug this yourself:
ssh slave07
cd /home/jenkins/workspace/core-regress-seabase-cdh4.4/trafodion/core; gdb /home/jenkins/workspace/core-regress-seabase-cdh4.4/trafodion/core/sqf/export/bin64/shell ./sqf/sql/regress/core.slave07.2446.shell
----------------------------------------------