With stats enabled, after dcsstart, there are many mxosrvr cores with signal 6/11. stack traces below:
Node 14:
Core was generated by `mxosrvr -ZKHOST n013:2181,n014:2181,n015:2181 -RZ g4q0014.houston.hp.com:2:8 -Z'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007ffff7ff9000 in ?? ()
Missing separate debuginfos, use: debuginfo-install boost-filesystem-1.41.0-11.el6_1.2.x86_64 boost-program-options-1.41.0-11.el6_1.2.x86_64 boost-system-1.41.0-11.el6_1.2.x86_64 cyrus-sasl-lib-2.1.23-13.el6.x86_64 glibc-2.12-1.107.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.9-33.el6.x86_64 libcom_err-1.41.12-12.el6.x86_64 libgcc-4.4.6-4.el6.x86_64 libselinux-2.0.94-5.3.el6.x86_64 libstdc++-4.4.6-4.el6.x86_64 libuuid-2.17.2-12.7.el6.x86_64 nspr-4.9.2-1.el6.x86_64 nss-3.14.0.0-12.el6.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64 nss-util-3.14.0.0-2.el6.x86_64 openldap-2.4.23-26.el6.x86_64 openssl-1.0.0-20.el6_2.5.x86_64 qpid-cpp-client-0.14-22.el6_3.x86_64 zlib-1.2.3-27.el6.x86_64
(gdb) bt
#0 0x00007ffff7ff9000 in ?? ()
#1 0x00007ffff67e98c8 in log4cpp::Category::getChainedPriority (this=<value optimized out>) at Category.cpp:87
#2 0x00007ffff67e9881 in log4cpp::Category::isPriorityEnabled (this=<value optimized out>, priority=0) at Category.cpp:264
#3 0x00007ffff67e93ce in log4cpp::Category::fatal (this=0xeb8b80, message=
"Node Number: 0, CPU: -465227280, PIN: 32767, Process Name: ,,, Could not initialize CLI globals.ComRtGetProgramInfo returned an error :160., FILE: ../cli/Globals.cpp, LINE: 164 TRANSACTION ID: 11604 ") at Category.cpp:413
#4 0x000000000047c37f in CommonLogger::log1 (cat=0x7fffed0aa1cb "SQL", level=<value optimized out>,
cmsg=<value optimized out>, eventId=<value optimized out>)
at /opt/home/tharakc/Git/trafodion/core/sqf/commonLogger/CommonLogger.cpp:132
#5 0x00007fffed080a26 in QRLogger::log (cat=0x7fffed0aa1cb "SQL", level=LL_FATAL, logMsgTemplate=<value optimized out>)
at ../qmscommon/QRLogger.cpp:466
#6 0x00007ffff27d3b3d in SQLMXLoggingArea::logSQLMXAssertionFailureEvent (file=<value optimized out>, line=164,
msgTxt=<value optimized out>, condition=0x0, tid=<value optimized out>) at ../sqlmxevents/logmxevent_traf.cpp:328
#7 0x00007ffff14545cc in assert_botch_abend (f=0x7ffff4592c07 "../cli/Globals.cpp", l=164,
m=0x7fffe44556c0 "Could not initialize CLI globals.ComRtGetProgramInfo returned an error :160.", c=0x0)
at ../export/NAAbort.cpp:234
#8 0x00007ffff455f190 in CliGlobals::init (this=0xec8800, espProcess=0, statsGlobals=<value optimized out>)
at ../cli/Globals.cpp:164
#9 0x00007ffff4562e2d in CliGlobals::CliGlobals (this=0xec8800, espProcess=0) at ../cli/Globals.cpp:113
#10 0x00007ffff4562f40 in CliGlobals::createCliGlobals (espProcess=0) at ../cli/Globals.cpp:542
#11 0x00007ffff4588244 in CliNonPrivPrologue () at ../cli/CliExtern.cpp:920
#12 0x00007ffff458e86c in SQL_EXEC_ClearDiagnostics (statement_id=0x0) at ../cli/CliExtern.cpp:1348
#13 0x00007ffff6a73569 in SRVR::WSQL_EXEC_ClearDiagnostics (statement_id=<value optimized out>) at SQLWrapper.cpp:137
#14 0x00007ffff6a754f6 in SRVR::WSQL_EXEC_CreateContext (context_handle=0x7fffe4455d80, sqlAuthId=<value optimized out>,
future=<value optimized out>) at SQLWrapper.cpp:170
#15 0x00000000004cc1d8 in SessionWatchDog (arg=<value optimized out>) at SrvrConnect.cpp:385
#16 0x00007ffff47c8851 in start_thread () from /lib64/libpthread.so.0
#17 0x00007ffff4cfe90d in clone () from /lib64/libc.so.6
Node 15:
Core was generated by `mxosrvr -ZKHOST n013:2181,n014:2181,n015:2181 -RZ g4q0015.houston.hp.com:3:28 -'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000000ffffffff in ?? ()
Missing separate debuginfos, use: debuginfo-install boost-filesystem-1.41.0-11.el6_1.2.x86_64 boost-program-options-1.41.0-11.el6_1.2.x86_64 boost-system-1.41.0-11.el6_1.2.x86_64 cyrus-sasl-lib-2.1.23-13.el6.x86_64 glibc-2.12-1.107.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.9-33.el6.x86_64 libcom_err-1.41.12-12.el6.x86_64 libgcc-4.4.6-4.el6.x86_64 libselinux-2.0.94-5.3.el6.x86_64 libstdc++-4.4.6-4.el6.x86_64 libuuid-2.17.2-12.7.el6.x86_64 nspr-4.9.2-1.el6.x86_64 nss-3.14.0.0-12.el6.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64 nss-util-3.14.0.0-2.el6.x86_64 openldap-2.4.23-26.el6.x86_64 openssl-1.0.0-20.el6_2.5.x86_64 qpid-cpp-client-0.14-22.el6_3.x86_64 zlib-1.2.3-27.el6.x86_64
(gdb) bt
#0 0x00000000ffffffff in ?? ()
#1 0x00007fffed08067f in QRLogger::log (cat=0x7fffed0aa1cb "SQL", level=LL_FATAL, logMsgTemplate=0x7ffff27d3dd3 ",, %s")
at ../qmscommon/QRLogger.cpp:456
#2 0x00007ffff27d3b3d in SQLMXLoggingArea::logSQLMXAssertionFailureEvent (file=<value optimized out>, line=164,
msgTxt=<value optimized out>, condition=0x0, tid=<value optimized out>) at ../sqlmxevents/logmxevent_traf.cpp:328
#3 0x00007ffff14545cc in assert_botch_abend (f=0x7ffff4592c07 "../cli/Globals.cpp", l=164,
m=0x7fffe44556c0 "Could not initialize CLI globals.ComRtGetProgramInfo returned an error :160.", c=0x0)
at ../export/NAAbort.cpp:234
#4 0x00007ffff455f190 in CliGlobals::init (this=0xec8590, espProcess=0, statsGlobals=<value optimized out>)
at ../cli/Globals.cpp:164
#5 0x00007ffff4562e2d in CliGlobals::CliGlobals (this=0xec8590, espProcess=0) at ../cli/Globals.cpp:113
#6 0x00007ffff4562f40 in CliGlobals::createCliGlobals (espProcess=0) at ../cli/Globals.cpp:542
#7 0x00007ffff4588244 in CliNonPrivPrologue () at ../cli/CliExtern.cpp:920
#8 0x00007ffff458e86c in SQL_EXEC_ClearDiagnostics (statement_id=0x0) at ../cli/CliExtern.cpp:1348
#9 0x00007ffff6a73569 in SRVR::WSQL_EXEC_ClearDiagnostics (statement_id=<value optimized out>) at SQLWrapper.cpp:137
#10 0x00007ffff6a754f6 in SRVR::WSQL_EXEC_CreateContext (context_handle=0x7fffe4455d80, sqlAuthId=<value optimized out>,
future=<value optimized out>) at SQLWrapper.cpp:170
#11 0x00000000004cc1d8 in SessionWatchDog (arg=<value optimized out>) at SrvrConnect.cpp:385
#12 0x00007ffff47c8851 in start_thread () from /lib64/libpthread.so.0
#13 0x00007ffff4cfe90d in clone () from /lib64/libc.so.6
Core was generated by `mxosrvr -ZKHOST n013:2181,n014:2181,n015:2181 -RZ g4q0015.houston.hp.com:3:31 -'.
Program terminated with signal 11, Segmentation fault.
#0 size (this=0x7ffff6a0f500, name=...)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/basic_string.h:629
629 /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/basic_string.h: No such file or directory.
in /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/basic_string.h
Missing separate debuginfos, use: debuginfo-install boost-filesystem-1.41.0-11.el6_1.2.x86_64 boost-program-options-1.41.0-11.el6_1.2.x86_64 boost-system-1.41.0-11.el6_1.2.x86_64 cyrus-sasl-lib-2.1.23-13.el6.x86_64 glibc-2.12-1.107.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.9-33.el6.x86_64 libcom_err-1.41.12-12.el6.x86_64 libgcc-4.4.6-4.el6.x86_64 libselinux-2.0.94-5.3.el6.x86_64 libstdc++-4.4.6-4.el6.x86_64 libuuid-2.17.2-12.7.el6.x86_64 nspr-4.9.2-1.el6.x86_64 nss-3.14.0.0-12.el6.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64 nss-util-3.14.0.0-2.el6.x86_64 openldap-2.4.23-26.el6.x86_64 openssl-1.0.0-20.el6_2.5.x86_64 qpid-cpp-client-0.14-22.el6_3.x86_64 zlib-1.2.3-27.el6.x86_64
(gdb) bt
#0 size (this=0x7ffff6a0f500, name="SQL")
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/basic_string.h:629
#1 compare (this=0x7ffff6a0f500, name="SQL")
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/basic_string.h:2021
#2 operator< <char, std::char_traits<char>, std::allocator<char> > (this=0x7ffff6a0f500, name="SQL")
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/basic_string.h:2317
#3 operator() (this=0x7ffff6a0f500, name="SQL")
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/stl_function.h:230
#4 _M_lower_bound (this=0x7ffff6a0f500, name="SQL")
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/stl_tree.h:986
#5 find (this=0x7ffff6a0f500, name="SQL")
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/stl_tree.h:1421
#6 find (this=0x7ffff6a0f500, name="SQL")
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/stl_map.h:659
#7 log4cpp::HierarchyMaintainer::_getExistingInstance (this=0x7ffff6a0f500, name="SQL") at HierarchyMaintainer.cpp:47
#8 0x00007ffff67ee338 in log4cpp::HierarchyMaintainer::_getInstance (this=0x7ffff6a0f500, name="SQL")
at HierarchyMaintainer.cpp:63
#9 0x00007ffff67ede23 in log4cpp::HierarchyMaintainer::getInstance (this=<value optimized out>, name="SQL")
at HierarchyMaintainer.cpp:57
#10 0x00007fffed08065e in QRLogger::log (cat=0x7fffed0aa1cb "SQL", level=LL_FATAL, logMsgTemplate=0x7ffff27d3dd3 ",, %s")
at ../qmscommon/QRLogger.cpp:455
#11 0x00007ffff27d3b3d in SQLMXLoggingArea::logSQLMXAssertionFailureEvent (file=<value optimized out>, line=164,
msgTxt=<value optimized out>, condition=0x0, tid=<value optimized out>) at ../sqlmxevents/logmxevent_traf.cpp:328
#12 0x00007ffff14545cc in assert_botch_abend (f=0x7ffff4592c07 "../cli/Globals.cpp", l=164,
m=0x7fffe44556c0 "Could not initialize CLI globals.ComRtGetProgramInfo returned an error :160.", c=0x0)
at ../export/NAAbort.cpp:234
#13 0x00007ffff455f190 in CliGlobals::init (this=0xec38e0, espProcess=0, statsGlobals=<value optimized out>)
at ../cli/Globals.cpp:164
#14 0x00007ffff4562e2d in CliGlobals::CliGlobals (this=0xec38e0, espProcess=0) at ../cli/Globals.cpp:113
#15 0x00007ffff4562f40 in CliGlobals::createCliGlobals (espProcess=0) at ../cli/Globals.cpp:542
#16 0x00007ffff4588244 in CliNonPrivPrologue () at ../cli/CliExtern.cpp:920
#17 0x00007ffff458e86c in SQL_EXEC_ClearDiagnostics (statement_id=0x0) at ../cli/CliExtern.cpp:1348
#18 0x00007ffff6a73569 in SRVR::WSQL_EXEC_ClearDiagnostics (statement_id=<value optimized out>) at SQLWrapper.cpp:137
#19 0x00007ffff6a754f6 in SRVR::WSQL_EXEC_CreateContext (context_handle=0x7fffe4455d80, sqlAuthId=<value optimized out>,
future=<value optimized out>) at SQLWrapper.cpp:170
#20 0x00000000004cc1d8 in SessionWatchDog (arg=<value optimized out>) at SrvrConnect.cpp:385
---Type <return> to continue, or q <return> to quit---
#21 0x00007ffff47c8851 in start_thread () from /lib64/libpthread.so.0
#22 0x00007ffff4cfe90d in clone () from /lib64/libc.so.6
(gdb)
Core was generated by `mxosrvr -ZKHOST n013:2181,n014:2181,n015:2181 -RZ g4q0013.houston.hp.com:1:31 -'.
Program terminated with signal 11, Segmentation fault.
#0 size (this=0x7ffff6a0f500, name=...)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/basic_string.h:629
629 /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/basic_string.h: No such file or directory.
in /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/basic_string.h
Missing separate debuginfos, use: debuginfo-install boost-filesystem-1.41.0-11.el6_1.2.x86_64 boost-program-options-1.41.0-11.el6_1.2.x86_64 boost-system-1.41.0-11.el6_1.2.x86_64 cyrus-sasl-lib-2.1.23-13.el6.x86_64 glibc-2.12-1.107.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.9-33.el6.x86_64 libcom_err-1.41.12-12.el6.x86_64 libgcc-4.4.6-4.el6.x86_64 libselinux-2.0.94-5.3.el6.x86_64 libstdc++-4.4.6-4.el6.x86_64 libuuid-2.17.2-12.7.el6.x86_64 nspr-4.9.2-1.el6.x86_64 nss-3.14.0.0-12.el6.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64 nss-util-3.14.0.0-2.el6.x86_64 openldap-2.4.23-26.el6.x86_64 openssl-1.0.0-20.el6_2.5.x86_64 qpid-cpp-client-0.14-22.el6_3.x86_64 zlib-1.2.3-27.el6.x86_64
(gdb) bt
#0 size (this=0x7ffff6a0f500, name="")
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/basic_string.h:629
#1 compare (this=0x7ffff6a0f500, name="")
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/basic_string.h:2021
#2 operator< <char, std::char_traits<char>, std::allocator<char> > (this=0x7ffff6a0f500, name="")
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/basic_string.h:2317
#3 operator() (this=0x7ffff6a0f500, name="")
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/stl_function.h:230
#4 _M_lower_bound (this=0x7ffff6a0f500, name="")
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/stl_tree.h:986
#5 find (this=0x7ffff6a0f500, name="")
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/stl_tree.h:1421
#6 find (this=0x7ffff6a0f500, name="")
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/stl_map.h:659
#7 log4cpp::HierarchyMaintainer::_getExistingInstance (this=0x7ffff6a0f500, name="") at HierarchyMaintainer.cpp:47
#8 0x00007ffff67ee338 in log4cpp::HierarchyMaintainer::_getInstance (this=0x7ffff6a0f500, name="")
at HierarchyMaintainer.cpp:63
#9 0x00007ffff67ee3e3 in log4cpp::HierarchyMaintainer::_getInstance (this=0x7ffff6a0f500, name="SQL")
at HierarchyMaintainer.cpp:76
#10 0x00007ffff67ede23 in log4cpp::HierarchyMaintainer::getInstance (this=<value optimized out>, name="SQL")
at HierarchyMaintainer.cpp:57
#11 0x00007fffed08065e in QRLogger::log (cat=0x7fffed0aa1cb "SQL", level=LL_FATAL, logMsgTemplate=0x7ffff27d3dd3 ",, %s")
at ../qmscommon/QRLogger.cpp:455
#12 0x00007ffff27d3b3d in SQLMXLoggingArea::logSQLMXAssertionFailureEvent (file=<value optimized out>, line=164,
msgTxt=<value optimized out>, condition=0x0, tid=<value optimized out>) at ../sqlmxevents/logmxevent_traf.cpp:328
#13 0x00007ffff14545cc in assert_botch_abend (f=0x7ffff4592c07 "../cli/Globals.cpp", l=164,
m=0x7fffe44556c0 "Could not initialize CLI globals.ComRtGetProgramInfo returned an error :160.", c=0x0)
at ../export/NAAbort.cpp:234
#14 0x00007ffff455f190 in CliGlobals::init (this=0xec3870, espProcess=0, statsGlobals=<value optimized out>)
at ../cli/Globals.cpp:164
#15 0x00007ffff4562e2d in CliGlobals::CliGlobals (this=0xec3870, espProcess=0) at ../cli/Globals.cpp:113
#16 0x00007ffff4562f40 in CliGlobals::createCliGlobals (espProcess=0) at ../cli/Globals.cpp:542
#17 0x00007ffff4588244 in CliNonPrivPrologue () at ../cli/CliExtern.cpp:920
#18 0x00007ffff458e86c in SQL_EXEC_ClearDiagnostics (statement_id=0x0) at ../cli/CliExtern.cpp:1348
#19 0x00007ffff6a73569 in SRVR::WSQL_EXEC_ClearDiagnostics (statement_id=<value optimized out>) at SQLWrapper.cpp:137
#20 0x00007ffff6a754f6 in SRVR::WSQL_EXEC_CreateContext (context_handle=0x7fffe4455d80, sqlAuthId=<value optimized out>,
---Type <return> to continue, or q <return> to quit---
future=<value optimized out>) at SQLWrapper.cpp:170
#21 0x00000000004cc1d8 in SessionWatchDog (arg=<value optimized out>) at SrvrConnect.cpp:385
#22 0x00007ffff47c8851 in start_thread () from /lib64/libpthread.so.0
#23 0x00007ffff4cfe90d in clone () from /lib64/libc.so.6
Node 16:
Core was generated by `mxosrvr -ZKHOST n013:2181,n014:2181,n015:2181 -RZ g4q0016.houston.hp.com:4:25 -'.
Program terminated with signal 6, Aborted.
#0 0x00007ffff4c488a5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install boost-filesystem-1.41.0-11.el6_1.2.x86_64 boost-program-options-1.41.0-11.el6_1.2.x86_64 boost-system-1.41.0-11.el6_1.2.x86_64 cyrus-sasl-lib-2.1.23-13.el6.x86_64 glibc-2.12-1.107.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.9-33.el6.x86_64 libcom_err-1.41.12-12.el6.x86_64 libgcc-4.4.6-4.el6.x86_64 libselinux-2.0.94-5.3.el6.x86_64 libstdc++-4.4.6-4.el6.x86_64 libuuid-2.17.2-12.7.el6.x86_64 nspr-4.9.2-1.el6.x86_64 nss-3.14.0.0-12.el6.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64 nss-util-3.14.0.0-2.el6.x86_64 openldap-2.4.23-26.el6.x86_64 openssl-1.0.0-20.el6_2.5.x86_64 qpid-cpp-client-0.14-22.el6_3.x86_64 zlib-1.2.3-27.el6.x86_64
(gdb) bt
#0 0x00007ffff4c488a5 in raise () from /lib64/libc.so.6
#1 0x00007ffff4c4a085 in abort () from /lib64/libc.so.6
#2 0x00007ffff4c41a1e in __assert_fail_base () from /lib64/libc.so.6
#3 0x00007ffff4c41ae0 in __assert_fail () from /lib64/libc.so.6
#4 0x00007ffff70e637e in EnterCriticalSection (pCriticalSection=<value optimized out>) at traf_misc.cpp:107
#5 0x00007ffff4588217 in get () at ../cli/CliSemaphore.h:63
#6 CliNonPrivPrologue () at ../cli/CliExtern.cpp:913
#7 0x00007ffff458e86c in SQL_EXEC_ClearDiagnostics (statement_id=0x0) at ../cli/CliExtern.cpp:1348
#8 0x00007ffff6a73569 in SRVR::WSQL_EXEC_ClearDiagnostics (statement_id=<value optimized out>) at SQLWrapper.cpp:137
#9 0x00007ffff6a754f6 in SRVR::WSQL_EXEC_CreateContext (context_handle=0x7fffe4455d80, sqlAuthId=<value optimized out>,
future=<value optimized out>) at SQLWrapper.cpp:170
#10 0x00000000004cc1d8 in SessionWatchDog (arg=<value optimized out>) at SrvrConnect.cpp:385
#11 0x00007ffff47c8851 in start_thread () from /lib64/libpthread.so.0
#12 0x00007ffff4cfe90d in clone () from /lib64/libc.so.6
(gdb)
Fix proposed to branch: master /review. trafodion. org/948
Review: https:/