T2 phoenix test - ReadIsolationLevelTest - cores in NAHeap::unlinkLargeFragment

[Aruna Sadashiva]

T2 Phoenix test, ReadIsolationLevel test creates core. This is with build 20150108 with workaround, SQL_NO_REGISTER_CANCEL=1 in ms.env.

Program terminated with signal 6, Aborted.
#0 0x0000003c93032925 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-13.el6_3.1.x86_64 glibc-2.12-1.132.el6.x86_64 hadoop-2.4.0.2.1.1.0-390.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-15.el6_5.1.x86_64 libcom_err-1.41.12-18.el6.x86_64 libgcc-4.4.7-4.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 libstdc++-4.4.7-4.el6.x86_64 nspr-4.10.6-1.el6_5.x86_64 nss-3.16.1-4.el6_5.x86_64 nss-softokn-freebl-3.14.3-10.el6_5.x86_64 nss-util-3.16.1-1.el6_5.x86_64 openldap-2.4.23-34.el6_5.1.x86_64 openssl-1.0.1e-16.el6_5.7.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0 0x0000003c93032925 in raise () from /lib64/libc.so.6
#1 0x0000003c9303408d in abort () from /lib64/libc.so.6
#2 0x00007fe77dec3a55 in os::abort(bool) ()
   from /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007fe77e043f87 in VMError::report_and_die() ()
   from /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007fe77dec896f in JVM_handle_linux_signal ()
   from /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 <signal handler called>
#6 0x00007fe76588adf7 in NAHeap::unlinkLargeFragment (this=0x7fe75a231998,
    treeFrag=0x7fe752904bf8) at ../common/NAMemory.cpp:1878
#7 0x00007fe765888093 in unlinkFragment (this=0x7fe75a231998,
    addr=<value optimized out>) at ../common/NAMemory.cpp:1952
#8 NAHeap::deallocateHeapMemory (this=0x7fe75a231998,
    addr=<value optimized out>) at ../common/NAMemory.cpp:3615
#9 0x00007fe765842675 in ComSpace::freeBlocks (this=0x7fe7528f9dc0)
    at ../common/ComSpace.cpp:133
#10 0x00007fe7658426cf in ComSpace::~ComSpace (this=0x7fe7528f9dc0,
    __in_chrg=<value optimized out>) at ../common/ComSpace.cpp:90
#11 0x00007fe765842711 in ComSpace::~ComSpace (this=0x7fe7528f9dc0,
    __in_chrg=<value optimized out>) at ../common/ComSpace.cpp:91
#12 0x00007fe764ac4741 in CliStatement::~CliStatement (this=0x7fe7528f8bd0,
    __in_chrg=<value optimized out>) at ../cli/Statement.cpp:738
#13 0x00007fe764ac48d1 in CliStatement::~CliStatement (this=0x7fe7528f8bd0,
    __in_chrg=<value optimized out>) at ../cli/Statement.cpp:741
#14 0x00007fe764a8eb18 in ContextCli::deallocStmt (this=<value optimized out>,
    statement_id=0x1713030, deallocStaticStmt=0) at ../cli/Context.cpp:2329
#15 0x00007fe764a74faf in SQLCLI_DeallocStmt (
    cliGlobals=<value optimized out>, statement_id=0x1713030)
    at ../cli/Cli.cpp:1649
#16 0x00007fe764ad5ae6 in SQL_EXEC_DeallocStmt (statement_id=0x1713030)
    at ../cli/CliExtern.cpp:1823
#17 0x00007fe766b56c39 in FREESTATEMENT (pSrvrStmt=0x1712a20)
    at native/SqlInterface.cpp:1375
#18 0x00007fe766b50f3e in SRVR_CONNECT_HDL::sqlClose (this=0xccdc30)
    at native/CSrvrConnect.cpp:138
---Type <return> to continue, or q <return> to quit---
#19 0x00007fe766b6074e in Java_org_trafodion_jdbc_t2_SQLMXConnection_close (
    jenv=0xb469e8, jcls=<value optimized out>, server=<value optimized out>,
    dialogueId=13425712) at native/SQLMXConnection.cpp:252
#20 0x00007fe779d437f8 in ?? ()
#21 0x00007fe77d6a2350 in ?? ()
#22 0x00007fe77d6a23a8 in ?? ()
#23 0x00007fe77d6a23a8 in ?? ()
#24 0x00007fe779d37350 in ?? ()
#25 0x00007fe77d6a2350 in ?? ()
#26 0x0000000000000000 in ?? ()
(gdb)

[Selvaganesan Govindarajan (selva-ganesan)]

It looks like this particular T2 driver test involves multiple connections. With the multiple Cmp context per CLI context, we might be deleting the same CmpContext again and again. This issue have a different stack trace in debug mode. The debug mode asserts while deleting the CmpContext.

#0 0x0000003ce9e328a5 in raise () from /lib64/libc.so.6
#1 0x0000003ce9e34085 in abort () from /lib64/libc.so.6
#2 0x0000003ce9e2ba1e in __assert_fail_base () from /lib64/libc.so.6
#3 0x0000003ce9e2bae0 in __assert_fail () from /lib64/libc.so.6
#4 0x00007fffafd78c94 in NAHeap::doCheckAnyFragment (this=0x7fffa3b88f48,
    p=0x7fff9da12bf8) at ../common/NAMemory.cpp:3916
#5 0x00007fffafd78f0d in NAHeap::doCheckInuseFragment (this=0x7fffa3b88f48,
    p=0x7fff9da12bf8) at ../common/NAMemory.cpp:3945
#6 0x00007fffafd774c5 in NAHeap::deallocateHeapMemory (this=0x7fffa3b88f48,
    addr=0x7fff9da12c08) at ../common/NAMemory.cpp:3495
#7 0x00007fffafd72df3 in NAMemory::deallocateMemory (this=0x7fffa3b88f48,
    addr=0x7fff9da12c08) at ../common/NAMemory.cpp:1391
#8 0x00007fffb014df57 in CmpContext::deleteInstance (
    parentHeap=0x7fffa3b88f48) at ../arkcmp/CmpContext.cpp:364
#9 0x00007fffaed172f4 in ContextCli::deleteMe (this=0x7fffa3b88f38)
    at ../cli/Context.cpp:427
#10 0x00007fffaed4274b in CliGlobals::dropContext (this=0x1bbba90,
    context=0x7fffa3b88f38) at ../cli/Globals.cpp:611
#11 0x00007fffaece94be in SQLCLI_DropContext (cliGlobals=0x1bbba90,
    context_handle=2004) at ../cli/Cli.cpp:1818
#12 0x00007fffaece8924 in SQLCLI_DeleteContext (cliGlobals=0x1bbba90,
    context_handle=2004) at ../cli/Cli.cpp:1406
#13 0x00007fffaed79d03 in SQL_EXEC_DeleteContext (contextHandle=2004)
#14 0x00007fffb10ff425 in DISCONNECT (pSrvrConnect=0x2acd4e0)
    at native/SqlInterface.cpp:2844
#15 0x00007fffb10edf40 in SRVR_CONNECT_HDL::sqlClose (this=0x2acd4e0)
    at native/CSrvrConnect.cpp:156
#16 0x00007fffb110c60b in Java_org_trafodion_jdbc_t2_SQLMXConnection_close (
    jenv=0x6259e8, jcls=0x7ffff6b3e390, server=0x0, dialogueId=44881120)
    at native/SQLMXConnection.cpp:252
#17 0x00007ffff31cb7f8 in ?? ()
#18 0x00007ffff6b3e340 in ?? ()
#19 0x00007ffff6b3e398 in ?? ()
#20 0x00007ffff6b3e398 in ?? ()
#21 0x00007ffff31bf350 in ?? ()
#22 0x00007ffff6b3e340 in ?? ()
#23 0x0000000000000000 in ?? ()

In release mode, this assert is ignored due to the infamous change in NAAbort and hence we get into different stack trace confusing the whole issue. Assigning this issue back to Aruna to assign it to compiler team.

[Selvaganesan Govindarajan (selva-ganesan)]

Fixed. See https://review.trafodion.org/#/c/940/ for details