Trafodion

Alter rename cores mxosrvr at ExHbaseAccessTcb::setupUniqueKeyAndCols()

Bug #1395840 reported by Weishiun Tsai
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Trafodion
In Progress
Critical
Suresh Subbiah
Trafodion r2.0

Bug Description

This core was seen several times when running the following sequence of commands. This problem can’t always be reproduced, but when it occurred, it always happened with the same statement and the cores all showed the same stack.

This is seen on the v1119_0830 build installed on a 4-node cluster. It is a regression introduced between the v1108_0830 build and the v1119_0830 build.

------------------------------------------------------------------------------------------------------------------

Here is the stack of the core files:

#0 0x00007ffff527f8a5 in raise () from /lib64/libc.so.6
#1 0x00007ffff528100d in abort () from /lib64/libc.so.6
#2 0x00007ffff6388a55 in os::abort(bool) ()
   from /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff6508f87 in VMError::report_and_die() ()
   from /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff638d96f in JVM_handle_linux_signal ()
   from /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 <signal handler called>
#6 0x00007ffff3743a61 in ExHbaseAccessTcb::setupUniqueKeyAndCols (
    this=0x7fffce17ac50, doInit=0) at ../executor/ExHbaseAccess.cpp:1682
#7 0x00007ffff374ee5b in ExHbaseAccessSQRowsetTcb::work (this=0x7fffce17ac50)
    at ../executor/ExHbaseIUD.cpp:3759
#8 0x00007ffff37613c3 in ExScheduler::work (this=0x7fffce179758,
    prevWaitTime=<value optimized out>) at ../executor/ExScheduler.cpp:328
#9 0x00007ffff36d3542 in ex_root_tcb::execute (this=0x7fffce17ebd8,
    cliGlobals=0xf40b30, glob=0x7fffd029e780, input_desc=0x7fffd16bb090,
    diagsArea=@0x7fffe4646a00, reExecute=0) at ../executor/ex_root.cpp:1029
#10 0x00007ffff49c991c in CliStatement::execute (this=0x7fffce1bca38,
    cliGlobals=0xf40b30, input_desc=0x7fffd16bb090,
    diagsArea=<value optimized out>, execute_state=<value optimized out>,
    fixupOnly=0, cliflags=0) at ../cli/Statement.cpp:4779
#11 0x00007ffff498c22a in SQLCLI_PerformTasks(CliGlobals *, ULng32, SQLSTMT_ID *
, SQLDESC_ID *, SQLDESC_ID *, Lng32, Lng32, typedef __va_list_tag __va_list_tag *, SQLCLI_PTR_PAIRS *, SQLCLI_PTR_PAIRS *) (cliGlobals=0xf40b30, tasks=4882,
    statement_id=0x7fffcf38fe90, input_descriptor=0x7fffcf38ff00,
    output_descriptor=0x0, num_input_ptr_pairs=0, num_output_ptr_pairs=0,
    ap=0x7fffe4646ba0, input_ptr_pairs=0x0, output_ptr_pairs=0x0)
    at ../cli/Cli.cpp:3278
#12 0x00007ffff498cf52 in SQLCLI_Exec(CliGlobals *, SQLSTMT_ID *, SQLDESC_ID *, Lng32, typedef __va_list_tag __va_list_tag *, SQLCLI_PTR_PAIRS *) (
    cliGlobals=<value optimized out>, statement_id=<value optimized out>,
    input_descriptor=<value optimized out>,
    num_ptr_pairs=<value optimized out>, ap=<value optimized out>,
    ptr_pairs=<value optimized out>) at ../cli/Cli.cpp:3525
#13 0x00007ffff49d5ca5 in SQL_EXEC_Exec (statement_id=0x7fffcf38fe90,
    input_descriptor=0x7fffcf38ff00, num_ptr_pairs=0)
    at ../cli/CliExtern.cpp:2062
#14 0x00007ffff3710bd1 in ExeCliInterface::exec (this=0x7fffe4648fc0,
    inputBuf=<value optimized out>, inputBufLen=<value optimized out>)
    at ../executor/ExExeUtilCli.cpp:599
#15 0x00007ffff37144c5 in ExeCliInterface::executeImmediateExec (
    this=0x7fffe4648fc0, stmtStr=<value optimized out>, outputBuf=0x0,
    outputBufLen=0x0, nullTerminate=1, rowsAffected=0x0)
    at ../executor/ExExeUtilCli.cpp:898
#16 0x00007ffff371744c in ExeCliInterface::executeImmediate (
    this=0x7fffe4648fc0,
    stmtStr=0x7fffe4646e00 "update TRAFODION.\"_MD_\".OBJECTS set catalog_name = 'TRAFODION', schema_name = 'ALTER_RENAME_RENAME21', object_name = 'TAB_EMP_NEW_01' where object_uid = 103071044763640121 ", outputBuf=0x0, outputBufLen=0x0,
    nullTerminate=1, rowsAffected=0x0, monitorThis=0, globalDiags=0x0)
    at ../executor/ExExeUtilCli.cpp:993
#17 0x00007ffff01b9264 in CmpSeabaseDDL::updateObjectName (
    this=0x7fffe464ae30, cliInterface=0x7fffe4648fc0,
    objUID=103071044763640121, catName=0x7fffe4649848 "TRAFODION",
    schName=<value optimized out>, objName=<value optimized out>)
    at ../sqlcomp/CmpSeabaseDDLcommon.cpp:3636
#18 0x00007ffff01e9a1f in CmpSeabaseDDL::renameSeabaseTable (
    this=0x7fffe464ae30, renameTableNode=0x7fffe46488c0,
    currCatName=<value optimized out>, currSchName=<value optimized out>)
    at ../sqlcomp/CmpSeabaseDDLtable.cpp:2631
#19 0x00007ffff01daade in CmpSeabaseDDL::executeSeabaseDDL (
    this=0x7fffe464ae30, ddlExpr=0x7fffd199c520, ddlNode=0x7fffd199c338,
    currCatName=..., currSchName=...)
    at ../sqlcomp/CmpSeabaseDDLcommon.cpp:5696
#20 0x00007fffedfb240a in CmpStatement::process (this=<value optimized out>,
    statement=<value optimized out>) at ../arkcmp/CmpStatement.cpp:911
#21 0x00007fffedfa82d8 in CmpContext::compileDirect (this=0x7fffe25dd090,
    data=0x7fffd01abef8 "\200", data_len=224, outHeap=0x7fffe3be60f0,
    charset=15, op=CmpMessageObj::PROCESSDDL, gen_code=@0x7fffe464b6f0,
    gen_code_len=@0x7fffe464b6fc, parserFlags=4194304,
    diagsArea=0x7fffd01abfe0) at ../arkcmp/CmpContext.cpp:718
#22 0x00007ffff36954f0 in ExDDLTcb::work (this=0x7fffe316d0a8)
    at ../executor/ex_ddl.cpp:265
#23 0x00007ffff37613c3 in ExScheduler::work (this=0x7fffe316cc10,
    prevWaitTime=<value optimized out>) at ../executor/ExScheduler.cpp:328
#24 0x00007ffff36d3542 in ex_root_tcb::execute (this=0x7fffe316d680,
    cliGlobals=0xf40b30, glob=0x7fffd01ab8c0, input_desc=0x7fffe3180a40,
    diagsArea=@0x7fffe464ce10, reExecute=0) at ../executor/ex_root.cpp:1029
#25 0x00007ffff49c991c in CliStatement::execute (this=0x7fffd16ecd08,
    cliGlobals=0xf40b30, input_desc=0x7fffe3180a40,
    diagsArea=<value optimized out>, execute_state=<value optimized out>,
    fixupOnly=0, cliflags=0) at ../cli/Statement.cpp:4779
#26 0x00007ffff498c22a in SQLCLI_PerformTasks(CliGlobals *, ULng32, SQLSTMT_ID *, SQLDESC_ID *, SQLDESC_ID *, Lng32, Lng32, typedef __va_list_tag __va_list_tag *, SQLCLI_PTR_PAIRS *, SQLCLI_PTR_PAIRS *) (cliGlobals=0xf40b30, tasks=8063,
    statement_id=0x2deb558, input_descriptor=0x2deb588, output_descriptor=0x0,
    num_input_ptr_pairs=0, num_output_ptr_pairs=0, ap=0x7fffe464cfb0,
    input_ptr_pairs=0x0, output_ptr_pairs=0x0) at ../cli/Cli.cpp:3278
#27 0x00007ffff49d4f55 in SQL_EXEC_ClearExecFetchClose (
    statement_id=0x2deb558, input_descriptor=0x2deb588, output_descriptor=0x0,
    num_input_ptr_pairs=0, num_output_ptr_pairs=0, num_total_ptr_pairs=0)
    at ../cli/CliExtern.cpp:2618
#28 0x00007ffff6e79b0f in SRVR::WSQL_EXEC_ClearExecFetchClose (
    statement_id=0x2deb558, input_descriptor=<value optimized out>,
    output_descriptor=<value optimized out>,
    num_input_ptr_pairs=<value optimized out>,
    num_output_ptr_pairs=<value optimized out>,
    num_total_ptr_pairs=<value optimized out>) at SQLWrapper.cpp:459
#29 0x00007ffff6e7044f in SRVR::EXECUTE2 (pSrvrStmt=0x2deaf40)
    at sqlinterface.cpp:5521
#30 0x00007ffff6ea0a45 in odbc_SQLSvc_Execute2_sme_ (
    objtag_=<value optimized out>, call_id_=<value optimized out>,
    dialogueId=<value optimized out>, sqlAsyncEnable=<value optimized out>,
    queryTimeout=<value optimized out>, inputRowCnt=<value optimized out>,
    sqlStmtType=0, stmtHandle=48148288, cursorLength=0, cursorName=0x0,
    cursorCharset=1, holdableCursor=0, inValuesLength=0, inValues=0x0,
    returnCode=0x7fffe464d8a8, sqlWarningOrErrorLength=0x7fffe464d8a4,
    sqlWarningOrError=@0x7fffe464d880, rowsAffected=0x7fffe464d8a0,
    outValuesLength=0x7fffe464d894, outValues=@0x7fffe464d878)
    at srvrothers.cpp:1514
#31 0x00000000004d2685 in odbc_SQLSrvr_ExecDirect_ame_ (objtag_=0x31c8ce0,
    call_id_=0x31c8d38, dialogueId=1676333437,
    stmtLabel=<value optimized out>, cursorName=0x0,
    stmtExplainLabel=<value optimized out>, stmtType=0, sqlStmtType=0,
    sqlString=0x379d794 "alter table tab_emp_original_01 rename to tab_emp_new_01", sqlAsyncEnable=0, queryTimeout=0, inputRowCnt=0, txnID=0, holdableCursor=0)
    at SrvrConnect.cpp:7018
#32 0x00000000004a1a36 in SQLEXECUTE_IOMessage (objtag_=0x31c8ce0,
    call_id_=0x31c8d38, operation_id=3012) at Interface/odbcs_srvr.cpp:1734
#33 0x00000000004a1ae4 in DISPATCH_TCPIPRequest (objtag_=0x31c8ce0,
    call_id_=0x31c8d38, operation_id=<value optimized out>)
    at Interface/odbcs_srvr.cpp:1799
#34 0x0000000000432a02 in BUILD_TCPIP_REQUEST (pnode=0x31c8ce0)
    at ../Common/TCPIPSystemSrvr.cpp:603
#35 0x000000000043339d in PROCESS_TCPIP_REQUEST (pnode=0x31c8ce0)
    at ../Common/TCPIPSystemSrvr.cpp:581
#36 0x0000000000470ad6 in CNSKListenerSrvr::tcpip_listener (arg=0xde0f40)
    at Interface/linux/Listener_srvr_ps.cpp:397
#37 0x00007ffff50329f0 in sb_thread_sthr_disp (pp_arg=0xf13020)
    at threadl.cpp:253
#38 0x00007ffff4bf9851 in start_thread () from /lib64/libpthread.so.0
#39 0x00007ffff533590d in clone () from /lib64/libc.so.6

------------------------------------------------------------------------------------------------------------------

Here is the execution output from trafci. The last ‘alter table tab_emp_original_01 rename to tab_emp_new_01;’ statement cored mxosrvr and caused trafci to lose its connection whenever the problem was seen:

SQL>drop schema trafodion.alter_rename_rename21 cascade;
--- SQL operation complete.

SQL>create schema trafodion.alter_rename_rename21;
--- SQL operation complete.

SQL>set schema trafodion.alter_rename_rename21;
--- SQL operation complete.

SQL>set nametype ansi;
--- SQL operation complete.

SQL>create table tab_emp_original_01
(
sbin0_10 Numeric(18) signed,
char0_2 Character(8),
udec0_uniq Decimal(9) unsigned,
ubin0_uniq PIC 9(9) COMP,
sdec0_500 PIC S9(9),
varchar0_10 varchar(16),
varchar1_20 varchar(8),
sbin1_5000 Numeric(4) signed,
sdec1_4 Decimal(18) signed,
char1_4 Character(8)
)
no partition
location $DB0002
;
--- SQL operation complete.

SQL>create table tab_emp_original_02
(
sbin0_10 Numeric(18) signed,
char0_2 Character(8),
udec0_uniq Decimal(9) unsigned,
ubin0_uniq PIC 9(9) COMP,
sdec0_500 PIC S9(9),
varchar0_10 varchar(16),
varchar1_20 varchar(8),
sbin1_5000 Numeric(4) signed,
sdec1_4 Decimal(18) signed,
char1_4 Character(8)
)
no partition
location $DB0002
;
--- SQL operation complete.

SQL>create table tab_ten_original_03
(
sbin0_10 Numeric(18) signed,
char0_2 Character(8),
udec0_uniq Decimal(9) unsigned,
ubin0_uniq PIC 9(9) COMP,
sdec0_500 PIC S9(9),
varchar0_10 varchar(16),
varchar1_20 varchar(8),
sbin1_5000 Numeric(4) signed,
sdec1_4 Decimal(18) signed,
char1_4 Character(8)
)
no partition
location $DB0002
;
--- SQL operation complete.

SQL>create table tab_ten_original_04
(
sbin0_10 Numeric(18) signed,
char0_2 Character(8),
udec0_uniq Decimal(9) unsigned,
ubin0_uniq PIC 9(9) COMP,
sdec0_500 PIC S9(9),
varchar0_10 varchar(16),
varchar1_20 varchar(8),
sbin1_5000 Numeric(4) signed,
sdec1_4 Decimal(18) signed,
char1_4 Character(8)
)
no partition
location $DB0002
;
--- SQL operation complete.

SQL>create table tab_lrg_original_05
(
sbin0_10 Numeric(18) signed,
char0_2 Character(8),
udec0_uniq Decimal(9) unsigned,
ubin0_uniq PIC 9(9) COMP,
sdec0_500 PIC S9(9),
varchar0_10 varchar(16),
varchar1_20 varchar(8),
sbin1_5000 Numeric(4) signed,
sdec1_4 Decimal(18) signed,
char1_4 Character(8)
)
no partition
location $DB0002
;
--- SQL operation complete.

SQL>create table tab_lrg_original_06
(
sbin0_10 Numeric(18) signed,
char0_2 Character(8),
udec0_uniq Decimal(9) unsigned,
ubin0_uniq PIC 9(9) COMP,
sdec0_500 PIC S9(9),
varchar0_10 varchar(16),
varchar1_20 varchar(8),
sbin1_5000 Numeric(4) signed,
sdec1_4 Decimal(18) signed,
char1_4 Character(8)
)
no partition
location $DB0002
;
--- SQL operation complete.

SQL>insert into tab_ten_original_03 values (1,'ABCDEFGH',1001,1001,11,'IJKLMNOPQRSTUVWX','YZABCDEF',1001,11,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_03 values (2,'ABCDEFGH',1002,1002,12,'IJKLMNOPQRSTUVWX','YZABCDEF',1002,12,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_03 values (3,'ABCDEFGH',1003,1003,13,'IJKLMNOPQRSTUVWX','YZABCDEF',1003,13,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_03 values (4,'ABCDEFGH',1004,1004,14,'IJKLMNOPQRSTUVWX','YZABCDEF',1004,14,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_03 values (5,'ABCDEFGH',1005,1005,15,'IJKLMNOPQRSTUVWX','YZABCDEF',1005,15,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_03 values (6,'ABCDEFGH',1006,1006,16,'IJKLMNOPQRSTUVWX','YZABCDEF',1006,16,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_03 values (7,'ABCDEFGH',1007,1007,17,'IJKLMNOPQRSTUVWX','YZABCDEF',1007,17,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_03 values (8,'ABCDEFGH',1008,1008,18,'IJKLMNOPQRSTUVWX','YZABCDEF',1008,18,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_03 values (9,'ABCDEFGH',1009,1009,19,'IJKLMNOPQRSTUVWX','YZABCDEF',1009,19,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_03 values (10,'ABCDEFGH',1010,1010,20,'IJKLMNOPQRSTUVWX','YZABCDEF',1010,20,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_04 values (1,'ABCDEFGH',1001,1001,11,'IJKLMNOPQRSTUVWX','YZABCDEF',1001,11,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_04 values (2,'ABCDEFGH',1002,1002,12,'IJKLMNOPQRSTUVWX','YZABCDEF',1002,12,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_04 values (3,'ABCDEFGH',1003,1003,13,'IJKLMNOPQRSTUVWX','YZABCDEF',1003,13,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_04 values (4,'ABCDEFGH',1004,1004,14,'IJKLMNOPQRSTUVWX','YZABCDEF',1004,14,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_04 values (5,'ABCDEFGH',1005,1005,15,'IJKLMNOPQRSTUVWX','YZABCDEF',1005,15,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_04 values (6,'ABCDEFGH',1006,1006,16,'IJKLMNOPQRSTUVWX','YZABCDEF',1006,16,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_04 values (7,'ABCDEFGH',1007,1007,17,'IJKLMNOPQRSTUVWX','YZABCDEF',1007,17,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_04 values (8,'ABCDEFGH',1008,1008,18,'IJKLMNOPQRSTUVWX','YZABCDEF',1008,18,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_04 values (9,'ABCDEFGH',1009,1009,19,'IJKLMNOPQRSTUVWX','YZABCDEF',1009,19,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>insert into tab_ten_original_04 values (10,'ABCDEFGH',1010,1010,20,'IJKLMNOPQRSTUVWX','YZABCDEF',1010,20,'GHIJKLMN');
--- 1 row(s) inserted.

SQL>upsert using load into trafodion.alter_rename_rename21.tab_lrg_original_05 select * from trafodion.g_cmureg.btpnl17a;
--- SQL operation complete.

SQL>select count(*) from tab_lrg_original_05;
(EXPR)
--------------------
                2000

--- 1 row(s) selected.

SQL>update statistics for table tab_lrg_original_05 on every column;
--- SQL operation complete.

SQL>upsert using load into trafodion.alter_rename_rename21.tab_lrg_original_06 select * from trafodion.g_cmureg.btpnl17a;
--- SQL operation complete.

SQL>select count(*) from tab_lrg_original_06;
(EXPR)
--------------------
                2000

--- 1 row(s) selected.

SQL>update statistics for table tab_lrg_original_06 on every column;
--- SQL operation complete.

-------------------------------------------------------------------------------
== TEST: tcase.test001
-------------------------------------------------------------------------------
SQL>select * from tab_emp_original_01;
--- 0 row(s) selected.

SQL>showddl tab_emp_original_01;

CREATE TABLE TRAFODION.ALTER_RENAME_RENAME21.TAB_EMP_ORIGINAL_01
  (
    SBIN0_10 NUMERIC(18, 0) DEFAULT NULL
  , CHAR0_2 CHAR(8) CHARACTER SET ISO88591 COLLATE
      DEFAULT DEFAULT NULL
  , UDEC0_UNIQ DECIMAL(9, 0) UNSIGNED DEFAULT NULL
  , UBIN0_UNIQ NUMERIC(9, 0) UNSIGNED DEFAULT NULL
  , SDEC0_500 DECIMAL(9, 0) DEFAULT NULL
  , VARCHAR0_10 VARCHAR(16) CHARACTER SET ISO88591 COLLATE
      DEFAULT DEFAULT NULL
, VARCHAR1_20 VARCHAR(8) CHARACTER SET ISO88591 COLLATE
      DEFAULT DEFAULT NULL
  , SBIN1_5000 NUMERIC(4, 0) DEFAULT NULL
  , SDEC1_4 DECIMAL(18, 0) DEFAULT NULL
  , CHAR1_4 CHAR(8) CHARACTER SET ISO88591 COLLATE
      DEFAULT DEFAULT NULL
  )
;

--- SQL operation complete.

SQL>alter table tab_emp_original_01 rename to tab_emp_new_01;
*** ERROR[1] The message id: problem_with_server_read
*** ERROR[1] The message id: header_not_long_enough
*** ERROR[1] The message id: problem_with_server_read
*** ERROR[1] The message id: header_not_long_enough

Tags: sql-cmp
Sandhya Sundaresan (sandhya-sundaresan)
Changed in trafodion:
assignee: nobody → Justin Du (justin-du-2)
Revision history for this message
Justin Du (justin-du-2) wrote :

Looked at few cores for this bug and they all failed due to the same cause: at frame #6 ExHbaseAccess::keySubsetExeExpr_ was null.

The keySubsetExeExpr_ is constructed at the runtime based on ComTdbHbaseAccess::keyInfo_, but that field was null too.

executor/ExHbaseAccess.cpp:
    268 keySubsetExeExpr_ = NULL;
    269 keyMdamExeExpr_ = NULL;
    270 if (hbaseAccessTdb.keySubsetGen())
    271 {
    272 // this constructor will also do the fixups of underlying expressi ons
    273 keySubsetExeExpr_ = new(glob->getSpace())
    274 keySingleSubsetEx(*hbaseAccessTdb.keySubsetGen(),
    275 1, // version 1
    276 pool_,
    277 getGlobals(),
    278 getExpressionMode(), this);
    279 }

comexe/ComTdbHbaseAccess.h:
    472 keySingleSubsetGen * keySubsetGen() const
    473 {
    474 return
    475 (keyInfo_ && keyInfo_->castToKeySingleSubsetGen() ?
    476 keyInfo_->castToKeySingleSubsetGen() :
    477 NULL);
    478 }
    479

(gdb) p this->keySubsetExeExpr_
$1 = (keySingleSubsetEx *) 0x0

(gdb) p ((ComTdbHbaseAccess*)0x7fffbe4e6e58)->keyInfo_
$4 = {{ptr_ = 0x0, offset_ = 0}}

The query executed was like:
update TRAFODION.\"_MD_\".OBJECTS set catalog_name = 'TRAFODION', schema_name = 'ALTER_RENAME_RENAME21', object_name = 'TAB_EMP_NEW_01' where object_uid = 158802970814398584;

The objects meta data table has an index based on object_uid. If we use index in the plan, we should not use key subset scan.

Could this be a problem of inconsistent plan?

Revision history for this message
Justin Du (justin-du-2) wrote :

Here is the plan for the delete statement generated from v1202_0830

>>set parserflags 131072;

--- SQL operation complete.
>>prepare ss from
 +>update TRAFODION."_MD_".OBJECTS set catalog_name = 'TRAFODION', schema_name = 'ALTER_RENAME_RENAME21', object_name = 'TAB_EMP_NEW_01' where object_uid = 158802970814398584;

--- SQL command prepared.
>>explain options 'f' ss;

LC RC OP OPERATOR OPT DESCRIPTION CARD
---- ---- ---- -------------------- -------- -------------------- ---------

8 . 9 root x 1.00E+001
6 7 8 nested_join 1.00E+001
. . 7 trafodion_insert OBJECTS_UNIQ_IDX 1.00E+000
4 5 6 nested_join 1.00E+001
. . 5 trafodion_insert OBJECTS 1.00E+000
3 . 4 sort 1.00E+001
1 2 3 nested_anti_semi_joi 1.00E+001
. . 2 trafodion_vsbb_delet OBJECTS_UNIQ_IDX 1.00E+000
. . 1 trafodion_delete OBJECTS 1.00E+001

--- SQL operation complete.
>>

Could we get the plan from v1108_0830 build?

Changed in trafodion:
status: New → In Progress
Revision history for this message
Justin Du (justin-du-2) wrote :
Download full text (8.4 KiB)

Here is the full explain output from v1202_0830 build.

>>prepare ss from
+>update TRAFODION."_MD_".OBJECTS set catalog_name = 'TRAFODION', schema_name = 'ALTER_RENAME_RENAME21', object_name = 'TAB_EMP_NEW_01' where object_uid = 158802970814398584;

--- SQL command prepared.
>>explain ss;

------------------------------------------------------------------ PLAN SUMMARY
MODULE_NAME .............. DYNAMICALLY COMPILED
STATEMENT_NAME ........... SS
PLAN_ID .................. 212284413487002637
ROWS_OUT ................ 10
EST_TOTAL_COST ........... 0.01
STATEMENT ................ update TRAFODION."_MD_".OBJECTS
                           set catalog_name = 'TRAFODION',
                             schema_name = 'ALTER_RENAME_RENAME21',
                             object_name = 'TAB_EMP_NEW_01'
                           where object_uid = 158802970814398584;

------------------------------------------------------------------ NODE LISTING
ROOT ====================================== SEQ_NO 9 ONLY CHILD 8
REQUESTS_IN .............. 1
ROWS_OUT ................ 10
EST_OPER_COST ............ 0
EST_TOTAL_COST ........... 0.01
DESCRIPTION
  max_card_est .......... 10
  fragment_id ............ 0
  parent_frag ............ (none)
  fragment_type .......... master
  statement_index ........ 0
  affinity_value ......... 0
  max_max_cardinality ... 10
  total_overflow_size .... 0.00 KB
  upd_action_on_error .... xn_rollback
  xn_access_mode ......... read_only
  xn_autoabort_interval 0
  auto_query_retry ....... enabled for privilege checks
  plan_version ....... 2,600
  embedded_arkcmp ........ used
  self_referencing_update forced_sort
  IS_SQLCI ............... ON
  LDAP_USERNAME
  input_variables ........ %('TRAFODION'), %('ALTER_RENAME_RENAME21'),
                             %('TAB_EMP_NEW_01'), %('TRAFODION'),
                             %('ALTER_RENAME_RENAME21'), %('TAB_EMP_NEW_01'),
                             %(158802970814398584)

NESTED_JOIN =============================== SEQ_NO 8 CHILDREN 6, 7
REQUESTS_IN .............. 1
ROWS_OUT ................ 10
EST_OPER_COST ............ 0.01
EST_TOTAL_COST ........... 0.01
DESCRIPTION
  max_card_est .......... 10
  fragment_id ............ 0
  parent_frag ............ (none)
  fragment_type .......... master
  join_type .............. inner
  join_method ............ nested

TRAFODION_INSERT ========================== SEQ_NO 7 NO CHILDREN
TABLE_NAME ............... TRAFODION."_MD_".OBJECTS_UNIQ_IDX
REQUESTS_IN .............. 1
ROWS_OUT ................. 1
EST_OPER_COST ............ 0
EST_TOTAL_COST ........... 0
DESCRIPTION
  max_card_est .......... 10
  fragment_id ............ 0
  parent_frag ............ (none)
  fragment_type .......... master
  iud_type ............... index_trafodion_insert TRAFODION."_MD_".OBJECTS_UNIQ
                             _IDX
  new_rec_expr ........... (TRAFODION."_MD_".OBJECTS_UNIQ_IDX."OBJECT_UID@"
                             assign %(158802970814398584)),
                             (TRAFODION."_MD_".OBJECTS_UNIQ_IDX.CATALOG_NAME
                             assign TRAFODION."_MD_".OBJECTS.CATALOG_NAME),
                           ...

Read more...

Revision history for this message
Justin Du (justin-du-2) wrote :

BTW, the core was created when executing the operator #2 in the plan.

Revision history for this message
Justin Du (justin-du-2) wrote :

Debugged v1108_0830 code and the keyInfo_ was not generated for any of the 4 hbase access tdbs either. So the problem may still be at the runtime side.

Weishiun Tsai (wei-shiun-tsai)
Changed in trafodion:
milestone: none → r1.1
Revision history for this message
Weishiun Tsai (wei-shiun-tsai) wrote :
Download full text (8.2 KiB)

Just to provide some more updates on this problem. This core is still being seen in recent builds, generally once or twice in each build when tests were run. Here is one from the v0118 build:

(gdb) bt
#0 0x00007ffff4a458a5 in raise () from /lib64/libc.so.6
#1 0x00007ffff4a4700d in abort () from /lib64/libc.so.6
#2 0x00007ffff5b4ea55 in os::abort(bool) ()
   from /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5ccef87 in VMError::report_and_die() ()
   from /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5b5396f in JVM_handle_linux_signal ()
   from /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 <signal handler called>
#6 0x00007ffff2e1ca51 in ExHbaseAccessTcb::setupUniqueKeyAndCols (
    this=0x7fffcf4e9cb8, doInit=0) at ../executor/ExHbaseAccess.cpp:1709
#7 0x00007ffff2e2e50a in ExHbaseAccessSQRowsetTcb::work (this=0x7fffcf4e9cb8)
    at ../executor/ExHbaseIUD.cpp:3805
#8 0x00007ffff2e4b553 in ExScheduler::work (this=0x7fffcf4e87c0,
    prevWaitTime=<value optimized out>) at ../executor/ExScheduler.cpp:328
#9 0x00007ffff2d79462 in ex_root_tcb::execute (this=0x7fffcf4edc40,
    cliGlobals=0xeebba0, glob=0x7fffe296c3a8, input_desc=0x7fffcf423470,
    diagsArea=@0x7fffe3dd4930, reExecute=0) at ../executor/ex_root.cpp:1055
#10 0x00007ffff437c554 in CliStatement::execute (this=0x7fffcf4b4e40,
    cliGlobals=0xeebba0, input_desc=0x7fffcf423470,
    diagsArea=<value optimized out>, execute_state=<value optimized out>,
    fixupOnly=0, cliflags=0) at ../cli/Statement.cpp:4812
#11 0x00007ffff4324d0a in SQLCLI_PerformTasks(CliGlobals *, ULng32, SQLSTMT_ID *, SQLDESC_ID *, SQLDESC_ID *, Lng32, Lng32, typedef __va_list_tag __va_list_tag *, SQLCLI_PTR_PAIRS *, SQLCLI_PTR_PAIRS *) (cliGlobals=0xeebba0, tasks=4882,
    statement_id=0x7fffcf124f38, input_descriptor=0x7fffcf124fa8,
    output_descriptor=0x0, num_input_ptr_pairs=0, num_output_ptr_pairs=0,
    ap=0x7fffe3dd4ad0, input_ptr_pairs=0x0, output_ptr_pairs=0x0)
    at ../cli/Cli.cpp:3284
#12 0x00007ffff4325a32 in SQLCLI_Exec(CliGlobals *, SQLSTMT_ID *, SQLDESC_ID *, Lng32, typedef __va_list_tag __va_list_tag *, SQLCLI_PTR_PAIRS *) (
    cliGlobals=<value optimized out>, statement_id=<value optimized out>,
    input_descriptor=<value optimized out>,
    num_ptr_pairs=<value optimized out>, ap=<value optimized out>,
    ptr_pairs=<value optimized out>) at ../cli/Cli.cpp:3531
#13 0x00007ffff438b10a in SQL_EXEC_Exec (statement_id=0x7fffcf124f38,
    input_descriptor=0x7fffcf124fa8, num_ptr_pairs=0)
    at ../cli/CliExtern.cpp:2071
#14 0x00007ffff2de1261 in ExeCliInterface::exec (this=0x7fffe3dd6f20,
    inputBuf=<value optimized out>, inputBufLen=<value optimized out>)
    at ../executor/ExExeUtilCli.cpp:599
#15 0x00007ffff2dea155 in ExeCliInterface::executeImmediateExec (
    this=0x7fffe3dd6f20, stmtStr=<value optimized out>, outputBuf=0x0,
    outputBufLen=0x0, nullTerminate=1, rowsAffected=0x0)
    at ../executor/ExExeUtilCli.cpp:898
#16 0x00007ffff2dea3cc in ExeCliInterface::executeImmediate (
    this=0x7fffe3dd6f20,
    stmtStr=0x7fffe3dd4d30 "update TRAFODION.\"_MD_\".OBJECTS set catalog_name = 'TRAFODION', schema_name = 'ALTER_RENAME_RE...

Read more...

Revision history for this message
Weishiun Tsai (wei-shiun-tsai) wrote :
Download full text (16.2 KiB)

This problem was seen twice on the v1.0.0 rc3 build, one on a Cloudera system and another one on the HortonWorks system:

(gdb) bt
#0 0x00007ffff4a458a5 in raise () from /lib64/libc.so.6
#1 0x00007ffff4a4700d in abort () from /lib64/libc.so.6
#2 0x00007ffff5b4ea55 in os::abort(bool) ()
   from /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5ccef87 in VMError::report_and_die() ()
   from /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5b5396f in JVM_handle_linux_signal ()
   from /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 <signal handler called>
#6 0x00007ffff2e1c0b1 in ExHbaseAccessTcb::setupUniqueKeyAndCols (
    this=0x7fffcf1b70a0, doInit=0) at ../executor/ExHbaseAccess.cpp:1709
#7 0x00007ffff2e2db6a in ExHbaseAccessSQRowsetTcb::work (this=0x7fffcf1b70a0)
    at ../executor/ExHbaseIUD.cpp:3805
#8 0x00007ffff2e4abb3 in ExScheduler::work (this=0x7fffcf1b5ba8,
    prevWaitTime=<value optimized out>) at ../executor/ExScheduler.cpp:328
#9 0x00007ffff2d784b2 in ex_root_tcb::execute (this=0x7fffcf1bb028,
    cliGlobals=0xeebbb0, glob=0x7fffcf3d02d8, input_desc=0x7fffcf3b5cd8,
    diagsArea=@0x7fffe3dd2930, reExecute=0) at ../executor/ex_root.cpp:1055
#10 0x00007ffff437c554 in CliStatement::execute (this=0x7fffe295d958,
    cliGlobals=0xeebbb0, input_desc=0x7fffcf3b5cd8,
    diagsArea=<value optimized out>, execute_state=<value optimized out>,
    fixupOnly=0, cliflags=0) at ../cli/Statement.cpp:4812
#11 0x00007ffff4324d0a in SQLCLI_PerformTasks(CliGlobals *, ULng32, SQLSTMT_ID *, SQLDESC_ID *, SQLDESC_ID *, Lng32, Lng32, typedef __va_list_tag __va_list_tag *, SQLCLI_PTR_PAIRS *, SQLCLI_PTR_PAIRS *) (cliGlobals=0xeebbb0, tasks=4882,
    statement_id=0x7fffcef3cc98, input_descriptor=0x7fffd10935a0,
    output_descriptor=0x0, num_input_ptr_pairs=0, num_output_ptr_pairs=0,
    ap=0x7fffe3dd2ad0, input_ptr_pairs=0x0, output_ptr_pairs=0x0)
    at ../cli/Cli.cpp:3284
#12 0x00007ffff4325a32 in SQLCLI_Exec(CliGlobals *, SQLSTMT_ID *, SQLDESC_ID *, Lng32, typedef __va_list_tag __va_list_tag *, SQLCLI_PTR_PAIRS *) (
    cliGlobals=<value optimized out>, statement_id=<value optimized out>,
    input_descriptor=<value optimized out>,
    num_ptr_pairs=<value optimized out>, ap=<value optimized out>,
    ptr_pairs=<value optimized out>) at ../cli/Cli.cpp:3531
#13 0x00007ffff438b10a in SQL_EXEC_Exec (statement_id=0x7fffcef3cc98,
    input_descriptor=0x7fffd10935a0, num_ptr_pairs=0)
    at ../cli/CliExtern.cpp:2071
#14 0x00007ffff2de02b1 in ExeCliInterface::exec (this=0x7fffe3dd4f20,
    inputBuf=<value optimized out>, inputBufLen=<value optimized out>)
    at ../executor/ExExeUtilCli.cpp:599
#15 0x00007ffff2de91a5 in ExeCliInterface::executeImmediateExec (
    this=0x7fffe3dd4f20, stmtStr=<value optimized out>, outputBuf=0x0,
    outputBufLen=0x0, nullTerminate=1, rowsAffected=0x0)
    at ../executor/ExExeUtilCli.cpp:898
#16 0x00007ffff2de941c in ExeCliInterface::executeImmediate (
    this=0x7fffe3dd4f20,
    stmtStr=0x7fffe3dd2d30 "update TRAFODION.\"_MD_\".OBJECTS set catalog_name = 'TRAFODION', schema_name = 'ALTER_RENAME_RENAME21', object_name = 'TAB_EMP_NEW_01' where object_uid = 287675506816...

Revision history for this message
Weishiun Tsai (wei-shiun-tsai) wrote :
Download full text (8.1 KiB)

A 2nd core from this problem was seen on the v1.0.1rc1 build installed on a 4-node Hortonworks machine:

(gdb) bt
#0 0x00007ffff4a458a5 in raise () from /lib64/libc.so.6
#1 0x00007ffff4a4700d in abort () from /lib64/libc.so.6
#2 0x00007ffff5b4ea55 in os::abort(bool) ()
   from /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5ccef87 in VMError::report_and_die() ()
   from /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5b5396f in JVM_handle_linux_signal ()
   from /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 <signal handler called>
#6 0x00007ffff2e1c2e1 in ExHbaseAccessTcb::setupUniqueKeyAndCols (
    this=0x7fffcbda0078, doInit=0) at ../executor/ExHbaseAccess.cpp:1709
#7 0x00007ffff2e2dd9a in ExHbaseAccessSQRowsetTcb::work (this=0x7fffcbda0078)
    at ../executor/ExHbaseIUD.cpp:3805
#8 0x00007ffff2e4ade3 in ExScheduler::work (this=0x7fffcbd9eb80,
    prevWaitTime=<value optimized out>) at ../executor/ExScheduler.cpp:328
#9 0x00007ffff2d784e2 in ex_root_tcb::execute (this=0x7fffcbda4000,
    cliGlobals=0xeea730, glob=0x7fffc7ff0520, input_desc=0x7fffc09c7198,
    diagsArea=@0x7fffe41e6bb0, reExecute=0) at ../executor/ex_root.cpp:1055
#10 0x00007ffff437c554 in CliStatement::execute (this=0x7fffc09c7458,
    cliGlobals=0xeea730, input_desc=0x7fffc09c7198,
    diagsArea=<value optimized out>, execute_state=<value optimized out>,
    fixupOnly=0, cliflags=0) at ../cli/Statement.cpp:4812
#11 0x00007ffff4324d0a in SQLCLI_PerformTasks(CliGlobals *, ULng32, SQLSTMT_ID *, SQLDESC_ID *, SQLDESC_ID *, Lng32, Lng32, typedef __va_list_tag __va_list_tag *, SQLCLI_PTR_PAIRS *, SQLCLI_PTR_PAIRS *) (cliGlobals=0xeea730, tasks=4882,
    statement_id=0x7fffd12ed3d0, input_descriptor=0x7fffd12ed440,
    output_descriptor=0x0, num_input_ptr_pairs=0, num_output_ptr_pairs=0,
    ap=0x7fffe41e6d50, input_ptr_pairs=0x0, output_ptr_pairs=0x0)
    at ../cli/Cli.cpp:3284
#12 0x00007ffff4325a32 in SQLCLI_Exec(CliGlobals *, SQLSTMT_ID *, SQLDESC_ID *, Lng32, typedef __va_list_tag __va_list_tag *, SQLCLI_PTR_PAIRS *) (
    cliGlobals=<value optimized out>, statement_id=<value optimized out>,
    input_descriptor=<value optimized out>,
    num_ptr_pairs=<value optimized out>, ap=<value optimized out>,
    ptr_pairs=<value optimized out>) at ../cli/Cli.cpp:3531
#13 0x00007ffff438b10a in SQL_EXEC_Exec (statement_id=0x7fffd12ed3d0,
    input_descriptor=0x7fffd12ed440, num_ptr_pairs=0)
    at ../cli/CliExtern.cpp:2071
#14 0x00007ffff2de0311 in ExeCliInterface::exec (this=0x7fffe41e91a0,
    inputBuf=<value optimized out>, inputBufLen=<value optimized out>)
    at ../executor/ExExeUtilCli.cpp:611
#15 0x00007ffff2de9255 in ExeCliInterface::executeImmediateExec (
    this=0x7fffe41e91a0, stmtStr=<value optimized out>, outputBuf=0x0,
    outputBufLen=0x0, nullTerminate=1, rowsAffected=0x0)
    at ../executor/ExExeUtilCli.cpp:910
#16 0x00007ffff2de94cc in ExeCliInterface::executeImmediate (
    this=0x7fffe41e91a0,
    stmtStr=0x7fffe41e6fb0 "update TRAFODION.\"_MD_\".OBJECTS set catalog_name = 'TRAFODION', schema_name = 'FEATURES_RMS', object_name = 'F0102' where object_uid = 322628626383173833 ", outputBuf=0x0, outputBufLen...

Read more...

Revision history for this message
Justin Du (justin-du-2) wrote :
Download full text (7.1 KiB)

To debug the problem, I added few lines of code at the generator/GenRelUpdate.cpp to capture the moment when we have rowset access set but key info is not provided:

@@ -1291,8 +1291,11 @@ short HbaseDelete::codeGen(Generator * generator)
   if (canDoCheckAndUpdel())
     hbasescan_tdb->setCanDoCheckAndUpdel(TRUE);

- if (uniqueRowsetHbaseOper())
+ if (uniqueRowsetHbaseOper()) {
     hbasescan_tdb->setRowsetOper(TRUE);
+ if (!keyInfo) abort();
+ GenAssert(keyInfo, "Rowset operation must be supported by key");
+ }

Ran the following and got a core:
explain update TRAFODION."_MD_".OBJECTS set catalog_name = 'TRAFODION', schema_name = 'ALTER_RENAME_RENAME21', object_name = 'TAB_EMP_NEW_01' where object_uid = 385679018054280799;

(gdb) bt
#0 0x0000003aab4328a5 in raise () from /lib64/libc.so.6
#1 0x0000003aab43400d in abort () from /lib64/libc.so.6
#2 0x00007fffee679446 in HbaseDelete::codeGen (this=0x7fffd5eaf908, generator=
    0x7ffffffeab90) at ../generator/GenRelUpdate.cpp:1296
#3 0x00007fffee62e8aa in NestedJoin::codeGen (this=0x7fffd5fe0fb8,
    generator=0x7ffffffeab90) at ../generator/GenRelJoin.cpp:3369
#4 0x00007fffee63ddf8 in Sort::codeGen (this=0x7fffd5fdff28,
    generator=0x7ffffffeab90) at ../generator/GenRelMisc.cpp:3318
#5 0x00007fffee62e418 in NestedJoin::codeGen (this=0x7fffd5fdde80,
    generator=0x7ffffffeab90) at ../generator/GenRelJoin.cpp:3283
#6 0x00007fffee62e418 in NestedJoin::codeGen (this=0x7fffd5fda7a0,
    generator=0x7ffffffeab90) at ../generator/GenRelJoin.cpp:3283
#7 0x00007fffee6367df in RelRoot::codeGen (this=0x7fffd5fd87e0,
    generator=0x7ffffffeab90) at ../generator/GenRelMisc.cpp:1202
#8 0x00007fffee5419bf in Generator::genCode (this=0x7ffffffeab90,
    source=0x7fffd68e5f48 " update TRAFODION.\"_MD_\".OBJECTS set catalog_name = 'TRAFODION', schema_name = 'ALTER_RENAME_RENAME21', object_name = 'TAB_EMP_NEW_01' where object_uid = 385679018054280799;", expr_node=0x7fffd5fd87e0)
    at ../generator/Generator.cpp:528
#9 0x00007ffff0ad345c in CmpMain::compile (this=0x7ffffffee460,
    input_str=0x7fffd68e5f48 " update TRAFODION.\"_MD_\".OBJECTS set catalog_name = 'TRAFODION', schema_name = 'ALTER_RENAME_RENAME21', object_name = 'TAB_EMP_NEW_01' where object_uid = 385679018054280799;", charset=15,
    queryExpr=@0x7ffffffee348, gen_code=0x7fffd68e5b08,
    gen_code_len=0x7fffd68e5b00, heap=0x7fffe8b54c08, phase=CmpMain::END,
    fragmentDir=0x7ffffffee5b8, op=3004, useQueryCache=1,
    cacheable=0x7ffffffee334, begTime=0x7ffffffee350, shouldLog=0)
    at ../sqlcomp/CmpMain.cpp:2408
#10 0x00007ffff0ad1029 in CmpMain::sqlcomp (this=0x7ffffffee460,
    input_str=0x7fffd68e5f48 " update TRAFODION.\"_MD_\".OBJECTS set catalog_name = 'TRAFODION', schema_name = 'ALTER_RENAME_RENAME21', object_name = 'TAB_EMP_NEW_01' where object_uid = 385679018054280799;", charset=15,
    queryExpr=@0x7ffffffee348, gen_code=0x7fffd68e5b08,
    gen_code_len=0x7fffd68e5b00, heap=0x7fffe8b54c08, phase=CmpMain::END,
    fragmentDir=0x7ffffffee5b8, op=3004, useQueryCache=1,
    cacheable=0x7ffffffee334, begTime=0x7ffffffee350, shouldLog=0)
    at ../sqlcomp/CmpMain.cpp:1732
#11 0x00007ffff0ace734 in C...

Read more...

Revision history for this message
Justin Du (justin-du-2) wrote :

The script has one error. It should be
#!/bin/sh
if [ $# != 1 ] # <== should be 1, not 2
then
  echo " Usage: $0 <num_interation> "
  exit
fi

Revision history for this message
Justin Du (justin-du-2) wrote :

Tried cqs comp_bool_158 'on' as Ravisha suggested and the problem was still shown.

Change to sql_cmp for further analysis

tags: added: sql-cmp
removed: sql-exe
Revision history for this message
Weishiun Tsai (wei-shiun-tsai) wrote :

Sometimes, this problem shows up in a different form. Instead of a core, it returns a strange error 4033 complaining about CATALOG_NAME being a primary key and cannot be updated. CATALOG_NAME is not a column in the user table. Presumably this is an internal update to the _MD_ tables that is returning the error This is what was seen on the r1.0.1rc3 build:

SQL>alter table tab_emp_original_01 rename to tab_emp_new_01 cascade;
*** ERROR[4033] Column CATALOG_NAME is a primary or clustering key column and cannot be updated. [2015-02-26 17:45:49]

Revision history for this message
Weishiun Tsai (wei-shiun-tsai) wrote :
Download full text (16.2 KiB)

Here are 2 cores seen on the v0303 build installed on a 4-node HortonWorks cluster:

(gdb) bt
#0 0x00007ffff4a458a5 in raise () from /lib64/libc.so.6
#1 0x00007ffff4a4700d in abort () from /lib64/libc.so.6
#2 0x00007ffff5b4ea55 in os::abort(bool) ()
   from /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5ccef87 in VMError::report_and_die() ()
   from /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5b5396f in JVM_handle_linux_signal ()
   from /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
#5 <signal handler called>
#6 0x00007ffff2e16261 in ExHbaseAccessTcb::setupUniqueKeyAndCols (
    this=0x7fffcfa3ae78, doInit=0) at ../executor/ExHbaseAccess.cpp:1707
#7 0x00007ffff2e27e5a in ExHbaseAccessSQRowsetTcb::work (this=0x7fffcfa3ae78)
    at ../executor/ExHbaseIUD.cpp:3805
#8 0x00007ffff2e44ee3 in ExScheduler::work (this=0x7fffcfa39980,
    prevWaitTime=<value optimized out>) at ../executor/ExScheduler.cpp:328
#9 0x00007ffff2d70682 in ex_root_tcb::execute (this=0x7fffcfa3ee00,
    cliGlobals=0xee5280, glob=0x7fffcf9db618, input_desc=0x7fffcf7da038,
    diagsArea=@0x7fffe52347b0, reExecute=0) at ../executor/ex_root.cpp:1055
#10 0x00007ffff437d014 in CliStatement::execute (this=0x7fffe3d6e2f0,
    cliGlobals=0xee5280, input_desc=0x7fffcf7da038,
    diagsArea=<value optimized out>, execute_state=<value optimized out>,
    fixupOnly=0, cliflags=0) at ../cli/Statement.cpp:4570
#11 0x00007ffff432611a in SQLCLI_PerformTasks(CliGlobals *, ULng32, SQLSTMT_ID *, SQLDESC_ID *, SQLDESC_ID *, Lng32, Lng32, typedef __va_list_tag __va_list_tag *, SQLCLI_PTR_PAIRS *, SQLCLI_PTR_PAIRS *) (cliGlobals=0xee5280, tasks=4882,
    statement_id=0x7fffcdbeafa0, input_descriptor=0x7fffcdbeb010,
    output_descriptor=0x0, num_input_ptr_pairs=0, num_output_ptr_pairs=0,
    ap=0x7fffe5234950, input_ptr_pairs=0x0, output_ptr_pairs=0x0)
    at ../cli/Cli.cpp:3284
#12 0x00007ffff4326e42 in SQLCLI_Exec(CliGlobals *, SQLSTMT_ID *, SQLDESC_ID *, Lng32, typedef __va_list_tag __va_list_tag *, SQLCLI_PTR_PAIRS *) (
    cliGlobals=<value optimized out>, statement_id=<value optimized out>,
    input_descriptor=<value optimized out>,
    num_ptr_pairs=<value optimized out>, ap=<value optimized out>,
    ptr_pairs=<value optimized out>) at ../cli/Cli.cpp:3531
#13 0x00007ffff438b55a in SQL_EXEC_Exec (statement_id=0x7fffcdbeafa0,
    input_descriptor=0x7fffcdbeb010, num_ptr_pairs=0)
    at ../cli/CliExtern.cpp:2071
#14 0x00007ffff2dd8831 in ExeCliInterface::exec (this=0x7fffe5236da0,
    inputBuf=<value optimized out>, inputBufLen=<value optimized out>)
    at ../executor/ExExeUtilCli.cpp:616
#15 0x00007ffff2de1b55 in ExeCliInterface::executeImmediateExec (
    this=0x7fffe5236da0, stmtStr=<value optimized out>, outputBuf=0x0,
    outputBufLen=0x0, nullTerminate=1, rowsAffected=0x0)
    at ../executor/ExExeUtilCli.cpp:915
#16 0x00007ffff2de1dcc in ExeCliInterface::executeImmediate (
    this=0x7fffe5236da0,
    stmtStr=0x7fffe5234bb0 "update TRAFODION.\"_MD_\".OBJECTS set catalog_name = 'TRAFODION', schema_name = 'ALTER_RENAME_RENAME23', object_name = 'TAB_EMP_NEW_01' where object_uid = 284349763952323046 ", outputBuf=0x0, outputBufLen=0...

Revision history for this message
Weishiun Tsai (wei-shiun-tsai) wrote :

This bug has been failing several QA regression tests for more than 3 months now. Bump it to Critical for it to be fixed in 1.1.

Changed in trafodion:
importance: High → Critical
Revision history for this message
Justin Du (justin-du-2) wrote :

Please see my comments on 2015-2-12 for the reproduction script and other analysis.

Changed in trafodion:
assignee: Justin Du (justin-du-2) → QF Chen (qifan-chen)
Anoop Sharma (anoop-sharma)
Changed in trafodion:
assignee: QF Chen (qifan-chen) → Anoop Sharma (anoop-sharma)
Anoop Sharma (anoop-sharma)
Changed in trafodion:
assignee: Anoop Sharma (anoop-sharma) → Selvaganesan Govindarajan (selva-ganesan)
Sandhya Sundaresan (sandhya-sundaresan)
Changed in trafodion:
milestone: r1.1 → r2.0
Revision history for this message
Weishiun Tsai (wei-shiun-tsai) wrote :
Download full text (5.1 KiB)

2 observations from running both the v0505 build, and a v0505+UTT build that is supposed to stop sending CQDs to the compilers (which was originally thought to be the root cause of this problem):

(1) The problem of CATALOG_NAME 4033 error and the core may be 2 different problems, even though they often appear on the same statement. The cores have been seen several times on the v0505 build, as well as the v0505+UTT build.
(2) The situation has gotten worse with the v0505 build. We now are seeing this core even with an update statement:

 SQL>update M set(vch7,nint) =(select vch7,nint from ttf1 where nint=5) where (nsint=1);
*** ERROR[1] The message id: problem_with_server_read
*** ERROR[1] The message id: header_not_long_enough
*** ERROR[1] The message id: problem_with_server_read
*** ERROR[1] The message id: header_not_long_enough

(gdb) bt
#0 0x00007ffff4a3d625 in raise () from /lib64/libc.so.6
#1 0x00007ffff4a3ed8d in abort () from /lib64/libc.so.6
#2 0x00007ffff5b48c55 in os::abort(bool) ()
   from /usr/java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#3 0x00007ffff5ccacd7 in VMError::report_and_die() ()
   from /usr/java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#4 0x00007ffff5b4db6f in JVM_handle_linux_signal ()
   from /usr/java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
#5 <signal handler called>
#6 0x00007ffff2df0b99 in ExHbaseAccessTcb::setupUniqueKeyAndCols (
    this=0x7fffd273d5a0, doInit=0) at ../executor/ExHbaseAccess.cpp:1690
#7 0x00007ffff2e024a1 in ExHbaseAccessSQRowsetTcb::work (this=0x7fffd273d5a0)
    at ../executor/ExHbaseIUD.cpp:3674
#8 0x00007ffff2e20d03 in ExScheduler::work (this=0x7fffd273b138,
    prevWaitTime=<optimized out>) at ../executor/ExScheduler.cpp:328
#9 0x00007ffff2d46732 in ex_root_tcb::execute (this=0x7fffd273f0f0,
    cliGlobals=0xee7250, glob=0x7fffd2777af0, input_desc=0x7fffd278d160,
    diagsArea=@0x7fffe519bb10: 0x0, reExecute=0)
    at ../executor/ex_root.cpp:1055
#10 0x00007ffff43744e4 in CliStatement::execute (this=0x7fffc6b34330,
    cliGlobals=0xee7250, input_desc=0x7fffd278d160, diagsArea=...,
    execute_state=<optimized out>, fixupOnly=0, cliflags=0)
    at ../cli/Statement.cpp:4558
#11 0x00007ffff431b14c in SQLCLI_PerformTasks(CliGlobals *, ULng32, SQLSTMT_ID *, SQLDESC_ID *, SQLDESC_ID *, Lng32, Lng32, typedef __va_list_tag __va_list_tag *, SQLCLI_PTR_PAIRS *, SQLCLI_PTR_PAIRS *) (cliGlobals=0xee7250, tasks=8063,
    statement_id=0x9f10ba8, input_descriptor=0x9f10bd8, output_descriptor=0x0,
    num_input_ptr_pairs=0, num_output_ptr_pairs=0, ap=0x7fffe519bcb0,
    input_ptr_pairs=0x0, output_ptr_pairs=0x0) at ../cli/Cli.cpp:3285
#12 0x00007ffff4381209 in SQL_EXEC_ClearExecFetchClose (
    statement_id=0x9f10ba8, input_descriptor=0x9f10bd8, output_descriptor=0x0,
    num_input_ptr_pairs=0, num_output_ptr_pairs=0, num_total_ptr_pairs=0)
    at ../cli/CliExtern.cpp:2627
#13 0x00007ffff686e29f in SRVR::WSQL_EXEC_ClearExecFetchClose (
    statement_id=0x9f10ba8, input_descriptor=<optimized out>,
    output_descriptor=<optimized out>, num_input_ptr_pairs=<optimized out>,
    num_output_ptr_pairs=<optimized out>, num_total_ptr_pairs=<optimized out>)
    at SQLWrapper.cpp:459
#14 0x00...

Read more...

Revision history for this message
Selvaganesan Govindarajan (selva-ganesan) wrote :
Download full text (26.0 KiB)

It looks like the plan is missing key expressions for the index delete.

The plan is ok when the object_uid is parameterized.
When the object_uid is given a value, both vsbb_delete and delete seems to have problem.

Trafodion Conversational Interface 1.2.0
(c) Copyright 2014 Hewlett-Packard Development Company, LP.
>>set parserflags 131072 ;

--- SQL operation complete.
>>prepare s1 from
>>update "_MD_".objects set catalog_name = 'TRAFODION', schema_name = 'SCH', object_name = 'OBJECT1' where object_uid = ?
 ..
>>;
 ..

--- SQL command prepared.
>>explain options 'f' s1 ;

LC RC OP OPERATOR OPT DESCRIPTION CARD
---- ---- ---- -------------------- -------- -------------------- ---------

8 . 9 root x 1.00E+000
6 7 8 nested_join 1.00E+000
. . 7 trafodion_insert OBJECTS_UNIQ_IDX 1.00E+000
4 5 6 nested_join 1.00E+000
. . 5 trafodion_insert OBJECTS 1.00E+000
3 . 4 sort 1.00E+000
1 2 3 nested_anti_semi_joi 1.00E+000
. . 2 trafodion_vsbb_delet OBJECTS_UNIQ_IDX 1.00E+000
. . 1 trafodion_delete OBJECTS 1.00E+000

--- SQL operation complete.
>>explain s1;

------------------------------------------------------------------ PLAN SUMMARY
MODULE_NAME .............. DYNAMICALLY COMPILED
STATEMENT_NAME ........... S1
PLAN_ID .................. 212299007795473919
ROWS_OUT ................. 1
EST_TOTAL_COST ........... 0.01
STATEMENT ................ update "_MD_".objects
                           set catalog_name = 'TRAFODION', schema_name = 'SCH',
                             object_name = 'OBJECT1'
                           where object_uid = ? ;

------------------------------------------------------------------ NODE LISTING
ROOT ====================================== SEQ_NO 9 ONLY CHILD 8
REQUESTS_IN .............. 1
ROWS_OUT ................. 1
EST_OPER_COST ............ 0
EST_TOTAL_COST ........... 0.01
DESCRIPTION
  max_card_est ........... 1
  fragment_id ............ 0
  parent_frag ............ (none)
  fragment_type .......... master
  statement_index ........ 0
  affinity_value ......... 0
  max_max_cardinality .... 1
  total_overflow_size .... 0.00 KB
  upd_action_on_error .... xn_rollback
  xn_access_mode ......... read_only
  xn_autoabort_interval 0
  auto_query_retry ....... enabled for privilege checks
  plan_version ....... 2,600
  embedded_arkcmp ........ used
  self_referencing_update forced_sort
  IS_SQLCI ............... ON
  LDAP_USERNAME
  input_variables ........ ?, %('TRAFODION'), %('SCH'), %('OBJECT1'),
                             %('TRAFODION'), %('SCH'), %('OBJECT1')

NESTED_JOIN =============================== SEQ_NO 8 CHILDREN 6, 7
REQUESTS_IN .............. 1
ROWS_OUT ................. 1
EST_OPER_COST ............ 0.01
EST_TOTAL_COST ........... 0.01
DESCRIPTION
  max_card_est ...

Changed in trafodion:
assignee: Selvaganesan Govindarajan (selva-ganesan) → Suresh Subbiah (suresh-subbiah)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.