Phoenix tests run into several error 8810 when other tests are run in parallel with it

[Aruna Sadashiva]

Running phoenix and jdbc catalog api tests at the same time resulted in 16 failures in phoenix with error 8810, during ddl operations as shown below. All the jdbc catalog api tests passed.

Phoenix runs fine when no other tests are running on the system.

test.java.com.hp.phoenix.end2end.ToNumberFunctionTest
....*** ERROR[8810] Executor ran into an internal failure and returned an error without populating the diagnostics area. This error is being injected to indicate that. [2015-01-22 08:32:31]
E.E.............
Time: 701.811
There were 2 failures:
1) testKeyProjectionWithIntegerValue(test.java.com.hp.phoenix.end2end.ToNumberFunctionTest)
java.lang.AssertionError: Failed to drop object: table TO_NUMBER_TABLE
 at org.junit.Assert.fail(Assert.java:88)
 at test.java.com.hp.phoenix.end2end.BaseTest.dropTestObjects(BaseTest.java:180)
 at test.java.com.hp.phoenix.end2end.BaseTest.doBaseTestCleanup(BaseTest.java:112)

On Justin's sugegstion, added ABORT_ON_ERROR for 8810 and the core has this stack:

#0 0x00007ffff4a458a5 in raise () from /lib64/libc.so.6
#1 0x00007ffff4a4700d in abort () from /lib64/libc.so.6
#2 0x00007ffff1248114 in ComCondition::setSQLCODE (
    this=<value optimized out>, newSQLCODE=-8810)
    at ../export/ComDiags.cpp:1425
#3 0x00007ffff3f66e36 in operator<< (d=..., dgObj=...)
    at ../common/DgBaseType.cpp:138
#4 0x00007ffff437e4e3 in CliStatement::fetch (this=<value optimized out>,
    cliGlobals=0xeeade0, output_desc=<value optimized out>, diagsArea=...,
    newOperation=1) at ../cli/Statement.cpp:5310
#5 0x00007ffff4324e0f in SQLCLI_PerformTasks(CliGlobals *, ULng32, SQLSTMT_ID *, SQLDESC_ID *, SQLDESC_ID *, Lng32, Lng32, typedef __va_list_tag __va_list_tag *, SQLCLI_PTR_PAIRS *, SQLCLI_PTR_PAIRS *) (cliGlobals=0xeeade0, tasks=8063,
    statement_id=0x1fbea88, input_descriptor=0x1fbeab8, output_descriptor=0x0,
    num_input_ptr_pairs=0, num_output_ptr_pairs=0, ap=0x7fffe41ef030,
    input_ptr_pairs=0x0, output_ptr_pairs=0x0) at ../cli/Cli.cpp:3382
#6 0x00007ffff438a40b in SQL_EXEC_ClearExecFetchClose (
    statement_id=0x1fbea88, input_descriptor=0x1fbeab8, output_descriptor=0x0,
    num_input_ptr_pairs=0, num_output_ptr_pairs=0, num_total_ptr_pairs=0)
    at ../cli/CliExtern.cpp:2627
#7 0x00007ffff68703bf in SRVR::WSQL_EXEC_ClearExecFetchClose (
    statement_id=0x1fbea88, input_descriptor=<value optimized out>,
    output_descriptor=<value optimized out>,
    num_input_ptr_pairs=<value optimized out>,
    num_output_ptr_pairs=<value optimized out>,
    num_total_ptr_pairs=<value optimized out>) at SQLWrapper.cpp:459
#8 0x00007ffff6866cff in SRVR::EXECUTE2 (pSrvrStmt=0x1fbe470)
    at sqlinterface.cpp:5520
#9 0x00007ffff689733e in odbc_SQLSvc_Execute2_sme_ (
    objtag_=<value optimized out>, call_id_=<value optimized out>,
    dialogueId=<value optimized out>, sqlAsyncEnable=<value optimized out>,
---Type <return> to continue, or q <return> to quit---
    queryTimeout=<value optimized out>, inputRowCnt=<value optimized out>,
    sqlStmtType=128, stmtHandle=33285232, cursorLength=0, cursorName=0x0,
    cursorCharset=1, holdableCursor=0, inValuesLength=0, inValues=0x0,
    returnCode=0x7fffe41ef928, sqlWarningOrErrorLength=0x7fffe41ef924,
    sqlWarningOrError=@0x7fffe41ef900, rowsAffected=0x7fffe41ef920,
    outValuesLength=0x7fffe41ef914, outValues=@0x7fffe41ef8f8)
    at srvrothers.cpp:1517
#10 0x00000000004cbc42 in odbc_SQLSrvr_ExecDirect_ame_ (objtag_=0x24a84d0,
    call_id_=0x24a8528, dialogueId=1492150530,
    stmtLabel=<value optimized out>, cursorName=0x0,
    stmtExplainLabel=<value optimized out>, stmtType=0, sqlStmtType=128,
    sqlString=0x2d43ea4 "drop table PRODUCT_METRICS cascade",
    sqlAsyncEnable=0, queryTimeout=0, inputRowCnt=0, txnID=0, holdableCursor=0)
    at SrvrConnect.cpp:7636
#11 0x0000000000494086 in SQLEXECUTE_IOMessage (objtag_=0x24a84d0,
    call_id_=0x24a8528, operation_id=3012) at Interface/odbcs_srvr.cpp:1734
#12 0x0000000000494134 in DISPATCH_TCPIPRequest (objtag_=0x24a84d0,
    call_id_=0x24a8528, operation_id=<value optimized out>)
    at Interface/odbcs_srvr.cpp:1799
#13 0x0000000000433822 in BUILD_TCPIP_REQUEST (pnode=0x24a84d0)
    at ../Common/TCPIPSystemSrvr.cpp:603
#14 0x00000000004341bd in PROCESS_TCPIP_REQUEST (pnode=0x24a84d0)
    at ../Common/TCPIPSystemSrvr.cpp:581
#15 0x0000000000462396 in CNSKListenerSrvr::tcpip_listener (arg=0xda6310)
    at Interface/linux/Listener_srvr_ps.cpp:397
#16 0x00007ffff47f9290 in sb_thread_sthr_disp (pp_arg=0xeb5490)
    at threadl.cpp:253
#17 0x00007ffff45c5851 in start_thread () from /lib64/libpthread.so.0
#18 0x00007ffff4afb90d in clone () from /lib64/libc.so.6
(gdb)

[Justin Du (justin-du-2)]

The query uses ExDDLTcb::work() method, which calls CmpContext::compileDirect(). It is possible that compileDirect() returned with failure without populating the diagsArea.

Also, the failure seems transient.

Could add code in CmpContext::compieDirect() to check the diagsArea at failure case.

[Sandhya Sundaresan (sandhya-sundaresan)]

This issue is probably a dup of: https://bugs.launchpad.net/trafodion/+bug/1418142

Trafodion does not support DDLs in transactions at this moment. There is a blueprint in the DTM group to track this support: https://blueprints.launchpad.net/trafodion/+spec/dtm-ddl-support 'Support for SQL ddl operations within transactions'

Until that support is added by the DTM group, it's hard to sort out all these issues regarding concurrent executions of DDLs from the SQL layer at this moment. This case will remain open to document the issues that we are encountering in SQL right now. They will be reevaluated once DDL operations in transactions are supported by DTM.

Until then, QA tests will continue to be run in one single stream.