Comment 2 for bug 1422894

Quoting some email from Arvind:

From: Narain, Arvind
Sent: Tuesday, February 17, 2015 8:24 PM
To: Varshneya, Renu; Hanlon, Mike; Hegde, Anuradha; Govindarajan, Selvaganes; Du, Justin
Cc: Sundaresan, Sandhya; Capirala, Tharaknath; Le Rouzo, Christophe
Subject: RE: ZIRCON4: I've never seen this error message before

Hi Renu,

Yes it is related to enabling stats. Went thru the cores with Selva – current thinking is that this is happening because of updating of the statement list maintained by mxosrvr by two different threads - we need to make that code thread safe. We didn’t face these cores on Lava though we did run with the modified code since Feb.

These cores did happen earlier too on Zircon (Feb 13 run).

Regards
Arvind

From: Varshneya, Renu
Sent: Tuesday, February 17, 2015 4:22 PM
To: Narain, Arvind; Hanlon, Mike; Hegde, Anuradha; Govindarajan, Selvaganes; Du, Justin
Cc: Sundaresan, Sandhya; Capirala, Tharaknath
Subject: RE: ZIRCON4: I've never seen this error message before

Thanks Arvind. Even though different issues, are they related to stats?
Mike mentioned in his earlier email that the queries did not fail. Could they have been AQRed?

From: Narain, Arvind
Sent: Tuesday, February 17, 2015 4:17 PM
To: Hanlon, Mike; Varshneya, Renu; Hegde, Anuradha; Govindarajan, Selvaganes; Du, Justin
Cc: Sundaresan, Sandhya; Capirala, Tharaknath
Subject: RE: ZIRCON4: I've never seen this error message before

25567 should be a separate bug – seems like the context got messed up. In the repository context I also see CREATE TABLE LIBRARIES.

Regarding 61369, the change that went into mxosrvr was to delete an existing statement and create a new one for each of the inserts/updates done to the repository table. Granted we should be checking if pSrvrStmt is NULL before doing ExecDirect on the same (to avoid the core), but I’m seeing a pattern of 8802 and 8804 errors for the repository context:

                        pSrvrStmt = getSrvrStmt("STMT_PUBLICATION", FALSE);
                        if (pSrvrStmt != NULL)
                        {
                                pSrvrStmt->cleanupAll();
                                pSrvrStmt->Close(SQL_DROP);
                                pSrvrStmt = NULL;
                        }

                        pSrvrStmt = getSrvrStmt("STMT_PUBLICATION", TRUE);

                        ss.str("");
                        ss.clear();

This change was tested in Feb with OE and YCSB runs.

The messages that I currently see in the logs are:

n020: /opt/hp/squser4/tr101rc2/logs/master_exec_9_5019.log:2015-02-17 18:36:23,050, ERROR, MXOSRVR, Node Number: 9, CPU: 9, PIN:5019, Process Name:$Z09043E , , ,A SQL/MX query failed. SQLCODE: SQL statement allocate failed, sqlcode = -8804, sql error = *** ERROR[8804] The provided input statement does not exist in the current context. [2015-02-17 18:36:23].

n014: /opt/hp/squser4/tr101rc2/logs/master_exec_3_61369.log:2015-02-17 18:36:22,796, ERROR, MXOSRVR, Node Number: 3, CPU: 3, PIN:61369, Process Name:$Z031F3E , , ,A SQL/MX query failed. SQLCODE: SQL statement allocate failed, sqlcode = -8804, sql error = *** ERROR[8804] The provided input statement does not exist in the current context. [2015-02-17 18:36:22].
n014: /opt/hp/squser4/tr101rc2/logs/master_exec_3_61369.log:2015-02-17 18:37:11,118, ERROR, SQL, Node Number: 0, CPU: 3, PIN: 61369, Process Name: $Z031F3E, SQLCODE: 8802,, *** ERROR[8802] Statement allocation failed because the statement already exists in the current context.
n014: /opt/hp/squser4/tr101rc2/logs/master_exec_3_61369.log:2015-02-17 18:37:11,118, ERROR, MXOSRVR, Node Number: 3, CPU: 3, PIN:61369, Process Name:$Z031F3E , , ,A SQL/MX query failed. SQLCODE: SQL statement allocate failed, sqlcode = -8802, sql error = *** ERROR[8802] Statement allocation failed because the statement already exists in the current context. [2015-02-17 18:37:11].

n015: /opt/hp/squser4/tr101rc2/logs/master_exec_4_27661.log:2015-02-17 19:51:27,248, ERROR, MXOSRVR, Node Number: 4, CPU: 4, PIN:27661, Process Name:$Z040MKB , , ,A SQL/MX query failed. SQLCODE: SQL statement allocate failed, sqlcode = -8804, sql error = *** ERROR[8804] The provided input statement does not exist in the current context. [2015-02-17 19:51:27].
n015: /opt/hp/squser4/tr101rc2/logs/master_exec_4_56189.log:2015-02-17 23:12:17,436, ERROR, SQL, Node Number: 0, CPU: 4, PIN: 56189, Process Name: $Z041AVE, SQLCODE: 8802,, *** ERROR[8802] Statement allocation failed because the statement already exists in the current context.
n015: /opt/hp/squser4/tr101rc2/logs/master_exec_4_56189.log:2015-02-17 23:12:17,436, ERROR, MXOSRVR, Node Number: 4, CPU: 4, PIN:56189, Process Name:$Z041AVE , , ,A SQL/MX query failed. SQLCODE: SQL statement allocate failed, sqlcode = -8802, sql error = *** ERROR[8802] Statement allocation failed because the statement already exists in the current context. [2015-02-17 23:12:17].

n012: /opt/hp/squser4/tr101rc2/logs/master_exec_1_13057.log:2015-02-17 19:51:27,237, ERROR, MXOSRVR, Node Number: 1, CPU: 1, PIN:13057, Process Name:$Z010AN2 , , ,A SQL/MX query failed. SQLCODE: SQL statement allocate failed, sqlcode = -8804, sql error = *** ERROR[8804] The provided input statement does not exist in the current context. [2015-02-17 19:51:27].
n012: /opt/hp/squser4/tr101rc2/logs/master_exec_1_13057.log:2015-02-17 19:52:14,489, ERROR, SQL, Node Number: 0, CPU: 1, PIN: 13057, Process Name: $Z010AN2, SQLCODE: 8802,, *** ERROR[8802] Statement allocation failed because the statement already exists in the current context.
n012: /opt/hp/squser4/tr101rc2/logs/master_exec_1_13057.log:2015-02-17 19:52:14,489, ERROR, MXOSRVR, Node Number: 1, CPU: 1, PIN:13057, Process Name:$Z010AN2 , , ,A SQL/MX query failed. SQLCODE: SQL statement allocate failed, sqlcode = -8802, sql error = *** ERROR[8802] Statement allocation failed because the statement already exists in the current context. [2015-02-17 19:52:14].

Still checking – looking at the other cores too.