Parallel DDL operations sees error 8810

[Weishiun Tsai]

To be able to run SQL regression tests in parallel on the same instance is one of the goals that we would like to see happen for the post-1.0 release. In order to do this, Trafodion needs to be able to handle parallel execution with a workload that has a mixture of DDLs and DMLs together. Right now, Trafodion is not very robust when it comes to handling concurrent DDL executions. (or a DDL execution with another DML execution. It’s hard to tell if it has to be 2 DDLs to cause all the problems that we seeing.)

There are several problems in this area. The first noticeable one is this particular 8810 error. QA did an experiment last night by splitting the regression test suites into 2 parts and ran them together on a 4-node cluster. After both completed, we saw a total of 50 occurrences of this 8810 error:

-bash-4.1$ grep 8810 */*.log | grep ERROR | wc -l
33
-bash-4.1$ grep 8810 */*.log | grep ERROR | wc -l
17

A typical error looks like this:

SQL>drop table t10a104;
*** ERROR[8810] Executor ran into an internal failure and returned an error without populating the diagnostics area. This error is being injected to indicate that. [2015-02-04 07:12:04]

There is a bug report https://bugs.launchpad.net/trafodion/+bug/1413831 ‘Phoenix tests run into several error 8810 when other tests are run in parallel with it´ that describes a similar problem. But that one has a narrower scope focusing only on phoenix tests and the severity of the bug report is only High. This one is intended to cover a broader issue of running parallel DDL operations in general. We will mark it as Critical as we need to remove this obstacle first to see what other problems may lie beneath for executing DDLs in parallel.

[Weishiun Tsai (wei-shiun-tsai)]

Lower this bug report from Critical to High. Trafodion does not support DDLs in transactions at this moment. There is a blueprint in the DTM group to track this support: https://blueprints.launchpad.net/trafodion/+spec/dtm-ddl-support 'Support for SQL ddl operations within transactions'

Until that support is added by the DTM group, it's hard to sort out all these issues regarding concurrent executions of DDLs from the SQL layer at this moment. This case will remain open to document the issues that we are encountering in SQL right now. They will be reevaluated once DDL operations in transactions are supported by DTM.

Until then, QA tests will continue to be run in one single stream.