jdbc_test-cm5.3 failing with build timeouts

[Arvind Narain]

Errors seen with daily builds as well as gate tests:

http://logs.trafodion.org/01/1501/3/check/jdbc_test-cm5.3/140cdc4/

2015-04-15 15:52:24 Running TestBasic
2015-04-15 15:52:24 url: jdbc:t4jdbc://localhost:37800/
2015-04-15 15:52:24 usr: qa001
2015-04-15 15:52:24 password: Neoview2009
2015-04-15 15:52:24 catalog: TRAFODION
2015-04-15 15:52:24 schema: T4QA
2015-04-15 15:52:24 hpjdbc_version : org.trafodion.jdbc.t4.T4Driver
2015-04-15 15:53:51 Batch : total of 6000 rows inserted
2015-04-15 15:53:57 Batch : Passed
2015-04-15 16:35:53 Build timed out (after 90 minutes). Marking the build as failed.
2015-04-15 16:35:53 Build was aborted
2015-04-15 16:35:53 [PostBuildScript] - Execution post build scripts.

====

hbase master logs show the following during this period:

http://logs.trafodion.org/01/1501/3/check/jdbc_test-cm5.3/140cdc4/hbase-logs/hbase-cmf-trafhbase-MASTER-slave-cm53.trafodion.org.log.out

2015-04-15 15:57:08,006 INFO org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler: Splitting hbase:meta logs for slave-cm53.trafodion.org,60020,1429112752431
2015-04-15 15:57:08,011 INFO org.apache.hadoop.hbase.master.SplitLogManager: dead splitlog workers [slave-cm53.trafodion.org,60020,1429112752431]
2015-04-15 15:57:08,013 INFO org.apache.hadoop.hbase.master.SplitLogManager: started splitting 1 logs in [hdfs://slave-cm53.trafodion.org:8020/hbase/WALs/slave-cm53.trafodion.org,60020,1429112752431-splitting]
2015-04-15 15:57:08,855 INFO org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned = 1 tasks={/hbase/splitWAL/WALs%2Fslave-cm53.trafodion.org%2C60020%2C1429112752431-splitting%2Fslave-cm53.trafodion.org%252C60020%252C1429112752431.1429112768947.meta=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}
2015-04-15 15:57:13,856 INFO org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned = 1 tasks={/hbase/splitWAL/WALs%2Fslave-cm53.trafodion.org%2C60020%2C1429112752431-splitting%2Fslave-cm53.trafodion.org%252C60020%252C1429112752431.1429112768947.meta=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}
2015-04-15 15:57:18,857 INFO org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned = 1 tasks={/hbase/splitWAL/WALs%2Fslave-cm53.trafodion.org%2C60020%2C1429112752431-splitting%2Fslave-cm53.trafodion.org%252C60020%252C1429112752431.1429112768947.meta=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}

....

2015-04-15 16:36:01,643 INFO org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned = 1 tasks={/hbase/splitWAL/WALs%2Fslave-cm53.trafodion.org%2C60020%2C1429112752431-splitting%2Fslave-cm53.trafodion.org%252C60020%252C1429112752431.1429112768947.meta=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}
2015-04-15 16:36:07,644 INFO org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned = 1 tasks={/hbase/splitWAL/WALs%2Fslave-cm53.trafodion.org%2C60020%2C1429112752431-splitting%2Fslave-cm53.trafodion.org%252C60020%252C1429112752431.1429112768947.meta=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}
2015-04-15 16:36:08,646 INFO org.apache.hadoop.hbase.master.SplitLogManager: task /hbase/splitWAL/RESCAN0000000013 entered state: DONE slave-cm53.trafodion.org,60000,1429112752445
2015-04-15 16:36:12,645 INFO org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned = 1 tasks={/hbase/splitWAL/WALs%2Fslave-cm53.trafodion.org%2C60020%2C1429112752431-splitting%2Fslave-cm53.trafodion.org%252C60020%252C1429112752431.1429112768947.meta=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}
2015-04-15 16:36:18,645 INFO org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned = 1 tasks={/hbase/splitWAL/WALs%2Fslave-cm53.trafodion.org%2C60020%2C1429112752431-splitting%2Fslave-cm53.trafodion.org%252C60020%252C1429112752431.1429112768947.meta=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}
2015-04-15 16:36:24,645 INFO org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned = 1 tasks={/hbase/splitWAL/WALs%2Fslave-cm53.trafodion.org%2C60020%2C1429112752431-splitting%2Fslave-cm53.trafodion.org%252C60020%252C1429112752431.1429112768947.meta=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}

[Arvind Narain (arvind-narain)]

This is the normal timeline - "Batch All" - Test JDBCBasic26 in TestBasic.java (jdbc_test/src/test/java/org/trafodion/jdbc_test) should take about 2-3 minutes

2015-04-15 15:56:50 Running TestBasic
2015-04-15 15:56:50 url: jdbc:t4jdbc://localhost:37800/
2015-04-15 15:56:50 usr: qa001
2015-04-15 15:56:50 password: Neoview2009
2015-04-15 15:56:50 catalog: TRAFODION
2015-04-15 15:56:50 schema: T4QA
2015-04-15 15:56:50 hpjdbc_version : org.trafodion.jdbc.t4.T4Driver
2015-04-15 15:58:06 Batch : total of 6000 rows inserted
2015-04-15 15:58:12 Batch : Passed
2015-04-15 16:00:03 Batch All : Passed
2015-04-15 16:01:08 JDBCUpsert : Passed