snapshot scan fails in hive/test018 after recent chechins

[khaled Bouaziz]

After rebasing to the latest changes in the mainline we started seeing an error error with test hive/test018. one of the queries that failed is below
>>UNLOAD
WITH PURGEDATA FROM TARGET
 NEW SNAPSHOT HAVING SUFFIX 'SNAP'
INTO '/user/hive/exttables/unload_customer_and_address'
select * from trafodion.hbase.customer_salt c join trafodion.hbase.customer_address ca on c.c_current_addr_sk = ca.ca_address_sk ;
+>+>+>+>Task: UNLOAD Status: Started
Task: EMPTY TARGET Status: Started
Task: EMPTY TARGET Status: Ended ET: 00:00:00.010

Task: CREATE SNAPSHO Status: Started
       Snapshots created: 2
Task: CREATE SNAPSHO Status: Ended ET: 00:00:02.850
Task: EXTRACT Status: Started

*** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::nextRow returned error HBASE_ACCESS_ERROR(-705). Cause:
java.util.concurrent.ExecutionException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
java.util.concurrent.FutureTask.report(FutureTask.java:122)
java.util.concurrent.FutureTask.get(FutureTask.java:188)
org.trafodion.sql.HBaseAccess.HTableClient.fetchRows(HTableClient.java:575)
.

--- 0 row(s) unloaded.
>>>>

while doing analysis and debugging i was able to reproduce the problem with a select statement using snapshot scan.

set schema hbase;
cqd TRAF_TABLE_SNAPSHOT_SCAN 'on';
cqd TRAF_TABLE_SNAPSHOT_SCAN_SNAP_SUFFIX 'SNAP';
cqd TRAF_TABLE_SNAPSHOT_SCAN_TMP_LOCATION '/bulklaod/';
select * from trafodion.hbase.customer_salt c join trafodion.hbase.customer_address ca on c.c_current_addr_sk = ca.ca_address_sk ;
.......

*** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::nextRow returned error HBASE_ACCESS_ERROR(-705). Cause:
java.util.concurrent.ExecutionException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
java.util.concurrent.FutureTask.report(FutureTask.java:122)
java.util.concurrent.FutureTask.get(FutureTask.java:188)
org.trafodion.sql.HBaseAccess.HTableClient.fetchRows(HTableClient.java:575)
.

--- 40029 row(s) selected.
>>

After undoing the HTableClient.java changes (in my test environment) related to ExecutorService the problem went away and the hive/test018 ran again without any issues. The changes that seem to cause the issue are
--------------
commit 8ec8d50e6dda94591ea1d096f20db1732622f01e
Author: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Date: Sat Feb 28 08:39:12 2015 -0800

    Reduced Java objects allocation in the client side
-----------------
maybe we need to adjust the changes in HTableClient.java so that it works with both regular scan and snapshot scan

[Selvaganesan Govindarajan (selva-ganesan)]

Fixed. See https://review.trafodion.org/#/c/1241/