ERROR: Unable to access Hbase interface.

[Chirag Bhalgami]

From trafci, querying table works fine but dropping throws following error message:

-----------------------------------------------------------------------------

SQL>show tables;
TABLE NAMES
----------------------
ICEWALLTEST_MD5 ICEWALLTEST_PLAIN ICEWALLTEST_PLAIN2

SQL>select count(*) from ICEWALLTEST_PLAIN2;
(EXPR)
--------------------
                   3
--- 1 row(s) selected.

SQL>drop table ICEWALLTEST_PLAIN2;
*** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::nextRow returned error HBASE_ACCESS_ERROR(-705). Cause:
java.util.concurrent.ExecutionException: java.io.IOException: PerformScan error on coprocessor call, scannerID: 0 java.io.IOException: peformScan encountered Exception txID: 17179870191 Exception: org.apache.hadoop.hbase.UnknownScannerException: ScannerId: 0, was scanner already closed?, transaction id 17179870191, nextCallSeq was 1, for region TRAFODION._MD_.VIEWS_USAGE,,1426027968880.03bee199768c6d4c25c1b757d6f71fd2.
java.util.concurrent.FutureTask.report(FutureTask.java:122)
java.util.concurrent.FutureTask.get(FutureTask.java:188)
org.trafodion.sql.HBaseAccess.HTableClient.fetchRows(HTableClient.java:575)
. [2015-03-25 18:59:17]
*** ERROR[8839] Transaction was aborted. [2015-03-25 18:59:17]

-----------------------------------------------------------------------------

Trafodion Build: Version 1.0.1 Release 1.1.0 (Build release [goyalna], branch a94de6e-master, date 06Mar15)
Hadoop Distro: HDP 2.2

Everything is green at Ambari admin.

[Chirag Bhalgami (chirag-bhalgami)]

Additional info:
From sqlci it also throws below error message:

--------------------------------------------------------------------------------
>>drop table ICEWALLTEST_PLAIN2;
*** ERROR[8604] Transaction subsystem TMF returned error 82 while starting a transaction.
--- SQL operation failed with errors.
--------------------------------------------------------------------------------

[Joanie Cooper (joanie-cooper)]

Chirag provided a DDL/DML script to reproduce the problem. I tried this script on a workstation with the latest (032615) R1.1.0 build. The script ran without any "performScan" errors.

It was suggested that the Trafodion installed version be upgraded to the latest build and tried again on the test cluster.

We have seen "performScan" errors when the various components are not all up to date and in-sync.

There is also one remaining "performScan" problem being addressed in the TrxRegionEndpoint coprocessor.

This problem can be confirmed to be the same issue by setting the HBase tracing to TRACE.

We'll wait for the Trafodion instance upgrade and see if the problem can be reproduced.

[Joanie Cooper (joanie-cooper)]

Some additional information provided by Chirag on reproducing the problem:

Executing the provided DDL/DML won't reproduce the issue. It was provided to recreate the schema in your environment and anlayze it. Trafodion went down when execution of the ODB utility (which uses Traf ODBC drivers with 3rd party driver manager unixODBC) was interupted - which somehow brought HBase into weird state. Also, both trafci and sqlci were successfully able to query the table but not succsesfully dropping it. So the bug was opened to analyze what caused HBase to go into this weird state where Trafodion can query (select statements), but the DB but cannot drop the table.

[Joanie Cooper (joanie-cooper)]

From: Bhalgami, Chirag
Sent: Monday, March 30, 2015 10:44 PM
To: Moran, Amanda; Cooper, Joanie; Bucaojit, Oliver; Anderson, Marvin; Mishra, Atanu
Cc: Messavussu, John; Goyal, Narendra (Trafodion R&D); Pham, Van; Broeder, Sean
Subject: RE: [Bug 1436526] [NEW] ERROR: Unable to access Hbase interface.

Hi Amanda,

Can you please tell us what type of permission issues it had? Do you think changing those permissions were part of the fresh hortonworks installation after wiping everything and reinstalling it from scratch?

Just curious, how everything worked without issues earlier if there were permission issues. Also, it will help with QA of backup/restore scripts and also in debugging https://bugs.launchpad.net/trafodion/+bug/1436526.

Thanks,
- Chirag

From: Moran, Amanda
Sent: Monday, March 30, 2015 10:39 PM
To: Cooper, Joanie; Bucaojit, Oliver; Bhalgami, Chirag; Anderson, Marvin; Mishra, Atanu
Cc: Messavussu, John; Goyal, Narendra (Trafodion R&D); Pham, Van; Broeder, Sean
Subject: RE: [Bug 1436526] [NEW] ERROR: Unable to access Hbase interface.

HI there All-

After many permissions issues with Hortonworks, it is finally up and running.

Installed Trafodion with no issues.

[root@n019 installer]# su trafodion
[trafodion@n019 ~]$ sqcheck
Checking if processes are up.
Checking attempt: 1; user specified max: 2. Execution time in seconds: 0.

The SQ environment is up!

Process Configured Actual Down
------- ---------- ------ ----
DTM 6 6
RMS 12 12
MXOSRVR 12 12

Thanks all!

--Amanda

[Joanie Cooper (joanie-cooper)]

Thanks Amanda for installing a new HDP 2.2 and Trafodion install.

The current node list is “-w n024 –w n023 –w n022 –w n021 –w n020 –w n019”.

We had some questions if reversing the order of nodes was causing a problem.

A clean install seems to work fine. But, it might still be worth investigating
if reversing the order has any impact on systems undergoing an upgrade.

I have left the LP bug at “In progress” for a bit.

If any changes need to be addressed by the TDI team for permission issues,
then this bug could be reassigned to that team.

I have updated the bug report with the latest status.

[Joanie Cooper (joanie-cooper)]

We confirmed that a fresh HDP 2.2 and Trafodion installation resolved the HBase connection issues.
Other HDP upgrade installations with Trafodion R1.1.0 have not had these configuration issues.
We are closing out this LP bug as a HDP configuration issue which was resolved by reinstalling HDP 2.2.