I reproduced the “OutOfOrderScannerNextException” matching
LP 1394702 (failure during trafodion upgrade) this morning.

[trafodion@getafix-1 logs]$ sqlci
Trafodion Conversational Interface 0.9.1
(c) Copyright 2014 Hewlett-Packard Development Company, LP.
>>initialize trafodion, upgrade;
Metadata Upgrade: started

Version Check: started
  Metadata need to be upgraded from Version 2.3 to 3.0
Version Check: done

Drop Old Metadata: started
Drop Old Metadata: done

Backup Current Metadata: started
Backup Current Metadata: done

Drop Current Metadata: started
Drop Current Metadata: done

Initialize New Metadata: started
Initialize New Metadata: done

Copy Old Metadata: started
Restore from Old Metadata: started
Restore from Old Metadata: done

Drop Old Metadata: started
Drop Old Metadata: done

Metadata Upgrade: failed


*** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::nextRow returned error HBASE_ACCESS_ERROR(-705). Cause: 
java.util.concurrent.ExecutionException: java.io.IOException: PerformScanResponse exception org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 0 But the nextCallSeq got from client: 1
java.util.concurrent.FutureTask.report(FutureTask.java:122)
java.util.concurrent.FutureTask.get(FutureTask.java:188)
org.trafodion.sql.HBaseAccess.HTableClient.fetchRows(HTableClient.java:415)
.

--- SQL operation failed with errors.
>>

I moved in a later hbase-trx HDP 0.9.1 jar file matching a December 10th date.
This date had some additional logging changes and multiple hbase-trx
changes from the DTM team.

With this version, the “initialize trafodion, upgrade” succeeded.

[trafodion@getafix-1 logs]$ sqlci
Trafodion Conversational Interface 0.9.1
(c) Copyright 2014 Hewlett-Packard Development Company, LP.
>>initialize trafodion, upgrade;
Metadata Upgrade: started

Version Check: started
  Metadata need to be upgraded from Version 2.3 to 3.0
Version Check: done

Drop Old Metadata: started
Drop Old Metadata: done

Backup Current Metadata: started
Backup Current Metadata: done

Drop Current Metadata: started
Drop Current Metadata: done

Initialize New Metadata: started
Initialize New Metadata: done

Copy Old Metadata: started
Copy Old Metadata: done

Validate Metadata Copy: started
Validate Metadata Copy: done

Customize New Metadata: started
  Start: Update COLUMNS
  End: Update COLUMNS
  Start: Update TABLES
  End: Update TABLES
  Start: Update TEXT
  End: Update TEXT
  Start: Update SEQ_GEN
  End: Update SEQ_GEN
Customize New Metadata: done

Delete Old Metadata Info: started
Delete Old Metadata Info: done

Update Metadata Views: started
Update Metadata Views: done

Update Priv Mgr: started
Update Priv Mgr: done

Update Metadata Version: started
Update Metadata Version: done

Drop Old Metadata from Hbase: started
Drop Old Metadata from Hbase: done

Metadata Upgrade to Version 3.0: done


--- SQL operation complete.
>>

I did have full logging on for both runs.

I’ve kept the logs that demonstrate the failures.
Unfortunately, these logs do not contain the additional logging statements,
so it’s not possible to really confirm what is the problem.

The problem did not reproduce with the enhanced logging statements.

We should continue some Trafodion upgrade testing with the latest Trafodion build to ensure this problem does not reoccur.