Eureka. This is trivially explainable and not due to any race.
The code running on the customer machine had a worker thread that could, potentially, exit without closing the transaction (no commit, rollback, or end). That leads to an error in the CleanupManager (see comment #2) but not before the manager marks that status as notified. Since it got notified, the next cleanup of the bucket will put it onto the free list in a locked state. This is how we wind up with a locked status on the free list, making that bucket essentially unusable.
This quick test code demonstrates all three aspects:
public static class EmptyTxnThread extends Thread {
private final Persistit persistit;
private final boolean doCommit;
public Throwable error;
@Test
public void abandonedTest() throws Throwable {
for(int i = 0; ; ++i) { EmptyTxnThread thread = new EmptyTxnThread(_persistit, i > 0); thread.start(); while(thread.isAlive()) { Thread.sleep(10);
} if(thread.error != null) { throw thread.error;
}
}
}
And output from running it locally:
Abandoning thread: Thread-2
[CLEANUP_MANAGER] ERROR java.lang.IllegalMonitorStateException
at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:155)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260)
at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:460)
at com.persistit.TransactionStatus.wwUnlock(TransactionStatus.java:378)
at com.persistit.TransactionIndexBucket.notifyCompleted(TransactionIndexBucket.java:263)
at com.persistit.TransactionIndex.notifyCompleted(TransactionIndex.java:614)
at com.persistit.Transaction.rollback(Transaction.java:726)
at com.persistit.Transaction.close(Transaction.java:511)
at com.persistit.Persistit.cleanup(Persistit.java:1512)
at com.persistit.CleanupManager.poll(CleanupManager.java:156)
at com.persistit.CleanupManager.runTask(CleanupManager.java:87)
at com.persistit.IOTaskRunnable.run(IOTaskRunnable.java:143)
at java.lang.Thread.run(Thread.java:722)
java.lang.AssertionError: status:<<ts=115 tc=ABORTED mvv=0>> wwLock:<java.util.concurrent.locks.ReentrantLock@51037076[Locked by thread Thread-2]> curThread:<Thread-240>
at com.persistit.TransactionIndexBucket.allocateTransactionStatus(TransactionIndexBucket.java:180)
at com.persistit.TransactionIndex.registerTransaction(TransactionIndex.java:560)
at com.persistit.TransactionIndex.registerTransaction(TransactionIndex.java:527)
at com.persistit.Transaction.begin(Transaction.java:596)
at com.persistit.unit.ExchangeTest$EmptyTxnThread.run(ExchangeTest.java:427)
The only other modification made was to change the assert in allocateTransactionStatus() to print the info from the wwLock to prove the theory.
Eureka. This is trivially explainable and not due to any race.
The code running on the customer machine had a worker thread that could, potentially, exit without closing the transaction (no commit, rollback, or end). That leads to an error in the CleanupManager (see comment #2) but not before the manager marks that status as notified. Since it got notified, the next cleanup of the bucket will put it onto the free list in a locked state. This is how we wind up with a locked status on the free list, making that bucket essentially unusable.
This quick test code demonstrates all three aspects:
public static class EmptyTxnThread extends Thread {
private final Persistit persistit;
private final boolean doCommit;
public Throwable error;
public EmptyTxnThread( Persistit persistit, boolean doCommit) {
this. persistit = persistit;
this. doCommit = doCommit;
}
@Override
Transactio n txn = persistit. getTransaction( );
txn.begin( );
if(doCommit) {
txn. commit( );
txn. end();
System. out.println( "Abandoning thread: " + Thread. currentThread( ).getName( ));
error = t;
public void run() {
try {
} else {
}
} catch(Throwable t) {
}
}
}
@Test
EmptyTxnTh read thread = new EmptyTxnThread( _persistit, i > 0);
thread. start() ;
while( thread. isAlive( )) {
Thread. sleep(10) ;
if( thread. error != null) {
throw thread.error;
public void abandonedTest() throws Throwable {
for(int i = 0; ; ++i) {
}
}
}
}
And output from running it locally: IllegalMonitorS tateException concurrent. locks.Reentrant Lock$Sync. tryRelease( ReentrantLock. java:155) concurrent. locks.AbstractQ ueuedSynchroniz er.release( AbstractQueuedS ynchronizer. java:1260) concurrent. locks.Reentrant Lock.unlock( ReentrantLock. java:460) TransactionStat us.wwUnlock( TransactionStat us.java: 378) TransactionInde xBucket. notifyCompleted (TransactionInd exBucket. java:263) TransactionInde x.notifyComplet ed(TransactionI ndex.java: 614) Transaction. rollback( Transaction. java:726) Transaction. close(Transacti on.java: 511) Persistit. cleanup( Persistit. java:1512) CleanupManager. poll(CleanupMan ager.java: 156) CleanupManager. runTask( CleanupManager. java:87) IOTaskRunnable. run(IOTaskRunna ble.java: 143) Thread. run(Thread. java:722)
Abandoning thread: Thread-2
[CLEANUP_MANAGER] ERROR java.lang.
at java.util.
at java.util.
at java.util.
at com.persistit.
at com.persistit.
at com.persistit.
at com.persistit.
at com.persistit.
at com.persistit.
at com.persistit.
at com.persistit.
at com.persistit.
at java.lang.
java.lang. AssertionError: status:<<ts=115 tc=ABORTED mvv=0>> wwLock: <java.util. concurrent. locks.Reentrant Lock@51037076[ Locked by thread Thread-2]> curThread: <Thread- 240> TransactionInde xBucket. allocateTransac tionStatus( TransactionInde xBucket. java:180) TransactionInde x.registerTrans action( TransactionInde x.java: 560) TransactionInde x.registerTrans action( TransactionInde x.java: 527) Transaction. begin(Transacti on.java: 596) unit.ExchangeTe st$EmptyTxnThre ad.run( ExchangeTest. java:427)
at com.persistit.
at com.persistit.
at com.persistit.
at com.persistit.
at com.persistit.
The only other modification made was to change the assert in allocateTransac tionStatus( ) to print the info from the wwLock to prove the theory.