OutOfMemoryError Direct buffer memory

[Peter Beaman]

Encountered while running 50-terminal TPCC on dev01:

Exception in thread "Thread-21" java.lang.OutOfMemoryError: Direct buffer memory
 at java.nio.Bits.reserveMemory(Bits.java:659)
 at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:113)
 at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:305)
 at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:75)
 at sun.nio.ch.IOUtil.write(IOUtil.java:87)
 at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:689)
 at com.persistit.MediatedFileChannel.write(MediatedFileChannel.java:238)
 at com.persistit.JournalManager.flush(JournalManager.java:1236)
 at com.persistit.JournalManager.prepareWriteBuffer(JournalManager.java:1318)
 at com.persistit.JournalManager.writeTransactionToJournal(JournalManager.java:1063)
 at com.persistit.Transaction.flushTransactionBuffer(Transaction.java:1018)
 at com.persistit.Transaction.commit(Transaction.java:723)
 at com.persistit.Transaction.commit(Transaction.java:642)
 at com.persistit.tpcc.PersistitTransactionEngine.newOrderTransaction(PersistitTransactionEngine.java:632)
 at com.persistit.tpcc.jTPCCTerminal.executeTransaction(jTPCCTerminal.java:147)
 at com.persistit.tpcc.jTPCCTerminal.executeTransactions(jTPCCTerminal.java:93)
 at com.persistit.tpcc.jTPCCTerminal.run(jTPCCTerminal.java:50)
 at java.lang.Thread.run(Thread.java:679)
Exception in thread "Thread-32" java.lang.OutOfMemoryError: Direct buffer memory
 at java.nio.Bits.reserveMemory(Bits.java:659)
 at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:113)
 at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:305)
 at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:75)
 at sun.nio.ch.IOUtil.write(IOUtil.java:87)
 at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:689)
 at com.persistit.MediatedFileChannel.write(MediatedFileChannel.java:238)
 at com.persistit.JournalManager.flush(JournalManager.java:1236)
 at com.persistit.JournalManager.prepareWriteBuffer(JournalManager.java:1318)
 at com.persistit.JournalManager.writeTransactionToJournal(JournalManager.java:1063)
 at com.persistit.Transaction.flushTransactionBuffer(Transaction.java:1018)
 at com.persistit.Transaction.commit(Transaction.java:723)
 at com.persistit.Transaction.commit(Transaction.java:642)
 at com.persistit.tpcc.PersistitTransactionEngine.paymentTransaction(PersistitTransactionEngine.java:1116)
 at com.persistit.tpcc.jTPCCTerminal.executeTransaction(jTPCCTerminal.java:185)
 at com.persistit.tpcc.jTPCCTerminal.executeTransactions(jTPCCTerminal.java:81)
 at com.persistit.tpcc.jTPCCTerminal.run(jTPCCTerminal.java:50)
 at java.lang.Thread.run(Thread.java:679)

Need to determine whether this is merely due to a configuration issue or is a genuine bug. I have never seen this before and have used this configuration on dev01 many times.

[Peter Beaman (pbeaman)]

Upon further investigation, it appears that IOUtil.java potentially allocates and caches a direct ByteBuffer for each thread. The cached buffers are head in SoftReferences, but I assume multiple threads could be backed up waiting to write to the journal, and therefore multiple threads could temporarily hold strong references to their DirectByteBuffers; this could exhaust the address space available for direct buffer allocation?

Bottom line is that the JournalManager's _writeBuffer field should be allocated as a direct buffer to avoid all the extra allocation and copying. There is a reference to the array() method in readFully that will need to change; the arraycopy operation can be done instead by a carefully executed call to put(bb) where bb is the destination ByteBuffer.

This change could improve performance, due primarily to gc and to a lesser extend, avoided array copying.

[Peter Beaman (pbeaman)]

Clarification: the code that allocates and caches direct ByteBuffer instances is sun.nio.ch.IOUtil.

[Peter Beaman (pbeaman)]

In all the testing we've done since this was first observed we have never seen a recurrence, and the likelihood is very small. I'm lowering the importance of this to LOW because we have more immediate issues to deal with.