OutOfMemoryError Direct buffer memory

Bug #985117 reported by Peter Beaman
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Akiban Persistit
Confirmed
Low
Peter Beaman

Bug Description

Encountered while running 50-terminal TPCC on dev01:

Exception in thread "Thread-21" java.lang.OutOfMemoryError: Direct buffer memory
 at java.nio.Bits.reserveMemory(Bits.java:659)
 at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:113)
 at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:305)
 at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:75)
 at sun.nio.ch.IOUtil.write(IOUtil.java:87)
 at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:689)
 at com.persistit.MediatedFileChannel.write(MediatedFileChannel.java:238)
 at com.persistit.JournalManager.flush(JournalManager.java:1236)
 at com.persistit.JournalManager.prepareWriteBuffer(JournalManager.java:1318)
 at com.persistit.JournalManager.writeTransactionToJournal(JournalManager.java:1063)
 at com.persistit.Transaction.flushTransactionBuffer(Transaction.java:1018)
 at com.persistit.Transaction.commit(Transaction.java:723)
 at com.persistit.Transaction.commit(Transaction.java:642)
 at com.persistit.tpcc.PersistitTransactionEngine.newOrderTransaction(PersistitTransactionEngine.java:632)
 at com.persistit.tpcc.jTPCCTerminal.executeTransaction(jTPCCTerminal.java:147)
 at com.persistit.tpcc.jTPCCTerminal.executeTransactions(jTPCCTerminal.java:93)
 at com.persistit.tpcc.jTPCCTerminal.run(jTPCCTerminal.java:50)
 at java.lang.Thread.run(Thread.java:679)
Exception in thread "Thread-32" java.lang.OutOfMemoryError: Direct buffer memory
 at java.nio.Bits.reserveMemory(Bits.java:659)
 at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:113)
 at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:305)
 at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:75)
 at sun.nio.ch.IOUtil.write(IOUtil.java:87)
 at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:689)
 at com.persistit.MediatedFileChannel.write(MediatedFileChannel.java:238)
 at com.persistit.JournalManager.flush(JournalManager.java:1236)
 at com.persistit.JournalManager.prepareWriteBuffer(JournalManager.java:1318)
 at com.persistit.JournalManager.writeTransactionToJournal(JournalManager.java:1063)
 at com.persistit.Transaction.flushTransactionBuffer(Transaction.java:1018)
 at com.persistit.Transaction.commit(Transaction.java:723)
 at com.persistit.Transaction.commit(Transaction.java:642)
 at com.persistit.tpcc.PersistitTransactionEngine.paymentTransaction(PersistitTransactionEngine.java:1116)
 at com.persistit.tpcc.jTPCCTerminal.executeTransaction(jTPCCTerminal.java:185)
 at com.persistit.tpcc.jTPCCTerminal.executeTransactions(jTPCCTerminal.java:81)
 at com.persistit.tpcc.jTPCCTerminal.run(jTPCCTerminal.java:50)
 at java.lang.Thread.run(Thread.java:679)

Need to determine whether this is merely due to a configuration issue or is a genuine bug. I have never seen this before and have used this configuration on dev01 many times.

Peter Beaman (pbeaman)
description: updated
Revision history for this message
Peter Beaman (pbeaman) wrote :

Upon further investigation, it appears that IOUtil.java potentially allocates and caches a direct ByteBuffer for each thread. The cached buffers are head in SoftReferences, but I assume multiple threads could be backed up waiting to write to the journal, and therefore multiple threads could temporarily hold strong references to their DirectByteBuffers; this could exhaust the address space available for direct buffer allocation?

Bottom line is that the JournalManager's _writeBuffer field should be allocated as a direct buffer to avoid all the extra allocation and copying. There is a reference to the array() method in readFully that will need to change; the arraycopy operation can be done instead by a carefully executed call to put(bb) where bb is the destination ByteBuffer.

This change could improve performance, due primarily to gc and to a lesser extend, avoided array copying.

Revision history for this message
Peter Beaman (pbeaman) wrote :

Clarification: the code that allocates and caches direct ByteBuffer instances is sun.nio.ch.IOUtil.

Peter Beaman (pbeaman)
Changed in akiban-persistit:
status: Confirmed → In Progress
Peter Beaman (pbeaman)
Changed in akiban-persistit:
status: In Progress → Confirmed
Changed in akiban-persistit:
milestone: none → 3.1.8
importance: Medium → High
visibility: private → public
Changed in akiban-persistit:
milestone: 3.2.0 → 3.2.1
Changed in akiban-persistit:
milestone: 3.2.1 → future
Revision history for this message
Peter Beaman (pbeaman) wrote :

In all the testing we've done since this was first observed we have never seen a recurrence, and the likelihood is very small. I'm lowering the importance of this to LOW because we have more immediate issues to deal with.

Changed in akiban-persistit:
importance: High → Low
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.