Fine generator can use a lot of memory

Bug #1406367 reported by Bill Erickson on 2014-12-29
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Evergreen
Medium
Unassigned

Bug Description

Circa 2.7, affects all versions.

When the fine generator has to process a lot of transactions, it can use a significant amount of memory.

As an extreme example, a fired up the fine generator on a test server with a large data set which had not run for several days. The storage process jumped to 3.6G RAM processing 388k transactions. For a more practical example, processing ~60k transactions a day puts the process into the ~600MB range, which is still quite high.

The main culprit seems to be that full objects for all relevant transactions are loaded into memory first thing in the fine generator. A secondary concern is that many of the transactions loaded are eventually ignored, since they do not accumulate fines (e.g. staff circulations). For us, this is about 10% of the overall transaction count.

Testing a patched version of the fine generator which collects IDs instead of rows in the initial query, then fetches rows as needed, and skips transactions with recurring/max fines of 0, dropped the memory consumption from 3.6G to 330M on the aforementioned test server.

Testing the code on a smaller server w/ 475 billable transactions showed a very slight (5%) decrease in overall processing time, though that's probably not significant enough to draw any timing conclusions. I have not tested a full run on a larger data set yet.

Will post the code shortly.

Bill Erickson (berick) wrote :
Changed in evergreen:
assignee: Bill Erickson (erickson-esilibrary) → nobody
milestone: none → 2.next
Bill Erickson (berick) on 2014-12-29
tags: added: billing pullrequest
Bill Erickson (berick) wrote :

Pushed a follow-up patch to avoid processing zero-fines transactions when running the fine generator in parallel mode.

Also note that when running in parallel mode, which already processes transactions by ID, the first commit to reduce memory will have no effect.

Changed in evergreen:
status: New → Confirmed
importance: Undecided → Medium
assignee: nobody → Jason Stephenson (jstephenson)
milestone: 2.next → 2.8-beta
Jason Stephenson (jstephenson) wrote :

Been running this on my development system for a week or so. Fines look good and it does seem to have improved the performance. We have a lot of zero fine circulations.

Pushed to master for great justice.

Changed in evergreen:
status: Confirmed → Fix Committed
assignee: Jason Stephenson (jstephenson) → nobody
Changed in evergreen:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers