Fine generator can use a lot of memory

[Bill Erickson]

Circa 2.7, affects all versions.

When the fine generator has to process a lot of transactions, it can use a significant amount of memory.

As an extreme example, a fired up the fine generator on a test server with a large data set which had not run for several days. The storage process jumped to 3.6G RAM processing 388k transactions. For a more practical example, processing ~60k transactions a day puts the process into the ~600MB range, which is still quite high.

The main culprit seems to be that full objects for all relevant transactions are loaded into memory first thing in the fine generator. A secondary concern is that many of the transactions loaded are eventually ignored, since they do not accumulate fines (e.g. staff circulations). For us, this is about 10% of the overall transaction count.

Testing a patched version of the fine generator which collects IDs instead of rows in the initial query, then fetches rows as needed, and skips transactions with recurring/max fines of 0, dropped the memory consumption from 3.6G to 330M on the aforementioned test server.

Testing the code on a smaller server w/ 475 billable transactions showed a very slight (5%) decrease in overall processing time, though that's probably not significant enough to draw any timing conclusions. I have not tested a full run on a larger data set yet.

Will post the code shortly.

[Bill Erickson (berick)]

Two commits pushed to

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/berick/lp1406367-fine-generator-bloat-control

[Bill Erickson (berick)]

Pushed a follow-up patch to avoid processing zero-fines transactions when running the fine generator in parallel mode.

Also note that when running in parallel mode, which already processes transactions by ID, the first commit to reduce memory will have no effect.

[Jason Stephenson (jstephenson)]

Been running this on my development system for a week or so. Fines look good and it does seem to have improved the performance. We have a lot of zero fine circulations.

Pushed to master for great justice.