fine generator can leak memory when run often
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Evergreen |
Fix Released
|
Medium
|
Unassigned | ||
2.3 |
Fix Released
|
Medium
|
Unassigned | ||
2.4 |
Fix Released
|
Medium
|
Unassigned |
Bug Description
Evergreen 2.1+
From the commit:
Calling "next" from within a "try" block results in a memory leak, presumably because "try" is a tangled nest of subs and evals. Replacing the "try" with a good ol' "eval" avoids the leak.
This can be reproduced with the following:
---------
use Error qw/:try/;
foreach (0..200000) {
try {
next;
} catch Error with {
};
}
---------
This particular leak in the fine generator is onerous when the fine generator is run often (e.g. every 15 mins), which means circs that have already been processed for the day are re-analzyed over and over, causing the code to continue early (next) to the next loop iteration for large numbers of circs. It also happens when a circs are skipped because they have no fine interval, rate, or max fine.
You know this is happening because you will see something like this in the storage stderr log:
Exiting eval via next at
/usr/local/
line 820.
Exiting subroutine via next at
/usr/local/
line 820.
This patch does not avoid the "exiting eval via next" warning, since we're still next'ing out of the eval. It just avoids the memory leak (and the "Exiting subroutine" warning). More extensive refactoring is needed to to completely remove the second warning.
=========
With the right combination of factors (running the fine gen often, re-using the same storage drones over and over to process fines), open-ils.storage processes can grow to consume Gigs of memory in a matter of days, prompting manual service restarts.
Changed in evergreen: | |
status: | New → Triaged |
Changed in evergreen: | |
milestone: | none → 2.4.0-rc |
importance: | Undecided → Medium |
Changed in evergreen: | |
milestone: | 2.4.0-rc → none |
Changed in evergreen: | |
milestone: | none → 2.5.0-m1 |
Changed in evergreen: | |
milestone: | 2.5.0-m1 → 2.5.0-m2 |
Changed in evergreen: | |
status: | Fix Committed → Fix Released |
Let's stop kicking this leaky can down the road. Thanks, Bill!