Autorenewal Can Overwhelm open-ils.trigger Service Drones
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Evergreen |
Fix Released
|
Medium
|
Unassigned | ||
3.10 |
Fix Released
|
Medium
|
Unassigned | ||
3.11 |
Fix Released
|
Medium
|
Unassigned |
Bug Description
Evergreen version: 3.10.3+
OpenSRF version: 3.2.3
PostgreSQL version: 10+ (N/A)
The way that the autorenewal code creates the follow up event to notify the patron of the outcome of the autorenewal attempt can overwhelm the open-ils.trigger service leading to the server running out of trigger drones or to drones crashing leading to errors that can cascade and affect multiple autorenewals. The following log messages are indicative of these two occurrences, respectively:
[2023-08-08 04:06:06] open-ils.trigger [WARN:1901:
and
[2023-08-09 03:53:04] open-ils.trigger [ERR :1516974:
Either condition can lead to autorenewal events getting the error state. In many cases, these events can be reset to pending and the action_
These conditions can occur when thousands of renewal events are being processed. The code on line 88 of the AutoRenew.pm module calls code to create the autorenewal notification event and does not wait for it to return before the process continues on to the next autorenewal event. When there are enough events this can lead to the exhaustion of drones and to drones possibly crashing or shutting down without properly notifying the listener.
As discussed in IRC, the solution appears to be to alter the code that creates the autorenewal notification event so that it waits on the event creation to complete before moving on to the next event: http://
A patch branch will be forthcoming.
Changed in evergreen: | |
status: | New → Confirmed |
Changed in evergreen: | |
importance: | Undecided → Medium |
Changed in evergreen: | |
assignee: | nobody → Jason Stephenson (jstephenson) |
Changed in evergreen: | |
assignee: | nobody → Jason Stephenson (jstephenson) |
Changed in evergreen: | |
status: | Confirmed → Fix Committed |
assignee: | Jason Stephenson (jstephenson) → nobody |
Changed in evergreen: | |
status: | Fix Committed → Fix Released |
Confirmed