Autorenewal Events Get Stuck in collected and reacting states

[Jason Stephenson]

Evergreen 3.5.3 (and 3.2.10)
OpenSRF 3.2.1
PostgreSQL 9.6

We occasionally have autorenewal events get stuck in collected or reacting state. It seems intermittent and apparently random. I'm attaching results of a query to count the number of autorenewal events by date and there state where the state was not complete, error, or invalid for the past 6 months. You can see that it doesn't happen all that often.

I have encountered issues with Ejabberd max_stanza_size and these events in the past. Our max_stanza_size is set to 10MB on the utility server that runs the action trigger runner. (I think I've mentioned a/t in the chunking/bundling bug.)

Here is some IRC discussion about this issue that was the impetus for me opening this bug:

http://irc.evergreen-ils.org/evergreen/2021-07-13#i_486805

[Ruth Frasur Davis (redavis)]

We see this issue in 3.4.4. Currently, we have a workaround to check for these failures and address. It would be nice to not have to have a work around.

[Jason Stephenson (jstephenson)]

An interesting thing happened this morning/last night. We had 33 autorenewal events get left in the reacting state, but the circulations were renewed, and the autorenewal email notices were generated, i.e. the number of complete email notice events matches the total of complete and reacting autorenewal events.

I used the following query to determine that the reacting events had actually renewed the ciruclations:

select circulation.*
from action.circulation
join action_trigger.event
on circulation.parent_circ = event.target
and event_def = 124
and event.state = 'reacting'
and event.add_time::date = now()::date;