EDI: fetcher repeatedly downloads and processes problematic message files

[Galen Charlton]

The EDI fetcher process compares the names of incoming files against files that have already been downloaded and stored in acq.edi_message by checking host, username, password, directory, file name... AND whether the acq.edi_message row has a status of 'processed'.

This means that if a given message can never be successfully processed for whatever reason, and if the materials vendor never deletes the file from their server, each run of edi_fetcher.pl will cause the same problematic file to be retrieved and (unsuccessfully) processed, thereby causing acq.edi_message to continue to grow. In the wild, this has resulted in a situation where one database had over 15,000,000 useless rows in acq.edi_message.

Evergreen master

[Galen Charlton (gmc)]

A patch is available in the user/gmcharlt/lp1662902_edi_fetcher branch of the working/Evergreen repository:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/gmcharlt/lp1662902_edi_fetcher

[Bill Erickson (berick)]

Hi Galen, any concerns about also adding "trans_error" to the list of filtered statuses? (I can push a patch). This will prevent duplicate rows for unparseable EDI messages.

[Galen Charlton (gmc)]

OK with me. IIRC, 'trans_error' could indicate either a permanent issue with the parseability of the EDI message -- or simply that the Ruby translator process wasn't running for whatever reason. The latter could be automatically recovered from, but I've no problem making recovering from that issue manual, as I don't think I've ever seen the Ruby translator randomly crash.

[Bill Erickson (berick)]

Pushed a sign-off for Galen's commit and pushed a secondary commit to filter on trans_error messages as well:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/berick/lp1662902_edi_fetcher

I avoided squashing them to make rolling back just the trans_error change easier if necessary. Included in the commit is the hopefully helpful comment:

In the event parsing failed due to a temporary condition (e.g. Ruby translator was not running), messages can be reprocessed by either deleting the offending edi_message row or setting its status to 'retry'.

[Mike Rylander (mrylander)]

Committed to 2.10-master. Thanks, Galen and Bill!