EDI: fetcher repeatedly downloads and processes problematic message files

Bug #1662902 reported by Galen Charlton on 2017-02-08
This bug affects 1 person
Affects Status Importance Assigned to Milestone

Bug Description

The EDI fetcher process compares the names of incoming files against files that have already been downloaded and stored in acq.edi_message by checking host, username, password, directory, file name... AND whether the acq.edi_message row has a status of 'processed'.

This means that if a given message can never be successfully processed for whatever reason, and if the materials vendor never deletes the file from their server, each run of edi_fetcher.pl will cause the same problematic file to be retrieved and (unsuccessfully) processed, thereby causing acq.edi_message to continue to grow. In the wild, this has resulted in a situation where one database had over 15,000,000 useless rows in acq.edi_message.

Evergreen master

Galen Charlton (gmc) on 2017-02-08
tags: added: acq edi
Changed in evergreen:
importance: Undecided → Medium
Galen Charlton (gmc) on 2017-02-09
Changed in evergreen:
milestone: none → 2.12-beta
Galen Charlton (gmc) wrote :

A patch is available in the user/gmcharlt/lp1662902_edi_fetcher branch of the working/Evergreen repository:


tags: added: pullrequest
Bill Erickson (berick) wrote :

Hi Galen, any concerns about also adding "trans_error" to the list of filtered statuses? (I can push a patch). This will prevent duplicate rows for unparseable EDI messages.

Galen Charlton (gmc) wrote :

OK with me. IIRC, 'trans_error' could indicate either a permanent issue with the parseability of the EDI message -- or simply that the Ruby translator process wasn't running for whatever reason. The latter could be automatically recovered from, but I've no problem making recovering from that issue manual, as I don't think I've ever seen the Ruby translator randomly crash.

Bill Erickson (berick) wrote :

Pushed a sign-off for Galen's commit and pushed a secondary commit to filter on trans_error messages as well:


I avoided squashing them to make rolling back just the trans_error change easier if necessary. Included in the commit is the hopefully helpful comment:

In the event parsing failed due to a temporary condition (e.g. Ruby translator was not running), messages can be reprocessed by either deleting the offending edi_message row or setting its status to 'retry'.

Mike Rylander (mrylander) wrote :

Committed to 2.10-master. Thanks, Galen and Bill!

Changed in evergreen:
assignee: nobody → Mike Rylander (mrylander)
status: New → Fix Committed
assignee: Mike Rylander (mrylander) → nobody
Changed in evergreen:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers