EDI File Transfer Needs to be Smarter

Bug #1836908 reported by Jason Stephenson
This bug affects 5 people
Affects Status Importance Assigned to Milestone

Bug Description

The EDI pusher and pullers (yes, even the new one) need to be made smarter when they use FTP to get or put EDI order files. We have one vendor with 106 configured EDI accounts in our system. Seventy-two of these are for 1 library and all have the same remote username and password. The fetcher, therefore, makes 106 FTP connections to this vendor every time it runs, and 72 of those connections are all for the same "user." The pushers will only connect if they have a file to send, but they can be affected by the fetcher's behavior.

This one vendor also apparently has rate limiting in their firewall and/or FTP server. We have lately been getting complaints from the one library in particular that their EDI orders are not being sent to the vendor. When I check the logs, I see lots of successful connections and lots of failed connections. After spending several hours with the log messages, I see what appears to be a pattern of accepted messages and failures that suggest a connection limit is being imposed on the vendor's side, or somewhere in between us and the vendor. The pusher often fails to connect because of the timing with the massive fetcher runs. The fetcher sometimes is still running after the pusher starts, or finished so soon before the pusher starts, that our connection attempts are blocked by the vendor.

I am going to adjust our crontab to see if I can alleviate this situation, BUT:

FTP can make a single connection to a server, login, do the appropriate get and put and other commands, logout, login as a different user, do its things, etc., and then disconnect. FTP does not need a separate connection for every user/account combination in the configuration.

The SFTP and SSH protocols, also supported by the EDI pusher and fetcher, cannot do this, but this does not mean that FTP should not be different. We should take advantage of its features where we can. Also, we do not have any vendors using SSH or SFTP, and I am not aware of any vendors who do. It would appear that it is still 1989 on the
Internet as far as library vendors are concerned.

All versions of Evergreen are affected by this "bug."

Lynn Floyd (lfloyd)
Changed in evergreen:
status: New → Confirmed
tags: added: acq-edi
removed: edi
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers