EDI File Transfer Needs to be Smarter

[Jason Stephenson]

The EDI pusher and pullers (yes, even the new one) need to be made smarter when they use FTP to get or put EDI order files. We have one vendor with 106 configured EDI accounts in our system. Seventy-two of these are for 1 library and all have the same remote username and password. The fetcher, therefore, makes 106 FTP connections to this vendor every time it runs, and 72 of those connections are all for the same "user." The pushers will only connect if they have a file to send, but they can be affected by the fetcher's behavior.

This one vendor also apparently has rate limiting in their firewall and/or FTP server. We have lately been getting complaints from the one library in particular that their EDI orders are not being sent to the vendor. When I check the logs, I see lots of successful connections and lots of failed connections. After spending several hours with the log messages, I see what appears to be a pattern of accepted messages and failures that suggest a connection limit is being imposed on the vendor's side, or somewhere in between us and the vendor. The pusher often fails to connect because of the timing with the massive fetcher runs. The fetcher sometimes is still running after the pusher starts, or finished so soon before the pusher starts, that our connection attempts are blocked by the vendor.

I am going to adjust our crontab to see if I can alleviate this situation, BUT:

FTP can make a single connection to a server, login, do the appropriate get and put and other commands, logout, login as a different user, do its things, etc., and then disconnect. FTP does not need a separate connection for every user/account combination in the configuration.

The SFTP and SSH protocols, also supported by the EDI pusher and fetcher, cannot do this, but this does not mean that FTP should not be different. We should take advantage of its features where we can. Also, we do not have any vendors using SSH or SFTP, and I am not aware of any vendors who do. It would appear that it is still 1989 on the
Internet as far as library vendors are concerned.

All versions of Evergreen are affected by this "bug."

[Jason Stephenson (jstephenson)]

For those of you suffering along with the B&T FTP server issues, here's a patch that will hardcode the FTP timeout to 15 seconds in OpenILS::Utils::RemoteAccount. For sites with lots of B&T accounts (we have 148!), this should reduce the time spent trying to connect to B&T.

The default timeout for Net::FTP is 2 minutes, and O::U::RemoteAccount does not set the timeout. Each failed attempt to connect to B&T (or any FTP server that is offline) will take 2 minutes. With 148 accounts...you do the math....

I'm posting a patch and not making a branch because this is a quick 'n' dirty patch not intended for inclusion in Evergreen. The timeout value ought to be configurable somewhere or at least better exposed in the code so it more easily adjusted.

[Jason Stephenson (jstephenson)]

The thing about SFTP not being able to do multiple things on a single login is incorrect. It's also incorrect for SSH depending on how that is implemented. I must have been out of my mind on 2019-07-17.

Since Ingram is going to begin requiring SFTP for EDI on 2024-04-15 according to emails that I have seen, I'm going to bump the importance of this bug to Medium.

While it is technically a wishlist item, I think it ought to be backported to 3.10 and 3.11 (as well as 3.12 if that is released before the SFTP code is ready). If anyone disagrees with that assessment, please comment here and we can talk about it. I do think that users of Evergreen will want this feature before upgrading to 3.12.

[Chris Sharp (chrissharp123)]

I just opened bug 2040514 to address the non-functionality of SFTP in current EDI stuff.