Missing close() after shutdown() in EGCatloader/Record.pm FD Leak

[Josh Stompro]

Running EG 2.8.2, OpenSRF 2.4.1, Debian Jessie, OpenVZ Container on a Proxmox VE 3.4 host.

See OpenILS-General list posts about this issue as I worked through trying to figure out what the issue was.
http://libmail.georgialibraries.org/pipermail/open-ils-general/2015-July/011950.html

The Summary is that in the EGCatLoader/Record.pm there are some functions that make requests to load added content so it can later be used by the template toolkit, in those functions the http requests are shutdown to get rid of them, but they are not closed also. When a socket is just shutdown the file descriptor isn't destoyed, so it just sticks around until apache is restarted.

I noticed this because OpenVZ has various resource limits, one of them counts/limits TCP sockets, which I kept hitting for no apparent reason. On a non virtualized host, or a host that doesn't have that sort of limit the problem isn't really a problem if the FD limits are set fairly high. I was seeing about 5 orphaned sockets per request of a record detail page.

The orphaned sockets show up when 'lsof" is run, as type "sock" with a message of " can't identify protocol"

To confirm this problem, run "lsof | grep " sock " | wc -l" A fixed system will have 2 x <Num Apache Processes> + ~7 of these orphaned sockets. Each apache process has 2 from startup, when a socket call is made without a connect, probably for IPV4/IPV6 setup reasons.

My system would show several hundred to several thousand of the orphaned sockets after a day of running. After restarting apache the number would go back down. And then after loading a record detail page (http://egcatalog/eg/opac/record/10) the number would climb after each request.

The fix is really simple, just call close after the shutdown. Patch on the way.

Josh

[Josh Stompro (u-launchpad-stompro-org)]

Branch at
/user/stompro/lp1478123_socket_fd_leak_added_content

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/stompro/lp1478123_socket_fd_leak_added_content

[Galen Charlton (gmc)]

Pushed to master, rel_2_8, and rel_2_7. Thanks, Josh!