jujud leaking file handles

[John A Meinel]

Still needs further investigation as to the root cause. However we are seeing 250,000 open file handles (according to lsof) for Jujud on a production server. It is currently failing to connect to the API server because of "too many open file handles".

From what we can tell the start of the machine-1.log has "failure to connect" because of too many open file handles. So likely the operational issue is that if the API servers are down, we slowly leak file descriptors. And after long enough time we can no longer allocate new ones so it will always fail to connect.

The lsof output shows ~99% of the open file handles as stuck in CLOSE_WAIT which should be TCP saying "the remote side has closed your connection, but you haven't closed it yet."

[Cheryl Jennings (cherylj)]

Appears to be a dup of 1420057. Taking a look...

[Cheryl Jennings (cherylj)]

There were two causes of the leaked file descriptors. One in go.net, and one in the jujud machine agent code. Bug #1420057 tracked the changes to go.net. I'll use this bug for the changes to jujud's machine agent.

The good news is that this problem is far less likely to occur on 1.22 and later as 1.20 (where the problem was originally seen) had a significantly larger timing window for this leak to occur, and the changes made to the agent code in 1.22 made this problem much less likely to be seen.

[John A Meinel (jameinel)]

Assigned across the versions to Cheryl, feel free to hand it off to someone, but someone should be responsible to make sure your change lands in the various releases.