Librarian sometimes leaks Swift sockets

[William Grant]

The librarian sometimes leaks HTTP connections to swift-proxy.

First seen this week, after we upgraded from r17203 to r17287 and enabled feed-swift's --rename option, so fallbacks from Swift can now fail.

It doesn't affect all instances equally, there are no corresponding inbound connections left open, and they don't appear to correspond to errors. This is only noticeable once requests start failing with EMFILE.

[William Grant (wgrant)]

Only librarian6 has hit EMFILE so far, but librarian3 is well on the way.

Total open FDs over the hours after the librarian6 restart:

10:50:25 < foli> wgrant: librarian1: 114, librarian2: 184, librarian3: 707, librarian4: 155, librarian5: 261, librarian6: 58
13:44:19 < foli> wgrant: librarian1: 175, librarian2: 175, librarian3: 852, librarian4: 246, librarian5: 259, librarian6: 165

The number of Swift connections varies significantly:

librarian1 FD: 229, librarian2 FD: 250, librarian3 FD: 893, librarian4 FD: 299, librarian5 FD: 62, librarian6 FD: 40,
librarian1 swift: 185, librarian2 swift: 202, librarian3 swift: 845, librarian4 swift: 252, librarian5 swift: 16, librarian6 swift: 15,

It's not clear whether the 1/2/4 behaviour is normal, but I suspect they're in some trouble.

[Stuart Bishop (stub)]

lp:~stub/launchpad/trivial is up for review and moves us to swiftclient 2.3.1, with extra bug fixes, and explicit closing of HTTP connections wherever possible.

https://twistedmatrix.com/trac/ticket/3853 is potentially the source of the leak. If so, the above branch should fix the file descriptor leaks (but we would still be leaking stackframes).

[Launchpad QA Bot (lpqabot)]

r17301 in stable (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/17301) is part of this bug's fix.