Librarian sometimes leaks Swift sockets

Bug #1408557 reported by William Grant on 2015-01-08
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Critical
William Grant

Bug Description

The librarian sometimes leaks HTTP connections to swift-proxy.

First seen this week, after we upgraded from r17203 to r17287 and enabled feed-swift's --rename option, so fallbacks from Swift can now fail.

It doesn't affect all instances equally, there are no corresponding inbound connections left open, and they don't appear to correspond to errors. This is only noticeable once requests start failing with EMFILE.

Related branches

William Grant (wgrant) wrote :

Only librarian6 has hit EMFILE so far, but librarian3 is well on the way.

Total open FDs over the hours after the librarian6 restart:

10:50:25 < foli> wgrant: librarian1: 114, librarian2: 184, librarian3: 707, librarian4: 155, librarian5: 261, librarian6: 58
13:44:19 < foli> wgrant: librarian1: 175, librarian2: 175, librarian3: 852, librarian4: 246, librarian5: 259, librarian6: 165

The number of Swift connections varies significantly:

librarian1 FD: 229, librarian2 FD: 250, librarian3 FD: 893, librarian4 FD: 299, librarian5 FD: 62, librarian6 FD: 40,
librarian1 swift: 185, librarian2 swift: 202, librarian3 swift: 845, librarian4 swift: 252, librarian5 swift: 16, librarian6 swift: 15,

It's not clear whether the 1/2/4 behaviour is normal, but I suspect they're in some trouble.

Stuart Bishop (stub) wrote :

lp:~stub/launchpad/trivial is up for review and moves us to swiftclient 2.3.1, with extra bug fixes, and explicit closing of HTTP connections wherever possible.

https://twistedmatrix.com/trac/ticket/3853 is potentially the source of the leak. If so, the above branch should fix the file descriptor leaks (but we would still be leaking stackframes).

Launchpad QA Bot (lpqabot) wrote :
Changed in launchpad:
assignee: nobody → William Grant (wgrant)
tags: added: qa-needstesting
Changed in launchpad:
status: Triaged → In Progress
William Grant (wgrant) on 2015-01-14
tags: added: qa-ok
removed: qa-needstesting
William Grant (wgrant) on 2015-01-14
Changed in launchpad:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.