Librarian sometimes leaks Swift sockets
Bug #1408557 reported by
William Grant
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Launchpad itself |
Fix Released
|
Critical
|
William Grant |
Bug Description
The librarian sometimes leaks HTTP connections to swift-proxy.
First seen this week, after we upgraded from r17203 to r17287 and enabled feed-swift's --rename option, so fallbacks from Swift can now fail.
It doesn't affect all instances equally, there are no corresponding inbound connections left open, and they don't appear to correspond to errors. This is only noticeable once requests start failing with EMFILE.
Related branches
lp:~stub/launchpad/trivial
Rejected
for merging
into
lp:launchpad
- William Grant (community): Disapprove
-
Diff: 224 lines (+80/-19)5 files modifiedlib/lp/services/librarianserver/storage.py (+5/-1)
lib/lp/services/librarianserver/swift.py (+64/-5)
lib/lp/testing/swift/tests/test_fixture.py (+3/-2)
lib/lp_sitecustomize.py (+3/-8)
versions.cfg (+5/-3)
lp:~wgrant/launchpad/bug-1408557-mitigate
- William Grant (community): Approve (code)
-
Diff: 14 lines (+3/-1)1 file modifiedlib/lp/services/librarianserver/storage.py (+3/-1)
tags: |
added: qa-ok removed: qa-needstesting |
Changed in launchpad: | |
status: | In Progress → Fix Released |
To post a comment you must log in.
Only librarian6 has hit EMFILE so far, but librarian3 is well on the way.
Total open FDs over the hours after the librarian6 restart:
10:50:25 < foli> wgrant: librarian1: 114, librarian2: 184, librarian3: 707, librarian4: 155, librarian5: 261, librarian6: 58
13:44:19 < foli> wgrant: librarian1: 175, librarian2: 175, librarian3: 852, librarian4: 246, librarian5: 259, librarian6: 165
The number of Swift connections varies significantly:
librarian1 FD: 229, librarian2 FD: 250, librarian3 FD: 893, librarian4 FD: 299, librarian5 FD: 62, librarian6 FD: 40,
librarian1 swift: 185, librarian2 swift: 202, librarian3 swift: 845, librarian4 swift: 252, librarian5 swift: 16, librarian6 swift: 15,
It's not clear whether the 1/2/4 behaviour is normal, but I suspect they're in some trouble.