Comment 9 for bug 1493303
Revision history for this message
| Matthew Oliver (matt-0) wrote Re: [Bug 1493303] Re: Swift proxy memory leak on unfinished read | #9 |
© 2004
Canonical Ltd.
•
Terms of use
•
Data privacy
•
Contact Launchpad Support
•
Blog
•
Careers
•
System status
•
0e1f616
(Get the code!)
Sam and Romain was the author of patch 12d8a53, the patch makes sure
connections are closed. Here is the commit message of the patch:
commit 12d8a53fffea6e4
Author: Samuel Merritt <email address hidden>
Date: Thu Jun 18 12:58:03 2015 -0700
Get better at closing WSGI iterables.
PEP 333 (WSGI) says: "If the iterable returned by the application has
a close() method, the server or gateway must call that method upon
completion of the current request[.]"
There's a bunch of places where we weren't doing that; some of them
matter more than others. Calling .close() can prevent a connection
leak in some cases. In others, it just provides a certain pedantic
smugness. Either way, we should do what WSGI requires.
Noteworthy goofs include:
* If a client is downloading a large object and disconnects halfway
through, a proxy -> obj connection may be leaked. In this case,
the WSGI iterable is a SegmentedIterable, which lacked a close()
method. Thus, when the WSGI server noticed the client disconnect,
it had no way of telling the SegmentedIterable about it, and so
the underlying iterable for the segment's data didn't get
closed.
Here, it seems likely (though unproven) that the object server
would time out and kill the connection, or that a
leaked connection would eventually go away. However, a flurry of
client disconnects could leave a big pile of useless connections.
* If a conditional request receives a 304 or 412, the underlying
app_iter is not closed. This mostly affects conditional requests
for large objects.
The leaked connections were noticed by this patch's co-author, who
made the changes to SegmentedIterable. Those changes helped, but did
not completely fix, the issue. The rest of the patch is an attempt to
plug the rest of the holes.
Co-Authored-By: Romain LE DISEZ <email address hidden>
Change-Id: I168e147aae7c17
Closes-Bug: #1466549
The first noteworthy goof looks like what we're seeing in this bug. This
patch is already in Liberty, but it was applied well after 2.2.0 (Juno) and
Kilo. So it only needs to be backported.
My 2 cents,
Matt
On Wed, Oct 14, 2015 at 7:59 PM, Christian Schwede <
<email address hidden>> wrote:
> Just to add more light on this: most of the connections on pre-12d8a53
> are still not closed after 50 minutes. That definetely does not happen
> on a branch on or after commit 12d8a53.
>
> --
> You received this bug notification because you are a member of Swift
> Core security contacts, which is subscribed to the bug report.
> https:/
>
> Title:
> Swift proxy memory leak on unfinished read
>
> Status in ubuntu-
> New
> Status in OpenStack Security Advisory:
> Incomplete
> Status in OpenStack Object Storage (swift):
> New
> Status in swift package in Ubuntu:
> New
>
> Bug description:
> This issue is being treated as a potential security risk under
> embargo. Please do not make any public mention of embargoed (private)
> security vulnerabilities before their coordinated publication by the
> OpenStack Vulnerability Management Team in the form of an official
> OpenStack Security Advisory. This includes discussion of the bug or
> associated fixes in public forums such as mailing lists, code review
> systems and bug trackers. Please also avoid private disclosure to
> other individuals not already approved for access to this information,
> and provide this same reminder to those who are made aware of the
> issue prior to publication. All discussion should remain confined to
> this private bug report, and any proposed fixes should be added to the
> bug as attachments.
>
> It looks like the Swift proxy will leak memory if the connection is
> closed and the full response is not read. This opens for a potential
> DoS attacks.
>
> Reproduce:
>
> $ swift -A http://
> --segment-size 1048576 <container> <big-file>
> $ curl -H'X-Auth-Token: AUTH_...' "http://
> -m 0.001 > /dev/null
>
> Repeat the curl command a couple of times and you will have more
> information in netstat and sockstat. The important part is the -m
> which sets the max time curl spends at downloading. After that point,
> it'll close the connection.
>
> $ sudo netstat -ant -p | grep :6000
> $ cat /proc/net/sockstat
>
> tcp 0 0 127.0.0.1:6000 0.0.0.0:*
> LISTEN 1358/python
> tcp 0 43221 127.0.0.1:6000 127.0.0.1:48350
> FIN_WAIT1 -
> tcp 0 43221 127.0.0.1:6000 127.0.0.1:48882
> FIN_WAIT1 -
> tcp 939820 0 127.0.0.1:48350 127.0.0.1:6000
> ESTABLISHED 17897/python
> tcp 939820 0 127.0.0.1:48882 127.0.0.1:6000
> ESTABLISHED 17890/python
> tcp 983041 0 127.0.0.1:48191 127.0.0.1:6000
> CLOSE_WAIT 17897/python
> tcp 983041 0 127.0.0.1:48948 127.0.0.1:6000
> CLOSE_WAIT 17892/python
>
> Restarting the proxy frees up the lingering memory.
>
> This problem did not exist in 2.2.0.
>
> ProblemType: Bug
> DistroRelease: Ubuntu 14.04
> Package: swift 2.2.2-0ubuntu1~
> ProcVersionSign
> Uname: Linux 3.16.0-48-generic x86_64
> ApportVersion: 2.14.1-0ubuntu3.12
> Architecture: amd64
> CrashDB:
> {
> "impl": "launchpad",
> "project": "cloud-archive",
> "bug_pattern_url": "
> http://
> }
> Date: Tue Sep 8 09:55:05 2015
> InstallationDate: Installed on 2015-06-22 (77 days ago)
> InstallationMedia: Ubuntu-Server 14.04.2 LTS "Trusty Tahr" - Release
> amd64 (20150218.1)
> PackageArchitec
> SourcePackage: swift
> UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications about this bug go to:
> https:/
>