Eventlet green threads not released back to the pool leading to choking of new requests
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
High
|
Abhishek Kekane | |||
Icehouse |
High
|
Abhishek Kekane | |||
Juno |
Undecided
|
Unassigned | |||
Glance |
Medium
|
Abhishek Kekane | |||
Icehouse |
Undecided
|
Abhishek Kekane | |||
Juno |
Undecided
|
Unassigned | |||
OpenStack Compute (nova) |
High
|
Abhishek Kekane | |||
Icehouse |
High
|
Abhishek Kekane | |||
OpenStack Heat |
Fix Released
|
Medium
|
Thomas Herve | ||
Kilo |
Fix Released
|
Medium
|
Angus Salkeld | ||
OpenStack Identity (keystone) |
Medium
|
Abhishek Kekane | |||
Juno |
Medium
|
Abhishek Kekane | |||
Kilo |
Undecided
|
Unassigned | |||
OpenStack Security Advisory |
Undecided
|
Unassigned | |||
OpenStack Security Notes |
Undecided
|
Travis McPeak | |||
OpenStack Shared File Systems Service (Manila) |
Medium
|
Valeriy Ponomaryov | |||
Sahara |
Fix Released
|
Medium
|
Sergey Reshetnyak | ||
neutron |
Undecided
|
Abhishek Kekane | |||
Icehouse |
High
|
Abhishek Kekane | |||
Juno |
Undecided
|
Unassigned |
Bug Description
Currently reproduced on Juno milestone 2. but this issue should be reproducible in all releases since its inception.
It is possible to choke OpenStack API controller services using wsgi+eventlet library by simply not closing the client socket connection. Whenever a request is received by any OpenStack API service for example nova api service, eventlet library creates a green thread from the pool and starts processing the request. Even after the response is sent to the caller, the green thread is not returned back to the pool until the client socket connection is closed. This way, any malicious user can send many API requests to the API controller node and determine the wsgi pool size configured for the given service and then send those many requests to the service and after receiving the response, wait there infinitely doing nothing leading to disrupting services for other tenants. Even when service providers have enabled rate limiting feature, it is possible to choke the API services with a group (many tenants) attack.
Following program illustrates choking of nova-api services (but this problem is omnipresent in all other OpenStack API Services using wsgi+eventlet)
Note: I have explicitly set the wsi_default_
After you run the below program, you should try to invoke API
=======
import time
import requests
from multiprocessing import Process
def request(number):
#Port is important here
path = 'http://
try:
response = requests.get(path)
print "RESPONSE %s-%d" % (response.
#during this sleep time, check if the client socket connection is released or not on the API controller node.
print “Thread %d complete" % number
except requests.
print “Exception occurred %d-%s" % (number, str(ex))
if __name__ == '__main__':
processes = []
for number in range(40):
p = Process(
p.start()
for p in processes:
p.join()
=======
Presently, the wsgi server allows persist connections if you configure keepalive to True which is default.
In order to close the client socket connection explicitly after the response is sent and read successfully by the client, you simply have to set keepalive to False when you create a wsgi server.
Additional information: By default eventlet passes “Connection: keepalive” if keepalive is set to True when a response is sent to the client. But it doesn’t have capability to set the timeout and max parameter.
For example.
Keep-Alive: timeout=10, max=5
Note: After we have disabled keepalive in all the OpenStack API service using wsgi library, then it might impact all existing applications built with the assumptions that OpenStack API services uses persistent connections. They might need to modify their applications if reconnection logic is not in place and also they might experience the performance has slowed down as it will need to reestablish the http connection for every request.
Tushar Patil (tpatil) wrote : | #1 |
Tushar Patil (tpatil) wrote : | #3 |
Presently, we are trying to fix on one issue in Glance [1] and during running functional tests, we have found out some of the tests were failing because greenthreads are not released back to the eventlet green pool (client socket connection is not closed) due to which the glance api service doesn't shutdown gracefully. Fixing functional tests in glance without fixing this issue would be temporary solution (we will need to pass “connection: close” to the header for every request made in the functional tests).
Could you please suggest how to go about this security issue and the glance patch?
[1] (glance Patch) https:/
Thierry Carrez (ttx) wrote : | #4 |
So the trick here is, I'm not sure disabling keepalive is an acceptable patch, as it breaks backward compatibility for existing users of the API. It also affects all OpenStack services. So I expect this patch to take a bit of discussion before we agree on the way forward.
So my suggestion would be to fix the test with a temporary solution, if possible silently enough so that we don't reveal this issue while doing so.
Tushar Patil (tpatil) wrote : | #5 |
Hi Mark Washenberger:
Would you please suggest what should be done with this patch https:/
Presently, glance doesn't handle SIGHUP signal properly (glance process exits) and because of it, it is not possible to reload the config changes. We really want this patch to land in Juno-3 and are ready to do whatever you suggest.
Note: Please ignore https:/
Sean Dague (sdague) wrote : | #6 |
@ttx how do you see disabling keepalive breaking backwards compat? Keepalive should just be an optimization, not impact the application semantics.
Thierry Carrez (ttx) wrote : | #7 |
I based that comment on the "Note" in the bug description: "it might impact all existing applications built with the assumptions that OpenStack API services uses persistent connections" -- do you think it's a non-existant concern?
Sean Dague (sdague) wrote : | #8 |
I think it's a non existant concern. HTTP keep alive exists below the application level. It's a pretty standard scaling technique in Apache configuration to disable HTTP keep alive because it means slow clients take proportionally more resource than fast clients (basically inverts your ability to service max throughput).
I think blanket disable of keepalive is probably the sane thing to do.
Abhishek Kekane (abhishek-kekane) wrote : | #9 |
Hi,
We have done some testing using python-clients, httplib and requests api and haven't seen any issue.
With the same connection, its processing multiple api's, one after another.
Thierry Carrez (ttx) wrote : | #10 |
OK, that sounds like a sane move then.
I'm still on the fence on the need for an advisory here, though, as this makes DoS harder but not impossible... so it sounds like a good strengthening move, but it's not really closing a vulnerability, just mitigating one we have to live with anyway ?
Changed in nova: | |
status: | New → Confirmed |
Sean Dague (sdague) wrote : | #11 |
Realistically, all the service launch code has been refactored into Oslo code now, so there is probably a common fix there. This actually isn't familiar enough to me any more to be able to propose the fix.
Sean Dague (sdague) wrote : | #12 |
Never mind, this isn't in the common service layer, here would be the Nova patch. Probably would make it configurable by the operator.
Abhishek Kekane (abhishek-kekane) wrote : | #13 |
Hi Sean,
IMO, there is no need to keep this keepalive as a configurable parameter. Because if someone has configured the keepalive as True then probably he will end-up with this issue.
Abhishek Kekane (abhishek-kekane) wrote : | #14 |
This is a cinder patch to fix this issue.
Abhishek Kekane (abhishek-kekane) wrote : | #15 |
This is a glance patch to fix this issue.
Abhishek Kekane (abhishek-kekane) wrote : | #16 |
This is a keystone patch to fix this issue.
Abhishek Kekane (abhishek-kekane) wrote : | #17 |
This is a neutron patch to fix this issue.
Sean Dague (sdague) wrote : | #18 |
Abhishek, for internal well defined clouds, keepalive could be a useful optimization. So config option would probably be fine.
It would be good if someone who has working tests for this sees if the eventlet change fixes things.
Thierry Carrez (ttx) wrote : | #19 |
Adding OSSG-coresec to get a wider perspective on this.
This is one of those bugs where it's difficult to backport the fix, since it changes established behavior and is therefore not really acceptable as a stable branch patch. If there is a workaround for icehouse/havana, then we can document it in a security note. If there isn't a workaround, we might want to backport the option, and default it to current behavior, and then issue an OSSN about the issue and how you can manually switch that option on.
Abhishek Kekane (abhishek-kekane) wrote : | #20 |
This is the cinder patch with configurable keepalive option.
Abhishek Kekane (abhishek-kekane) wrote : | #21 |
This is the glance patch with configurable keepalive option.
Abhishek Kekane (abhishek-kekane) wrote : | #22 |
This is the keystone patch with configurable keepalive option.
Abhishek Kekane (abhishek-kekane) wrote : | #23 |
This is the neutron patch with configurable keepalive option.
Abhishek Kekane (abhishek-kekane) wrote : | #24 |
This is the nova patch with configurable keepalive option.
Bryan D. Payne (bdpayne) wrote : | #25 |
So my preference would be to backport the fix as this seems like a pretty trivial DoS. If that is deemed unacceptable, then I would support the solution that Thierry proposed in msg #19. Specifically:
"we might want to backport the option, and default it to current behavior, and then issue an OSSN about the issue and how you can manually switch that option on."
Changed in ossa: | |
status: | Incomplete → Confirmed |
Changed in keystone: | |
status: | New → Confirmed |
Thierry Carrez (ttx) wrote : | #26 |
Adding Ihar, Alan and Adam to get the pulse on the stable side of things.
The question is: would you consider it OK to disable keepalive as part of a stable update ?
Ihar Hrachyshka (ihar-hrachyshka) wrote : | #27 |
General rule states that 'Incompatible config file changes' are forbidden. It's not clear what *incompatible* config file changes are though. If keepalive is considered to be speed optimization and does not change the way clouds behave on API level (I guess the answer to both assumptions is 'yes'?), then I think we won't introduce any incompatibilities in existing deployments by disabling it. So I'm in favour of flipping the default value to avoid shipping code that is insecure by default.
That said, the change will probably require explicit notice in release notes, and there should be a way to re-enable keepalive for those who know what they are doing.
Also, Havana was mentioned in the comments. The thing is that the last Havana release was tagged this Monday, so there is no way for us to deliver the fix to users, and since Monday we don't consider Havana as a supported branch anyway. So in terms of backports, we should be interested in Icehouse only.
Still, I think it's worth to mention Havana in CVE, so that affected distributions are explicitly notified to backport those patches to their downstream repositories.
Adam Gandelman (gandelman-a) wrote : | #28 |
I'm personally fine with the proposed patches going to stable/icehouse (keep it enabled by default, provide a config option and notes in release notes for operators to consider) That is, assuming we are okay with adding the new config option to stable (I am) I dont' have an example off the top of my head, but we've done similar in the past. We need to provide the toggle, but whether we keep it enabled by default is where it gets sticky. We can either keep it enabled by default and reference the bug as we normally do, or change behavior + disable it and be more vocal about it in the release notes. TBH I could go either way if people feel strongly, but the patches propose LGTM as-is.
Changed in ossa: | |
importance: | Undecided → High |
Thierry Carrez (ttx) wrote : | #29 |
OK, let's just backport this, and try to get it all out in time for Juno release.
We need an impact desc, and we need coresec reviews on proposed patches.
Changed in cinder: | |
status: | New → In Progress |
Changed in glance: | |
status: | New → In Progress |
Changed in keystone: | |
status: | Confirmed → In Progress |
Changed in nova: | |
status: | Confirmed → In Progress |
Changed in neutron: | |
status: | New → In Progress |
Stuart McLaren (stuart-mclaren) wrote : | #30 |
Is there a second aspect to this?
A client can currently open a connection and do nothing with it.
The socket will never be closed (even with the above keepalive change in place).
(This can be alleviated to some degree by timing out such idle client sockets https:/
Abhishek Kekane (abhishek-kekane) wrote : | #31 |
Hi,
Should I need to resubmit the patches by setting default value for keepalive as False?
Please advise.
Thierry Carrez (ttx) wrote : | #32 |
@Abhishek: no, master patches look good. We'll need the same patches for the stable/icehouse branches, and get them reviewed by core reviewers here on this bug, before we can coordinate their publication.
Stuart McLaren (stuart-mclaren) wrote : | #33 |
@ttx what do you think of the second aspect to this? Namely that a client can choke the system by opening connections to the server and doing nothing with them. Unlike the keepalive case this doesn't require authentication.
Thierry Carrez (ttx) wrote : | #34 |
@Stuart: If I understand correctly, you're describing a Slowloris attack, for which there is no great solution, only mitigation... But then, why close a keepalive-based hole if we keep a simpler one open ? If we require security proxies on open API ports anyway, why close the keepalive hole ? I'll discuss this with the Vulnerability Management Team today.
Robert Clark (robert-clark) wrote : | #35 |
In general I think we should be striving to create a system that's secure by default. An open-source project that's only secure when you buy x,y,z to put in front of it is less than ideal. Though I accept that in cases like this Sloworis style attacks can potentially be mitigated by some open-source proxies too.
Maybe an OSSN or even a chapter on the Security Guide directly address this sort of setup might make sense ? - I'm considering only the aspects discussed by @ttx and @stuart directly above...
Jeremy Stanley (fungi) wrote : | #36 |
So just to confirm... we have a potential for a denial of service condition here. The proposed patch mitigates it for authenticated sessions, but does nothing to solve the more general issue which also can be triggered without authentication and for which we can't reasonably backport a solution.
This sounds more like something we need to document and hope it can be solved more completely in the future.
Tushar Patil (tpatil) wrote : | #37 |
If you pass socket_timeout parameter to the wsgi server, then the telnet client connection is closed automatically by the server when the specified timeout elapses.
Stuart Mclaren has proposed this patch for the cinder project.
https:/
So to recap, considering both keep-alive disabling and eventlet socket timeout, it now boils down to how effective these changes are:
* Can a single user still reserve all the resource as shown in this bug description ?
* Is the default timeout proposed (900 seconds) enough to prevent trivial attacks as shown in this bug description ?
Combined together these patches might warrant an embargoed advisory if they makes this type of attack really more difficult,
else we should consider marking this public security and focus on a good documentation (OSSN and/or security guide).
Also, for what it worth, as long as the relationship between the client side resources and server side resources is a linear one, I guess we should not consider an OSSA, especially an embargoed OSSA that will make the coordination much more difficult...
Eoghan Glynn (eglynn) wrote : | #39 |
Just a datapoint for ceilometer WRT the proposed approach of setting the eventlet.
Ceilometer does not currently use eventlet.
Abhishek Kekane (abhishek-kekane) wrote : | #40 |
Hi Thierry,
Should I submit the patches with socket_timeout parameter as well?
Please advise.
Thierry Carrez (ttx) wrote : | #41 |
@Abishek: I think we need to decide first if it makes sense to fix a specific case when the general case is left open. It will probably be solved after Juno release, since this is hard to coordinate one week away from release.
Thierry Carrez (ttx) wrote : | #42 |
Still no clear way forward on this one. Does it make sense to close the authenticated thread exhaustion through keepalive while unauthenticated users can just get the same effect as easily using slowloris to starve connections ?
Nathan Kinder (nkinder) wrote : | #43 |
@ttx
I think it makes sense to address the 'authenticated thread exhaustion' issue regardless. Even from a non-security standpoint, it could help to prevent an accidental DoS by a poorly written client.
Thierry Carrez (ttx) wrote : | #44 |
@Nathan: Oh, we should definitely fix this, there is no question about that. The question is, do we consider it a vulnerability (and issue an OSSA for it), do we need to keep this embargoed (while opening it up would hugely facilitate the fix), and do we need to backport it to previous releases (and bend our stable rules to do so).
If we consider it a security enhancement rather than a vulnerability, then the answer to those 3 above questions is "no". Since there are more convenient ways to achieve the same effect (DoS on endpoints) and we wouldn't patch those, it feels like considering this one a specific vulnerability that this fix would "close" is at best misleading. So my vote goes to considering it a security enhancement.
Nathan Kinder (nkinder) wrote : | #45 |
@ttx
I'm in agreement with you that this should be considered a security enhancement and not treated as an embargoed vulnerability. I do think that this is a case where backporting to stable releases would be ideal.
Thierry Carrez (ttx) wrote : | #46 |
OK, so unless someone complains, we'll switch this to public bug and start pushing patches publicly on Thursday to harden the future releases.
Changed in cinder: | |
assignee: | nobody → Abhishek Kekane (abhishek-kekane) |
Changed in glance: | |
assignee: | nobody → Abhishek Kekane (abhishek-kekane) |
Changed in keystone: | |
assignee: | nobody → Abhishek Kekane (abhishek-kekane) |
Changed in nova: | |
assignee: | nobody → Abhishek Kekane (abhishek-kekane) |
Changed in neutron: | |
assignee: | nobody → Abhishek Kekane (abhishek-kekane) |
information type: | Private Security → Public |
tags: | added: security |
Changed in ossa: | |
importance: | High → Undecided |
status: | Confirmed → Won't Fix |
Changed in cinder: | |
importance: | Undecided → High |
Fix proposed to branch: master
Review: https:/
Fix proposed to branch: master
Review: https:/
Fix proposed to branch: master
Review: https:/
Fix proposed to branch: master
Review: https:/
Fix proposed to branch: master
Review: https:/
Morgan Fainberg (mdrnstm) wrote : | #52 |
It is recommended Keystone is deployed under mod_wsgi, not eventlet, this still does affect Keystone's eventlet deployment model.
Changed in keystone: | |
importance: | Undecided → Medium |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit fc87da7eeb3451e
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 02:31:15 2014 -0700
Eventlet green threads not released back to pool
Presently, the wsgi server allows persist connections hence even after
the response is sent to the client, it doesn't close the client socket
connection.
Because of this problem, the green thread is not released back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
DocImpact:
Added wsgi_keep_alive option (default=True).
In order to maintain the backward compatibility, setting wsgi_keep_alive
as True by default. Recommended is set it to False.
SecurityImpact
Closes-Bug: #1361360
Change-Id: Ic57b2aceb136e8
Changed in cinder: | |
status: | In Progress → Fix Committed |
Fix proposed to branch: stable/juno
Review: https:/
Changed in nova: | |
importance: | Undecided → High |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/juno
commit b57c024bfb5f542
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 02:31:15 2014 -0700
Eventlet green threads not released back to pool
Presently, the wsgi server allows persist connections hence even after
the response is sent to the client, it doesn't close the client socket
connection.
Because of this problem, the green thread is not released back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
DocImpact:
Added wsgi_keep_alive option (default=True).
In order to maintain the backward compatibility, setting wsgi_keep_alive
as True by default. Recommended is set it to False.
Conflicts:
SecurityImpact
Closes-Bug: #1361360
Change-Id: Ic57b2aceb136e8
(cherry picked from commit fc87da7eeb3451e
Changed in neutron: | |
assignee: | Abhishek Kekane (abhishek-kekane) → Assaf Muller (amuller) |
Changed in neutron: | |
assignee: | Assaf Muller (amuller) → Abhishek Kekane (abhishek-kekane) |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit 04d7a724fdf80db
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 01:37:42 2014 -0700
Eventlet green threads not released back to pool
Presently, the wsgi server allows persist connections hence even after
the response is sent to the client, it doesn't close the client socket
connection.
Because of this problem, the green thread is not released back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
Add a parameter to take advantage of the new(ish) eventlet socket timeout
behaviour. Allows closing idle client connections after a period of
time, eg:
$ time nc localhost 8776
real 1m0.063s
Setting 'client_
DocImpact:
Added wsgi_keep_alive option (default=True).
Added client_
SecurityImpact
Closes-Bug: #1361360
Change-Id: I399b812f6d4522
Changed in nova: | |
status: | In Progress → Fix Committed |
Fix proposed to branch: stable/icehouse
Review: https:/
Fix proposed to branch: stable/juno
Review: https:/
Fix proposed to branch: stable/icehouse
Review: https:/
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/icehouse
commit 4041d30611baa47
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 02:31:15 2014 -0700
Eventlet green threads not released back to pool
Presently, the wsgi server allows persist connections hence even after
the response is sent to the client, it doesn't close the client socket
connection.
Because of this problem, the green thread is not released back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
DocImpact:
Added wsgi_keep_alive option (default=True).
In order to maintain the backward compatibility, setting wsgi_keep_alive
as True by default. Recommended is set it to False.
Conflicts:
SecurityImpact
Closes-Bug: #1361360
Change-Id: Ic57b2aceb136e8
(cherry picked from commit fc87da7eeb3451e
Fix proposed to branch: stable/juno
Review: https:/
Changed in cinder: | |
milestone: | none → kilo-1 |
status: | Fix Committed → Fix Released |
Changed in nova: | |
milestone: | none → kilo-1 |
status: | Fix Committed → Fix Released |
Changed in heat: | |
assignee: | nobody → Xurong Yang (idopra) |
Changed in sahara: | |
assignee: | nobody → Xurong Yang (idopra) |
Fix proposed to branch: master
Review: https:/
Changed in heat: | |
status: | New → In Progress |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit 8e7a0dbb12082f6
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:15:15 2014 -0700
Eventlet green threads not released back to pool
Presently, the wsgi server allows persist connections. Hence even after
the response is sent to the client, it doesn't close the client socket
connection. Because of this problem, the green thread is not released
back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
Added a parameter to take advantage of the new(ish) eventlet socket timeout
behaviour. Allows closing idle client connections after a period of time, eg:
$ time nc localhost 8776
real 1m0.063s
Setting 'client_
DocImpact:
Added wsgi_keep_alive option (default=True).
Added client_
SecurityImpact
Closes-Bug: #1361360
Change-Id: I3a361d6590d180
Changed in neutron: | |
status: | In Progress → Fix Committed |
Fix proposed to branch: stable/juno
Review: https:/
Fix proposed to branch: stable/icehouse
Review: https:/
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/juno
commit a89ae1bd93dea2d
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:15:15 2014 -0700
Eventlet green threads not released back to pool
Presently, the wsgi server allows persist connections. Hence even after
the response is sent to the client, it doesn't close the client socket
connection. Because of this problem, the green thread is not released
back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
Added a parameter to take advantage of the new(ish) eventlet socket timeout
behaviour. Allows closing idle client connections after a period of time, eg:
$ time nc localhost 8776
real 1m0.063s
Setting 'client_
DocImpact:
Added wsgi_keep_alive option (default=True).
Added client_
SecurityImpact
Closes-Bug: #1361360
Change-Id: I3a361d6590d180
(cherry picked from commit 8e7a0dbb12082f6
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/juno
commit f88dc8495f7ec5d
Author: Stuart McLaren <email address hidden>
Date: Fri Sep 5 12:48:04 2014 +0000
Add client_
Add a parameter to take advantage of the new(ish) eventlet socket timeout
behaviour. Allows closing idle client connections after a period of
time, eg:
$ time nc localhost 8776
real 1m0.063s
Setting 'client_
DocImpact:
Added client_
Conflicts:
Note: This patch is not 1:1 cherry-pick, I have changed the default value
of client_
Change-Id: If492810a2f10fa
Closes-bug: #1361360
Closes-bug: #1371022
(cherry picked from commit 08bfa77aeccb8ca
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/juno
commit 965fd3059eff0fd
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 01:37:42 2014 -0700
Eventlet green threads not released back to pool
Presently, the wsgi server allows persist connections hence even after
the response is sent to the client, it doesn't close the client socket
connection.
Because of this problem, the green thread is not released back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
Add a parameter to take advantage of the new(ish) eventlet socket timeout
behaviour. Allows closing idle client connections after a period of
time, eg:
$ time nc localhost 8776
real 1m0.063s
Setting 'client_
DocImpact:
Added wsgi_keep_alive option (default=True).
Added client_
Conflicts:
Note: The required unit-tests are manually added to the below path,
as new path for unit-tests is not present in stable/juno release.
nova/
This patch is not 1:1 cherry-pick, I have changed the default value
of client_
stable branches.
(https:/
SecurityImpact
Closes-Bug: #1361360
Change-Id: I399b812f6d4522
(cherry picked from commit 04d7a724fdf80db
Changed in neutron: | |
milestone: | none → kilo-2 |
status: | Fix Committed → Fix Released |
information type: | Public → Public Security |
information type: | Public Security → Public |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/icehouse
commit a657582c5cd8a39
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 01:37:42 2014 -0700
Eventlet green threads not released back to pool
Presently, the wsgi server allows persist connections hence even after
the response is sent to the client, it doesn't close the client socket
connection.
Because of this problem, the green thread is not released back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
Add a parameter to take advantage of the new(ish) eventlet socket timeout
behaviour. Allows closing idle client connections after a period of
time, eg:
$ time nc localhost 8776
real 1m0.063s
Setting 'client_
DocImpact:
Added wsgi_keep_alive option (default=True).
Added client_
Conflicts:
Note: The required unit-tests are manually added to the below path,
as new path for unit-tests is not present in stable/icehouse release.
nova/
This patch is not 1:1 cherry-pick, I have changed the default value
of client_
stable branches.
(https:/
SecurityImpact
Closes-Bug: #1361360
Change-Id: I399b812f6d4522
(cherry picked from commit 04d7a724fdf80db
Changed in glance: | |
importance: | Undecided → Medium |
milestone: | none → kilo-3 |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit 16a821e00d15520
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:39:59 2014 -0700
Eventlet green threads not released back to pool
Presently, the wsgi server allows persist connections. Hence even after
the response is sent to the client, it doesn't close the client socket
connection. Because of this problem, the green thread is not released
back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
DocImpact:
Added http_keepalive option (default=True).
SecurityImpact
Closes-Bug: #1361360
Change-Id: I93aaca24935a4f
Changed in glance: | |
status: | In Progress → Fix Committed |
Adam Gandelman (gandelman-a) wrote : | #71 |
This should not be committed to icehouse branches. It is incompatible with the minimum eventlet version described in icehouse's globall-
Ihar Hrachyshka (ihar-hrachyshka) wrote : | #72 |
@Adam, we have Cinder patch applied. Does it mean we should revert it?
Fix proposed to branch: stable/juno
Review: https:/
Abhishek Kekane (abhishek-kekane) wrote : | #74 |
Hi Adam, Ihar,
In global-
The patch which was backported to stable/icehouse for cinder [1] does not include 'socket_timeout' applied to wsgi server, so there is no need to revert this patch.
Alan Pevec (apevec) wrote : | #75 |
If keepalive is enough to fix this bug, nova/neutron Icehouse backports could be amended to drop socket_timeout part.
If not, Cinder Icehouse fix is incomplete.
Abhishek Kekane (abhishek-kekane) wrote : | #76 |
Hi Alan,
keepalive is enough to fix this bug.
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/icehouse
commit 4cdbefc202acc24
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:15:15 2014 -0700
Eventlet green threads not released back to pool
Presently, the wsgi server allows persist connections. Hence even after
the response is sent to the client, it doesn't close the client socket
connection. Because of this problem, the green thread is not released
back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
Icehouse backport note: socket_timeout was dropped, it was introduced
in 0.14[*] and Icehouse eventlet lower bound is 0.13
[*] https:/
DocImpact:
Added wsgi_keep_alive option (default=True).
SecurityImpact
Closes-Bug: #1361360
Change-Id: I3a361d6590d180
(cherry picked from commit 8e7a0dbb12082f6
Fix proposed to branch: stable/icehouse
Review: https:/
Changed in glance: | |
status: | Fix Committed → Fix Released |
tags: | added: kilo-rc-potential |
Changed in heat: | |
importance: | Undecided → Medium |
milestone: | none → kilo-rc1 |
Changed in heat: | |
assignee: | Xurong Yang (idopra) → Angus Salkeld (asalkeld) |
Changed in sahara: | |
milestone: | none → liberty-1 |
Changed in heat: | |
milestone: | kilo-rc1 → liberty-1 |
Changed in sahara: | |
importance: | Undecided → Medium |
milestone: | liberty-1 → kilo-rc1 |
status: | New → Confirmed |
Changed in sahara: | |
milestone: | kilo-rc1 → liberty-1 |
Morgan Fainberg (mdrnstm) wrote : | #79 |
Eventlet is deprecated for keystone and not the primary deployment method. For many reasons it is recommended that keystone not be deployed under event let
Changed in keystone: | |
status: | In Progress → Won't Fix |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit 3b08644eb9cf4c5
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:10:57 2014 -0700
Eventlet green threads not released back to pool
Presently, the wsgi server allows persist connections hence even after
the response is sent to the client, it doesn't close the client socket
connection.
Because of this problem, the green thread is not released back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
Add a parameter to take advantage of the new(ish) eventlet socket timeout
behaviour. Allows closing idle client connections after a period of
time, eg:
$ time nc localhost 8776
real 1m0.063s
Setting 'client_
DocImpact:
Added wsgi_keep_alive option (default=True).
Added client_
SecurityImpact
Closes-Bug: #1361360
Change-Id: I03b9c5c64f4bd8
Changed in keystone: | |
status: | Won't Fix → Fix Committed |
Fix proposed to branch: stable/kilo
Review: https:/
Fix proposed to branch: stable/juno
Review: https:/
Changed in glance: | |
milestone: | kilo-3 → 2015.1.0 |
Changed in nova: | |
milestone: | kilo-1 → 2015.1.0 |
Changed in neutron: | |
milestone: | kilo-2 → 2015.1.0 |
Changed in cinder: | |
milestone: | kilo-1 → 2015.1.0 |
tags: | removed: kilo-rc-potential |
tags: | added: kilo-backport-potential |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/kilo
commit 67cda0ccae04471
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:10:57 2014 -0700
Eventlet green threads not released back to pool
Presently, the wsgi server allows persist connections hence even after
the response is sent to the client, it doesn't close the client socket
connection.
Because of this problem, the green thread is not released back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
Add a parameter to take advantage of the new(ish) eventlet socket timeout
behaviour. Allows closing idle client connections after a period of
time, eg:
$ time nc localhost 8776
real 1m0.063s
Setting 'client_
DocImpact:
Added wsgi_keep_alive option (default=True).
Added client_
SecurityImpact
Closes-Bug: #1361360
Change-Id: I03b9c5c64f4bd8
(cherry picked from commit 3b08644eb9cf4c5
tags: | added: in-stable-kilo |
Changed in heat: | |
assignee: | Angus Salkeld (asalkeld) → Thomas Herve (therve) |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/icehouse
commit 392dc228034bbd8
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 01:37:42 2014 -0700
Eventlet green threads not released back to pool
Presently, the wsgi server allows persist connections hence even after
the response is sent to the client, it doesn't close the client socket
connection.
Because of this problem, the green thread is not released back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
Icehouse backport note: socket_timeout was dropped, it was introduced
in 0.14[*] and Icehouse eventlet lower bound is 0.13
[*] https:/
Note: The required unit-tests are manually added to the below path,
as new path for unit-tests is not present in stable/icehouse release.
nova/
DocImpact:
Added wsgi_keep_alive option (default=True).
SecurityImpact
Conflicts:
Closes-Bug: #1361360
(cherry picked from commit 04d7a724fdf80db
Change-Id: I3b14a37edbe4bd
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit 2be72f42f9debf9
Author: yangxurong <email address hidden>
Date: Mon Mar 30 14:39:17 2015 +1000
Eventlet green threads not released back to pool
Presently, the wsgi server allows persist connections hence even after
the response is sent to the client, it doesn't close the client socket
connection.
Because of this problem, the green thread is not released back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
Add a parameter to take advantage of the new(ish) eventlet socket timeout
behaviour. Allows closing idle client connections after a period of
time, eg:
$ time nc localhost 8776
real 1m0.063s
Setting 'client_
DocImpact:
Added wsgi_keep_alive option (default=True).
Added client_
SecurityImpact
Change-Id: I303d87addeed8b
Closes-Bug: #1361360
Changed in heat: | |
status: | In Progress → Fix Committed |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/juno
commit d569ed9db9dc194
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:39:59 2014 -0700
Eventlet green threads not released back to pool
Presently, the wsgi server allows persist connections. Hence even after
the response is sent to the client, it doesn't close the client socket
connection. Because of this problem, the green thread is not released
back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
DocImpact:
Added http_keepalive option (default=True).
Conflicts:
SecurityImpact
Closes-Bug: #1361360
Change-Id: I93aaca24935a4f
(cherry picked from commit 16a821e00d15520
Changed in sahara: | |
milestone: | liberty-1 → liberty-2 |
Changed in keystone: | |
milestone: | none → liberty-1 |
status: | Fix Committed → Fix Released |
Changed in heat: | |
status: | Fix Committed → Fix Released |
Changed in manila: | |
assignee: | nobody → Valeriy Ponomaryov (vponomaryov) |
status: | New → In Progress |
Fix proposed to branch: master
Review: https:/
Travis McPeak (travis-mcpeak) wrote : | #88 |
Since there is unlikely to ever be a patch for the slowloris side of this, I'm opening an OSSN task.
Changed in manila: | |
importance: | Undecided → Medium |
milestone: | none → liberty-2 |
Changed in sahara: | |
milestone: | liberty-2 → liberty-3 |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit accb273c7aa369f
Author: Valeriy Ponomaryov <email address hidden>
Date: Wed Jul 22 20:30:42 2015 +0300
Eventlet green threads not released back to pool
Presently, the wsgi server allows persist connections hence even after the
response is sent to the client, it doesn't close the client socket connection.
Because of this problem, the green thread is not released back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
DocImpact:
Added wsgi_keep_alive option (default=True).
In order to maintain the backward compatibility, setting wsgi_keep_alive
as True by default. Recommended is set it to False.
This is port of Cinder change - [1]
[1] Ic57b2aceb136e8
SecurityImpact
Closes-Bug: #1361360
Change-Id: If9241e6f6ba105
Changed in manila: | |
status: | In Progress → Fix Committed |
Changed in manila: | |
status: | Fix Committed → Fix Released |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/juno
commit d2d6aba069ea310
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:10:57 2014 -0700
Eventlet green threads not released back to pool
Presently, the wsgi server allows persist connections hence even after
the response is sent to the client, it doesn't close the client socket
connection.
Because of this problem, the green thread is not released back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
Add a parameter to take advantage of the new(ish) eventlet socket timeout
behaviour. Allows closing idle client connections after a period of
time, eg:
$ time nc localhost 8776
real 1m0.063s
Setting 'client_
DocImpact:
Added wsgi_keep_alive option (default=True).
Added client_
SecurityImpact
Conflicts:
NOTE: This is not 1:1 cherry-pick because 'eventlet_server' config
group is not present in juno.
Closes-Bug: #1361360
Change-Id: I03b9c5c64f4bd8
(cherry picked from commit 3b08644eb9cf4c5
(cherry picked from commit 67cda0ccae04471
Changed in sahara: | |
milestone: | liberty-3 → liberty-rc1 |
Changed in ossn: | |
assignee: | nobody → Travis McPeak (travis-mcpeak) |
Travis McPeak (travis-mcpeak) wrote : | #91 |
There isn't anything specific to OpenStack about this. Since we don't really have any good advice other than limiting connections, I'm closing the OSSN for this. Anybody interested should look into "slowloris attacks".
Changed in ossn: | |
status: | New → Won't Fix |
tags: | removed: in-stable-kilo kilo-backport-potential |
Fix proposed to branch: stable/kilo
Review: https:/
Changed in sahara: | |
milestone: | liberty-rc1 → next |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/kilo
commit 5f86d05c50e02a8
Author: yangxurong <email address hidden>
Date: Mon Mar 30 14:39:17 2015 +1000
Eventlet green threads not released back to pool
Presently, the wsgi server allows persist connections hence even after
the response is sent to the client, it doesn't close the client socket
connection.
Because of this problem, the green thread is not released back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
Add a parameter to take advantage of the new(ish) eventlet socket timeout
behaviour. Allows closing idle client connections after a period of
time, eg:
$ time nc localhost 8776
real 1m0.063s
Setting 'client_
DocImpact:
Added wsgi_keep_alive option (default=True).
Added client_
SecurityImpact
Change-Id: I303d87addeed8b
Closes-Bug: #1361360
(cherry picked from commit 2be72f42f9debf9
no longer affects: | keystone/icehouse |
Changed in sahara: | |
assignee: | Xurong Yang (idopra) → Sergey Reshetnyak (sreshetniak) |
milestone: | next → mitaka-1 |
Changed in sahara: | |
status: | Confirmed → In Progress |
Changed in keystone: | |
milestone: | liberty-1 → 8.0.0 |
Changed in heat: | |
milestone: | liberty-1 → 5.0.0 |
Changed in manila: | |
milestone: | liberty-2 → 1.0.0 |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit 64583e4b839d96e
Author: Sergey Reshetnyak <email address hidden>
Date: Wed Oct 7 16:24:35 2015 +0300
Use api-paste.ini for loading middleware
Changes:
* use api-paste.ini config for loading middleware
* refactoring middlewares to support loading via pastedeploy
* use debug middleware from oslo_middleware library instead log_exchange
middleware
Closes-bug: #1503983
Closes-bug: #1361360
Change-Id: I444c1799ef53db
Changed in sahara: | |
status: | In Progress → Fix Committed |
Changed in sahara: | |
status: | Fix Committed → Fix Released |
Thanks for the report! The OSSA task is set to incomplete pending additional details from nova-coresec.
It sounds like a slowloris DoS like scenario... What are the side effect of disabling keepalive ?