Eventlet green threads not released back to the pool leading to choking of new requests

[Tushar Patil]

Currently reproduced on Juno milestone 2. but this issue should be reproducible in all releases since its inception.

It is possible to choke OpenStack API controller services using wsgi+eventlet library by simply not closing the client socket connection. Whenever a request is received by any OpenStack API service for example nova api service, eventlet library creates a green thread from the pool and starts processing the request. Even after the response is sent to the caller, the green thread is not returned back to the pool until the client socket connection is closed. This way, any malicious user can send many API requests to the API controller node and determine the wsgi pool size configured for the given service and then send those many requests to the service and after receiving the response, wait there infinitely doing nothing leading to disrupting services for other tenants. Even when service providers have enabled rate limiting feature, it is possible to choke the API services with a group (many tenants) attack.

Following program illustrates choking of nova-api services (but this problem is omnipresent in all other OpenStack API Services using wsgi+eventlet)

Note: I have explicitly set the wsi_default_pool_size default value to 10 in order to reproduce this problem in nova/wsgi.py.
After you run the below program, you should try to invoke API
============================================================================================
import time
import requests
from multiprocessing import Process

def request(number):
   #Port is important here
   path = 'http://127.0.0.1:8774/servers'
    try:
        response = requests.get(path)
        print "RESPONSE %s-%d" % (response.status_code, number)
        #during this sleep time, check if the client socket connection is released or not on the API controller node.
        time.sleep(1000)
        print “Thread %d complete" % number
    except requests.exceptions.RequestException as ex:
        print “Exception occurred %d-%s" % (number, str(ex))

if __name__ == '__main__':
    processes = []
    for number in range(40):
        p = Process(target=request, args=(number,))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()

================================================================================================

Presently, the wsgi server allows persist connections if you configure keepalive to True which is default.
In order to close the client socket connection explicitly after the response is sent and read successfully by the client, you simply have to set keepalive to False when you create a wsgi server.

Additional information: By default eventlet passes “Connection: keepalive” if keepalive is set to True when a response is sent to the client. But it doesn’t have capability to set the timeout and max parameter.
For example.
Keep-Alive: timeout=10, max=5

Note: After we have disabled keepalive in all the OpenStack API service using wsgi library, then it might impact all existing applications built with the assumptions that OpenStack API services uses persistent connections. They might need to modify their applications if reconnection logic is not in place and also they might experience the performance has slowed down as it will need to reestablish the http connection for every request.

[Tristan Cacqueray (tristan-cacqueray)]

Thanks for the report! The OSSA task is set to incomplete pending additional details from nova-coresec.

It sounds like a slowloris DoS like scenario... What are the side effect of disabling keepalive ?

[Tushar Patil (tpatil)]

Presently, we are trying to fix on one issue in Glance [1] and during running functional tests, we have found out some of the tests were failing because greenthreads are not released back to the eventlet green pool (client socket connection is not closed) due to which the glance api service doesn't shutdown gracefully. Fixing functional tests in glance without fixing this issue would be temporary solution (we will need to pass “connection: close” to the header for every request made in the functional tests).

Could you please suggest how to go about this security issue and the glance patch?

[1] (glance Patch) https://review.openstack.org/#/c/110012/

[Thierry Carrez (ttx)]

So the trick here is, I'm not sure disabling keepalive is an acceptable patch, as it breaks backward compatibility for existing users of the API. It also affects all OpenStack services. So I expect this patch to take a bit of discussion before we agree on the way forward.

So my suggestion would be to fix the test with a temporary solution, if possible silently enough so that we don't reveal this issue while doing so.

[Tushar Patil (tpatil)]

Hi Mark Washenberger:

Would you please suggest what should be done with this patch https://review.openstack.org/#/c/117191/?
Presently, glance doesn't handle SIGHUP signal properly (glance process exits) and because of it, it is not possible to reload the config changes. We really want this patch to land in Juno-3 and are ready to do whatever you suggest.

Note: Please ignore https://review.openstack.org/#/c/110012/ patch as my team member wanted to set dependency and he couldn't do it correctly.

[Sean Dague (sdague)]

@ttx how do you see disabling keepalive breaking backwards compat? Keepalive should just be an optimization, not impact the application semantics.

[Thierry Carrez (ttx)]

I based that comment on the "Note" in the bug description: "it might impact all existing applications built with the assumptions that OpenStack API services uses persistent connections" -- do you think it's a non-existant concern?

[Sean Dague (sdague)]

I think it's a non existant concern. HTTP keep alive exists below the application level. It's a pretty standard scaling technique in Apache configuration to disable HTTP keep alive because it means slow clients take proportionally more resource than fast clients (basically inverts your ability to service max throughput).

I think blanket disable of keepalive is probably the sane thing to do.

[Abhishek Kekane (abhishek-kekane)]

Hi,

We have done some testing using python-clients, httplib and requests api and haven't seen any issue.
With the same connection, its processing multiple api's, one after another.

[Thierry Carrez (ttx)]

OK, that sounds like a sane move then.

I'm still on the fence on the need for an advisory here, though, as this makes DoS harder but not impossible... so it sounds like a good strengthening move, but it's not really closing a vulnerability, just mitigating one we have to live with anyway ?

[Sean Dague (sdague)]

Realistically, all the service launch code has been refactored into Oslo code now, so there is probably a common fix there. This actually isn't familiar enough to me any more to be able to propose the fix.

[Sean Dague (sdague)]

Never mind, this isn't in the common service layer, here would be the Nova patch. Probably would make it configurable by the operator.

[Abhishek Kekane (abhishek-kekane)]

Hi Sean,

IMO, there is no need to keep this keepalive as a configurable parameter. Because if someone has configured the keepalive as True then probably he will end-up with this issue.

[Abhishek Kekane (abhishek-kekane)]

This is a cinder patch to fix this issue.

[Abhishek Kekane (abhishek-kekane)]

This is a glance patch to fix this issue.

[Abhishek Kekane (abhishek-kekane)]

This is a keystone patch to fix this issue.

[Abhishek Kekane (abhishek-kekane)]

This is a neutron patch to fix this issue.

[Sean Dague (sdague)]

Abhishek, for internal well defined clouds, keepalive could be a useful optimization. So config option would probably be fine.

It would be good if someone who has working tests for this sees if the eventlet change fixes things.

[Thierry Carrez (ttx)]

Adding OSSG-coresec to get a wider perspective on this.

This is one of those bugs where it's difficult to backport the fix, since it changes established behavior and is therefore not really acceptable as a stable branch patch. If there is a workaround for icehouse/havana, then we can document it in a security note. If there isn't a workaround, we might want to backport the option, and default it to current behavior, and then issue an OSSN about the issue and how you can manually switch that option on.

[Abhishek Kekane (abhishek-kekane)]

This is the cinder patch with configurable keepalive option.

[Abhishek Kekane (abhishek-kekane)]

This is the glance patch with configurable keepalive option.

[Abhishek Kekane (abhishek-kekane)]

This is the keystone patch with configurable keepalive option.

[Abhishek Kekane (abhishek-kekane)]

This is the neutron patch with configurable keepalive option.

[Abhishek Kekane (abhishek-kekane)]

This is the nova patch with configurable keepalive option.

[Bryan D. Payne (bdpayne)]

So my preference would be to backport the fix as this seems like a pretty trivial DoS. If that is deemed unacceptable, then I would support the solution that Thierry proposed in msg #19. Specifically:

"we might want to backport the option, and default it to current behavior, and then issue an OSSN about the issue and how you can manually switch that option on."

[Thierry Carrez (ttx)]

Adding Ihar, Alan and Adam to get the pulse on the stable side of things.
The question is: would you consider it OK to disable keepalive as part of a stable update ?

[Ihar Hrachyshka (ihar-hrachyshka)]

General rule states that 'Incompatible config file changes' are forbidden. It's not clear what *incompatible* config file changes are though. If keepalive is considered to be speed optimization and does not change the way clouds behave on API level (I guess the answer to both assumptions is 'yes'?), then I think we won't introduce any incompatibilities in existing deployments by disabling it. So I'm in favour of flipping the default value to avoid shipping code that is insecure by default.

That said, the change will probably require explicit notice in release notes, and there should be a way to re-enable keepalive for those who know what they are doing.

Also, Havana was mentioned in the comments. The thing is that the last Havana release was tagged this Monday, so there is no way for us to deliver the fix to users, and since Monday we don't consider Havana as a supported branch anyway. So in terms of backports, we should be interested in Icehouse only.

Still, I think it's worth to mention Havana in CVE, so that affected distributions are explicitly notified to backport those patches to their downstream repositories.

[Adam Gandelman (gandelman-a)]

I'm personally fine with the proposed patches going to stable/icehouse (keep it enabled by default, provide a config option and notes in release notes for operators to consider) That is, assuming we are okay with adding the new config option to stable (I am) I dont' have an example off the top of my head, but we've done similar in the past. We need to provide the toggle, but whether we keep it enabled by default is where it gets sticky. We can either keep it enabled by default and reference the bug as we normally do, or change behavior + disable it and be more vocal about it in the release notes. TBH I could go either way if people feel strongly, but the patches propose LGTM as-is.

[Thierry Carrez (ttx)]

OK, let's just backport this, and try to get it all out in time for Juno release.
We need an impact desc, and we need coresec reviews on proposed patches.

[Stuart McLaren (stuart-mclaren)]

Is there a second aspect to this?

A client can currently open a connection and do nothing with it.

The socket will never be closed (even with the above keepalive change in place).

(This can be alleviated to some degree by timing out such idle client sockets https://review.openstack.org/#/c/119365/)

[Abhishek Kekane (abhishek-kekane)]

Hi,

Should I need to resubmit the patches by setting default value for keepalive as False?

Please advise.

[Thierry Carrez (ttx)]

@Abhishek: no, master patches look good. We'll need the same patches for the stable/icehouse branches, and get them reviewed by core reviewers here on this bug, before we can coordinate their publication.

[Stuart McLaren (stuart-mclaren)]

@ttx what do you think of the second aspect to this? Namely that a client can choke the system by opening connections to the server and doing nothing with them. Unlike the keepalive case this doesn't require authentication.

[Thierry Carrez (ttx)]

@Stuart: If I understand correctly, you're describing a Slowloris attack, for which there is no great solution, only mitigation... But then, why close a keepalive-based hole if we keep a simpler one open ? If we require security proxies on open API ports anyway, why close the keepalive hole ? I'll discuss this with the Vulnerability Management Team today.

[Robert Clark (robert-clark)]

In general I think we should be striving to create a system that's secure by default. An open-source project that's only secure when you buy x,y,z to put in front of it is less than ideal. Though I accept that in cases like this Sloworis style attacks can potentially be mitigated by some open-source proxies too.

Maybe an OSSN or even a chapter on the Security Guide directly address this sort of setup might make sense ? - I'm considering only the aspects discussed by @ttx and @stuart directly above...

[Jeremy Stanley (fungi)]

So just to confirm... we have a potential for a denial of service condition here. The proposed patch mitigates it for authenticated sessions, but does nothing to solve the more general issue which also can be triggered without authentication and for which we can't reasonably backport a solution.

This sounds more like something we need to document and hope it can be solved more completely in the future.

[Tushar Patil (tpatil)]

If you pass socket_timeout parameter to the wsgi server, then the telnet client connection is closed automatically by the server when the specified timeout elapses.

Stuart Mclaren has proposed this patch for the cinder project.
https://review.openstack.org/#/c/119365/

[Tristan Cacqueray (tristan-cacqueray)]

So to recap, considering both keep-alive disabling and eventlet socket timeout, it now boils down to how effective these changes are:
* Can a single user still reserve all the resource as shown in this bug description ?
* Is the default timeout proposed (900 seconds) enough to prevent trivial attacks as shown in this bug description ?

Combined together these patches might warrant an embargoed advisory if they makes this type of attack really more difficult,
else we should consider marking this public security and focus on a good documentation (OSSN and/or security guide).

Also, for what it worth, as long as the relationship between the client side resources and server side resources is a linear one, I guess we should not consider an OSSA, especially an embargoed OSSA that will make the coordination much more difficult...

[Eoghan Glynn (eglynn)]

Just a datapoint for ceilometer WRT the proposed approach of setting the eventlet.wsgi.server keep-alive to false in order to mitigate this issue.

Ceilometer does not currently use eventlet.wsgi.server ... instead we use wsgiref.simple_server with pecan, and AFAICS an analogous means of disabling keep-alive is not surfaced in that case.

[Abhishek Kekane (abhishek-kekane)]

Hi Thierry,

Should I submit the patches with socket_timeout parameter as well?

Please advise.

[Thierry Carrez (ttx)]

@Abishek: I think we need to decide first if it makes sense to fix a specific case when the general case is left open. It will probably be solved after Juno release, since this is hard to coordinate one week away from release.

[Thierry Carrez (ttx)]

Still no clear way forward on this one. Does it make sense to close the authenticated thread exhaustion through keepalive while unauthenticated users can just get the same effect as easily using slowloris to starve connections ?

[Nathan Kinder (nkinder)]

@ttx
I think it makes sense to address the 'authenticated thread exhaustion' issue regardless. Even from a non-security standpoint, it could help to prevent an accidental DoS by a poorly written client.

[Thierry Carrez (ttx)]

@Nathan: Oh, we should definitely fix this, there is no question about that. The question is, do we consider it a vulnerability (and issue an OSSA for it), do we need to keep this embargoed (while opening it up would hugely facilitate the fix), and do we need to backport it to previous releases (and bend our stable rules to do so).

If we consider it a security enhancement rather than a vulnerability, then the answer to those 3 above questions is "no". Since there are more convenient ways to achieve the same effect (DoS on endpoints) and we wouldn't patch those, it feels like considering this one a specific vulnerability that this fix would "close" is at best misleading. So my vote goes to considering it a security enhancement.

[Nathan Kinder (nkinder)]

@ttx
I'm in agreement with you that this should be considered a security enhancement and not treated as an embargoed vulnerability. I do think that this is a case where backporting to stable releases would be ideal.

[Thierry Carrez (ttx)]

OK, so unless someone complains, we'll switch this to public bug and start pushing patches publicly on Thursday to harden the future releases.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/130821

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/130824

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/130834

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/130839

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/130843

[Morgan Fainberg (mdrnstm)]

It is recommended Keystone is deployed under mod_wsgi, not eventlet, this still does affect Keystone's eventlet deployment model.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/130821
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=fc87da7eeb3451e139ee71b31095d0b9093332ce
Submitter: Jenkins
Branch: master

commit fc87da7eeb3451e139ee71b31095d0b9093332ce
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 02:31:15 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    In order to maintain the backward compatibility, setting wsgi_keep_alive
    as True by default. Recommended is set it to False.

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: Ic57b2aceb136e8626388cfe4df72b2f47cb0661c

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/131987

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/131987
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=b57c024bfb5f542e0412f268038f38bf52e06456
Submitter: Jenkins
Branch: stable/juno

commit b57c024bfb5f542e0412f268038f38bf52e06456
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 02:31:15 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    In order to maintain the backward compatibility, setting wsgi_keep_alive
    as True by default. Recommended is set it to False.

    Conflicts:
            cinder/wsgi.py
            etc/cinder/cinder.conf.sample

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: Ic57b2aceb136e8626388cfe4df72b2f47cb0661c
    (cherry picked from commit fc87da7eeb3451e139ee71b31095d0b9093332ce)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/130843
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=04d7a724fdf80db51e73f12c5b8c982db9310742
Submitter: Jenkins
Branch: master

commit 04d7a724fdf80db51e73f12c5b8c982db9310742
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 01:37:42 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:

    $ time nc localhost 8776
    real 1m0.063s

    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=900).

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I399b812f6d452226fd306c423de8dcea8520d2aa

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/138365

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/138368

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/138811

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/138365
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=4041d30611baa476d33627f5078d5bcc12ef50eb
Submitter: Jenkins
Branch: stable/icehouse

commit 4041d30611baa476d33627f5078d5bcc12ef50eb
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 02:31:15 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    In order to maintain the backward compatibility, setting wsgi_keep_alive
    as True by default. Recommended is set it to False.

    Conflicts:
            cinder/wsgi.py
            etc/cinder/cinder.conf.sample

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: Ic57b2aceb136e8626388cfe4df72b2f47cb0661c
    (cherry picked from commit fc87da7eeb3451e139ee71b31095d0b9093332ce)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/140304

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/144074

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/130834
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8e7a0dbb12082f6159d98a4628fb8a6fcd05e95a
Submitter: Jenkins
Branch: master

commit 8e7a0dbb12082f6159d98a4628fb8a6fcd05e95a
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:15:15 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections. Hence even after
    the response is sent to the client, it doesn't close the client socket
    connection. Because of this problem, the green thread is not released
    back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Added a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of time, eg:

    $ time nc localhost 8776
    real 1m0.063s

    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=900).

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I3a361d6590d1800b85791f23ac1cdfd79815341b

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/144708

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/144714

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/144708
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a89ae1bd93dea2dd1902c1d49572329f6646abcc
Submitter: Jenkins
Branch: stable/juno

commit a89ae1bd93dea2dd1902c1d49572329f6646abcc
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:15:15 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections. Hence even after
    the response is sent to the client, it doesn't close the client socket
    connection. Because of this problem, the green thread is not released
    back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Added a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of time, eg:

    $ time nc localhost 8776
    real 1m0.063s

    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=900).

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I3a361d6590d1800b85791f23ac1cdfd79815341b
    (cherry picked from commit 8e7a0dbb12082f6159d98a4628fb8a6fcd05e95a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140304
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=f88dc8495f7ec5dc7f1b01c43f0237141af6b98f
Submitter: Jenkins
Branch: stable/juno

commit f88dc8495f7ec5dc7f1b01c43f0237141af6b98f
Author: Stuart McLaren <email address hidden>
Date: Fri Sep 5 12:48:04 2014 +0000

    Add client_socket_timeout option

    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:

    $ time nc localhost 8776
    real 1m0.063s

    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added client_socket_timeout option (default=0).

    Conflicts:
            cinder/wsgi.py
            etc/cinder/cinder.conf.sample

    Note: This patch is not 1:1 cherry-pick, I have changed the default value
    of client_socket_timeout to 0.

    Change-Id: If492810a2f10fa5954f8c8bb708b14be0b77fb90
    Closes-bug: #1361360
    Closes-bug: #1371022
    (cherry picked from commit 08bfa77aeccb8ca589e3fb5cf9771879818f59de)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/138368
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=965fd3059eff0fdc7228754742042bbff3abf705
Submitter: Jenkins
Branch: stable/juno

commit 965fd3059eff0fdc7228754742042bbff3abf705
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 01:37:42 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:

    $ time nc localhost 8776
    real 1m0.063s

    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=0).

    Conflicts:
            nova/tests/unit/test_wsgi.py

    Note: The required unit-tests are manually added to the below path,
    as new path for unit-tests is not present in stable/juno release.
    nova/tests/compute/test_host_api.py

    This patch is not 1:1 cherry-pick, I have changed the default value
    of client_socket_timeout to 0, as per the policy for changes to
    stable branches.
    (https://wiki.openstack.org/wiki/StableBranch#Appropriate_Fixes)

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I399b812f6d452226fd306c423de8dcea8520d2aa
    (cherry picked from commit 04d7a724fdf80db51e73f12c5b8c982db9310742)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/138811
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a657582c5cd8a39580c44ad401fd3e69870068b1
Submitter: Jenkins
Branch: stable/icehouse

commit a657582c5cd8a39580c44ad401fd3e69870068b1
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 01:37:42 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:

    $ time nc localhost 8776
    real 1m0.063s

    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=0).

    Conflicts:
            nova/tests/unit/test_wsgi.py

    Note: The required unit-tests are manually added to the below path,
    as new path for unit-tests is not present in stable/icehouse release.
    nova/tests/compute/test_host_api.py

    This patch is not 1:1 cherry-pick, I have changed the default value
    of client_socket_timeout to 0, as per the policy for changes to
    stable branches.
    (https://wiki.openstack.org/wiki/StableBranch#Appropriate_Fixes)

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I399b812f6d452226fd306c423de8dcea8520d2aa
    (cherry picked from commit 04d7a724fdf80db51e73f12c5b8c982db9310742)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/130839
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=16a821e00d15520d2f6e940e184bd289b8782620
Submitter: Jenkins
Branch: master

commit 16a821e00d15520d2f6e940e184bd289b8782620
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:39:59 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections. Hence even after
    the response is sent to the client, it doesn't close the client socket
    connection. Because of this problem, the green thread is not released
    back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    DocImpact:
    Added http_keepalive option (default=True).

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I93aaca24935a4f3096210233097dd6b8c5440176

[Adam Gandelman (gandelman-a)]

This should not be committed to icehouse branches. It is incompatible with the minimum eventlet version described in icehouse's globall-requirements.txt

[Ihar Hrachyshka (ihar-hrachyshka)]

@Adam, we have Cinder patch applied. Does it mean we should revert it?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/162964

[Abhishek Kekane (abhishek-kekane)]

Hi Adam, Ihar,

In global-requirements.txt eventlet version is mentioned as 'eventlet>=0.13.0,<0.16.0' and keepalive option is availabe in version 0.13.2 but socket_timeout is not available in the same version.

The patch which was backported to stable/icehouse for cinder [1] does not include 'socket_timeout' applied to wsgi server, so there is no need to revert this patch.

[1] https://review.openstack.org/#/c/138365/

[Alan Pevec (apevec)]

If keepalive is enough to fix this bug, nova/neutron Icehouse backports could be amended to drop socket_timeout part.
If not, Cinder Icehouse fix is incomplete.

[Abhishek Kekane (abhishek-kekane)]

Hi Alan,

keepalive is enough to fix this bug.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/144714
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4cdbefc202acc2499c253da625db48d6165b7c84
Submitter: Jenkins
Branch: stable/icehouse

commit 4cdbefc202acc2499c253da625db48d6165b7c84
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:15:15 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections. Hence even after
    the response is sent to the client, it doesn't close the client socket
    connection. Because of this problem, the green thread is not released
    back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Icehouse backport note: socket_timeout was dropped, it was introduced
    in 0.14[*] and Icehouse eventlet lower bound is 0.13

    [*] https://github.com/eventlet/eventlet/commit/7d4916f01462de09cb58853d9de2e85777c2ad5b

    DocImpact:
    Added wsgi_keep_alive option (default=True).

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I3a361d6590d1800b85791f23ac1cdfd79815341b
    (cherry picked from commit 8e7a0dbb12082f6159d98a4628fb8a6fcd05e95a)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/164643

[Morgan Fainberg (mdrnstm)]

Eventlet is deprecated for keystone and not the primary deployment method. For many reasons it is recommended that keystone not be deployed under event let

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/130824
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=3b08644eb9cf4c5aca51a36250ae93105c17f6c4
Submitter: Jenkins
Branch: master

commit 3b08644eb9cf4c5aca51a36250ae93105c17f6c4
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:10:57 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:

    $ time nc localhost 8776
    real 1m0.063s

    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=900).

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I03b9c5c64f4bd8bca78dfc83199ef17d9a7ea5b7

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/177670

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/177686

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/177670
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=67cda0ccae04471340bcada099d945d90979e64d
Submitter: Jenkins
Branch: stable/kilo

commit 67cda0ccae04471340bcada099d945d90979e64d
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:10:57 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:

    $ time nc localhost 8776
    real 1m0.063s

    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=900).

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I03b9c5c64f4bd8bca78dfc83199ef17d9a7ea5b7
    (cherry picked from commit 3b08644eb9cf4c5aca51a36250ae93105c17f6c4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/164643
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=392dc228034bbd8968f4c65ddfce6343bff938ea
Submitter: Jenkins
Branch: stable/icehouse

commit 392dc228034bbd8968f4c65ddfce6343bff938ea
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 01:37:42 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Icehouse backport note: socket_timeout was dropped, it was introduced
    in 0.14[*] and Icehouse eventlet lower bound is 0.13

    [*] https://github.com/eventlet/eventlet/commit/7d4916f01462de09cb58853d9de2e85777c2ad5b

    Note: The required unit-tests are manually added to the below path,
    as new path for unit-tests is not present in stable/icehouse release.
    nova/tests/test_wsgi.py

    DocImpact:
    Added wsgi_keep_alive option (default=True).

    SecurityImpact

    Conflicts:
            nova/tests/unit/test_wsgi.py

    Closes-Bug: #1361360
    (cherry picked from commit 04d7a724fdf80db51e73f12c5b8c982db9310742)

    Change-Id: I3b14a37edbe4bdc5db31ff4f08f78e78b60077ff

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/144074
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=2be72f42f9debf9c2f186212dd27cb4f860d3399
Submitter: Jenkins
Branch: master

commit 2be72f42f9debf9c2f186212dd27cb4f860d3399
Author: yangxurong <email address hidden>
Date: Mon Mar 30 14:39:17 2015 +1000

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:

    $ time nc localhost 8776
    real 1m0.063s
    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=900).
    SecurityImpact

    Change-Id: I303d87addeed8b103eeb26dbcc48e3acce06ee6a
    Closes-Bug: #1361360

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/162964
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=d569ed9db9dc1941ef74d38f85f8f67a85ff10b0
Submitter: Jenkins
Branch: stable/juno

commit d569ed9db9dc1941ef74d38f85f8f67a85ff10b0
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:39:59 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections. Hence even after
    the response is sent to the client, it doesn't close the client socket
    connection. Because of this problem, the green thread is not released
    back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    DocImpact:
    Added http_keepalive option (default=True).

    Conflicts:
            doc/source/configuring.rst
            etc/glance-api.conf
            glance/common/wsgi.py
            glance/tests/unit/test_opts.py

    SecurityImpact

    Closes-Bug: #1361360
    Change-Id: I93aaca24935a4f3096210233097dd6b8c5440176
    (cherry picked from commit 16a821e00d15520d2f6e940e184bd289b8782620)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/204705

[Travis McPeak (travis-mcpeak)]

Since there is unlikely to ever be a patch for the slowloris side of this, I'm opening an OSSN task.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/204705
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=accb273c7aa369f624e8062137b356dc3e5589cf
Submitter: Jenkins
Branch: master

commit accb273c7aa369f624e8062137b356dc3e5589cf
Author: Valeriy Ponomaryov <email address hidden>
Date: Wed Jul 22 20:30:42 2015 +0300

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after the
    response is sent to the client, it doesn't close the client socket connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    In order to maintain the backward compatibility, setting wsgi_keep_alive
    as True by default. Recommended is set it to False.

    This is port of Cinder change - [1]

    [1] Ic57b2aceb136e8626388cfe4df72b2f47cb0661c

    SecurityImpact
    Closes-Bug: #1361360

    Change-Id: If9241e6f6ba10592a64ca312cb479e8cea929913

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/177686
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=d2d6aba069ea3101dfbc3363689eb6142ffb6d1f
Submitter: Jenkins
Branch: stable/juno

commit d2d6aba069ea3101dfbc3363689eb6142ffb6d1f
Author: abhishekkekane <email address hidden>
Date: Tue Oct 21 04:10:57 2014 -0700

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:

    $ time nc localhost 8776
    real 1m0.063s

    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=900).

    SecurityImpact

    Conflicts:
            keystone/common/config.py
            keystone/common/environment/eventlet_server.py

    NOTE: This is not 1:1 cherry-pick because 'eventlet_server' config
    group is not present in juno.

    Closes-Bug: #1361360
    Change-Id: I03b9c5c64f4bd8bca78dfc83199ef17d9a7ea5b7
    (cherry picked from commit 3b08644eb9cf4c5aca51a36250ae93105c17f6c4)
    (cherry picked from commit 67cda0ccae04471340bcada099d945d90979e64d)

[Travis McPeak (travis-mcpeak)]

There isn't anything specific to OpenStack about this. Since we don't really have any good advice other than limiting connections, I'm closing the OSSN for this. Anybody interested should look into "slowloris attacks".

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/224941

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/224941
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=5f86d05c50e02a895093d4cb6e4285dce4ba93a2
Submitter: Jenkins
Branch: stable/kilo

commit 5f86d05c50e02a895093d4cb6e4285dce4ba93a2
Author: yangxurong <email address hidden>
Date: Mon Mar 30 14:39:17 2015 +1000

    Eventlet green threads not released back to pool

    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.

    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.

    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:

    $ time nc localhost 8776
    real 1m0.063s
    Setting 'client_socket_timeout = 0' means do not timeout.

    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=900).
    SecurityImpact

    Change-Id: I303d87addeed8b103eeb26dbcc48e3acce06ee6a
    Closes-Bug: #1361360
    (cherry picked from commit 2be72f42f9debf9c2f186212dd27cb4f860d3399)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/231989
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=64583e4b839d96e2cbe7dcba7d0f08852b40bcca
Submitter: Jenkins
Branch: master

commit 64583e4b839d96e2cbe7dcba7d0f08852b40bcca
Author: Sergey Reshetnyak <email address hidden>
Date: Wed Oct 7 16:24:35 2015 +0300

    Use api-paste.ini for loading middleware

    Changes:
    * use api-paste.ini config for loading middleware
    * refactoring middlewares to support loading via pastedeploy
    * use debug middleware from oslo_middleware library instead log_exchange
      middleware

    Closes-bug: #1503983
    Closes-bug: #1361360

    Change-Id: I444c1799ef53dbb19a601e51dd95cd8509fb1c0c