[OSSA-2021-006] Routes middleware memory leak for nonexistent controllers (CVE-2021-40797)

Bug #1942179 reported by Slawek Kaplonski
264
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Security Advisory
Fix Released
Medium
Unassigned
neutron
Fix Released
Critical
Unassigned

Bug Description

Authorized cloud user may do API requests to neutron to not existing endpoints, like e.g.:

curl -g -i -X GET http://10.120.0.30:9696/v2.0/blabla -H "Accept: application/json" -H "User-Agent: openstacksdk/0.59.0 keystoneauth1/4.3.1 python-requests/2.26.0 CPython/3.6.8" -H "X-Auth-Token: $token"

and each such request will increase memory consumption of the neutron-api worker process.

What I did was:

* start neutron server with just one api worker (easier to calculate memory consumption but it would be the same leak in case of more workers too). Memory consumption was:

sudo pmap 212436 | tail -n 1
 total 183736K

* now run command like:

$ i=1; while [ $i -lt 2000 ]; do echo "Request $i"; curl -g -i -X GET http://10.120.0.30:9696/v2.0/blabla -H "Accept: application/json" -H "User-Agent: openstacksdk/0.59.0 keystoneauth1/4.3.1 python-requests/2.26.0 CPython/3.6.8" -H "X-Auth-Token: $token" 2>1 >/dev/null; i=$(( i+1 )); sleep 0.01; done

* check memory consumption of the same api worker now:

sudo pmap 212436 | tail -n 1
 total 457896K

CVE References

Revision history for this message
Jeremy Stanley (fungi) wrote :

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

description: updated
Changed in ossa:
status: New → Incomplete
Revision history for this message
Jeremy Stanley (fungi) wrote :

This seems like it could represent a fairly quick means of memory consumption on servers where the API worker is running. I guess the impact is service degredation (as it starts to thrash swapspace) and eventually denial of service for the API once the kernel OOM killer gets involved and takes down the Python process?

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

@fungi - yes, it looks like that for me too. We are still investigating this issue with Rodolfo

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

After long investigation with Rodolfo on that issue, there is finally proposed fix/workaround for that memory leak in Neutron.

Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hello:

Slawek found the place where Neutron was leaking memory. This is in the "routes.middleware.RoutesMiddleware" object. When using "singleton=True" (default value), a routes._RequestConfig() is always created [1]. This object has a thread safe variable to store the context information for each request.

The problem is that we use evenlet and we monkey patch the "threading" module (among many other ones) at the beginning of the Neutron server. The equivalent to "threading.local()" [1] is "eventlet.corolocal.local()". This object stores the greenthread-safe parameters in a WeakKeyDictionary() located in "eventlet.corolocal.local()._local__greens".

The problem we see is that this dictionary is not released. The patch provided in the next comment fixes this issue (this is just for testing, this patch could not land "routes" repository).

However, I think this is over engineering. The patch proposed by Slawek, using "singleton-=False" is a valid solution when using greenthreads.

Regards.

[1]https://github.com/bbangert/routes/blob/c4d5a5fb693ce8dc7cf5dbc591861acfc49d5c23/routes/middleware.py#L98
[2]https://github.com/bbangert/routes/blob/c4d5a5fb693ce8dc7cf5dbc591861acfc49d5c23/routes/__init__.py#L12

Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

diff --git a/routes/middleware.py b/routes/middleware.py
index 649aa25..d06fd70 100644
--- a/routes/middleware.py
+++ b/routes/middleware.py
@@ -3,10 +3,12 @@ import re
 import logging

 from webob import Request
+from eventlet import greenthread

 from routes.base import request_config
 from routes.util import URLGenerator

+
 log = logging.getLogger('routes.middleware')

@@ -155,6 +157,9 @@ class RoutesMiddleware(object):
         # Wrapped in try as in rare cases the attribute will be gone already
         try:
             del self.mapper.environ
+ if self.singleton:
+ cur = greenthread.getcurrent()
+ del config._RequestConfig__shared_state._local__greens[cur]
         except AttributeError:
             pass
         return response

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

I'm not sure if we should make it public now, or should we treat it as private for now? I can propose fixes to stable branches here, that's not a problem at all but I think we should move on with that issue into some direction :)

Revision history for this message
Jeremy Stanley (fungi) wrote :

This risk is only exploitable by authenticated users, right? With the worst impact from it being service degradation/denial of service, operators should be able to fairly quickly identify and revoke the malicious account. Given that, I think continuing to work this one in public would be a reasonable choice.

Revision history for this message
Lajos Katona (lajos-katona) wrote :

I agree, let's make and fix it publicly.

Revision history for this message
Jeremy Stanley (fungi) wrote :

I checked with Rodoldo via IRC and he's okay with switching to our public security workflow for this, so I'll do that now.

description: updated
information type: Private Security → Public Security
Changed in ossa:
status: Incomplete → Confirmed
importance: Undecided → Medium
assignee: nobody → Jeremy Stanley (fungi)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/807335

Changed in neutron:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/807335
Committed: https://opendev.org/openstack/neutron/commit/e610a5eb9e71aa2549fb11e2139370d227787da2
Submitter: "Zuul (22348)"
Branch: master

commit e610a5eb9e71aa2549fb11e2139370d227787da2
Author: Slawek Kaplonski <email address hidden>
Date: Fri Sep 3 16:04:02 2021 +0200

    Don't use singleton in routes.middleware.RoutesMiddleware

    It seems that using default singleton=True in the
    routes.middleware.RoutesMiddleware which is leading to use thread-local
    RequestConfig singleton object is not working well with eventlet
    monkeypatching of threading library which we are doing in Neutron.
    As a result it leaks memory in neutron-api workers every time when API
    request to not existing API endpoint is made by user.

    To avoid that memory leak, let's use singletone=False in that
    RoutesMiddleware object, at least until problem with thread-local
    singleton and eventlet monkey patching will be solved.

    Closes-Bug: #1942179
    Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/807632

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/807633

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/807634

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/807635

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/neutron/+/807636

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/c/openstack/neutron/+/807637

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/c/openstack/neutron/+/807638

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/807632
Committed: https://opendev.org/openstack/neutron/commit/c5e86f4f8f7e36233f383323e24e72dbe28efc04
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit c5e86f4f8f7e36233f383323e24e72dbe28efc04
Author: Slawek Kaplonski <email address hidden>
Date: Fri Sep 3 16:04:02 2021 +0200

    Don't use singleton in routes.middleware.RoutesMiddleware

    It seems that using default singleton=True in the
    routes.middleware.RoutesMiddleware which is leading to use thread-local
    RequestConfig singleton object is not working well with eventlet
    monkeypatching of threading library which we are doing in Neutron.
    As a result it leaks memory in neutron-api workers every time when API
    request to not existing API endpoint is made by user.

    To avoid that memory leak, let's use singletone=False in that
    RoutesMiddleware object, at least until problem with thread-local
    singleton and eventlet monkey patching will be solved.

    Closes-Bug: #1942179
    Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b
    (cherry picked from commit e610a5eb9e71aa2549fb11e2139370d227787da2)

tags: added: in-stable-wallaby
tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/807633
Committed: https://opendev.org/openstack/neutron/commit/d961731a73128125ebf03fae52b4b34f3e7abf27
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit d961731a73128125ebf03fae52b4b34f3e7abf27
Author: Slawek Kaplonski <email address hidden>
Date: Fri Sep 3 16:04:02 2021 +0200

    Don't use singleton in routes.middleware.RoutesMiddleware

    It seems that using default singleton=True in the
    routes.middleware.RoutesMiddleware which is leading to use thread-local
    RequestConfig singleton object is not working well with eventlet
    monkeypatching of threading library which we are doing in Neutron.
    As a result it leaks memory in neutron-api workers every time when API
    request to not existing API endpoint is made by user.

    To avoid that memory leak, let's use singletone=False in that
    RoutesMiddleware object, at least until problem with thread-local
    singleton and eventlet monkey patching will be solved.

    Closes-Bug: #1942179
    Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b
    (cherry picked from commit e610a5eb9e71aa2549fb11e2139370d227787da2)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/807634
Committed: https://opendev.org/openstack/neutron/commit/2f642e23e085aa0bd5a23b4b169b436da9ad455c
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 2f642e23e085aa0bd5a23b4b169b436da9ad455c
Author: Slawek Kaplonski <email address hidden>
Date: Fri Sep 3 16:04:02 2021 +0200

    Don't use singleton in routes.middleware.RoutesMiddleware

    It seems that using default singleton=True in the
    routes.middleware.RoutesMiddleware which is leading to use thread-local
    RequestConfig singleton object is not working well with eventlet
    monkeypatching of threading library which we are doing in Neutron.
    As a result it leaks memory in neutron-api workers every time when API
    request to not existing API endpoint is made by user.

    To avoid that memory leak, let's use singletone=False in that
    RoutesMiddleware object, at least until problem with thread-local
    singleton and eventlet monkey patching will be solved.

    Closes-Bug: #1942179
    Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b
    (cherry picked from commit e610a5eb9e71aa2549fb11e2139370d227787da2)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/807637
Committed: https://opendev.org/openstack/neutron/commit/6d773bb8b84b496a71e4ee6c879af967aa031e98
Submitter: "Zuul (22348)"
Branch: stable/rocky

commit 6d773bb8b84b496a71e4ee6c879af967aa031e98
Author: Slawek Kaplonski <email address hidden>
Date: Fri Sep 3 16:04:02 2021 +0200

    Don't use singleton in routes.middleware.RoutesMiddleware

    It seems that using default singleton=True in the
    routes.middleware.RoutesMiddleware which is leading to use thread-local
    RequestConfig singleton object is not working well with eventlet
    monkeypatching of threading library which we are doing in Neutron.
    As a result it leaks memory in neutron-api workers every time when API
    request to not existing API endpoint is made by user.

    To avoid that memory leak, let's use singletone=False in that
    RoutesMiddleware object, at least until problem with thread-local
    singleton and eventlet monkey patching will be solved.

    Closes-Bug: #1942179
    Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b
    (cherry picked from commit e610a5eb9e71aa2549fb11e2139370d227787da2)

tags: added: in-stable-rocky
Revision history for this message
Jeremy Stanley (fungi) wrote : Re: neutron api worker leaks memory when processing requests to not existing controllers

As fixes have merged to all of Neutron's maintained stable branches, we can (and probably should) issue an advisory for this defect. I propose we use the following impact description to request a CVE assignment from MITRE, but please let me know if this isn't entirely accurate so I can correct it:

Title: Routes middleware memory leak for nonexistent controllers
Reporter: Slawek Kaplonski (Red Hat)
Products: Neutron
Affects: <16.4.1, >=17.0.0 <17.2.1, >=18.0.0 <18.1.1

Description:
Slawek Kaplonski with Red Hat reported a vulnerability in Neutron's routes middleware. By making API requests involving nonexistent controllers, an authenticated user may cause the API worker to consume increasing amounts of memory, resulting in API performance degradation or denial of service. All Neutron deployments are affected.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/queens)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/807638
Committed: https://opendev.org/openstack/neutron/commit/dfbb9bcb5dae282c575b33171d691a5fca946beb
Submitter: "Zuul (22348)"
Branch: stable/queens

commit dfbb9bcb5dae282c575b33171d691a5fca946beb
Author: Slawek Kaplonski <email address hidden>
Date: Fri Sep 3 16:04:02 2021 +0200

    Don't use singleton in routes.middleware.RoutesMiddleware

    It seems that using default singleton=True in the
    routes.middleware.RoutesMiddleware which is leading to use thread-local
    RequestConfig singleton object is not working well with eventlet
    monkeypatching of threading library which we are doing in Neutron.
    As a result it leaks memory in neutron-api workers every time when API
    request to not existing API endpoint is made by user.

    To avoid that memory leak, let's use singletone=False in that
    RoutesMiddleware object, at least until problem with thread-local
    singleton and eventlet monkey patching will be solved.

    Closes-Bug: #1942179
    Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b
    (cherry picked from commit e610a5eb9e71aa2549fb11e2139370d227787da2)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/train)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/807635
Committed: https://opendev.org/openstack/neutron/commit/1977a8d9ba06e2361070f8894825ca388635c043
Submitter: "Zuul (22348)"
Branch: stable/train

commit 1977a8d9ba06e2361070f8894825ca388635c043
Author: Slawek Kaplonski <email address hidden>
Date: Fri Sep 3 16:04:02 2021 +0200

    Don't use singleton in routes.middleware.RoutesMiddleware

    It seems that using default singleton=True in the
    routes.middleware.RoutesMiddleware which is leading to use thread-local
    RequestConfig singleton object is not working well with eventlet
    monkeypatching of threading library which we are doing in Neutron.
    As a result it leaks memory in neutron-api workers every time when API
    request to not existing API endpoint is made by user.

    To avoid that memory leak, let's use singletone=False in that
    RoutesMiddleware object, at least until problem with thread-local
    singleton and eventlet monkey patching will be solved.

    Closes-Bug: #1942179
    Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b
    (cherry picked from commit e610a5eb9e71aa2549fb11e2139370d227787da2)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/stein)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/807636
Committed: https://opendev.org/openstack/neutron/commit/b08715862e360c45cc0abd8341889e37858207bb
Submitter: "Zuul (22348)"
Branch: stable/stein

commit b08715862e360c45cc0abd8341889e37858207bb
Author: Slawek Kaplonski <email address hidden>
Date: Fri Sep 3 16:04:02 2021 +0200

    Don't use singleton in routes.middleware.RoutesMiddleware

    It seems that using default singleton=True in the
    routes.middleware.RoutesMiddleware which is leading to use thread-local
    RequestConfig singleton object is not working well with eventlet
    monkeypatching of threading library which we are doing in Neutron.
    As a result it leaks memory in neutron-api workers every time when API
    request to not existing API endpoint is made by user.

    To avoid that memory leak, let's use singletone=False in that
    RoutesMiddleware object, at least until problem with thread-local
    singleton and eventlet monkey patching will be solved.

    Closes-Bug: #1942179
    Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b
    (cherry picked from commit e610a5eb9e71aa2549fb11e2139370d227787da2)

tags: added: in-stable-stein
Revision history for this message
Jeremy Stanley (fungi) wrote : Re: neutron api worker leaks memory when processing requests to not existing controllers

I checked with Slawek and Rodolfo in IRC, and they said they were good with the proposed impact description from comment #24, so I'm using that to submit a CVE request to MITRE now.

Jeremy Stanley (fungi)
summary: - neutron api worker leaks memory when processing requests to not existing
- controllers
+ Routes middleware memory leak for nonexistent controllers
+ (CVE-2021-40797)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ossa (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ossa/+/807942

Changed in ossa:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ossa (master)

Reviewed: https://review.opendev.org/c/openstack/ossa/+/807942
Committed: https://opendev.org/openstack/ossa/commit/4f5d81b664a81ad7ba4856fbabe1d3f1f12a14e8
Submitter: "Zuul (22348)"
Branch: master

commit 4f5d81b664a81ad7ba4856fbabe1d3f1f12a14e8
Author: Jeremy Stanley <email address hidden>
Date: Wed Sep 8 20:15:03 2021 +0000

    Add OSSA-2021-006 (CVE-2021-40797)

    Change-Id: Ie61b5ffbec78e8c90e5ad773c9479f0d7ae1b932
    Closes-Bug: #1942179

Changed in ossa:
status: In Progress → Fix Released
Jeremy Stanley (fungi)
summary: - Routes middleware memory leak for nonexistent controllers
- (CVE-2021-40797)
+ [OSSA-2021-006] Routes middleware memory leak for nonexistent
+ controllers (CVE-2021-40797)
Revision history for this message
Jeremy Stanley (fungi) wrote :

Notifications have been distributed to the usual mailing lists.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 16.4.1

This issue was fixed in the openstack/neutron 16.4.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 17.2.1

This issue was fixed in the openstack/neutron 17.2.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.1.1

This issue was fixed in the openstack/neutron 18.1.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 19.0.0.0rc1

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.

tags: added: neutron-proactive-backport-potential
Revision history for this message
test (bingdiancn) wrote (last edit ):

sorry,i can't reproduce the bug in the neutron 17.2.0。Could you please give me any help about it?

test (bingdiancn)
summary: [OSSA-2021-006] Routes middleware memory leak for nonexistent
- controllers (CVE-2021-40797)
+ controllers (CVE-2021-40797)dsd
summary: [OSSA-2021-006] Routes middleware memory leak for nonexistent
- controllers (CVE-2021-40797)dsd
+ controllers (CVE-2021-40797)
Changed in ossa:
assignee: Jeremy Stanley (fungi) → nobody
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hello:

Slawek provided a method to test it. Actually we tested it, checking the Neutron server memory consumption increased linearly every time we made a HTTP request.

Deploy OpenStack Neutron (without the fix, now present in all versions) and execute this (changing your server IP address):
$ i=1; while [ $i -lt 2000 ]; do echo "Request $i"; curl -g -i -X GET http://10.120.0.30:9696/v2.0/blabla -H "Accept: application/json" -H "User-Agent: openstacksdk/0.59.0 keystoneauth1/4.3.1 python-requests/2.26.0 CPython/3.6.8" -H "X-Auth-Token: $token" 2>1 >/dev/null; i=$(( i+1 )); sleep 0.01; done

You'll see how the Neutron server memory grows continuously.

Regards

Revision history for this message
test (bingdiancn) wrote :

I deployed it by Kolla-ansible and tested it by the method.But the memory keep unchanged. The process as attached pictures below。

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

Maybe You already have fix for that issue included? Did You check that?

tags: removed: neutron-proactive-backport-potential
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Bug attachments