Note: Keystone Request / Header Size Limits Required to Avoid DoS

[Robert Clark]

Requests with large POST body can crash Keystone or underlying services.
-----

### Summary ###
Concurrent Keystone POST requests with large body messages are held in memory without filtering or rate limiting, this can lead to resource exhaustion on the Keystone server.

### Affected Services / Software ###
Keystone, Databases

### Discussion ###

### Recommended Actions ###

### Contacts / References ###

[Robert Clark (robert-clark)]

https://bugs.launchpad.net/keystone/+bug/1098177

[Thierry Carrez (ttx)]

Note that Keystone in Grizzly is protected against that through the sizelimit middleware that was recently introduced. One option is to backport it to Folsom and attach the patch to the advisory... or just advise people to use proper filtering anyway.

[Robert Clark (robert-clark)]

Difficult to say which way to go other than the rather obvious 'do both'.

Large scale deployments will no doubt have the ability to rate limit at their border LB's smaller setups may need a middleware solution but they're less likely to have an attacker profile that would necessitate this in the first place.

My understanding was that from an architectural point of view the desire was to avoid putting extra security controls into OpenStack, instead looking for other compensating controls to be put in place. I'm reasonably happy either way, just wondering if this signifies a shift in policy.

My feeling is that Folsom users may be looking for this to be backported. That said the OSSG are happy to write up a few compensating controls (configure nginx/apache this way, setup your LB for X etc) and can reference a backported patch if one exists.

[Kurt Seifried (kseifried)]

Can you say roughly how how big the POST requests are (and how many are needed) to cause memory consumption of say 1 gig? 4 gigs? I'm wondering if this needs a CVE or if it's a case of" way to much stuff makes system slow. block insane POST upload sizes". Is it linear (e.g. a 1 meg post takes one second to process, a 600 megabyte post takes 10 minutes? or is it quadratic, exponential roughly, or? (e.g. a 100 meg request takes 100 minutes? 100 hours?). Thanks, I'm just trying to figure out if this needs a CVE or not, and if it can be blocked by enforcing a maximum post size.

[Thierry Carrez (ttx)]

Opened original bug @ https://bugs.launchpad.net/keystone/+bug/1098177

Next time, you can just open an additional "OSSN" task on the orginal bug so that the comments are on the same page :)

@Kurt: haven't tested myself, but original bug seems to point to a
echo 100M > /sys/fs/cgroup/memory/keystone/memory.limit_in_bytes (max memory is 100M for the keystone process)

2, echo $$ /sys/fs/cgroup/memory/keystone/tasks && keystone-all

3 send http post request with data large than 100M

[Thierry Carrez (ttx)]

Oops. I meant "seems to point to a rather linear memory consumption problem."

[Kurt Seifried (kseifried)]

Yeah if it's linear then it's not really a DoS. It's more of a "don't do that/allow that". Typically DoS is more for "one weirdly formatted packet eats all your cpu/ram". Considering security hardening for now and not a security vulnerability unless someone has new info/etc.

[Kurt Seifried (kseifried)]

Oh also can you confirm it's the OOM killing it? Thanks.

[Robert Clark (robert-clark)]

Requests with large POST body can crash Pre-Grizzly Keystone or underlying services.
-----

### Summary ###
Concurrent Keystone POST requests with large body messages are held in memory without filtering or rate limiting, this can lead to resource exhaustion on the Keystone server.

### Affected Services / Software ###
Keystone, Databases

### Discussion ###
Keystone stores POST messages in memory before validation, concurrent submission of multiple large POST messages can cause the Keystone process to be killed due to memory exhaustion, resulting in a remote Denial of Service.

In many cases Keystone will be deployed behind a load-balancer or proxy that can rate limit POST messages inbound to Keystone. Grizzly is protected against that through the sizelimit middleware.

### Recommended Actions ###
If you are in a situation where Keystone is directly exposed to incoming POST messages and not protected by the sizelimit middleware there are a number of load-balancing/proxy options, we suggest you consider one of the following:

Nginx [Need info, one line with link is fine]
Apache [Need info, one line with link is fine]
[Other Options]

### Contacts / References ###

[Jason Hullinger (jason-hullinger)]

Seems about right

On Apr 4, 2013, at 10:45 AM, "Robert Clark" <email address hidden> wrote:

> Requests with large POST body can crash Pre-Grizzly Keystone or underlying services.
> -----
>
> ### Summary ###
> Concurrent Keystone POST requests with large body messages are held in memory without filtering or rate limiting, this can lead to resource exhaustion on the Keystone server.
>
> ### Affected Services / Software ###
> Keystone, Databases
>
> ### Discussion ###
> Keystone stores POST messages in memory before validation, concurrent submission of multiple large POST messages can cause the Keystone process to be killed due to memory exhaustion, resulting in a remote Denial of Service.
>
> In many cases Keystone will be deployed behind a load-balancer or proxy
> that can rate limit POST messages inbound to Keystone. Grizzly is
> protected against that through the sizelimit middleware.
>
> ### Recommended Actions ###
> If you are in a situation where Keystone is directly exposed to incoming POST messages and not protected by the sizelimit middleware there are a number of load-balancing/proxy options, we suggest you consider one of the following:
>
> Nginx [Need info, one line with link is fine]
> Apache [Need info, one line with link is fine]
> [Other Options]
>
> ### Contacts / References ###
>
> --
> You received this bug notification because you are a member of OpenStack
> Security Group, which is subscribed to a duplicate bug report (1098177).
> https://bugs.launchpad.net/bugs/1155566
>
> Title:
> Note: Keystone Request / Header Size Limits Required to Avoid DoS
>
> Status in OpenStack Security Notes:
> Confirmed
>
> Bug description:
> Requests with large POST body can crash Keystone or underlying services.
> -----
>
> ### Summary ###
> Concurrent Keystone POST requests with large body messages are held in memory without filtering or rate limiting, this can lead to resource exhaustion on the Keystone server.
>
> ### Affected Services / Software ###
> Keystone, Databases
>
> ### Discussion ###
>
> ### Recommended Actions ###
>
> ### Contacts / References ###
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/osn/+bug/1155566/+subscriptions

[Bryan D. Payne (bdpayne)]

Nginx
http://wiki.nginx.org/HttpCoreModule#client_max_body_size

Apache
http://httpd.apache.org/docs/2.4/mod/core.html#limitrequestbody

I originally suggested that we provide info for haproxy as well, but further investigation leads me to believe that it is best to use one of the above.

[Robert Clark (robert-clark)]

Requests with large POST body can crash Pre-Grizzly Keystone or underlying services.
-----

### Summary ###
Concurrent Keystone POST requests with large body messages are held in memory without filtering or rate limiting, this can lead to resource exhaustion on the Keystone server.

### Affected Services / Software ###
Keystone, Databases

### Discussion ###
Keystone stores POST messages in memory before validation, concurrent submission of multiple large POST messages can cause the Keystone process to be killed due to memory exhaustion, resulting in a remote Denial of Service.

In many cases Keystone will be deployed behind a load-balancer or proxy that can rate limit POST messages inbound to Keystone. Grizzly is protected against that through the sizelimit middleware.

### Recommended Actions ###
If you are in a situation where Keystone is directly exposed to incoming POST messages and not protected by the sizelimit middleware there are a number of load-balancing/proxy options, we suggest you consider one of the following:

Nginx: Open-source, high-performance HTTP server and reverse proxy.
Nginx Config: http://wiki.nginx.org/HttpCoreModule#client_max_body_size

Apache: HTTP Server Project
Apache Config: http://httpd.apache.org/docs/2.4/mod/core.html#limitrequestbody
### Contacts / References ###
Original LaunchPad Bug : https://bugs.launchpad.net/keystone/+bug/1098177
OpenStack Security ML : <email address hidden>
OpenStack Security Group : https://launchpad.net/~openstack-ossg

[Robert Clark (robert-clark)]

Requests with large POST body can crash Pre-Grizzly Keystone or underlying services.
-----

### Summary ###
Concurrent Keystone POST requests with large body messages are held in memory without filtering or rate limiting, this can lead to resource exhaustion on the Keystone server.

### Affected Services / Software ###
Keystone, Databases

### Discussion ###
Keystone stores POST messages in memory before validation, concurrent submission of multiple large POST messages can cause the Keystone process to be killed due to memory exhaustion, resulting in a remote Denial of Service.

In many cases Keystone will be deployed behind a load-balancer or proxy that can rate limit POST messages inbound to Keystone. Grizzly is protected against that through the sizelimit middleware.

### Recommended Actions ###
If you are in a situation where Keystone is directly exposed to incoming POST messages and not protected by the sizelimit middleware there are a number of load-balancing/proxy options, we suggest you consider one of the following:

Nginx: Open-source, high-performance HTTP server and reverse proxy.
Nginx Config: http://wiki.nginx.org/HttpCoreModule#client_max_body_size

Apache: HTTP Server Project
Apache Config: http://httpd.apache.org/docs/2.4/mod/core.html#limitrequestbody

### Contacts / References ###
Original LaunchPad Bug : https://bugs.launchpad.net/keystone/+bug/1098177
OpenStack Security ML : <email address hidden>
OpenStack Security Group : https://launchpad.net/~openstack-ossg

[Robert Clark (robert-clark)]

HTTP POST limiting advised to avoid Essex/Folsom Keystone DoS
---

### Summary ###
Concurrent Keystone POST requests with large body messages are held in memory without filtering or rate limiting, this can lead to resource exhaustion on the Keystone server.

### Affected Services / Software ###
Keystone, Databases

### Discussion ###
Keystone stores POST messages in memory before validation, concurrent submission of multiple large POST messages can cause the Keystone process to be killed due to memory exhaustion, resulting in a remote Denial of Service.

In many cases Keystone will be deployed behind a load-balancer or proxy that can rate limit POST messages inbound to Keystone. Grizzly is protected against that through the sizelimit middleware.

### Recommended Actions ###
If you are in a situation where Keystone is directly exposed to incoming POST messages and not protected by the sizelimit middleware there are a number of load-balancing/proxy options, we suggest you consider one of the following:

Nginx: Open-source, high-performance HTTP server and reverse proxy.
Nginx Config: http://wiki.nginx.org/HttpCoreModule#client_max_body_size

Apache: HTTP Server Project
Apache Config: http://httpd.apache.org/docs/2.4/mod/core.html#limitrequestbody

### Contacts / References ###
This OSSN Bug: https://bugs.launchpad.net/ossn/+bug/1155566
Original LaunchPad Bug : https://bugs.launchpad.net/keystone/+bug/1098177
OpenStack Security ML : <email address hidden>
OpenStack Security Group : https://launchpad.net/~openstack-ossg

[Kurt Seifried (kseifried)]

I was wrong in comment #7, I misunderstood this issue slightly. Basically an attacker can use this to cause remote crashes, so DoS. Please use CVE-2013-2014 for this issue. Adding the CVE to the actual bug report.

[Robert Clark (robert-clark)]

Published to OpenStack and OpenStack-Dev on 23rd April