[FG-VD-16-011] Openstack Logging DoS Vulnerability Notification

[Tristan Cacqueray]

Reported by private mail:

Dear Openstack,

The following information pertains to information discovered by Fortinet's
FortiGuard Labs. It has been determined that a vulnerability exists in
Openstack. To streamline the disclosure process, we have created a
preliminary advisory which you can find below. This upcoming advisory is
purely intended as a reference, and does not contain sensitive information
such as proof of concept code.

As a mature corporation involved in security research, we strive to
responsibly disclose vulnerability information. We will not post an advisory
until we determine it is appropriate to do so in co-ordination with the
vendor unless a resolution cannot be reached. We will not disclose full
proof of concept, only details relevant to the advisory.

We look forward to working closely with you to resolve this issue, and
kindly ask for your co-operation during this time. Please let us know if you
have any further questions, and we will promptly respond to address any
issues.

Type of Vulnerability & Repercussions:
  DoS

Affected Software:
  Ubuntu 14.04.3 with latest repository installed
  # apt-get install software-properties-common
  # add-apt-repository cloud-archive:liberty

Upcoming Advisory Reference:
  http://www.fortiguard.com/advisory/UpcomingAdvisories.html

Credits:
  This vulnerability was discovered by Fortinet's FortiGuard Labs.

Proof of Concept/How to Reproduce:
    The vulnerability exists in Openstack server when dealing with many HTTP GET requests in a single connection. Please check PoC glance_large_request_2.txt. It's a single file containing many long HTTP GET requests. When you run "nc.exe Openstack_IP 9292 < glance_large_request_2.txt" on Windows, Openstack glance server deals with all HTTP requests in a single connection and records them in log file(/var/log/glance/glance-api.log). Run this PoC once, then Openstack generates about 140M log file. To re-produce the DoS issue, you can run the PoC multiple times until Openstack controller node space is exhausted.

  The vulnerability has high impact because it consumes the log space via a single connection. As we know, Apache also records full URLs in log file, but in order to consume its space, hundreds or thousands connections with long URL should be created. This behavour is easily detected and prevented. But in this PoC, it only needs one connection to server.

  The attack doesn't need authentication.

Additional Information:
  Extract from glance_large_request:
GET /v2/images/cirros/file?test=$(python -c 'print "A"*6744')&quot;%22%u0022&lt;<script>alert(5)</script>&gt; HTTP/1.1

[Tristan Cacqueray (tristan-cacqueray)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

I fail to see why a single connection is more difficult to detect compared to multiple connection... This seems to be a class B2 type of bug according to VMT taxonomy ( https://security.openstack.org/vmt-process.html#incident-report-taxonomy ), the bad design being the lack of rate-limiting.

[Stuart McLaren (stuart-mclaren)]

In glance-api.conf:

# If False, server will return the header "Connection: close", If
# True, server will return "Connection: Keep-Alive" in its responses.
# In order to close the client socket connection explicitly after the
# response is sent and read successfully by the client, you simply
# have to set this option to False when you create a wsgi server.
# (boolean value)
#http_keepalive = true

This is passed to eventlet's wsgi server:

 http://eventlet.net/doc/modules/wsgi.html

 keepalive – If set to False, disables keepalives on the server; all connections will be closed after serving one request.

If you set this I think you need more than one connection in order to send all the GET requests. (But I'm not 100% sure why that's considered more secure?)

You can also increase the default log level, eg to WARN. Then the unauthenticated requests will not create a log entry at all.

> the lack of rate-limiting.

I think there's some consensus in the community that rate limiting should happen outside of the openstack service (you probably want 'C' not 'Python').

[Tristan Cacqueray (tristan-cacqueray)]

Thank stuart for the analysis. Then the B2 class still holds, I'm waiting for feedback from reporter before removing the OSSA task and privacy settings.

[chro eric (chrorxu)]

The logging exhaust DoS exist in all openstack server, such as neutron, nava, these server also have risk.

[chro eric (chrorxu)]

more efficent python PoC, and when install openstack, it is almost maybe bring the security risk to system, if have additional server configure parameter can protect from the vulnerability, at least, openstack should post a blog to declare the issue.

[chro eric (chrorxu)]

The openstack logging DoS vulnerability key cause is openstack server keep record every generated log while repeat do GET request with long url, and openstack server does not reset the long connection, untill openstack log is exhausted.

I test apache with repeat python submit GET request, apache response with 1. after about 14000 request,the server reset the connection; 2.apache only record some repeat log, it does not generate every log record in logger file.

[chro eric (chrorxu)]

I think only via configure server http keepalive false to protect from the attack is not enough, because the options have negative impact for openstack server performance after set the keepalive option false, so it seems is not good choice, The apache server protect from the attack is 1: too many connection reset, 2: apache access log not record all repeat log. I think apache deal with keepalive method is better.

[Flavio Percoco (flaper87)]

The way I'm reading Chro's analysis makes me think the issue might not be in Glance itself. Resetting the connection properly is something that, unless I'm misinterpreting the problem, should be taken care of by eventlet.

As far as not repeating log lines, this is something that oslo.log should probably do. At least, provide an API to tell it what lines can/should be filtered when duplicated. Doing this in oslo.log would also help other projects and not just Glance.

Does this make sense?

[Tristan Cacqueray (tristan-cacqueray)]

Chro, the PoC you attached is incomplete. Could you explain why this behavior is more easily detected/prevented when keepalive is disabled ? And about repeating logs, what if the attacker use randomized http payload ?

In short, the vulnerability described here is that unauthenticated and inexpensive calls to the service can generate very large amounts of log data. This type of protection are probably better implemented before the call is even processed by OpenStack's services and this could be documented as a Security Note or as part of the Security Guide.

I've subscribed OSSG-coresec to discuss an eventual document about rate limiting.
If nobody objects, I'd like to close the OSSA task and remove the privacy setting by the end of this week.

[chro eric (chrorxu)]

I think send many connection with large url is have performance impact for openstack server when keepalive is disabled, so easily cause pay attention and take action such timely empty log, so no result in logging DoS.

If url large payload with randomized, I think there is not better means.

[Travis McPeak (travis-mcpeak)]

Just to understand the impact - is there an asymmetric aspect to this? I expect most deployments won't suffer an actual DoS as log rotation should be configured, but this may of course cause legitimate log information to be lost.

I also agree rate limiting shouldn't be Glance's job, this seems like the kind of thing we want a universal solution for (something in Oslo as Flavio suggested).

[Travis McPeak (travis-mcpeak)]

Is there any legitimate reason to allow URLs beyond a reasonable value (for example 4K)? I know most browsers do this already. This seems like a reasonable mitigation for this attack.

[Robert Clark (robert-clark)]

Travis' comment seems appropriate. Needs a Glance person to weigh in.

[Tristan Cacqueray (tristan-cacqueray)]

As long as the client needs to repeatedly send the request, the load seems symmetric and this report will likely not warrant an advisory. Any concern if we remove the privacy settings and close the OSSA task ?