DOS : API_RESULT_LIMIT does not work for swift objects

Bug #1724598 reported by Yves-Gwenael Bourhis on 2017-10-18
278
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Dashboard (Horizon)
Undecided
Unassigned
OpenStack Security Advisory
Undecided
Unassigned

Bug Description

A user can make the horizon apache process crash.

Indeed, API_RESULT_LIMIT does not work when `full_listing=False` is passed as argument to swiftclient.client.Connection.get_account or to swiftclient.client.Connection.get_container

Therefore When a customer has a very large amount of objects, the full production server crashes and stops responding.

To reproduce : slowly upload a million small objects on one container, then view this container : The server crashes.

Rob Cresswell (robcresswell) wrote :

Is it intended that the result limit doesn't work when full_listing is passed? It's not entirely clear to me why there are 2 different parameters controlling API result quantity. Its seems like a valid bug, however.

NOTE: all active branches are affected.

Jeremy Stanley (fungi) wrote :

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

Changed in ossa:
status: New → Incomplete
description: updated
Jeremy Stanley (fungi) wrote :

So, just to clarify the claimed exploit scenario: an authenticated normal user can upload an excessive number of objects into a Swift container, then request a full listing of that container through Horizon and effect a denial of service against Horizon itself even if the administrator of the deployment has intentionally set API_RESULT_LIMIT to mitigate such attacks?

@Jeremy Stanley Yes, exactly this scenario.

Even if the administrator of the deployment has intentionally set API_RESULT_LIMIT to mitigate such attacks, this limit does not work and an authenticated normal user can upload an excessive number of objects into a Swift container, and then crash the horizon server while viewing this container.

More info, here is the apache LOG when crash occurs:

    [Mon Oct 16 15:41:28.695638 2017] [core:error] [pid 21470:tid 139843967366912] [client 10.12.0.39:46851] End of script output before headers: horizon.wsgi, referer: https://<Sorry_if_I_hide_the_affected_domain>/project/containers/container/<Sorry_if_I_hide_the_customer_container_name>
    [Mon Oct 16 15:41:55.935525 2017] [:error] [pid 24160:tid 139844023068416] [remote 10.12.0.39:28931] mod_wsgi (pid=24160): Exception occurred processing WSGI script '/opt/console-cw/lib/python2.7/site-packages/openstack_dashboard/wsgi/horizon.wsgi'.
    [Mon Oct 16 15:41:55.937407 2017] [:error] [pid 24160:tid 139844023068416] [remote 10.12.0.39:28931] IOError: failed to write data

The crash lead to DOS and was produced by a legit customer who uploaded a million objects in a container.

We have multiple loadballanced horizon instances, but when the crash occurred, the customer hit "F5" and crashed all loadballanced nodes... And there was no more service (DOS).

The provided patch solved the issue.

We deploy Horizon from source "almost" this way (Updated a bit since):
https://dev.cloudwatt.com/en/blog/deploy-horizon-from-source-with-apache-and-ssl.html

So it's not a distro packaging issue, however packaged versions are theoretically impacted too.

Jeremy Stanley (fungi) wrote :

Thanks for the updated details. We're still waiting for Horizon security reviewers to confirm their bugtask, evaluate the provided patch in comment #1 or suggest an alternate fix, and determine which stable branches will require corresponding backports.

Rob Cresswell (robcresswell) wrote :

The fix is valid; it seems like some overly ambitious caching on Horizons side. I think the patch will cause a regression in behaviour without any further work, however. Since security is more important, I think the only way forward would be to fix and add a release note documenting this, unless someone volunteers their time for further investigation.

Jeremy Stanley (fungi) wrote :

Does the issue apply to all currently supported stable branches, and is the proposed patch backportable to them (what sort of regression are you anticipating)?

Rob Cresswell (robcresswell) wrote :

Its valid for every active branch. My concern is that the Swift interface was written expecting all containers to be available for the client to page through; so without the full_listing parameter, it will only return the first 10000 and stop there.

This seems like an acceptable regression, a user having that many container shouldn't be using a web interface to manage them individually anyway right?

Yves-Gwenael, would you mind attaching a properly formatted patch as explained here: https://security.openstack.org/#how-to-propose-and-review-a-security-patch .

Hello Tristan,

Here is the patch.

Thanks Yves-Gwenael,

@horizon-coresec, so is this an acceptable regression that could be backported down to stable/ocata?

Jeremy Stanley (fungi) wrote :

In keeping with recent OpenStack vulnerability management policy changes, no report should remain under private embargo for more than 90 days. Because this report predates the change in policy, the deadline for public disclosure is being set to 90 days from today. If the report is not resolved within the next 90 days, it will revert to our public workflow as of 2020-05-27. Please see http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012721.html for further details.

description: updated
Jeremy Stanley (fungi) wrote :

It doesn't look like this report has seen any activity since my update two months ago, so consider this a friendly reminder:

The embargo for this report is due to expire one month from today, on May 27, and will be switched public on or shortly after that day if it is not already resolved sooner.

Thanks!

Jeremy Stanley (fungi) on 2020-05-19
description: updated
Jeremy Stanley (fungi) wrote :

The embargo for this report has expired and is now lifted, so it's acceptable to discuss further in public.

description: updated
information type: Private Security → Public Security
To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Other bug subscribers