No limit in length of "image description " parameter results in DOS attack

[Jeremy Stanley]

Members of the VMT received the following report by E-mail:

Dear VMT

I have identified a vulnerability in the OpenStack system and would like to report it to the OpenStack Vulnerability Management Team.:

While using OpenStack, I attempted to edit an image and noticed that the web frontend interface restricts the character limit of the image description to 256 characters. However, when I intercepted the request using Burp Suite and filled the description with more than 256 characters, the server did not reject the request. Therefore, the frontend restriction is not enforced on the backend. This vulnerability could potentially allow an attacker to quickly fill up storage, leading to a denial-of-service (DoS) attack. Additionally, I want to mention that after discovering this issue, I tested other modules for similar problems and have not found any. Their frontend and backend implementations appear to be consistent.

Attached is a video demonstrating the reproduction of the vulnerability.

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

[Jeremy Stanley (fungi)]

I've also subscribed lujiefsi, the original reporter.

[Abhishek Kekane (abhishek-kekane)]

I am afraid that this is issue from horizon side since we do not accept/store any description for image in our database.

[Jeremy Stanley (fungi)]

Thanks for clarifying, I suppose the "backend" in this case is the Horizon API rather than the Glance API. I'll add the appropriate component.

[Brian Rosmaita (brian-rosmaita)]

Looking at horizon, I think it passes the 'description' to glance as an additional image property:

https://opendev.org/openstack/horizon/src/commit/c528bfe84f25eef17d808d635ff17cf95e7a8136/openstack_dashboard/api/glance.py#L410-L426

@lujiefsi: have you tried to do this directly to the Image API without involving horizon?

Glance has a system-wide 'image_property_quota' setting (default: 128) that limits the number of properties that can be added to a particular image. Additionally, there is a per-user 'image_count_total' that controls how many images can be held by a project. So there are some levers in place to control how much garbage someone can stuff into the DB. Since an unauthorized user can't create/update and image, and a user must be in some project, I don't know how much of a DOS you can do with this vector (though the default image property quota may be a bit high).

[lujiefsi (lujiefsi)]

Hi:i am trying to do this directly to the Image API without involving horizon.
i will post python script later.

[lujiefsi (lujiefsi)]

@Brian Rosmaita,

I have tried to populate all the properties using the Image REST API without involving Horizon. Each property can be up to 64KB in size. By default, each image can have a maximum of 128 properties, allowing for a total of 8MB of property data per image. With the default configuration, each user can have 128 images, allowing for nearly 1GB of junk data to be filled per user. If the default configuration is modified to allow for larger images or a user has multiple projects, there is a possibility for even more junk data to be filled by a single user.

Furthermore, I have observed that accessing image details becomes very slow when we populate the properties up to the 8MB limit. I simulated 50 threads on a single machine, using the same user's cookies to access image details. It was found that requests from other users were significantly delayed, especially when trying to access the images through Horizon. It was almost impossible for other users to access the images successfully.

[lujiefsi (lujiefsi)]

This is another file that is used to fork 50 threads to access image details.

[lujiefsi (lujiefsi)]

I have checked some other properties, such as name, tags, and owner. All of them have been verified and are limited to 255 characters.

I think this issue can be resolved by limiting the length of additional properties to 255. To prevent any users from needing more than 255, we can offer a configuration option for administrators to decide.

[Brian Rosmaita (brian-rosmaita)]

@lujiefsi: Thanks for investigating and reporting back.

There was actually a discussion on the mailing list a few months back about limiting the length of image additional properties:

https://<email address hidden>/thread/B7UET4JKHQU5SHH44KLSKHFBMFN3ZZYV/#B7UET4JKHQU5SHH44KLSKHFBMFN3ZZYV

A former Glance PTL was against the proposal because it would be a backward-incompatible change, but I don't think the performance issue was discussed in that thread. Also, the original email in the thread asked for anyone who had a use case that would be affected by the change, and no one responded to that question.

Since glance, nova, cinder (and possibly, ironic) all store image metadata, this impacts all 3 (or 4) APIs. Nova is sort of unilaterally putting a 255 char (or byte -- I'm not sure) limit on image property values. The cinder change that prompted the above email thread is enforcing a 65535 byte limit via JSON schema validation.

I think the slowdown issue is something we should take seriously, but I don't know if 255 chars is too restrictive if, for example, people are storing a string of JSON in an image property that's used by some kind of deployment tool. We could maybe restrict to 512 or 1024 chars or something like that, and wouldn't have to migrate from a text field to varchar (I think the slowdown is from stuffing all those chars into JSON when generating the response, not from the way the data is stored in the DB).

Because this impacts 3 or 4 APIs, I don't think we want to go the config option route; I think we need to just decide what is a suitable length and enforce it. (But that's just my opinion.)

[Brian Rosmaita (brian-rosmaita)]

@fungi: my opinion is that the above-referenced email discussion + the existing ability to restrict the number of properties per image and images per project imply that this issue can be worked in public, but I may be missing something.

[Jeremy Stanley (fungi)]

Thanks to the both of you for all the additional details. I agree, we should switch this to public for now in order to allow for broader discussion. I'll also switch the Glance bugtask from invalid back to new, since this is in fact about image properties as far as the backend API is concerned.

[lujiefsi (lujiefsi)]

@Brian Rosmaita

"I think the slowdown is from stuffing all those chars into JSON when generating the response, not from the way the data is stored in the DB"

About this, you are right. I wrote a script to test reading data directly from DB and found that there was not much latency. However, when reading data from Horizon, the CPU usage of the backend Apache2 server reached 100%.

“but I don't think the performance issue was discussed in that thread.....,I think the slowdown issue is something we should take seriously,”

@fungi and @brian-rosmaita

Should we create a thread on Horizon to discuss this issue? In my opinion, Horizon does not need to read and return all the additional properties to the user? Horizon can choose to only return the additional properties (i.e., descriptions) that Horizon is capable of modifying, thus Horizon will not be affected by the length of properties .

 In general, I believe Horizon should protect itself instead of relying on Glance.

[lujiefsi (lujiefsi)]

I found some code https://github.com/openstack/horizon/blob/42915b6a78bb35b030371892665ce7ab30c6def0/openstack_dashboard/api/glance.py#L92, maybe related to filter additional properties. But i don't think it can meet our needs.