Instance actions history is unbounded

Bug #1175193 reported by Russell Bryant
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Opinion
Wishlist
Unassigned

Bug Description

Grizzly included a new feature to keep a history of actions performed on an instance. The history kept in the database is unbounded. As a result, a malicious user could perform actions in a loop and cause the database to grow without bounds. Some of the quicker actions that could be used to exploit this are pause, unpause, or change password.

Tags: security
Andrew Laski (alaski)
Changed in nova:
assignee: nobody → Andrew Laski (alaski)
Revision history for this message
Thierry Carrez (ttx) wrote :

I'm a bit reluctant to consider linear database growth due to normal usage a "vulnerability", but I guess if there is a quick and free action out there it could be easily exploited... Opinions ?

Revision history for this message
Andrew Laski (alaski) wrote :

Thierry, I would say that there's not a very large risk here. Pause/unpause, start/stop, and change password are fairly cheap actions that could be used, but as you pointed out the db isn't going to just balloon quickly.

Looking at other models in the database they seem to be bounded in some way, typically by quotas or physical resources, while instance_actions and events are not. So, though a small risk, it's still a risk.

Revision history for this message
Michael Still (mikal) wrote : Re: [Bug 1175193] Re: Instance actions history is unbounded

I agree. This feels to me like a bug, but not a security bug. A row in
this table is 1,331 bytes assuming ASCII strings, so a user would have
to change their password nearly 788 times to write a megabyte to the
database. Additionally, they wouldn't get that maximum row size unless
they carefully crafted the request to fill each column.

On Fri, May 3, 2013 at 5:35 AM, Andrew Laski <email address hidden> wrote:
> Thierry, I would say that there's not a very large risk here.
> Pause/unpause, start/stop, and change password are fairly cheap actions
> that could be used, but as you pointed out the db isn't going to just
> balloon quickly.
>
> Looking at other models in the database they seem to be bounded in some
> way, typically by quotas or physical resources, while instance_actions
> and events are not. So, though a small risk, it's still a risk.
>
> ** Attachment added: "patch"
> https://bugs.launchpad.net/nova/+bug/1175193/+attachment/3662969/+files/patch
>
> --
> You received this bug notification because you are a member of OpenStack
> Vulnerability Management team, which is subscribed to the bug report.
> https://bugs.launchpad.net/bugs/1175193
>
> Title:
> Instance actions history is unbounded
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/1175193/+subscriptions

Revision history for this message
Thierry Carrez (ttx) wrote :

I could go either way, but I'm tempted to consider this an automated database cleanup more than a vulnerability fix. Russell ?

Revision history for this message
Michael Still (mikal) wrote :

Yeah, I'd be interested in knowing what Russell thought about
strategies to cleanup this table. A periodic task? If so where would
it sit?

On Fri, May 3, 2013 at 10:45 PM, Thierry Carrez
<email address hidden> wrote:
> I could go either way, but I'm tempted to consider this an automated
> database cleanup more than a vulnerability fix. Russell ?
>
> --
> You received this bug notification because you are a member of OpenStack
> Vulnerability Management team, which is subscribed to the bug report.
> https://bugs.launchpad.net/bugs/1175193
>
> Title:
> Instance actions history is unbounded
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/1175193/+subscriptions

Revision history for this message
Russell Bryant (russellb) wrote :

This seems like a low severity security issue to me. A user is able to consume resources unbounded in a way that does not cost them any money. I think we need to treat that sort of thing as a vulnerability.

Revision history for this message
Andrew Laski (alaski) wrote :

Also note, that I did submit a patch on my earlier comment. That was one direction to go in, the other being a periodic task which could work as well though it still leaves a window of opportunity.

Revision history for this message
Chris Behrens (cbehrens) wrote :

I'm not sure I agree with this being a bug at all. :) One can grow log files without bound, too. Is that a bug?

Revision history for this message
Kurt Seifried (kseifried) wrote :
Download full text (3.9 KiB)

Ok so this is one of those lovely gray areas of informations ecurity, this is obviously a security related issue, but is this a security vulnerability (requiring CVE and full security treatment) or is this a security hardening issue (thus no CVE or full security treatment)?

So first let's look at trust boundaries. Now the end problem here is that the database gets filled up by spurious log entries. What level of access is needed to accomplish this? Luckily in this case it requires authenticated access, this cannot be done as a non authenicated user from what I am told (if this isn't true please let us know ASAP!). As such if an anonymous user was able to remotely flood the database that would be an obvious security problem, they could kill the system remotely which usually falls into the category of security vulnerability.

But in this case it requires authenticated access, and in general users are allowed to consume system resources, however typically we do not want to allow a single user, or even a group of users to consume resources in a way that renders the system unstable or unusable. In this case filling up the databse has a very negative effect on the system.

So this issue falls into the grey area of "is it security or hardening?" and then further falls down the rabbit hole of "how much and how quickly do we allow a user to consume system resources before we call it a security vulnerability?" So clearly the amount of data logged has a significant impact on this. For example if each pause/unpause/etc action generated say a megabyte of logged data this would be extremely easy to use to fill up the database (even a large database would fill quickly at 1meg/entry). In this case it's approximately 1k per entry, which is enough to make it interesting, assuming a few gigabytes are available then it would take some time for the attacker to exploit this enough times to cause a problem. Additionally because it requires authenticated access it is easy to track down who did it and censor them (e.g. tell them not to do it, remove their account, whatever).

So with this in mind I am inclined to classify this as a security hardening issue, and not a security vulnerability. This may of course change if new information comes to light (e.g. unauthenticated access allows exploitation, or the amount of data logged per entry can be increased somehow).

Now a note on fixing this: there are several strategies used in logging systems to deal with attacks like this. The first one is a classic syslog "the last message was repeated X times", this allows multiple identical log entries to be consolidated quickly and easily, reducing the amount of logging space required and can work well with flooding attacks. The second strategy is to rotate and archive the logs once they hit a certain size (log files typically compress really well). Of course neither of these solutions will completely stop resources from being used, but they will reduce the amount of some of the resources used (e.g. disk space) at the cost of CPU time (to rotate/compress logs).

There are other solutions that will prevent resources from being abused, but will compromise the integrity of th...

Read more...

Revision history for this message
Russell Bryant (russellb) wrote :

Thanks for the detailed opinion, Kurt! It looks like everyone agrees that we should treat this as hardening and open this up.

information type: Private Security → Public
tags: added: security
Revision history for this message
Thierry Carrez (ttx) wrote :

Awesome, thx Kurt! I definitely agree that it's a grey area... and that if it were wildly asymmetric or unauthenticated this would be much more "exploitable".

Changed in nova:
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Andrew Laski (alaski) wrote :

Moving to medium since I have not seen any reports of issues with this, nor have I seen this cropping up in a large deployment I help maintain.

Changed in nova:
importance: High → Medium
assignee: Andrew Laski (alaski) → nobody
Revision history for this message
Sean Dague (sdague) wrote :

It's not really clear that anyone is actually seeing this thing, and there seems to be no interest in fixing it. So it's a pretty theoretical bug right now.

Changed in nova:
importance: Medium → Wishlist
status: Confirmed → Opinion
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Patches

Remote bug watches

Bug watches keep track of this bug in other bug trackers.