Enabling soft-deletes opens a DOS on compute hosts

[Matthew Booth]

If the user sets reclaim_instance_interval to anything other than 0, then when a user requests an instance delete, it will instead be soft deleted. Soft delete explicitly releases the user's quota, but does not release the instance's resources until period task _reclaim_queued_deletes runs with a period of reclaim_instance_interval seconds.

A malicious authenticated user can repeatedly create and delete instances without limit, which will consume resources on the host without consuming their quota. If done quickly enough, this will exhaust host resources.

I'm not entirely sure what to suggest in remediation, as this seems to be a deliberate design. The most obvious fix would be to not release quota until the instance is reaped, but that would be a significant change in behaviour.

This is very similar to https://bugs.launchpad.net/bugs/cve/2015-3280 , except that we do it deliberately.

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Jeremy Stanley (fungi)]

This is a grey area we get into with denial of service "vulnerabilities" related to unbounded (or loosely bounded) resource consumption. I strongly suspect the "fix" for this will happen in documentation (noting that the soft delete option allows users to temporarily exhaust resources), a new configurable feature (such as a secondary soft quota buffer), or a behavior change (perhaps altering/removing the soft delete functionality).

Input from Nova security reviewers is appreciated, but if the solution space is limited to one or more of the above then this report is likely not going to have an associated advisory.

[Michael Still (mikal)]

This is definitely be design. That said I agree there is a DoS possible here.

It seems to me there is a tweak we could make where if a hypervisor becomes space constrained we delete earlier than the configured time, but that might be a surprise for administrators using a "fill first" scheduling methodology.

[Tristan Cacqueray (tristan-cacqueray)]

This sounds like a B1 or D type of bug (according to https://security.openstack.org/vmt-process.html#incident-report-taxonomy ).
Any objections if we open this bug by the end of the week ?

[Tony Breeds (o-tony)]

Actually I think it'll be an A class. The mitigation we have in mind will be effective on master and stable/* and will (we hope) leverage existing config options to reduce (not remove) the impact.

[Tristan Cacqueray (tristan-cacqueray)]

Any progress on that one Tony ?

[Tristan Cacqueray (tristan-cacqueray)]

Tony, would you mind if we open this bug report so that an eventual patch could get posted directly on gerrit ?

[Tony Breeds (o-tony)]

Yeah okay let's remove the embargo.

[Tristan Cacqueray (tristan-cacqueray)]

I've removed the privacy settings and put the OSSA tasks as Won't Fix based on comment #3. This can be put back to incomplete if the situation changes.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/386756

[Chris Martin (cm876n)]

Not sure why the assignment was changed. I have had a patch submitted to gerrit for 7 weeks but nobody has reviewed it. Would love some reviews on it.

[Sean Dague (sdague)]

This is really a design decision, it's not really clear that changing the expected behavior here is going to provide a good experience for operators. We punt on various classes of potential DOS (like api rate limiting).

[Sean Dague (sdague)]

Found open reviews for this bug in gerrit, setting to In Progress.

review: https://review.openstack.org/386756 in branch: master
review: https://review.openstack.org/407877 in branch: master

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/386756
Reason: This review is > 4 weeks without comment, and is not mergable in it's current state. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[huanhongda (hongda)]

Is anyone working on this?
Can we change the code not to release quota until the instance is reclaimed?

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/407877
Reason: Duplicate of https://review.openstack.org/#/c/386756/ which was abandoned and I'm going to abandon this also - as noted in that other review, this would be a change in behavior and requires wider discussion.