Quota usage is able to drop below zero

[Nate Bensimon]

Scenario:
1. Quota usage starts off @ 0 for existing accounts
2. Delete an instance on existing account

Expected:
1. Usage data gets updated with actual usage on the account

Actual:
1. Usage data is relative to the delete action and therefore dropped to a negative number

[Auston McReynolds (amcrn)]

Can you elaborate on how to reproduce this? I don't see how a user without an instance, can delete an instance? The backdoor-fix for this is trivially simple, but I'd prefer to find the root cause vs. doing something like "if impending_quota_usage < 0 then impending_quota_usage = 0".

[Dan Nguyen (daniel-a-nguyen)]

This has been reported by QA on our end as well.
QA has a test that concurrently creates new instances that puts the quotas and instance records out of whack in the trove db.

Here is the longer story.

1. Tenant quota limits are set to 10 for instances.
2. QA runs a concurrency test resulting in many records in trove instances (219)
3. Of the 219 only 10 correspond to actual nova instances
4. Quotas for the user are -38

We currently work around this issue by manually updating the database.

[Steve Leon (steve-leon)]

I think i may have found the root cause.

Lets say you start out with a quota usage of 0. When a create-instance fails in the API server (lets say due to security quota limit), the quota rolls back and stays at 0. However, the DB record created in this method does not get rolled back. So at the end, the user will see an instance with status 'BUILD' or 'ERROR'. When the user tries to delete this failed instance, the quota engine subtracts the instance and puts it in the pool (quota is now -1)

I will fix it by also rolling back the DB record created.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/49301

[Steve Leon (steve-leon)]

There are two issues that causes this bug. Once is what i described previously , where the instance fails when creating sec-group. The quota rolls back but the instance does not. This is being addressed in BP security-groups-workflow-update.

Another issue that could cause this is when two requests from the same tenant gets to the API. During the quota reservation, the quota reserve counter gets set, instead of being incremented. This can cause the reservation of one of the request be overwritten

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/49301
Committed: http://github.com/openstack/trove/commit/99f7c104022aeaa61fe0710d2a8f8fcc09c2b09a
Submitter: Jenkins
Branch: master

commit 99f7c104022aeaa61fe0710d2a8f8fcc09c2b09a
Author: Steve Leon <email address hidden>
Date: Wed Oct 2 12:35:39 2013 -0700

    Fix quota issue where usages can drop to negative value

    This is caused by an exception occuring in the model when creating
    and instance. When this happens, the quota rolls back but the DB
    record, created for that instance, does not. This issue is NOT fixed
    in this patch but depends on
    BP https://blueprints.launchpad.net/trove/+spec/security-groups-workflow-update

    What this fix does is adds a check that prevents quota usages drop below 0.

    Also it fixes a bug where the reserve quota was being set and not added.
    This causes problem when multiple requests are sent at the same time
    and causes race condition issues.

    Fixes bug 1178333

    Change-Id: I1c64941815b91e58d79b17d7a6e69c15c1daaf36