Abnormal changes of quota usage after instance restored by admin

Bug #1482444 reported by zhengyue
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
zhengyue

Bug Description

Nova version, output of 'git log -1':
    commit 676ba7bbc788a528b0fe4c87c1c4bf94b4bb6eb1
    Author: Dave McCowan <email address hidden>
    Date: Tue Feb 24 21:35:48 2015 -0500

    Websocket Proxy should verify Origin header

    If the Origin HTTP header passed in the WebSocket handshake does
    not match the host, this could indicate an attempt at a
    cross-site attack. This commit adds a check to verify
    the origin matches the host.

    Change-Id: Ica6ec23d6f69a236657d5ba0c3f51b693c633649
    Closes-Bug: 1409142
Reproduce steps:
1. Enable soft delete via set reclaim_instance_interval in nova.conf.
2. A normal project: ProjectA create a new instance and then delete it, then it's status change to SOFT_DELETED.
3. Now restore the instance by admin user in project: admin, the instance back to ACTIVE, but the quota usage of project: admin has changed, the flavor of that instance has added on admin project quota usage.

Tags: quotas nova
zhengyue (zhengyue-5)
description: updated
Changed in nova:
assignee: nobody → zhengyue (zhengyue-5)
assignee: zhengyue (zhengyue-5) → nobody
assignee: nobody → zhengyue (zhengyue-5)
Revision history for this message
Zhenyu Zheng (zhengzhenyu) wrote :

I didn't reproduce this bug, maybe you can explain more.

Revision history for this message
zhengyue (zhengyue-5) wrote :
Download full text (6.1 KiB)

The projects in my environment:
$ keystone tenant-list
+-------------------------------------------------------+--------------------+---------+
| id | name | enabled |
+-------------------------------------------------------+--------------------+---------+
| ba4448bdba0540409d2ec7901b919b30 | admin | True |
| 4065938305c847d3b72d89118935d1f0 | demo | True |

Detail of reproduce steps:
1. Enable soft delete set 'reclaim_instance_interval = 3600' in nova.conf;
2. Create a instance in project: demo, get the instance: $ nova list --all-tenants:
+--------------------------------------+------------------+----------------------------------+--------+------------+-------------+------------------+
| ID | Name | Tenant ID | Status | Task State | Power State | Networks |
+--------------------------------------+------------------+----------------------------------+--------+------------+-------------+------------------+
| acaacb74-3fdf-40a7-8f47-dd00b949254f | test_soft_delete | 4065938305c847d3b72d89118935d1f0 | ACTIVE | - | Running | private=10.0.0.2 |
+--------------------------------------+------------------+----------------------------------+--------+------------+-------------+------------------+
3. Show quota usage before action:
(1). Usage of project admin (Don't care the current value, please focus on the comparison with results at later):
nova limits --tenant ba4448bdba0540409d2ec7901b919b30
+--------------------+------+-------+
| Name | Used | Max |
+--------------------+------+-------+
| Cores | 5 | 20 |
| FloatingIps | 0 | 10 |
| ImageMeta | - | 128 |
| Instances | 5 | 10 |
| Keypairs | - | 100 |
| Personality | - | 5 |
| Personality Size | - | 10240 |
| RAM | 2560 | 51200 |
| SecurityGroupRules | - | 20 |
| SecurityGroups | 0 | 10 |
| Server Meta | - | 128 |
| ServerGroupMembers | - | 10 |
| ServerGroups | 0 | 10 |
+--------------------+------+-------+
 (2). Usage of project demo:
nova limits --tenant 4065938305c847d3b72d89118935d1f0
+--------------------+------+-------+
| Name | Used | Max |
+--------------------+------+-------+
| Cores | 2 | 20 |
| FloatingIps | 0 | 10 |
| ImageMeta | - | 128 |
| Instances | 2 | 10 |
| Keypairs | - | 100 |
| Personality | - | 5 |
| Personality Size | - | 10240 |
| RAM | 1024 | 51200 |
| SecurityGroupRules | - | 20 |
| SecurityGroups | 1 | 10 |
| Server Meta | - | 128 |
| ServerGroupMembers | - | 10 |
| ServerGroups | 0 | 10 |
+--------------------+------+-------+
4. Delete server belongs to project demo:
$ nova delete acaacb74-3fdf-40a7-8f47-dd00b949254f
Request to delete server acaacb74-3fdf-40a7-8f47-dd00b949254f has been accepted.
5. Check status of the instance i...

Read more...

Revision history for this message
zhengyue (zhengyue-5) wrote :

I think the reason of the problem is that the obtain of quota by context.
The code at nova/compute/api.py:
def _check_num_instances_quota(....
    ...
    quotas = objects.Quotas(context) // The context is admin
    quotas.reserve(instances=max_count, cores=req_cores, ram=req_ram)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/211432

Changed in nova:
status: New → In Progress
Alex Xu (xuhj)
Changed in nova:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/211432
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c474b44bdcdda223b0ec90b7fa1673501cf690e7
Submitter: Jenkins
Branch: master

commit c474b44bdcdda223b0ec90b7fa1673501cf690e7
Author: Zheng Yue <email address hidden>
Date: Tue Aug 11 10:59:43 2015 +0800

    Fix abnormal quota usage after restore by admin

    At environment which enabled soft delete at nova conf,
    if one instance of normal project be restored by admin,
    the usage of instance's flavor will be add to quota of admin.
    The reason is that obtain of quota from admin context.
    Now passing project_id and user_id which get from instance
    to amend owner of quota.

    Change-Id: I0e0f6085a6b0a0b9d1072cc2daffd85f54830fff
    Closes-bug: #1482444

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → liberty-rc1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: liberty-rc1 → 12.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.