quota-usage error in soft-delete

[hzxiongwenwu]

condition: reclaim_instance_interval > 0 in nova.conf

I am testing soft-delete , found that quota_usages table will lead to error
result. when an instance was soft-deleted , before it was deleted completely by period task i soft-delete it again, the quato_usages table will reduce double resource of this instance.

the reason is that every execution of soft-delete instance, the reservation will commit.

how to fixed it: we should make reservations=None when instance.vm_state='soft-deleted'.

how to reproduct it:

i am project_id='30528b0d602c4a9c9d8b4cd3d416d710', and I have an instance:

ubuntu@xfolsom:/opt/stack/nova$ nova list
+--------------------------------------+------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+------------------+
| 6f6c1258-6eda-43f1-9531-7a4eb0b44724 | test | ACTIVE | - | Running | private=10.0.0.2 |
+--------------------------------------+------+--------+------------+-------------+------------------+

1.first select from quota_usage, the result is :

mysql> select * from quota_usages;
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| created_at | updated_at | deleted_at | id | project_id | resource | in_use | reserved | until_refresh | deleted | user_id |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| 2014-06-20 08:24:35 | 2014-06-23 08:35:03 | NULL | 1 | 30528b0d602c4a9c9d8b4cd3d416d710 | instances | 1 | 0 | NULL | 0 | e522bb6fecaa4a69b6d7df69211dab13 |
| 2014-06-20 08:24:35 | 2014-06-23 08:35:03 | NULL | 2 | 30528b0d602c4a9c9d8b4cd3d416d710 | ram | 64 | 0 | NULL | 0 | e522bb6fecaa4a69b6d7df69211dab13 |
| 2014-06-20 08:24:35 | 2014-06-23 08:35:03 | NULL | 3 | 30528b0d602c4a9c9d8b4cd3d416d710 | cores | 1 | 0 | NULL | 0 | e522bb6fecaa4a69b6d7df69211dab13 |
| 2014-06-20 08:24:35 | 2014-06-20 08:24:35 | NULL | 4 | 30528b0d602c4a9c9d8b4cd3d416d710 | security_groups | 1 | 0 | NULL | 0 | e522bb6fecaa4a69b6d7df69211dab13 |
| 2014-06-20 08:24:36 | 2014-06-23 03:56:03 | NULL | 5 | 30528b0d602c4a9c9d8b4cd3d416d710 | fixed_ips | 1 | 0 | NULL | 0 | NULL |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
5 rows in set (0.00 sec)

2.using nova-network, set reclaim_instance_interval=600 in nova.conf.
3.nova delete 6f6c1258-6eda-43f1-9531-7a4eb0b44724
4. select from quota_usages, result is :

mysql> select * from quota_usages;
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| created_at | updated_at | deleted_at | id | project_id | resource | in_use | reserved | until_refresh | deleted | user_id |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| 2014-06-20 08:24:35 | 2014-06-23 08:42:30 | NULL | 1 | 30528b0d602c4a9c9d8b4cd3d416d710 | instances | 0 | 0 | NULL | 0 | e522bb6fecaa4a69b6d7df69211dab13 |
| 2014-06-20 08:24:35 | 2014-06-23 08:42:30 | NULL | 2 | 30528b0d602c4a9c9d8b4cd3d416d710 | ram | 0 | 0 | NULL | 0 | e522bb6fecaa4a69b6d7df69211dab13 |
| 2014-06-20 08:24:35 | 2014-06-23 08:42:30 | NULL | 3 | 30528b0d602c4a9c9d8b4cd3d416d710 | cores | 0 | 0 | NULL | 0 | e522bb6fecaa4a69b6d7df69211dab13 |
| 2014-06-20 08:24:35 | 2014-06-20 08:24:35 | NULL | 4 | 30528b0d602c4a9c9d8b4cd3d416d710 | security_groups | 1 | 0 | NULL | 0 | e522bb6fecaa4a69b6d7df69211dab13 |
| 2014-06-20 08:24:36 | 2014-06-23 03:56:03 | NULL | 5 | 30528b0d602c4a9c9d8b4cd3d416d710 | fixed_ips | 1 | 0 | NULL | 0 | NULL |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
5 rows in set (0.00 sec)

5. nova delete 6f6c1258-6eda-43f1-9531-7a4eb0b44724 again

then select from quota_usages, result is :

mysql> select * from quota_usages;
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| created_at | updated_at | deleted_at | id | project_id | resource | in_use | reserved | until_refresh | deleted | user_id |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| 2014-06-20 08:24:35 | 2014-06-23 08:43:38 | NULL | 1 | 30528b0d602c4a9c9d8b4cd3d416d710 | instances | -1 | 0 | NULL | 0 | e522bb6fecaa4a69b6d7df69211dab13 |
| 2014-06-20 08:24:35 | 2014-06-23 08:43:38 | NULL | 2 | 30528b0d602c4a9c9d8b4cd3d416d710 | ram | -64 | 0 | NULL | 0 | e522bb6fecaa4a69b6d7df69211dab13 |
| 2014-06-20 08:24:35 | 2014-06-23 08:43:38 | NULL | 3 | 30528b0d602c4a9c9d8b4cd3d416d710 | cores | -1 | 0 | NULL | 0 | e522bb6fecaa4a69b6d7df69211dab13 |
| 2014-06-20 08:24:35 | 2014-06-20 08:24:35 | NULL | 4 | 30528b0d602c4a9c9d8b4cd3d416d710 | security_groups | 1 | 0 | NULL | 0 | e522bb6fecaa4a69b6d7df69211dab13 |
| 2014-06-20 08:24:36 | 2014-06-23 03:56:03 | NULL | 5 | 30528b0d602c4a9c9d8b4cd3d416d710 | fixed_ips | 1 | 0 | NULL | 0 | NULL |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
5 rows in set (0.00 sec)

oh my god!!! instance in_use is -1 and ram in_use is -64Mb, do you think this is right?

6.nova restore 6f6c1258-6eda-43f1-9531-7a4eb0b44724
 now the db result is:

mysql> select * from quota_usages;
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| created_at | updated_at | deleted_at | id | project_id | resource | in_use | reserved | until_refresh | deleted | user_id |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| 2014-06-20 08:24:35 | 2014-06-23 08:44:41 | NULL | 1 | 30528b0d602c4a9c9d8b4cd3d416d710 | instances | 2 | 0 | NULL | 0 | e522bb6fecaa4a69b6d7df69211dab13 |
| 2014-06-20 08:24:35 | 2014-06-23 08:44:41 | NULL | 2 | 30528b0d602c4a9c9d8b4cd3d416d710 | ram | 128 | 0 | NULL | 0 | e522bb6fecaa4a69b6d7df69211dab13 |
| 2014-06-20 08:24:35 | 2014-06-23 08:44:41 | NULL | 3 | 30528b0d602c4a9c9d8b4cd3d416d710 | cores | 2 | 0 | NULL | 0 | e522bb6fecaa4a69b6d7df69211dab13 |
| 2014-06-20 08:24:35 | 2014-06-20 08:24:35 | NULL | 4 | 30528b0d602c4a9c9d8b4cd3d416d710 | security_groups | 1 | 0 | NULL | 0 | e522bb6fecaa4a69b6d7df69211dab13 |
| 2014-06-20 08:24:36 | 2014-06-23 03:56:03 | NULL | 5 | 30528b0d602c4a9c9d8b4cd3d416d710 | fixed_ips | 1 | 0 | NULL | 0 | NULL |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
5 rows in set (0.00 sec)

obviously, the result in db is error.now I have only one instance, but in db, i have used 2 instances, 2 cores , 128Mb ram and so on.

The correct way is that we should not allow deleting instance which status in soft-delete.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/102153

[OpenStack Infra (hudson-openstack)]

Change abandoned by hzxiongwenwu (<email address hidden>) on branch: master
Review: https://review.openstack.org/101855

[Sean Dague (sdague)]

Patch abandoned, not in progress any more

[OpenStack Infra (hudson-openstack)]

Change abandoned by Joe Gordon (<email address hidden>) on branch: master
Review: https://review.openstack.org/102153
Reason: Patch is stalled waiting for the author, looks like this has been abandoned. Feel free to restore.

[Sergey Nikitin (snikitin)]

Fix here https://review.openstack.org/#/c/128848/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/128848
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4546ce0012b526cd370533ba5a3f2f4107d249a2
Submitter: Jenkins
Branch: master

commit 4546ce0012b526cd370533ba5a3f2f4107d249a2
Author: Sergey Nikitin <email address hidden>
Date: Thu Oct 16 12:31:13 2014 +0400

    Fixed quotas double decreasing problem

    When we delete vm with status "soft-delete" we shouldn't change
    quota values because they were changed during soft-deleting.

    Closes-bug: #1333145
    Change-Id: I3ed7b3245b8908faa02903c74503918f061016bd

[Patrick Crews (patrick-crews)]

possibly related to: https://bugs.launchpad.net/nova/+bug/1284424?