Quota usage is not updated if an instance is deleted while booting

[Lisa Zangrando]

OpenStack version: Ocata

Description
===========
If you delete an instance while booting (i.e. vm_state=building, task_state=scheduling), such instance is not listed anymore by the "nova list" but the quota usage is not updated (i.e. it should be decreased by the flavor size).

Steps to reproduce
==================
1) edit /etc/nova/nova-api.conf

$ cat /etc/nova/nova-api.conf
[conductor]
topic=my_topic

and restart nova-api service
# systemctl restart openstack-nova-api.service

2) check the project usage (project_id="01ab8de5387547d093aa8ae6b85bd8b1")

MariaDB [nova]> select * from quota_usages where project_id="01ab8de5387547d093aa8ae6b85bd8b1";
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| created_at | updated_at | deleted_at | id | project_id | resource | in_use | reserved | until_refresh | deleted | user_id |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| 2017-05-23 09:42:02 | 2017-07-25 10:18:22 | NULL | 1 | 01ab8de5387547d093aa8ae6b85bd8b1 | instances | 0 | 0 | NULL | 0 | 4469ff06d1e247e0bbfc23668e92bd66 |
| 2017-05-23 09:42:02 | 2017-07-25 10:18:22 | NULL | 2 | 01ab8de5387547d093aa8ae6b85bd8b1 | ram | 0 | 0 | NULL | 0 | 4469ff06d1e247e0bbfc23668e92bd66 |
| 2017-05-23 09:42:02 | 2017-07-25 10:18:22 | NULL | 3 | 01ab8de5387547d093aa8ae6b85bd8b1 | cores | 0 | 0 | NULL | 0 | 4469ff06d1e247e0bbfc23668e92bd66 |
| 2017-05-23 13:21:26 | 2017-05-23 13:21:26 | NULL | 4 | 01ab8de5387547d093aa8ae6b85bd8b1 | security_groups | 0 | 0 | 0 | 0 | 4469ff06d1e247e0bbfc23668e92bd66 |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
4 rows in set (0,00 sec)

3) create a new instance (the task_state will be "scheduling" and vm_state = "building")

$ openstack server create --flavor m1.tiny --image cirros --nic net-id=admin_net --security-group default test
+-------------------------------------+------------------------------------------------+
| Field | Value |
+-------------------------------------+------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | 4i8zTGZZ3u9w |
| config_drive | |
| created | 2017-07-25T10:18:21Z |
| flavor | m1.tiny (5cdecdda-111d-4659-acc3-506609e5fadc) |
| hostId | |
| id | aab25cc8-9df6-4f3b-9585-3cf099bb1adb |
| image | cirros (03d54ef8-f0ac-4ad2-92a0-95835d77d2b5) |
| key_name | None |
| name | test |
| progress | 0 |
| project_id | 01ab8de5387547d093aa8ae6b85bd8b1 |
| properties | |
| security_groups | name='default' |
| status | BUILD |
| updated | 2017-07-25T10:18:23Z |
| user_id | 4469ff06d1e247e0bbfc23668e92bd66 |
| volumes_attached | |
+-------------------------------------+------------------------------------------------+

$ openstack server show aab25cc8-9df6-4f3b-9585-3cf099bb1adb
+-------------------------------------+------------------------------------------------+
| Field | Value |
+-------------------------------------+------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| config_drive | |
| created | 2017-07-25T10:18:21Z |
| flavor | m1.tiny (5cdecdda-111d-4659-acc3-506609e5fadc) |
| hostId | |
| id | aab25cc8-9df6-4f3b-9585-3cf099bb1adb |
| image | cirros (03d54ef8-f0ac-4ad2-92a0-95835d77d2b5) |
| key_name | None |
| name | test |
| progress | 0 |
| project_id | 01ab8de5387547d093aa8ae6b85bd8b1 |
| properties | |
| status | BUILD |
| updated | 2017-07-25T10:19:25Z |
| user_id | 4469ff06d1e247e0bbfc23668e92bd66 |
| volumes_attached | |
+-------------------------------------+------------------------------------------------+

4) check the quota usage for project_id="01ab8de5387547d093aa8ae6b85bd8b1"

MariaDB [nova]> select * from quota_usages where project_id="01ab8de5387547d093aa8ae6b85bd8b1";
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| created_at | updated_at | deleted_at | id | project_id | resource | in_use | reserved | until_refresh | deleted | user_id |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| 2017-05-23 09:42:02 | 2017-07-25 10:18:22 | NULL | 1 | 01ab8de5387547d093aa8ae6b85bd8b1 | instances | 1 | 0 | NULL | 0 | 4469ff06d1e247e0bbfc23668e92bd66 |
| 2017-05-23 09:42:02 | 2017-07-25 10:18:22 | NULL | 2 | 01ab8de5387547d093aa8ae6b85bd8b1 | ram | 512 | 0 | NULL | 0 | 4469ff06d1e247e0bbfc23668e92bd66 |
| 2017-05-23 09:42:02 | 2017-07-25 10:18:22 | NULL | 3 | 01ab8de5387547d093aa8ae6b85bd8b1 | cores | 1 | 0 | NULL | 0 | 4469ff06d1e247e0bbfc23668e92bd66 |
| 2017-05-23 13:21:26 | 2017-05-23 13:21:26 | NULL | 4 | 01ab8de5387547d093aa8ae6b85bd8b1 | security_groups | 0 | 0 | 0 | 0 | 4469ff06d1e247e0bbfc23668e92bd66 |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
4 rows in set (0,00 sec)

5) delete the instance

$ openstack server delete aab25cc8-9df6-4f3b-9585-3cf099bb1adb
$ openstack server show aab25cc8-9df6-4f3b-9585-3cf099bb1adb
No server with a name or ID of 'aab25cc8-9df6-4f3b-9585-3cf099bb1adb' exists.

6) check again the quota usage
MariaDB [nova]> select * from quota_usages where project_id="01ab8de5387547d093aa8ae6b85bd8b1";
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| created_at | updated_at | deleted_at | id | project_id | resource | in_use | reserved | until_refresh | deleted | user_id |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| 2017-05-23 09:42:02 | 2017-07-25 10:18:22 | NULL | 1 | 01ab8de5387547d093aa8ae6b85bd8b1 | instances | 1 | 0 | NULL | 0 | 4469ff06d1e247e0bbfc23668e92bd66 |
| 2017-05-23 09:42:02 | 2017-07-25 10:18:22 | NULL | 2 | 01ab8de5387547d093aa8ae6b85bd8b1 | ram | 512 | 0 | NULL | 0 | 4469ff06d1e247e0bbfc23668e92bd66 |
| 2017-05-23 09:42:02 | 2017-07-25 10:18:22 | NULL | 3 | 01ab8de5387547d093aa8ae6b85bd8b1 | cores | 1 | 0 | NULL | 0 | 4469ff06d1e247e0bbfc23668e92bd66 |
| 2017-05-23 13:21:26 | 2017-05-23 13:21:26 | NULL | 4 | 01ab8de5387547d093aa8ae6b85bd8b1 | security_groups | 0 | 0 | 0 | 0 | 4469ff06d1e247e0bbfc23668e92bd66 |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
4 rows in set (0,00 sec)

Expected result
===============
The quota usage should be decreased

[Sean Dague (sdague)]

I feel like this might have been addressed with melanie's new quota work

[Matt Riedemann (mriedem)]

Can you reproduce this with debug logging enabled and then see if you can find anything in the logs related to the instance when it gets deleted? At the point that you deleted the instance, it didn't have a host set, so it wasn't done building on a compute node.

But things get tricky with delete while booting because the instance may or may not have existed in a cell at that point, and if it didn't exist in a cell yet, then we're really deleting the build request (in the API DB build_requests table) rather than the instance. But this "local delete" code should be handling it:

https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L1785-L1841

There were some bugs in this area when Ocata 15.0.0 was released, but it's not clear what version of Nova you're running. Please provide the version and/or git hash for the nova-api service?

https://docs.openstack.org/releasenotes/nova/ocata.html

For example: https://bugs.launchpad.net/nova/+bug/1670627

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/487104

[Lisa Zangrando (lisa-zangrando)]

Hi Matt,

I'm referring to the Ocata version 15.0.0.4.
I know you have fixed a similar bug in https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L1785-L1841

I debugged the issue and the problem occurs in
https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L1788

 if self._delete_while_booting(context, instance):
     return

if True, the function exits without decreasing the quota usage.
I committed my fix, it seems to work fine. Please take e look.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lee Yarwood (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/487104
Reason: Abandoning this review after 5 weeks without any updates from the owner, please feel free to reopen if you're still working on this.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]