| 2014-12-10 00:57:34 |
Patrick Crews |
description |
I am still working on diagnosing the exact steps to reproduce, but on test systems it is possible to get nova's quota tracking into an inaccurate state.
This was triggered via randomized testing with a focus on create server images, rebooting, and resizing them.
In the example below, after running some randomized tests, there are no active servers on the project, yet 'nova absolute-limits' output lists cores as being used (the quotas-usage table data reflects this).
No cores are listed as used in the admin view for hypervisor usage, but this inaccurate state will prevent the project from spinning up machines.
nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
pcrews@erlking-dev:~/git/rannsaka$ nova absolute-limits
+-------------------------+-------+
| Name | Value |
+-------------------------+-------+
| maxServerMeta | 128 |
| maxPersonality | 5 |
| totalServerGroupsUsed | 0 |
| maxImageMeta | 128 |
| maxPersonalitySize | 10240 |
| maxTotalRAMSize | 51200 |
| maxTotalKeypairs | 100 |
| maxSecurityGroupRules | 20 |
| maxServerGroups | 10 |
| totalCoresUsed | 8 |
| totalRAMUsed | 26368 |
| maxSecurityGroups | 10 |
| totalFloatingIpsUsed | 0 |
| totalInstancesUsed | 0 |
| totalSecurityGroupsUsed | 1 |
| maxTotalFloatingIps | 10 |
| maxTotalInstances | 20 |
| maxTotalCores | 20 |
| maxServerGroupMembers | 10 |
+-------------------------+-------+
mysql> select * from quota_usages;
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| created_at | updated_at | deleted_at | id | project_id | resource | in_use | reserved | until_refresh | deleted | user_id |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| 2014-12-09 23:28:12 | 2014-12-10 00:40:06 | NULL | 1 | 078e0e1371f44e2e9e6d9691342ed02d | instances | 0 | 0 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:12 | 2014-12-10 00:40:06 | NULL | 2 | 078e0e1371f44e2e9e6d9691342ed02d | ram | 26368 | 3968 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:12 | 2014-12-10 00:40:06 | NULL | 3 | 078e0e1371f44e2e9e6d9691342ed02d | cores | 8 | 2 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:12 | 2014-12-09 23:28:12 | NULL | 4 | 078e0e1371f44e2e9e6d9691342ed02d | security_groups | 1 | 0 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:14 | 2014-12-10 00:40:05 | NULL | 5 | 078e0e1371f44e2e9e6d9691342ed02d | fixed_ips | 0 | 0 | NULL | 0 | NULL |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+ |
I am still working on diagnosing the exact steps to reproduce, but on test systems it is possible to get nova's quota tracking into an inaccurate state.
This was triggered via randomized testing with a focus on create server images as well as rebooting and resizing the test servers.
In the example below, after running some randomized tests, there are no active servers on the project, yet 'nova absolute-limits' output lists cores as being used (the quotas-usage table data reflects this).
No cores are listed as used in the admin view for hypervisor usage, but this inaccurate state will prevent the project from spinning up machines.
nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
pcrews@erlking-dev:~/git/rannsaka$ nova absolute-limits
+-------------------------+-------+
| Name | Value |
+-------------------------+-------+
| maxServerMeta | 128 |
| maxPersonality | 5 |
| totalServerGroupsUsed | 0 |
| maxImageMeta | 128 |
| maxPersonalitySize | 10240 |
| maxTotalRAMSize | 51200 |
| maxTotalKeypairs | 100 |
| maxSecurityGroupRules | 20 |
| maxServerGroups | 10 |
| totalCoresUsed | 8 |
| totalRAMUsed | 26368 |
| maxSecurityGroups | 10 |
| totalFloatingIpsUsed | 0 |
| totalInstancesUsed | 0 |
| totalSecurityGroupsUsed | 1 |
| maxTotalFloatingIps | 10 |
| maxTotalInstances | 20 |
| maxTotalCores | 20 |
| maxServerGroupMembers | 10 |
+-------------------------+-------+
mysql> select * from quota_usages;
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| created_at | updated_at | deleted_at | id | project_id | resource | in_use | reserved | until_refresh | deleted | user_id |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+
| 2014-12-09 23:28:12 | 2014-12-10 00:40:06 | NULL | 1 | 078e0e1371f44e2e9e6d9691342ed02d | instances | 0 | 0 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:12 | 2014-12-10 00:40:06 | NULL | 2 | 078e0e1371f44e2e9e6d9691342ed02d | ram | 26368 | 3968 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:12 | 2014-12-10 00:40:06 | NULL | 3 | 078e0e1371f44e2e9e6d9691342ed02d | cores | 8 | 2 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:12 | 2014-12-09 23:28:12 | NULL | 4 | 078e0e1371f44e2e9e6d9691342ed02d | security_groups | 1 | 0 | NULL | 0 | d2197b1accca4a51b2dbb964d9fc7683 |
| 2014-12-09 23:28:14 | 2014-12-10 00:40:05 | NULL | 5 | 078e0e1371f44e2e9e6d9691342ed02d | fixed_ips | 0 | 0 | NULL | 0 | NULL |
+---------------------+---------------------+------------+----+----------------------------------+-----------------+--------+----------+---------------+---------+----------------------------------+ |
|
