Cinder

Cinder Group Quota Incorrect Functionality

Bug #1714216 reported by Shashi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Medium
Unassigned

Bug Description

Observation - Group-quota incorrect functionality

Steps to reproduce -
 1. Set Group Quota to 5 and Try to create more than 5 groups.
 2. Quota exception is seen when we try to create 6th group which as expected.
 3. Now increase group quota to 15 and again try to create 6th group.
 4. Same quota exception is seen though quota is extended from 5 to 15.

Please see detailed output below.
-- UPDATED CINDER GROUP-QUOTS TO 5 --
stack@cld6b16:~$ cinder --os-volume-api-version 3.14 quota-update --groups 5 admin
+-----------------------+-------+
| Property | Value |
+-----------------------+-------+
| backup_gigabytes | 1000 |
| backups | 10 |
| gigabytes | 1000 |
| gigabytes_3pariscsi | -1 |
| gigabytes_lvmdriver-1 | -1 |
| groups | 5 |
| per_volume_gigabytes | -1 |
| snapshots | 10 |
| snapshots_3pariscsi | -1 |
| snapshots_lvmdriver-1 | -1 |
| volumes | 15 |
| volumes_3pariscsi | -1 |
| volumes_lvmdriver-1 | -1 |
+-----------------------+-------+
stack@cld6b16:~$ cinder quota-show admin
+-----------------------+-------+
| Property | Value |
+-----------------------+-------+
| backup_gigabytes | 1000 |
| backups | 10 |
| gigabytes | 1000 |
| gigabytes_3pariscsi | -1 |
| gigabytes_lvmdriver-1 | -1 |
| groups | 5 |
| per_volume_gigabytes | -1 |
| snapshots | 10 |
| snapshots_3pariscsi | -1 |
| snapshots_lvmdriver-1 | -1 |
| volumes | 15 |
| volumes_3pariscsi | -1 |
| volumes_lvmdriver-1 | -1 |
+-----------------------+-------+

-- TRIED TO CREATE MORE THAN 5 GROUPS --

stack@cld6b16:~$ cinder --os-volume-api-version 3.14 group-create --name GROUP-1 3pariscsi-group-type 3pariscsi
+-------------------+-------------------------------------------+
| Property | Value |
+-------------------+-------------------------------------------+
| availability_zone | nova |
| created_at | 2017-08-31T09:51:49.000000 |
| description | None |
| group_snapshot_id | None |
| group_type | 259641e2-fea8-4216-8585-710b6eb378ac |
| id | d7997566-964c-4191-b5e6-e23e17a6a50d |
| name | GROUP-1 |
| source_group_id | None |
| status | creating |
| volume_types | [u'eb58f329-8c84-4ca0-87df-55a7579db634'] |
+-------------------+-------------------------------------------+
stack@cld6b16:~$ cinder --os-volume-api-version 3.14 group-create --name GROUP-2 3pariscsi-group-type 3pariscsi
+-------------------+-------------------------------------------+
| Property | Value |
+-------------------+-------------------------------------------+
| availability_zone | nova |
| created_at | 2017-08-31T09:51:59.000000 |
| description | None |
| group_snapshot_id | None |
| group_type | 259641e2-fea8-4216-8585-710b6eb378ac |
| id | 312689c0-5667-4be7-b85d-fb6422942fea |
| name | GROUP-2 |
| source_group_id | None |
| status | creating |
| volume_types | [u'eb58f329-8c84-4ca0-87df-55a7579db634'] |
+-------------------+-------------------------------------------+
stack@cld6b16:~$ cinder --os-volume-api-version 3.14 group-create --name GROUP-3 3pariscsi-group-type 3pariscsi
+-------------------+-------------------------------------------+
| Property | Value |
+-------------------+-------------------------------------------+
| availability_zone | nova |
| created_at | 2017-08-31T09:52:09.000000 |
| description | None |
| group_snapshot_id | None |
| group_type | 259641e2-fea8-4216-8585-710b6eb378ac |
| id | 8d69be83-3e3f-48d9-9d13-4102978a1abd |
| name | GROUP-3 |
| source_group_id | None |
| status | creating |
| volume_types | [u'eb58f329-8c84-4ca0-87df-55a7579db634'] |
+-------------------+-------------------------------------------+
stack@cld6b16:~$ cinder --os-volume-api-version 3.14 group-create --name GROUP-4 3pariscsi-group-type 3pariscsi
+-------------------+-------------------------------------------+
| Property | Value |
+-------------------+-------------------------------------------+
| availability_zone | nova |
| created_at | 2017-08-31T09:52:22.000000 |
| description | None |
| group_snapshot_id | None |
| group_type | 259641e2-fea8-4216-8585-710b6eb378ac |
| id | 8592fc4b-cba3-4672-a341-30a74e02c27f |
| name | GROUP-4 |
| source_group_id | None |
| status | creating |
| volume_types | [u'eb58f329-8c84-4ca0-87df-55a7579db634'] |
+-------------------+-------------------------------------------+
stack@cld6b16:~$ cinder --os-volume-api-version 3.14 group-create --name GROUP-5 3pariscsi-group-type 3pariscsi
+-------------------+-------------------------------------------+
| Property | Value |
+-------------------+-------------------------------------------+
| availability_zone | nova |
| created_at | 2017-08-31T09:52:34.000000 |
| description | None |
| group_snapshot_id | None |
| group_type | 259641e2-fea8-4216-8585-710b6eb378ac |
| id | 214ef050-37fc-4cc0-a541-79e84a6f13ae |
| name | GROUP-5 |
| source_group_id | None |
| status | creating |
| volume_types | [u'eb58f329-8c84-4ca0-87df-55a7579db634'] |
+-------------------+-------------------------------------------+
stack@cld6b16:~$ cinder --os-volume-api-version 3.14 group-create --name GROUP-6 3pariscsi-group-type 3pariscsi
ERROR: GroupLimitExceeded: Maximum number of groups allowed (6) exceeded (HTTP 413) (Request-ID: req-1fc7419b-e5c4-4d52-93a4-c8bc5353d79a)

--EXPECT ERROR.
--NOW UPDATE QUOTA FROM 5 TO 15 AND TRY TO CREATE GROUP AGAIN --

stack@cld6b16:~$ cinder --os-volume-api-version 3.14 quota-update --groups 15 admin
+-----------------------+-------+
| Property | Value |
+-----------------------+-------+
| backup_gigabytes | 1000 |
| backups | 10 |
| gigabytes | 1000 |
| gigabytes_3pariscsi | -1 |
| gigabytes_lvmdriver-1 | -1 |
| groups | 15 |
| per_volume_gigabytes | -1 |
| snapshots | 10 |
| snapshots_3pariscsi | -1 |
| snapshots_lvmdriver-1 | -1 |
| volumes | 15 |
| volumes_3pariscsi | -1 |
| volumes_lvmdriver-1 | -1 |
+-----------------------+-------+
stack@cld6b16:~$ cinder quota-show admin
+-----------------------+-------+
| Property | Value |
+-----------------------+-------+
| backup_gigabytes | 1000 |
| backups | 10 |
| gigabytes | 1000 |
| gigabytes_3pariscsi | -1 |
| gigabytes_lvmdriver-1 | -1 |
| groups | 15 |
| per_volume_gigabytes | -1 |
| snapshots | 10 |
| snapshots_3pariscsi | -1 |
| snapshots_lvmdriver-1 | -1 |
| volumes | 15 |
| volumes_3pariscsi | -1 |
| volumes_lvmdriver-1 | -1 |
+-----------------------+-------+
stack@cld6b16:~$ cinder --os-volume-api-version 3.14 group-create --name GROUP-6 3pariscsi-group-type 3pariscsi
ERROR: GroupLimitExceeded: Maximum number of groups allowed (6) exceeded (HTTP 413) (Request-ID: req-3c1a0d53-1bc9-4c12-a198-4e49889c9023)

-- WE CAN SEE THE SAME ERROR--
ATTACHING C-VOL AND C-API FOR YOUR REFERENCE.

Revision history for this message
Shashi (sonawane-shashikant) wrote :
Revision history for this message
Shashi (sonawane-shashikant) wrote :

ATTACHING C-VOL.LOG

Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

Hmm, trying to reproduce locally and I actually have a problem that it's set to 5 but I've been able to create 7 groups.

Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

Original default group quota was 10. I was able to create 9 groups before I got the error.

Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

Raised from 5 to 15, still get an error on trying to create GROUP-10.

Would appear something is being cached somewhere.

Changed in cinder:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

Likely a quota refresh issue here:

https://github.com/openstack/cinder/blob/e5081da98f1b46070ecf861bb78dcdd3ed7c1f35/cinder/db/sqlalchemy/api.py#L1146

Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

As a workaround, the "reservation_expire" config option is currently defaulted to 1 day (86400 seconds). This can be changed to a smaller value so the quota cache gets refreshed more frequently.

Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

Finally noticed what's actually going on here. The quota-update command takes a project ID, not a project name.

Since quotas are stored in Cinder and project info is something in keystone, plus the fact that IDs are not required to be UUIDs, when you tell Cinder to set the quota for the "admin" project it dutifully stored that in the database that project ID admin has the new quota value.

Then later, when you attempt to create a group past the original default, the code looks at your context and gets your project ID (something like d68334af53424ea089736f61aa8af542) and looks for a set quota. Since there is a quota set for "admin" but not "d68334af53424ea089736f61aa8af542", it then falls back to going with the default quota values and fails the request with the over quota error.

Changed in cinder:
status: Confirmed → Invalid
Revision history for this message
Shashi (sonawane-shashikant) wrote :
Download full text (16.2 KiB)

Using tenant_id, group quota could be updated appropriately, but when we try to create groups as per the quota limit, we found that we can create 1 group lesser compared to the set quota.
Eg.: When we set group quota to 10, we can create only 9 groups. Or when we set group quota to 15, only 14 groups are allowed to create.

--USING TENANT ID UPDATED QUOTA FROM 6 TO 10--

stack@cld6b16:~$ cinder --os-volume-api-version 3.14 quota-show 1eecfa0581b04a76989f0c154dbe3f87
+-----------------------+-------+
| Property | Value |
 +-----------------------+-------+
| backup_gigabytes | 1000 |
| backups | 10 |
| gigabytes | 1000 |
| gigabytes_3pariscsi | -1 |
| gigabytes_lvmdriver-1 | -1 |
| groups | 6 |
| per_volume_gigabytes | -1 |
| snapshots | 10 |
| snapshots_3pariscsi | -1 |
| snapshots_lvmdriver-1 | -1 |
| volumes | 15 |
| volumes_3pariscsi | -1 |
| volumes_lvmdriver-1 | -1 |
 +-----------------------+-------+
stack@cld6b16:~$ cinder --os-volume-api-version 3.14 quota-update --groups 10 1eecfa0581b04a76989f0c154dbe3f87
+-----------------------+-------+
| Property | Value |
 +-----------------------+-------+
| backup_gigabytes | 1000 |
| backups | 10 |
| gigabytes | 1000 |
| gigabytes_3pariscsi | -1 |
| gigabytes_lvmdriver-1 | -1 |
| groups | 10 |
| per_volume_gigabytes | -1 |
| snapshots | 10 |
| snapshots_3pariscsi | -1 |
| snapshots_lvmdriver-1 | -1 |
| volumes | 15 |
| volumes_3pariscsi | -1 |
| volumes_lvmdriver-1 | -1 |
 +-----------------------+-------+
stack@cld6b16:~$ cinder --os-volume-api-version 3.14 quota-show 1eecfa0581b04a76989f0c154dbe3f87
+-----------------------+-------+
| Property | Value |
 +-----------------------+-------+
| backup_gigabytes | 1000 |
| backups | 10 |
| gigabytes | 1000 |
| gigabytes_3pariscsi | -1 |
| gigabytes_lvmdriver-1 | -1 |
| groups | 10 |
| per_volume_gigabytes | -1 |
| snapshots | 10 |
| snapshots_3pariscsi | -1 |
| snapshots_lvmdriver-1 | -1 |
| volumes | 15 |
| volumes_3pariscsi | -1 |
| volumes_lvmdriver-1 | -1 |
 +-----------------------+-------+

-- STARTED CREATING GROUPS. 5 GROUPS WERE ALREADY PRESENT. --

stack@cld6b16:~$ cinder --os-volume-api-version 3.14 group-list
+--------------------------------------+-----------+---------+
| ID | Status | Name |
 +--------------------------------------+-----------+---------+
| 214ef050-37fc-4cc0-a541-79e84a6f13ae | available | GROUP-5 |
| 312689c0-5667-4be7-b85d-fb6422942fea | available | GROUP-2 |
| 8592fc4b-cba3-4672-a341-30a74e02c27f | available | GROUP-4 |
| 8d69be83-3e3f-48d9-9d13-4102978a1abd | available | GROUP-3 |
| d7997566-964c-4191-b5e6-e23e17a6a50d | available | GROUP-1 |
 +--------------------------------------+-----------+---------+
stack@cld6b16:~$ cinder --os-volume-api-version 3.14 quota-show 1eecfa0581b04a76989f0c154dbe...

Revision history for this message
Shashi (sonawane-shashikant) wrote :

Changing status to New as per comment https://bugs.launchpad.net/cinder/+bug/1714216/comments/9

Changed in cinder:
status: Invalid → New
Revision history for this message
jialiang gao (chialiang) wrote :

Set Group Quota to 16,I can create 16 groups.we try to create 17th group which as expected

stack@openstack:~$ cinder --os-volume-api-version 3.14 quota-update --groups 16 18fce2b7ad2f4b91bfd2eb8010ae4d67
+-----------------------+-------+
| Property | Value |
 +-----------------------+-------+
| backup_gigabytes | 1000 |
| backups | 10 |
| gigabytes | 1000 |
| gigabytes_ceph | -1 |
| gigabytes_lvmdriver-1 | -1 |
| groups | 16 |
| per_volume_gigabytes | -1 |
| snapshots | 10 |
| snapshots_ceph | -1 |
| snapshots_lvmdriver-1 | -1 |
| volumes | 10 |
| volumes_ceph | -1 |
| volumes_lvmdriver-1 | -1 |
 +-----------------------+-------+
stack@openstack:~$
stack@openstack:~$ cinder --os-volume-api-version 3.14 group-create --name GROUP-16 test_group_type ceph
+-------------------+-------------------------------------------+
| Property | Value |
 +-------------------+-------------------------------------------+
| availability_zone | nova |
| created_at | 2017-11-13T08:33:54.000000 |
| description | None |
| group_snapshot_id | None |
| group_type | fcda3cd1-6904-43e9-8d71-efdc7c50cfb3 |
| id | f7c60456-af28-443b-8687-b6e40af67a22 |
| name | GROUP-16 |
| source_group_id | None |
| status | creating |
| volume_types | [u'02bd3415-08ba-4ce0-9c7d-bcde910107f5'] |
 +-------------------+-------------------------------------------+
stack@openstack:~$ cinder --os-volume-api-version 3.14 group-create --name GROUP-17 test_group_type ceph
ERROR: 发生异常 GroupLimitExceeded：原因 Maximum number of groups allowed (16) exceeded (HTTP 413) (Request-ID: req-829a8158-c6ac-4e79-9edf-2c9c7f6f3af5)

Changed in cinder:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.