Deleted Cinder Volumes also taking the quota limit

[Shanthakumar K]

I'm using openstack grizzly with MYSQL DB.

I have only one volume in my cinder, but when i'm trying to create the volume i'm getting the following error

"ERROR: VolumeLimitExceeded: Maximum number of volumes allowed (20) exceeded"

root@grzrc3:~# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 617158ed-176b-4c99-88b5-8070b04f9096 | available | vol1 | 1 | None | false | |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
root@grzrc3:~#
root@grzrc3:~#
root@grzrc3:~# cinder create --display-name test 2
ERROR: VolumeLimitExceeded: Maximum number of volumes allowed (20) exceeded
root@grzrc3:~#

All the volumes that I have deleted upto now is also taking the quota limit.

[John Griffith (john-griffith)]

Wondering how you deleted these? I just went through for the demo account with stable/grizzly and did:

cinder --os-username admin --os-password xxxxx quota-update --volumes 3 0197b403ecf14245a89fa4c973754564
cinder create 1
cinder create 1
cinder create 1
cinder create 1 (Fails as expected for over-quota)

cinder delete xxxxxxx
cinder create 1 (successfully created)

Success... inspection of DB matches expected results each time as well.

[Shanthakumar K (shantha-kumar)]

Thanks for your response.

I have deleted using cinder delete & cinder force-delete commands. I belive this is correct way of deleting the volume still i face same problem. So I now increased the quota limit and proceeding.

What would be the cause of this problem?

[Alexander Saint Croix (alexsaintx)]

I'm having this exact same problem right now. However, raising the quota limits for the tenant through the Horizon UI is apparently having no effect. It seems to be using the default quota limits, instead of those assigned to the tenant, which might actually be a separate bug.

In my example, even though the project has 10000 volumes in its quota, and 0 in use, I'm still getting the message:

"VolumeLimitExceeded: Maximum number of volumes allowed (10) exceeded"

Might this be two closely interrelated bugs?

[John Griffith (john-griffith)]

Shausy, the process you've described is correct, I can't quite figure out why I'm unable to reproduce this though. There is a bug in the snapshot side of things that I found and addressed here: bug: 1173365

But I am not able to determine what's happening in your case. Did you try duplicating the steps that I've outlined above?

Is it at all possible that you've got a mix of tenants being used?

[John Griffith (john-griffith)]

Alexander,

Could you please check the cinder database and make sure it's updated properly and that the tenant id is as expected?

The way to do this would be:
    select * from quotas;

You should see something like this for the tenants you've modified quotas for:

mysql> select * from quotas;
+----+---------------------+---------------------+------------+---------+----------------------------------+-----------+------------+
| id | created_at | updated_at | deleted_at | deleted | project_id | resource | hard_limit |
+----+---------------------+---------------------+------------+---------+----------------------------------+-----------+------------+
| 1 | 2013-04-13 04:43:16 | 2013-04-13 04:45:28 | NULL | 0 | d39dce92f23c474b8e2c9c4a2cb3e14b | volumes | 8 |
| 2 | 2013-04-13 04:45:28 | NULL | NULL | 0 | d39dce92f23c474b8e2c9c4a2cb3e14b | gigabytes | 1000 |
+----+---------------------+---------------------+------------+---------+----------------------------------+-----------+------------+

[Vincent Hou (houshengbo)]

Shausy and, Alexander, I have not reproduced this bug either. I tried the following steps:

1. The default quota limit for volumes is 10.
I tried created 10 volumes with "cinder create 1" and I received the VolumeLimitExceeded error when I created the11th volume.
Then, I deleted one volume, and created one. It was successful.

2. I tried to increase the quota limit to 12, and the the new quota took effect. I was able to create 12 volumes.

3. I tried to decrease the quota limit to 8, while I already had 12 volumes created. The interesting thing was that the quota 8 took effect, showing that I had 12 volumes out of my 8 volumes' limit(12/8 volumes showing in Horizon). In this case, I was unable to create any volumes. I had to delete 5 volumes(12 -5 = 7, then I cloud create one to reach the quota limit 8), before I could create one successfully. I guess this might be something similar happening in your case. Please check with "select * from quotas".

[Shanthakumar K (shantha-kumar)]

Then its databases related issue, I'm using Mysql.

I got the same problem with snapshot also.

If you want i can reproduce and send you the results.

[Duncan Thomas (duncan-thomas)]

Shausy:

Ideally, when you reproduce, can you post the output of:

SELECT * FROM quotas;
SELECT * FROM volumes;

This should help us see where the issue might be

[Heiko Krämer (foexle)]

I've the exactly same problem,

it's maybe a caching problem ? I've take a look into the MySQL database and all values are correct, all old valumes are "deleted" and not present anymore.

I've write a mail to the mailing list with a log abstract:
https://lists.launchpad.net/openstack/msg24347.html

If you need more informations please let me know

[Ollie Leahy (oliver-leahy-l)]

On Fri 14 Jun 2013 09:15:39 IST, Heiko Krämer wrote:
> My DB looks clean:
>
> mysql> select SUM(size) from volumes where deleted=0\G
> *************************** 1. row ***************************
> SUM(size): 88
> 1 row in set (0.00 sec)
>
> mysql> select SUM(volume_size) from snapshots where deleted=0\G
> *************************** 1. row ***************************
> SUM(volume_size): 53
> 1 row in set (0.00 sec)
>
I think 'cinder list' showed that you had only 38G used in volumes, one likely reason
for the value of 88G given here is that there is another project with volumes created in the
database.

>
> That's the entries of my mysql db but it seems that cinder will sumarize
> all new created volumes to the "usage" (that's correct) even they will
> delete after.
I don't think that deleted volumes are included in usage counts, if that's what you mean. Possible,
but I haven't seen it happen. Quota usage is recorded in a table called 'quota_usages'. I believe that
it is possible that quota_usage has not been decremented when volumes were deleted. Can you
post the output of these sql commands

mysql> select project_id from volumes where display_name='bosh';

or instead of 'bosh' you can use any of the volume names that are displayed when you run 'cinder list'
this will give your project id, and then

mysql> select SUM(size) from volumes where project_id='your project id' and deleted=0;
mysql> select SUM(volume_size) from snapshots where project_id='your project id' and deleted=0;
mysql> select * from quota_usages;

This will tell us whether the quota_usages table is out of sync with the volumes table.

Thanks

[Sam Morrison (sorrison)]

Hi All,

My guess is this is due to wrong data in the quota_usages table, this gets out of sync for us a lot.

We set max_age and until_fresh to very low settings so this cache is updates frequently.

There are a lot of bugs in cinder that relate to modifying this value at wrong times.

Eg: https://bugs.launchpad.net/cinder/+bug/1078668

Nova also suffers the same issues. I have lodged a bug about it at https://bugs.launchpad.net/nova/+bug/1202896 and will link it to cinder too.

[Milan Detroja (miland)]

even i faced same problem while running some tempest tests. then i solved by deleting those volume from database. after doing this it was working fine for me. for me it was tempest problem, in that case it was not showing on "cinder list" but actually some volume were present on DB. so i will suggest to do this once and retry.

[John Griffith (john-griffith)]

Milan,
One thing to make sure of particularly when running Tempest tests is make certain that the resource was actually cleaned up and deleted properly. There are a number of situations in the Tempest tests where this doesn't happen properly, and given that the tests are run using a "hidden" tenant it's often missed.

[Amol Chobe (amol-chobe)]

we have same issue in havana , we keep on hitting the quota limit and after cleaning up we get volume error message .
each time we have to go to database and delete entries from volume and volume admin tables .

[Mike Perez (thingee)]

Amol, when you're able to reproduce the problem, can you give the output of

SELECT * FROM quotas;
SELECT * FROM volumes;

[Rarylson Freitas (rarylson)]

I have the issue.

I discover a workaround for the bug listed by "Heiko Krämer (foexle) wrote on 2013-06-12". The workaround consistis in "delete" the entries in the 'quota_usages' table (new entries will be created after).

Thanks for the "Sam Morrison (sorrison) wrote on 2013-07-30" tip.

```sql
UPDATE quota_usages set deleted_at=NOW(), deleted="1";
```

[guevara (kyang-happy)]

1.enter mysql and use cinder
2.modify the value of in_use in quota_usages
 command:
update quota_usages set in_use=0 where (project_id='23ead8fe1f434d33930a686f04b8d686'&&resource='volumes');

project_id is what you use to create volume (eg: admin or demo)

in_use=0(the value is actually you use in your opensatck, use 'cinder list ' command to show how many volumes in use)

[Gokul (gokul-venkatachalam)]

How to decrease usage count in MySql database while deleting the volume?
can any one help.