Cinder API response with code 500 when creating a new volume when quota is exceeded

[Ron]

Hello.

On Queens I receive a code 500 response from the cinder API when I try to create a new volume when the quota for the project is already exceeded:

{"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

I can nail down the ERROR from the logs:

- Quota exceeded for xxx, tried to create volume (50 volumes already consumed).: OverQuota: Quota exceeded for resources: ['volumes']

- Task 'cinder.volume.flows.api.create_volume.QuotaReserveTask;volume:create' (...) transitioned into state 'FAILURE' from state 'RUNNING'

- Task 'cinder.volume.flows.api.create_volume.QuotaReserveTask;volume:create' (...) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'

- Task 'cinder.volume.flows.api.create_volume.ExtractVolumeRequestTask;volume:create' (.. ) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'

- Flow 'volume_create_api' (...) transitioned into state 'REVERTED' from state 'RUNNING'

- ERROR cinder.api.middleware.fault [req-id and so on ...] Caught error: <type 'exceptions.TypeError'> 'NoneType' object is not iterable: TypeError: 'NoneType' object is not iterable

What I see also after the Error is another WARNING which probably is related to https://bugs.launchpad.net/cinder/+bug/1188189. But afterwards I see another INFO with the 500 code:

- WARNING py.warnings [...] /var/lib/python2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

- INFO cinder.api.middleware.fault [...] http://MYCINDERAPI:443/v2/MYID/volumes returned with HTTP 500

Of course there are a plenty of other DEBUG and INFO messages but they don't look interesting.

I read in some other bug reports of other projects that this seems to be a "general" exception handler, e.g. here: https://git.openstack.org/cgit/openstack/ironic/commit/?id=2b5849b49e24f310b7cf7f7c2e9a8b14f11da2a7

The question here is if quota exceeded actions raise such an code 500 error with that general exception or if it is something specific on my side. I think the expected behaviour would another response code, e.g. 4xx.

Best regards, Ron

[Rajat Dhasmana (whoami-rajat)]

Hi Ron,

Cinder currently (on master/stein) returns 413[1] status when we exceed the quota limit[2].
It was implemented by the commit [3], and the code exists in queens too.
The exception called are here[4]. Could you please check again?

[1] https://github.com/openstack/cinder/blob/master/cinder/exception.py#L627
[2] https://github.com/openstack/cinder/blob/master/cinder/exception.py#L632
[3] https://github.com/openstack/cinder/commit/87c5749a3fe88e42277c3dcf631b510a45995935#diff-568ee7fd302847948453bea53c42c72f
[4] https://github.com/openstack/cinder/blob/master/cinder/quota_utils.py#L245-L248

[Ron (unmerkbar)]

Hi Rajat.

Thank you very much for your feedback and the links. I checked our source code and the code you referred to is also available on our implementation. So I re-tested again and it seems that the problem is gone. I receive now a status code 413 after the quota is exceeded. I made a deep dive and it seems that the issue was on our side. I can nail it down to a longer list of commits between 2018-11-26 and today where I should find the issue.

We can close the bug report.

Thank you for your help and best regards