Cinder backend for glance raises 500 if quota of volume exceeds

[Abhishek Kekane]

If you are using cinder backend for glance and quota of volumes is exceeded it raises 500 error on new image creation.

Expected error:
40* error response (mostly 401 or 403)

Actual error:
HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation.

glance-api logs:

Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi Traceback (most recent call last):
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/common/wsgi.py", line 1347, in __call__
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi request, **action_args)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/common/wsgi.py", line 1390, in dispatch
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi return method(*args, **kwargs)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/common/utils.py", line 416, in wrapped
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/api/v2/image_data.py", line 299, in upload
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi self._restore(image_repo, image)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi self.force_reraise()
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi raise value
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/api/v2/image_data.py", line 164, in upload
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi image.set_data(data, size, backend=backend)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/domain/proxy.py", line 198, in set_data
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/notifier.py", line 501, in set_data
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi self.force_reraise()
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi raise value
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/notifier.py", line 448, in set_data
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi set_active=set_active)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/api/policy.py", line 204, in set_data
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/quota/__init__.py", line 319, in set_data
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi set_active=set_active)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/location.py", line 559, in set_data
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/location.py", line 486, in _upload_to_store
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi verifier=verifier)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/glance_store/backend.py", line 491, in add_to_backend_with_multihash
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/glance_store/backend.py", line 468, in store_add_to_backend_with_multihash
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/glance_store/driver.py", line 279, in add_adapter
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/glance_store/capabilities.py", line 176, in op_checker
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/glance_store/_drivers/cinder/cinder.py", line 780, in add
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi volume_type=volume_type)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/cinderclient/v3/volumes.py", line 125, in create
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi return self._create('/volumes', body, 'volume')
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/cinderclient/base.py", line 302, in _create
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi resp, body = self.api.client.post(url, body=body)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/cinderclient/client.py", line 476, in post
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi return self._cs_request(url, 'POST', **kwargs)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/cinderclient/client.py", line 429, in _cs_request
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi resp, body = self.request(url, method, **kwargs)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi File "/usr/local/lib/python3.6/dist-packages/cinderclient/client.py", line 411, in request
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi raise exceptions.from_response(resp, body)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi cinderclient.exceptions.OverLimit: VolumeLimitExceeded: Maximum number of volumes allowed (10) exceeded for quota 'volumes'. (HTTP 413) (Request-ID: req-44598b3d-fee7-403b-8ce4-a59e695a1953)
Apr 27 15:15:46 victoria-cinder glance-api[29768]: ERROR glance.common.wsgi
Apr 27 15:15:46 victoria-cinder glance-api[29768]: INFO eventlet.wsgi.server [None req-d2dd6858-73c9-405f-9833-1c6142f2d7ab admin admin] 192.168.100.9 - - [27/Apr/2020 15:15:46] "PUT /v2/images/e815975d-0836-40aa-b4b9-9b290e287562/file HTTP/1.1" 500 454 0.164477

[Rajat Dhasmana (whoami-rajat)]

If it's related to devstack environment, the fix has landed in devstack https://review.opendev.org/c/openstack/devstack/+/774172