Noticed during a cinder multistore test run, we hit a quota not found error. It looks like we don't handle this well, which causes nova to see a 503: Proxy Error. I dunno if there's anything better can do than raise a 5xx, but we should probably explain in the error what happened when we know, as we clearly do here.
From this:
https://cbff25b854b00bc0ff99-8ce5690b0835baabd00baac02d43f418.ssl.cf5.rackcdn.com/770629/5/check/glance-multistore-cinder-import/7c71502/controller/logs/screen-g-api.txt
this log text (see the end):
Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi Traceback (most recent call last):
Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/common/wsgi.py", line 1347, in __call__
Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action,
Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/common/wsgi.py", line 1391, in dispatch
Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi return method(*args, **kwargs)
Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/common/utils.py", line 416, in wrapped
Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/api/v2/image_data.py", line 299, in upload
Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi self._restore(image_repo, image)
Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi self.force_reraise()
Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Feb 04 21:07:13.368998 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi raise value
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/api/v2/image_data.py", line 164, in upload
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi image.set_data(data, size, backend=backend)
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/domain/proxy.py", line 208, in set_data
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active)
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/notifier.py", line 501, in set_data
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi self.force_reraise()
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi raise value
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/notifier.py", line 447, in set_data
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend,
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/api/policy.py", line 206, in set_data
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs)
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/quota/__init__.py", line 322, in set_data
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend,
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/location.py", line 567, in set_data
Feb 04 21:07:13.370209 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size)
Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/location.py", line 458, in _upload_to_store
Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash(
Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/glance_store/multi_backend.py", line 398, in add_with_multihash
Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi return store_add_to_backend_with_multihash(
Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/glance_store/multi_backend.py", line 480, in store_add_to_backend_with_multihash
Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add(
Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/glance_store/driver.py", line 279, in add_adapter
Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs)
Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/glance_store/capabilities.py", line 176, in op_checker
Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/glance_store/_drivers/cinder.py", line 852, in add
Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi volume = client.volumes.create(size_gb, name=name, metadata=metadata,
Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/cinderclient/v3/volumes.py", line 125, in create
Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi return self._create('/volumes', body, 'volume')
Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/cinderclient/base.py", line 302, in _create
Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi resp, body = self.api.client.post(url, body=body)
Feb 04 21:07:13.371474 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/cinderclient/client.py", line 485, in post
Feb 04 21:07:13.372519 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi return self._cs_request(url, 'POST', **kwargs)
Feb 04 21:07:13.372519 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/cinderclient/client.py", line 438, in _cs_request
Feb 04 21:07:13.372519 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi resp, body = self.request(url, method, **kwargs)
Feb 04 21:07:13.372519 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi File "/usr/local/lib/python3.8/dist-packages/cinderclient/client.py", line 420, in request
Feb 04 21:07:13.372519 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi raise exceptions.from_response(resp, body)
Feb 04 21:07:13.372519 ubuntu-focal-limestone-regionone-0022873642 <email address hidden>[93292]: ERROR glance.common.wsgi cinderclient.exceptions.OverLimit: VolumeLimitExceeded: Maximum number of volumes allowed (10) exceeded for quota 'volumes'. (HTTP 413) (Request-ID: req-5a3e84d0-70fb-425c-b8b2-fdddc14b7bf1)
Patch proposed https:/