BackendException: Failed to add object to Swift. - While trying to upload image size of around 400 MB

[Sukesh Behera]

2017-03-15 22:36:29.268 418 ERROR glance_store._drivers.swift.store [req-96a417c4-7dcf-4406-a2d0-54681b54cca7 33efe02c6fbf41afaaf911f811444fd0 ff56560910c64b68b540eb0bd47d76aa - default default] Failed
to add object to Swift.
Got error from Swift: put_object('glance_images', '00840e88-2589-4543-a5dd-e2b35d0f0440-00001', ...) failure and no ability to reset contents for reupload..
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data [req-96a417c4-7dcf-4406-a2d0-54681b54cca7 33efe02c6fbf41afaaf911f811444fd0 ff56560910c64b68b540eb0bd47d76aa - default default] Failed to upload
 image data due to internal error
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data Traceback (most recent call last):
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 115, in upload
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data image.set_data(data, size)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/domain/proxy.py", line 195, in set_data
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data self.base.set_data(data, size)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/notifier.py", line 479, in set_data
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data _send_notification(notify_error, 'image.upload', msg)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data self.force_reraise()
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data six.reraise(self.type_, self.value, self.tb)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/notifier.py", line 426, in set_data
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data self.repo.set_data(data, size)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/api/policy.py", line 185, in set_data
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data return self.image.set_data(*args, **kwargs)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/quota/__init__.py", line 304, in set_data
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data self.image.set_data(data, size=size)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/location.py", line 440, in set_data
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data verifier=verifier)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance_store/backend.py", line 447, in add_to_backend
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data verifier)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance_store/backend.py", line 420, in store_add_to_backend
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data verifier=verifier)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance_store/capabilities.py", line 225, in op_checker
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data return store_op_fun(store, *args, **kwargs)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance_store/_drivers/swift/store.py", line 966, in add
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data raise glance_store.BackendException(msg)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data BackendException: Failed to add object to Swift.
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data Got error from Swift: put_object('glance_images', '00840e88-2589-4543-a5dd-e2b35d0f0440-00001', ...) failure and no ability to reset contents f
or reupload..
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi [req-96a417c4-7dcf-4406-a2d0-54681b54cca7 33efe02c6fbf41afaaf911f811444fd0 ff56560910c64b68b540eb0bd47d76aa - default default] Caught error: Failed t
o add object to Swift.
Got error from Swift: put_object('glance_images', '00840e88-2589-4543-a5dd-e2b35d0f0440-00001', ...) failure and no ability to reset contents for reupload..
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi Traceback (most recent call last):
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/common/wsgi.py", line 1113, in __call__
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi request, **action_args)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/common/wsgi.py", line 1152, in dispatch
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi return method(*args, **kwargs)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/common/utils.py", line 363, in wrapped
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 250, in upload
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi self._restore(image_repo, image)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi self.force_reraise()
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 115, in upload
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi image.set_data(data, size)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/domain/proxy.py", line 195, in set_data
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi self.base.set_data(data, size)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/notifier.py", line 479, in set_data
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi self.force_reraise()
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/notifier.py", line 426, in set_data
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi self.repo.set_data(data, size)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/api/policy.py", line 185, in set_data
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/quota/__init__.py", line 304, in set_data
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi self.image.set_data(data, size=size)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/location.py", line 440, in set_data
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi verifier=verifier)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance_store/backend.py", line 447, in add_to_backend
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi verifier)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance_store/backend.py", line 420, in store_add_to_backend
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi verifier=verifier)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance_store/capabilities.py", line 225, in op_checker
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance_store/_drivers/swift/store.py", line 966, in add
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi raise glance_store.BackendException(msg)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi BackendException: Failed to add object to Swift.
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi Got error from Swift: put_object('glance_images', '00840e88-2589-4543-a5dd-e2b35d0f0440-00001', ...) failure and no ability to reset contents for reu
pload..
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi

When I try to upload an image of lesser size (around 30 MB), it get thru successfully. I am not sure what might be causing this failure. Any timeout set anywhere?

This is my glance-api.conf looks like :

[glance_store]
default_store = swift

stores = swift,http,cinder

swift_store_config_file = /etc/glance/glance-swift-store.conf
default_swift_reference = swift1
swift_store_auth_insecure = True
swift_store_region = RegionOne
swift_store_container = glance_images
swift_store_create_container_on_put = True
swift_store_large_object_size = 5120
swift_store_large_object_chunk_size = 2048
swift_store_retry_get_count = 5
swift_store_endpoint_type = publicURL

[Brian Rosmaita (brian-rosmaita)]

Is this still occurring? We haven't heard about it from anyone else, and have not seen it ourselves. Have been able to upload 2GB images in recent devstack with swift backend.

[Rajiv Mucheli (rajiv.mucheli)]

I keep getting this error irrespective image sizes, logs are similar as well. Any suggestions on how to fix this ?

[Oliver (ciwi)]

I'm getting this error for snapshots above a certain size, e.g. no error when taking a snapshot of a 'fresh' Ubuntu 20.04 instance but it occurs after 'apt upgrade' was run on the instance. I'm using OpenStack Victoria.

[Harald Kornmayer (hakor)]

Somehow connected to the error above.
When creating an Image, we reach the function add() of class class BaseStore(driver.Store) in the file store.py.
- We get for the parameter "image_size" the value 0.

- We use swift as the backend for our image/snapshot files.

- We can create images with a size below swift_store_large_object_size, as the service will save the new image as one file/chunk

- But for image with a size above, the function fails, as it can not determine the number of chunks due to image_size parameter of 0(zero)

- We did not find a "workaround" to determine the correct image_size within the function add().

As we need to fix this urgently.

* which function is calling the BaseStore.add() function?
* were is the image_size calculated? where does the value originate from?

Any hint is welcome!

[Erno Kuvaja (jokke)]

I'm pretty sure this is related to https://bugs.launchpad.net/glance/+bug/1916052 and the whole trust mechanism being broken is causing any over 1 segment uploads to swift fail.