Failed to add object to Swift. causes glance to return 503

[Joe Gordon]

http://logs.openstack.org/23/49623/3/check/check-tempest-devstack-vm-neutron/8a1a42d/logs/screen-g-api.txt.gz?#_2013-10-04_16_31_15_067

2013-10-04 16:31:15.067 32574 ERROR glance.store.swift [7ba13d89-a2f1-4ed6-96f9-013286e74edc cafcbf642bd4498b9ca6aa7f387af5ee 03102de58497423db8082bd91188d60a] Failed to add object to Swift.
Got error from Swift: put_object('glance', '6d938c1c-3295-4ac4-81af-630b6b906f46', ...) failure and no ability to reset contents for reupload.
2013-10-04 16:31:15.067 32574 ERROR glance.api.v1.upload_utils [7ba13d89-a2f1-4ed6-96f9-013286e74edc cafcbf642bd4498b9ca6aa7f387af5ee 03102de58497423db8082bd91188d60a] Failed to upload image 6d938c1c-3295-4ac4-81af-630b6b906f46
2013-10-04 16:31:15.067 32574 TRACE glance.api.v1.upload_utils Traceback (most recent call last):
2013-10-04 16:31:15.067 32574 TRACE glance.api.v1.upload_utils File "/opt/stack/new/glance/glance/api/v1/upload_utils.py", line 101, in upload_data_to_store
2013-10-04 16:31:15.067 32574 TRACE glance.api.v1.upload_utils store)
2013-10-04 16:31:15.067 32574 TRACE glance.api.v1.upload_utils File "/opt/stack/new/glance/glance/store/__init__.py", line 333, in store_add_to_backend
2013-10-04 16:31:15.067 32574 TRACE glance.api.v1.upload_utils (location, size, checksum, metadata) = store.add(image_id, data, size)
2013-10-04 16:31:15.067 32574 TRACE glance.api.v1.upload_utils File "/opt/stack/new/glance/glance/store/swift.py", line 447, in add
2013-10-04 16:31:15.067 32574 TRACE glance.api.v1.upload_utils raise glance.store.BackendException(msg)
2013-10-04 16:31:15.067 32574 TRACE glance.api.v1.upload_utils BackendException: Failed to add object to Swift.
2013-10-04 16:31:15.067 32574 TRACE glance.api.v1.upload_utils Got error from Swift: put_object('glance', '6d938c1c-3295-4ac4-81af-630b6b906f46', ...) failure and no ability to reset contents for reupload.
2013-10-04 16:31:15.067 32574 TRACE glance.api.v1.upload_utils

logstash.openstack.org shows at least 4 cases of this in the past week

http://logstash.openstack.org/#eyJzZWFyY2giOiJAbWVzc2FnZTpcIkdvdCBlcnJvciBmcm9tIFN3aWZ0OiBwdXRfb2JqZWN0XCIgQU5EIEBmaWVsZHMuZmlsZW5hbWU6XCJsb2dzL3NjcmVlbi1nLWFwaS50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4MDkwOTQ1MDAwMCwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

[Joe Gordon (jogo)]

Discovered in https://review.openstack.org/#/c/49623/3/

[Erno Kuvaja (jokke)]

IMHO 503 "Service Unavailable" is the right response at this scenario, could this be closed off?

[Brian Elliott (belliott)]

This is the associated traceback:

2014-08-18 00:36:38.165 32653 ERROR glance.api.v1.upload_utils [45bf8677-93da-4b43-a1d3-7080cd85479f 174895 665889] Failed to upload image 9ad21fc1-5a50-4604-8056-23f593d2eb52
2014-08-18 00:36:38.165 32653 TRACE glance.api.v1.upload_utils Traceback (most recent call last):
2014-08-18 00:36:38.165 32653 TRACE glance.api.v1.upload_utils File "/opt/rackstack/615.44/glance/lib/python2.6/site-packages/glance/api/v1/upload_utils.py", line 99, in upload_data_to_store
2014-08-18 00:36:38.165 32653 TRACE glance.api.v1.upload_utils store)
2014-08-18 00:36:38.165 32653 TRACE glance.api.v1.upload_utils File "/opt/rackstack/615.44/glance/lib/python2.6/site-packages/glance/store/__init__.py", line 366, in store_add_to_backend
2014-08-18 00:36:38.165 32653 TRACE glance.api.v1.upload_utils (location, size, checksum, metadata) = store.add(image_id, data, size)
2014-08-18 00:36:38.165 32653 TRACE glance.api.v1.upload_utils File "/opt/rackstack/615.44/glance/lib/python2.6/site-packages/glance/store/swift.py", line 503, in add
2014-08-18 00:36:38.165 32653 TRACE glance.api.v1.upload_utils raise glance.store.BackendException(msg)
2014-08-18 00:36:38.165 32653 TRACE glance.api.v1.upload_utils BackendException: Failed to add object to Swift.
2014-08-18 00:36:38.165 32653 TRACE glance.api.v1.upload_utils Got error from Swift: put_object('ord-images', '9ad21fc1-5a50-4604-8056-23f593d2eb52-00080', ...) failure and no ability to reset contents for reupload.
2014-08-18 00:36:38.165 32653 TRACE glance.api.v1.upload_utils

[Brian Elliott (belliott)]

What's happening here is an error during the upload of a segment (chunk) of a large object to Swift. A single error response to a put_object request will trigger a failure to add the object. Glance should be resilient enough to tolerate and recover from a single chunk upload failure without aborting the entire upload that could have been in progress for a long time.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/115111

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/120866

[OpenStack Infra (hudson-openstack)]

Change abandoned by Brian Elliott (<email address hidden>) on branch: master
Review: https://review.openstack.org/115111
Reason: Superseded by: https://review.openstack.org/#/c/120866/

[Ian Cordasco (icordasc)]

This hasn't been updated in over 2 years. I'm going to close this in a week if no one can confirm this is still a valid issue for glance_store.

[Ian Cordasco (icordasc)]

Closing as promised