Hi,
I see the below error during image upload :
2020/10/09 12:04:17 [error] 19#19: *3044066 upstream timed out (110: Operation timed out) while sending request to upstream, client: XXX , server: objectstore-3, request: "PUT /v1/AUTH_53d8861350d946f8a83e4f84f8c951c8/glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8/bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098 HTTP/1.1", upstream: "http://127.0.0.1:8080/v1/AUTH_53d8861350d946f8a83e4f84f8c951c8/glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8/bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098", host: "objectstore-3"
Got error from Swift: put_object('glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8', 'bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098', ...) failure and no ability to reset contents for reupload..: glance_store.exceptions.BackendException: Failed to add object to Swift.
Got error from Swift: put_object('glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8', 'bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098', ...) failure and no ability to reset contents for reupload..
2020-10-09 12:04:38,772.772 44 ERROR glance.common.wsgi swiftclient.exceptions.ClientException: put_object('glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8', 'bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098', ...) failure and no ability to reset contents for reupload.
2020-10-09 12:04:38,772.772 44 ERROR glance.common.wsgi Got error from Swift: put_object('glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8', 'bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098', ...) failure and no ability to reset contents for reupload..
Got error from Swift: put_object('glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8', 'bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098', ...) failure and no ability to reset contents for reupload..: swiftclient.exceptions.ClientException: put_object('glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8', 'bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098', ...) failure and no ability to reset contents for reupload.
2020-10-09 12:04:23,464.464 44 ERROR glance_store._drivers.swift.store [req-4a8ab54a-0826-48d3-8a4d-54c71af100de 16fbf18dd4ee5d415412c19df7d1f3ba6942d6a7f2f58b411815eae206f4a1ca 53d8861350d946f8a83e4f84f8c951c8 - cc7f99d95b14466a84faab103a7b8a52 cc7f99d95b14466a84faab103a7b8a52] Error during chunked upload to backend, deleting stale chunks.: swiftclient.exceptions.ClientException: put_object('glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8', 'bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098', ...) failure and no ability to reset contents for reupload.
Openstack Glance is on Train release
Openstack Glance backend is Swift (Ussuri Release)
This error is generated after 7mins timeout set in swift, this occurs at different chunk# and images, since this error occurs sporadically i am unable to capture tcpdump to determine why glance cannot send data for 7mins or if there is another issue.
Please let me know if further data / information is required.
Regards,
Rajiv
