Hi Team,
I would like to understand why and how does glance-api pod generate HTTP 409 errors, i did look into the available documentation and code but had no conclusion. I referred the below :
https://github.com/openstack/glance/blob/stable/train/doc/source/user/glanceapi.rst
https://github.com/openstack/glance/blob/54329c6a21b0d3f845b09e79f710fc795976a175/releasenotes/source/locale/ja/LC_MESSAGES/releasenotes.po
https://bugs.launchpad.net/glance/+bug/1229823
https://docs.openstack.org/glance/pike/configuration/configuring.html#configuring-the-swift-storage-backend
I wonder the HTTP 409 conflicts occur due to the below flags, are the api calls parallel or sequential for deletion ? i.e. a HTTP 409 means a Conflict response status code, maybe the deletion calls dont wait until the deletion is completed ? :
swift_store_large_object_size = 5120
swift_store_large_object_chunk_size = 512 (default is 200Mb (i dont see HTTP 409), would it generate HTTP 409 if its increased to 500Mb)
glance-api logs :
2020-07-18 01:55:46,627.627 52 ERROR glance.common.wsgi [req-236a9c8c-396e-42a1-8987-f847923c7e13 f1083795e1da57ca00ff8c967ad0c3d80751fe341a1e64046869e0ae0770cc1d 7e49c7a15b4a4f149cae86a0c1366afa - ec213443e8834473b579f7bea9e8c194 ec213443e8834473b579f7bea9e8c194] Caught error: Container DELETE failed: https://xxx:443/v1/AUTH_7e49c7a15b4a4f149cae86a0c1366afa/glance_1352ddc3-12ba-4afe-9c89-304cefd90ef5 409 Conflict [first 60 chars of response] b'<html><h1>Conflict</h1><p>There was a conflict when trying t': swiftclient.exceptions.ClientException: Container DELETE failed: https://xxx:443/v1/AUTH_7e49c7a15b4a4f149cae86a0c1366afa/glance_1352ddc3-12ba-4afe-9c89-304cefd90ef5 409 Conflict [first 60 chars of response] b'<html><h1>Conflict</h1><p>There was a conflict when trying t'
2020-07-18 01:55:46,707.707 52 INFO eventlet.wsgi.server [req-236a9c8c-396e-42a1-8987-f847923c7e13 f1083795e1da57ca00ff8c967ad0c3d80751fe341a1e64046869e0ae0770cc1d 7e49c7a15b4a4f149cae86a0c1366afa - ec213443e8834473b579f7bea9e8c194 ec213443e8834473b579f7bea9e8c194] 10.46.14.92,100.85.0.29 - - [18/Jul/2020 01:55:46] "DELETE /v2/images/1352ddc3-12ba-4afe-9c89-304cefd90ef5 HTTP/1.1" 500 449 11.140969
Openstack Glance Version : Train
Glance-api.conf : https://github.com/sapcc/helm-charts/blob/master/openstack/glance/templates/etc/_glance-api.conf.tpl
Swift-api.conf
Swift conf file : https://github.com/sapcc/helm-charts/blob/master/openstack/swift/templates/etc/_proxy-server.conf.tpl
Glance api logs have been attached.