s3: Large images uploaded but then deleted

[Thaddeus Fortenberry]

We are trying to upload images to an on-premise minio-based S3 compatible storage cluster. We have been using it for ~30GB images for months. However when increasing the size to >30GB, the image is successfully uploaded and then glance seems to tell it to delete.

Successful log looks like this:
Sep 6, 2022 @ 08:40:22.812 1 1 rPR1E4MBBdpF5mC-rsYE test_logs - ttcloud-image ttcloud-image CompleteMultipartUpload CompleteMultipartUpload
Sep 6, 2022 @ 08:39:39.038 1 1 q_R1E4MBBdpF5mC-A8YF test_logs - ttcloud-image ttcloud-image PutObjectPart PutObjectPart
...
Whole bunch of PutObjectPart...

Failure log looks like this:
Sep 2, 2022 @ 18:11:46.067 1 1 PvTnAIMBBdpF5mC-XLv7 test_logs - ttcloud-image ttcloud-image DeleteObject DeleteObject
Sep 2, 2022 @ 18:11:45.987 1 1 PfTnAIMBBdpF5mC-XLur test_logs - ttcloud-image ttcloud-image HeadObject HeadObject
Sep 2, 2022 @ 18:11:45.927 1 1 PPTnAIMBBdpF5mC-XLtw test_logs - ttcloud-image ttcloud-image CompleteMultipartUpload CompleteMultipartUpload
Sep 2, 2022 @ 18:10:49.370 1 1 O_TmAIMBBdpF5mC-f7uE test_logs - ttcloud-image ttcloud-image PutObjectPart PutObjectPart

We were able to get the 38GB image to work by increasing the chunk size from 10MB to 100MB. However this does not help the 50GB or 100GB.

The storage system is correctly responding to all S3 request with either 200 or 204 replies. It appears that something important is happening with the HeadObject which is then causing glance to send a DeleteObject.

I've attached shortened versions of the S3 logs - I've deleted the majority of PutObjectPart entries to keep the size down.

We're not finding any information. What is causing this behavior?

[Brian Rosmaita (brian-rosmaita)]

I suspect this is a glance_store issue, but keeping both components until more research is done.