Deleting a key using S3 API throws an exception

[arpit]

When doing a delete using s3 API ,it first delete the key from swift but later throws an exception with message "The specified key
does not exist"

[Kota Tsuyuzaki (tsuyuzaki-kota)]

Do you mean a DELETE request for deleted object results in showing "The specified key does not exist"?

It sounds correct because if the specified key doesn't exist (somewhere, deleted correctly), S3 will return 404 Not Found with the body, "The specified key does not exist".

The error handing, whether regarding 404 as an error or not, depends on your client, though.

In detail:
http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

[arpit (erarpitagarwal)]

Kota, I am getting the exception even in the case when the keys exists on the swift server.

 So for an existing key on Swift server , the delete call will delete the file and then in the same call throws the "Key not found" exception".

[Kota Tsuyuzaki (tsuyuzaki-kota)]

On your case, perhaps the log of proxy-server show something like "Handoff requested", right?

Swift (<=2.1.0) had a chance 404 Not found when keys exist but some failures occurred. It will result in 404 Not Found at Swift3 as well as backend Swift.

If you uses newer version than above and there is no log for failures on proxy-node, please tell me your environment information (e.g. Swift version, Swift3 version and so on) and the reproduce process for the bug.

Thanks.

[arpit (erarpitagarwal)]

{code}

Jul 21 20:18:57 localhost account-server: 172.31.26.87 - - [21/Jul/2015:20:18:57 +0000] "HEAD /d0/3064/rubrik_key" 404 - "HEAD http://swift2.rubrik.com/v1/rubrik_key" "txfbc561e87747499c932a2-0055aea931" "proxy-server 27447" 0.0004 "-" 27374 -
Jul 21 20:18:57 localhost proxy-server: - - 21/Jul/2015/20/18/57 HEAD /v1/rubrik_key HTTP/1.0 204 - Swift - - - - txfbc561e87747499c932a2-0055aea931 - 0.0024 RL - 1437509937.908023119 1437509937.910382986 -
Jul 21 20:18:57 localhost container-server: 172.31.26.87 - - [21/Jul/2015:20:18:57 +0000] "DELETE /d0/52693/AUTH_arpit.agarwal/bucket1/rubrik_key" 204 - "DELETE http://swift2.rubrik.com/d0/63520/AUTH_arpit.agarwal/bucket1/rubrik_key" "txfbc561e87747499c932a2-0055aea931" "object-server 31362" 0.0004 "-" 31286 0
Jul 21 20:18:57 localhost object-server: 172.31.26.87 - - [21/Jul/2015:20:18:57 +0000] "DELETE /d0/63520/AUTH_arpit.agarwal/bucket1/rubrik_key" 204 - "DELETE http://swift2.rubrik.com/v1/AUTH_arpit.agarwal/bucket1/rubrik_key" "txfbc561e87747499c932a2-0055aea931" "proxy-server 27447" 0.0068 "-" 31362 0
Jul 21 20:18:57 localhost proxy-server: 127.0.0.1 127.0.0.1 21/Jul/2015/20/18/57 DELETE /bucket1/rubrik_key HTTP/1.0 204 - aws-sdk-java/1.9.34%20Linux/3.16.0-30-generic%20Java_HotSpot%28TM%29_64-Bit_Server_VM/24.80-b11/1.7.0_80%20com.amazonaws.services.s3.AmazonS3EncryptionClient/1.9.34 - - - - txfbc561e87747499c932a2-0055aea931 - 0.0132 - - 1437509937.907640934 1437509937.920881033 -
Jul 21 20:18:57 localhost account-server: 172.31.26.87 - - [21/Jul/2015:20:18:57 +0000] "HEAD /d0/60499/rubrik_key.instruction" 404 - "HEAD http://swift2.rubrik.com/v1/rubrik_key.instruction" "tx1e3b5cec22c94da4b8793-0055aea931" "proxy-server 27447" 0.0003 "-" 27374 -
Jul 21 20:18:57 localhost proxy-server: - - 21/Jul/2015/20/18/57 HEAD /v1/rubrik_key.instruction HTTP/1.0 204 - Swift - - - - tx1e3b5cec22c94da4b8793-0055aea931 - 0.0021 RL - 1437509937.931546926 1437509937.933619022 -
Jul 21 20:18:57 localhost container-server: 172.31.26.87 - - [21/Jul/2015:20:18:57 +0000] "DELETE /d0/52693/AUTH_arpit.agarwal/bucket1/rubrik_key.instruction" 204 - "DELETE http://swift2.rubrik.com/d0/57530/AUTH_arpit.agarwal/bucket1/rubrik_key.instruction" "tx1e3b5cec22c94da4b8793-0055aea931" "object-server 31362" 0.0004 "-" 31286 0
Jul 21 20:18:57 localhost object-server: 172.31.26.87 - - [21/Jul/2015:20:18:57 +0000] "DELETE /d0/57530/AUTH_arpit.agarwal/bucket1/rubrik_key.instruction" 404 - "DELETE http://swift2.rubrik.com/v1/AUTH_arpit.agarwal/bucket1/rubrik_key.instruction" "tx1e3b5cec22c94da4b8793-0055aea931" "proxy-server 27447" 0.0063 "-" 31362 0
Jul 21 20:18:57 localhost proxy-server: 127.0.0.1 127.0.0.1 21/Jul/2015/20/18/57 DELETE /bucket1/rubrik_key.instruction HTTP/1.0 404 - aws-sdk-java/1.9.34%20Linux/3.16.0-30-generic%20Java_HotSpot%28TM%29_64-Bit_Server_VM/24.80-b11/1.7.0_80 - - 218 - tx1e3b5cec22c94da4b8793-0055aea931 - 0.0122 - - 1437509937.931221008 1437509937.943407059 -

{code}

I am seeing the above log lines when I try to delete a key from swift server.

the swift version I am running is swift 2.4.0.1 and swift 3 version is 1.7.0.7. I am able to reproduce the above error by simply calling delete on AmazonS3C...

[Kota Tsuyuzaki (tsuyuzaki-kota)]

Interesting. You can use Swift 2.3.0 as the current newest one unless you have a time machine.

And I have a question, why "s3.deleteObject(bucketName, key)" causes 2 requests for proxy server, one is for "rubrik_key" and the other is for "rubrik_key.instruction".

Your log seems to show the delete request for "rubrik_key" succeeded as 204. This seems no problem. However the following request for "rubik_key.instruction" returns 404. It will make 404 Not Found response from Swfit3 as well as backend Swift.

I'm not sure because I'm not well about aws Java sdk but it seems a problem at client side (if you requested delete object just for "rubrik_key".)

In addition, Swift3 v1.7 is stale anymore (release was 2 years ago), could you please test again on v1.8?

[arpit (erarpitagarwal)]

I am using Swiftstack which supposedly is running swift 2.4.*.*

Also I have tested this against swift3 (1.8.0.dev8.ge1ac214) on swift version 2.3.0

[arpit (erarpitagarwal)]

Also the files that I am uploading are encrypted files which are being encrypted by s3 client and I think S3 client stores some metadat a regarding encryption in the instruction file. I have verified that I am not getting this error if I use the non encrypted verison of s3 client.