Complete Multipart Upload fails if the file or folder name contains multibyte characters

[Kaneoka Keiji]

I have encountered the following error, when I tried to put objects named or in folder named with multi-byte characters by multipart upload.
According to logs, the PUT of parted object itself is successfully done, but Complete Multipart Upload failed with status 400.

There is no problem found, when the PUT of the same object is not Multipart Upload.
I assume that it is caused by the following process;
swift3-1.12\swift3\controllers\multi_upload.py line 577
                info = objtable.get("%s/%s/%s" % (req.object_name, upload_id,
                                                  part_number))

If I replace the line as follows, the error has gone.
                info = objtable.get(u"%s/%s/%s" % (req.object_name, upload_id,
                                                  part_number))

-------------------------------------------------------------------------------------------------------------------------
note:
Object:test.txt is 20010342 bytes

[root@odddtst105 ver100]# s3cmd -v --multipart-chunk-size-mb=50 -q -c /root/ver100/s3cfg/.s3cfg92sndnp put /root/ver100/test.txt s3://test/テスト/test.txt 2>&1
INFO: No cache file found, creating it.
INFO: Compiling list of local files...
INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
INFO: Summary: 1 local files to upload
[root@odddtst105 ver100]# s3cmd -v --multipart-chunk-size-mb=10 -q -c /root/ver100/s3cfg/.s3cfg92sndnp put /root/ver100/test.txt s3://test/テスト/test.txt 2>&1
INFO: No cache file found, creating it.
INFO: Compiling list of local files...
INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
INFO: Summary: 1 local files to upload
ERROR: S3 error: 400 (InvalidPart): One or more of the specified parts could not be found. The part might not have been uploaded, or the specified entity tag might not have matched the part's entity tag.

***.***.***.***.38772-***.***.***.***.08080: POST /test/%E3%83%86%E3%82%B9%E3%83%88/test.txt?uploadId=MjNhODg4MTYtMzc0OS00NTNiLWI0ZTctZWUyNGMzMjYyZWMx HTTP/1.1
Host: ***.***.***.***:8080
Accept-Encoding: identity
x-amz-content-sha256: 38d4f5b4a294a2d96ce675062e4e8b8a8ce67d6b09c3a0268b19351911ca684c
content-length: 219
Authorization: AWS4-HMAC-SHA256 Credential=1120f817956c4544a0123bf7fdd1a455/20190419/US/s3/aws4_request,SignedHeaders=content-length;host;x-amz-content-sha256;x-amz-date,Signature=1a1c0e047ab46f42cdc56b6b1461fb2e7bb58ce2be366a238a9ca4ba45c7e22c
x-amz-date: 20190419T021444Z

<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>7a56a7fcb0484dc816e6faf0d25ac37e</ETag></Part><Part><PartNumber>2</PartNumber><ETag>d3d1c823d4a4100c43828c6e3f5c3e8b</ETag></Part></CompleteMultipartUpload>
***.***.***.***.08080-***.***.***.***.38772: HTTP/1.1 400 Bad Request
x-amz-id-2: txc5f023c0d005425c87577-005cb92f14
x-amz-request-id: txc5f023c0d005425c87577-005cb92f14
Content-Type: application/xml
X-Trans-Id: txc5f023c0d005425c87577-005cb92f14
X-Openstack-Request-Id: txc5f023c0d005425c87577-005cb92f14
Date: Fri, 19 Apr 2019 02:14:45 GMT
Transfer-Encoding: chunked

1a0
<?xml version='1.0' encoding='UTF-8'?>
<Error><Code>InvalidPart</Code><Message>One or more of the specified parts could not be found. The part might not have been uploaded, or the specified entity tag might not have matched the part's entity tag.</Message><RequestId>txc5f023c0d005425c87577-005cb92f14</RequestId><PartNumber>1</PartNumber><UploadId>MjNhODg4MTYtMzc0OS00NTNiLWI0ZTctZWUyNGMzMjYyZWMx</UploadId></Error>

note:
Object:%E3%83%86%E3%82%B9%E3%83%88 is 20010342 bytes
***.***.***.***.38772-***.***.***.***.08080: POST /test/%E3%83%86%E3%82%B9%E3%83%88?uploadId=MDI5ZjkyMTktMGYzMC00ZDk0LTllNmEtODYwNTJkYzQzZDBh HTTP/1.1
Host: ***.***.***.***:8080
Accept-Encoding: identity
x-amz-content-sha256: 38d4f5b4a294a2d96ce675062e4e8b8a8ce67d6b09c3a0268b19351911ca684c
content-length: 219
Authorization: AWS4-HMAC-SHA256 Credential=1120f817956c4544a0123bf7fdd1a455/20190419/US/s3/aws4_request,SignedHeaders=content-length;host;x-amz-content-sha256;x-amz-date,Signature=b1be9f93fac8ddd07b85ed22848c11825a07b33663bb7c1f0d69624ffa31aa56
x-amz-date: 20190419T052230Z

<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>7a56a7fcb0484dc816e6faf0d25ac37e</ETag></Part><Part><PartNumber>2</PartNumber><ETag>d3d1c823d4a4100c43828c6e3f5c3e8b</ETag></Part></CompleteMultipartUpload>
***.***.***.***.08080-***.***.***.***.56432: HTTP/1.1 400 Bad Request
x-amz-id-2: tx70c7426162f8492ebdec4-005cb95b16
x-amz-request-id: tx70c7426162f8492ebdec4-005cb95b16
Content-Type: application/xml
X-Trans-Id: tx70c7426162f8492ebdec4-005cb95b16
X-Openstack-Request-Id: tx70c7426162f8492ebdec4-005cb95b16
Date: Fri, 19 Apr 2019 05:22:30 GMT
Transfer-Encoding: chunked

1a0
<?xml version='1.0' encoding='UTF-8'?>
<Error><Code>InvalidPart</Code><Message>One or more of the specified parts could not be found. The part might not have been uploaded, or the specified entity tag might not have matched the part's entity tag.</Message><RequestId>tx70c7426162f8492ebdec4-005cb95b16</RequestId><PartNumber>1</PartNumber><UploadId>MDI5ZjkyMTktMGYzMC00ZDk0LTllNmEtODYwNTJkYzQzZDBh</UploadId></Error>

[Kota Tsuyuzaki (tsuyuzaki-kota)]

Just a small comment for this, Swift3 was merged to OpenStack Swift code base at [1] so that currently no one actively maintains *Swift3* itself so that please try the newer Swift and s3api middleware instead of Swift3.

Even, the bug exists still there, reporting at Swift team bug managements are better than leaving this here as it is.
And if you could share more strict information like, version number for Swift and python. That would be helpful to know what's going on.

FWIW, IIRC, Swift3 (and s3api) should handle all path and the name (account, container, object) included in the path as utf8 encoded value so that we should not make it as unicode. if objtable dict handles the keys as unicode, we should change it, just IMO.

Thanks.

1: https://github.com/openstack/swift/blob/master/CHANGELOG#L279-L281

[Kaneoka Keiji (kaneoka)]

Thank you for your promt comment.
As you mentioned, Swift3 has been no longer available and it was merged to Swift. I just forgot it and left this report here. It was not appropreate place to report.

I intend to check newer Swift/S3API and report the error to Swift team with more information if there still exist.

Thanks

[Yuxin Wang (chhyx2008)]

I have also encountered the same error when I was trying to put a large object with a Chinese object name using S3 Multipart Upload.

After the `obj_name` was decoded with `utf-8`, it seemed to work well.

```
                 if not isinstance(obj_name, six.text_type):
                     obj_name = obj_name.decode('utf-8')
```