Comment 11 for bug 1008940

Revision history for this message
gholt (gholt) wrote : Re: Unicode is not currently supported for object copy.

Yeah, I'm pretty sure there's just a misunderstanding somewhere along the line.

According to the code I see in Swift, it has supported Unicode names with COPY/Destination (and PUT/X-Copy-From) for quite some time (since the beginning of OpenStack?) but all strings sent to Swift need to be UTF-8 and URL encoded.

There was a problem with this with the large object support (X-Object-Manifest) but that was fixed about a month and half ago -- not exactly related to this particular bug.

The key here is: Path strings and header values sent to Swift need to be UTF-8 and URL encoded -- sometimes you can get away with not doing it, but it's not a supported thing when you do.

For instance, the previous example should really be using %E5%B9%B8%E7%A6%8F for the Unicode value. The following quick snippet should show this (but I guess it depends on your Terminal; mine UTF-8 encodes strings from the command line):

python -c 'import sys, urllib; print urllib.quote(sys.argv[-1])' 幸福
%E5%B9%B8%E7%A6%8F

With that as the UTF-8/URL-Encoded string, the example should've been:

$ echo test > OBJECT
$ swift upload CONTAINER OBJECT
OBJECT
$ swift post CONTAINER OBJECT -m X-Object-Meta-Orig-Filename:%E5%B9%B8%E7%A6%8F
$ swift stat CONTAINER OBJECT
       Account: AUTH_test
     Container: CONTAINER
        Object: OBJECT
  Content Type: application/octet-stream
Content Length: 5
 Last Modified: Wed, 15 Aug 2012 01:17:08 GMT
          ETag: d8e8fca2dc0f896fd7cb4cb0031ba249
Meta X-Object-Meta-Orig-Filename: %E5%B9%B8%E7%A6%8F
 Accept-Ranges: bytes
   X-Timestamp: 1344993428.16163

Let's do another example with the value as the object name, including a copy/destination and a put/x-copy-from.

$ curl -XPUT http://127.0.0.1:8080/v1/AUTH_test/CONTAINER/%E5%B9%B8%E7%A6%8F -Hx-auth-token:AUTH_tkc10eb54529004c0d91e5ca3310ee7a09 --data-binary 'test'
201 Created
$ curl -XCOPY http://127.0.0.1:8080/v1/AUTH_test/CONTAINER/%E5%B9%B8%E7%A6%8F -Hx-auth-token:AUTH_tkc10eb54529004c0d91e5ca3310ee7a09 -Hdestination:/CONTAINER/copied
201 Created
$ curl -XPUT http://127.0.0.1:8080/v1/AUTH_test/CONTAINER/x-copy-fromed -Hx-auth-token:AUTH_tkc10eb54529004c0d91e5ca3310ee7a09 -Hx-copy-from:/CONTAINER/%E5%B9%B8%E7%A6%8F -Hcontent-length:0
201 Created
$ curl http://127.0.0.1:8080/v1/AUTH_test/CONTAINER?format=json -Hx-auth-token:AUTH_tkc10eb54529004c0d91e5ca3310ee7a09 -s | python -mjson.tool
[
    {
        "bytes": 5,
        "content_type": "application/octet-stream",
        "hash": "d8e8fca2dc0f896fd7cb4cb0031ba249",
        "last_modified": "2012-08-15T01:17:08.161630",
        "name": "OBJECT"
    },
    {
        "bytes": 4,
        "content_type": "application/x-www-form-urlencoded",
        "hash": "098f6bcd4621d373cade4e832627b4f6",
        "last_modified": "2012-08-15T01:30:58.755590",
        "name": "copied"
    },
    {
        "bytes": 4,
        "content_type": "application/x-www-form-urlencoded",
        "hash": "098f6bcd4621d373cade4e832627b4f6",
        "last_modified": "2012-08-15T01:32:51.507340",
        "name": "x-copy-fromed"
    },
    {
        "bytes": 4,
        "content_type": "application/x-www-form-urlencoded",
        "hash": "098f6bcd4621d373cade4e832627b4f6",
        "last_modified": "2012-08-15T01:26:36.678980",
        "name": "\u5e78\u798f"
    }
]