container-server and account-server fail to deal with unicode key in metadata

[Fabien Boucher]

I don't know whether putting unicode meta key on account or on container is known
to work but I've experienced some problems.

Here is a test script that generate failure on container server
http://pastie.org/private/cr21pjzlczqelveycbspa. Test 4 and 6 fail on my swift installation (devstack).

Using python-swiftclient, I can set meta data on objects and retrieve meta with HEAD or GET even if meta data
key is unicode, but when I try to do the same on container or account the PUT or
POST with meta data (when key is unicode) work but next a HEAD or GET on it
return a 503 error from proxy server.

On client side I get the following traceback:
Traceback (most recent call last):
  File "/home/fabien/git/morucci/python-swiftclient/mytest/test_unicode_meta.py", line 57, in test_06_kvmeta_utf8_container
    self.c.head_container(self.cont)
  File "/home/fabien/venv-openstack-tools/local/lib/python2.7/site-packages/swiftclient/client.py", line 1043, in head_container
    return self._retry(None, head_container, container)
  File "/home/fabien/venv-openstack-tools/local/lib/python2.7/site-packages/swiftclient/client.py", line 1000, in _retry
    rv = func(self.url, self.token, *args, **kwargs)
  File "/home/fabien/venv-openstack-tools/local/lib/python2.7/site-packages/swiftclient/client.py", line 557, in head_container
    http_response_content=body)
ClientException: Container HEAD failed: http://192.168.56.102:8080/v1/AUTH_739d46c58fe149f9a296fa94b81d3e48/MBMCQSJ 503 Internal Server Error

And from the server side I get the following:
Apr 24 10:43:30 devstack-local container-server 127.0.0.1 - - [24/Apr/2013:08:43:30 +0000] "HEAD /sdb1/62/AUTH_17beb1fd308b40e29d18aaa40ca26f91/NEMPVKN" 204 - "tx18b22b589d3949c6bc10c38ed2d31797" "-" "-" 0.0042
Apr 24 10:43:30 devstack-local proxy-server ERROR 500 From Container Server 127.0.0.1:6011/sdb1 (txn: tx18b22b589d3949c6bc10c38ed2d31797) (client_ip: 192.168.56.1)
Apr 24 10:43:30 devstack-local proxy-server Container HEAD returning 503 for [500] (txn: tx18b22b589d3949c6bc10c38ed2d31797) (client_ip: 192.168.56.1)

Or sometime:
Apr 24 10:44:03 devstack-local container-server 127.0.0.1 - - [24/Apr/2013:08:44:03 +0000] "HEAD /sdb1/110/AUTH_17beb1fd308b40e29d18aaa40ca26f91/TFOEXSP" 204 - "tx13857280b2e946d9bca3397f24a37ae3" "-" "-" 0.0038
Apr 24 10:44:03 devstack-local proxy-server ERROR with Container server 127.0.0.1:6011/sdb1 re: Trying to HEAD /v1/AUTH_17beb1fd308b40e29d18aaa40ca26f91/TFOEXSP: #012Traceback (most recent call last):#012 File "/opt/stack/swift/swift/proxy/controllers/base.py", line 795, in GETorHEAD_base#012 possible_source = conn.getresponse()#012 File "/opt/stack/swift/swift/common/bufferedhttp.py", line 102, in getresponse#012 response = HTTPConnection.getresponse(self)#012 File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse#012 response.begin()#012 File "/usr/lib/python2.7/httplib.py", line 407, in begin#012 version, status, reason = self._read_status()#012 File "/usr/lib/python2.7/httplib.py", line 371, in _read_status#012 raise BadStatusLine(line)#012BadStatusLine: '' (txn: tx13857280b2e946d9bca3397f24a37ae3) (client_ip: 192.168.56.1)
Apr 24 10:44:03 devstack-local proxy-server Container HEAD returning 503 for [] (txn: tx13857280b2e946d9bca3397f24a37ae3) (client_ip: 192.168.56.1)

As you can see from server side logs we can't get a meaningful error, so I've tried to sniff the
connection between container-server and proxy-server and I got the following :
# container-server HEAD response grabbed with tshark
#sudo tshark -i lo -f "not port 22 and port 6011" -V -O http -w /tmp/a
#0040 37 e6 48 54 54 50 2f 31 2e 31 20 35 30 30 20 49 7.HTTP/1 .1 500 I
#0050 6e 74 65 72 6e 61 6c 20 53 65 72 76 65 72 20 45 nternal Server E
#0060 72 72 6f 72 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e rror..Co nnection
#0070 3a 20 63 6c 6f 73 65 0d 0a 43 6f 6e 74 65 6e 74 : close. .Content
#0080 2d 74 79 70 65 3a 20 74 65 78 74 2f 70 6c 61 69 -type: t ext/plai
#0090 6e 0d 0a 43 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 n..Conte nt-lengt
#00a0 68 3a 20 39 38 0d 0a 44 61 74 65 3a 20 4d 6f 6e h: 98..D ate: Mon
#00b0 2c 20 32 32 20 41 70 72 20 32 30 31 33 20 30 35 , 22 Apr 2013 05
#00c0 3a 30 35 3a 30 39 20 47 4d 54 0d 0a 0d 0a 49 6e :05:09 G MT....In
#00d0 74 65 72 6e 61 6c 20 53 65 72 76 65 72 20 45 72 ternal S erver Er
#00e0 72 6f 72 3a 20 77 73 67 69 20 61 70 70 6c 69 63 ror: wsg i applic
#00f0 61 74 69 6f 6e 20 70 61 73 73 65 64 20 61 20 75 ation pa ssed a u
#0100 6e 69 63 6f 64 65 20 6f 62 6a 65 63 74 20 74 6f nicode o bject to
#0110 20 74 68 65 20 73 65 72 76 65 72 20 69 6e 73 74 the ser ver inst
#0120 65 61 64 20 6f 66 20 61 20 73 74 72 69 6e 67 2e ead of a string.

I open the bug against swift as I never seen anywhere in documentation/code that adding utf-8 key on container
or account is not possible. I've tried to fix that behavior by the following patch http://pastie.org/private/wngnbalo29eenu8cuffow
and now swift passes successfully my tests, however I've not submitted that patch to review system as it brake some tests from
swift unit tests suite.

[clayg (clay-gerrard)]

Every time I try to figure out what the specification says about character encoding of Header Keys - I get confused :(

Either way, this is unfortunate:

# account works
clayg@swift:~$ swift stat -v
StorageURL: http://localhost:8080/v1/AUTH_test
Auth Token: AUTH_tkbccad49a6c8d4179995310859247b290
   Account: AUTH_test
Containers: 0
   Objects: 0
     Bytes: 0
Accept-Ranges: bytes
X-Timestamp: 1366816607.59358
Content-Type: text/plain; charset=utf-8

# post a meta key with a non-asci char
clayg@swift:~$ swift post -m x-ݢ-x:test

# my account broke!
clayg@swift:~$ curl -H 'x-auth-token: AUTH_tkbccad49a6c8d4179995310859247b290' http://localhost:8080/v1/AUTH_test -I
HTTP/1.1 503 Internal Server Error
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Wed, 24 Apr 2013 15:17:29 GMT

# eventlet is being snobby
clayg@swift:~$ curl http://127.0.0.1:6022/sdb2/160771/AUTH_test -v
* About to connect() to 127.0.0.1 port 6022 (#0)
* Trying 127.0.0.1... connected
> GET /sdb2/160771/AUTH_test HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: 127.0.0.1:6022
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Connection: close
< Content-type: text/plain
< Content-length: 98
< Date: Wed, 24 Apr 2013 15:17:44 GMT
<
* Closing connection #0
Internal Server Error: wsgi application passed a unicode object to the server instead of a string.

# account server thinks everything is peachy!
clayg@swift:~$ tail -n1 /var/log/syslog
Apr 24 08:17:44 swift account-server 127.0.0.1 - - [24/Apr/2013:15:17:44 +0000] "GET /sdb2/160771/AUTH_test" 204 - "-" "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3" 0.0012 ""

I wonder if the better place to fix this isn't in the broker? Do the sqlite bindings always return unicode? ... From the pasted patch I'm assuming eventlet is fine with non-ascii chars as long as they're in an encoded bytestring.

[clayg (clay-gerrard)]

Oh, it's json, lawl

    @property
    def metadata(self):
        """
        Returns the metadata dict for the database. The metadata dict values
        are tuples of (value, timestamp) where the timestamp indicates when
        that key was set to that value.
        """
        with self.get() as conn:
            try:
                metadata = conn.execute('SELECT metadata FROM %s_stat' %
                                        self.db_type).fetchone()[0]
            except sqlite3.OperationalError, err:
                if 'no such column: metadata' not in str(err):
                    raise
                metadata = ''
        if metadata:
            metadata = json.loads(metadata)
        else:
            metadata = {}
        return metadata

So it's just utf-8 in and unicode out, just encode on the way out - should be gravy.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/27446

[Fabien Boucher (fabien-boucher)]

Ok, thanks for helping, I've just send a review that encode unicode keys in metadata dictionary to utf-8 and this one pass unit test suite.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/27446
Committed: http://github.com/openstack/swift/commit/e7ce2909048fefc2d53c9278cdab23aa86b5f257
Submitter: Jenkins
Branch: master

commit e7ce2909048fefc2d53c9278cdab23aa86b5f257
Author: Fabien Boucher <email address hidden>
Date: Wed Apr 24 23:05:28 2013 +0200

    Metadata retrieving from sqlite must be str for key

    Encode metadata key as utf8 if key is unicode when
    retriving it from sqlite database.

    Change-Id: I4ba11543d1bed17098b5e52dd768c75b403188a1
    Fixes: bug #1172202