Encryption writes different paths for key_id on py2 and py3
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
Fix Released
|
High
|
Unassigned |
Bug Description
On py2, I see crypto meta like
Data crypto details: {
"body_key": {
"iv": "P8mgGrapX7D1+
"key": "mUp+9gJVOgJbsh
},
"cipher": "AES_CTR_256",
"iv": "at+NLMIYulxU5+
"key_id": {
"path": "/AUTH_
"secret_id": "2018",
"v": "2"
}
}
but on py3, it looks like
Data crypto details: {
"body_key": {
"iv": "tthtblP9/
"key": "pmkcVNE6wpo1f1
},
"cipher": "AES_CTR_256",
"iv": "q+D9/nJ3c/
"key_id": {
"path": "/AUTH_
"secret_id": "2018",
"v": "2"
}
}
That is, we passed the WSGI string to json.dumps(). Of course, this is going to cause upgrade issues when moving from py2 to py3. If you've got data written down on py2, upgrading your proxy to py3 gives tracebacks like
Jul 18 01:37:56 saio-py2 proxy-server: Path stored in meta ('/AUTH_test/c/🌴') does not match path from request ('/AUTH_
Jul 18 01:37:56 saio-py2 proxy-server: get_keys(): from callback: 'latin-1' codec can't encode character '\U0001f334' in position 13: ordinal not in range(256):
Traceback (most recent call last):
File "/usr/local/
keys = fetch_crypto_
File "/usr/local/
path, secret_
File "/usr/local/
return hmac.new(key, wsgi_to_
File "/usr/local/
return wsgi_str.
UnicodeEncodeError: 'latin-1' codec can't encode character '\U0001f334' in position 13: ordinal not in range(256) (txn: txc22c1da047394
Also worth noting: in a mixed py2/py3 cluster, data written through a proxy on py3 may be returned corrupted when read on py2:
vagrant@ saio:~/ swift$ curl -H x-auth- token:$ OS_AUTH_ TOKEN http:// saio/v1/ AUTH_test/🌴/🌴 -v | hd | head -n 2
Dload Upload Total Spent Left Speed token:AUTH_ tkb0683aed8c044 d12a4e02a9f678d d2d9 ���);�� ����v,: ���M��Ӻ" Meta-Mtime: c�V����I��T��+. octet-stream 1a0901b1- 005f15cddd Request- Id: tx0d09115108594 1a0901b1- 005f15cddd
% Total % Received % Xferd Average Speed Time Time Time Current
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 127.0.1.1...
* TCP_NODELAY set
* Connected to saio (127.0.1.1) port 80 (#0)
> GET /v1/AUTH_test/🌴/🌴 HTTP/1.1
> Host: saio
> User-Agent: curl/7.58.0
> Accept: */*
> x-auth-
>
< HTTP/1.1 200 OK
< Etag: "����C)
< X-Object-
< Content-Length: 4788
< Accept-Ranges: bytes
< Last-Modified: Mon, 20 Jul 2020 16:46:22 GMT
< X-Timestamp: 1595263581.06434
< Content-Type: application/
< X-Trans-Id: tx0d09115108594
< X-Openstack-
< Date: Mon, 20 Jul 2020 17:01:17 GMT
<
{ [4788 bytes data]
00000000 bd e4 ea 56 2e 0a fd 2f 8d 70 b5 d3 e8 42 c6 93 |...V.../.p...B..|
00000010 35 bb bc 0f 3d 3c 4b c4 54 e7 6f d2 27 41 3f d8 |5...=<K.T.o.'A?.|
100 4788 100 4788 0 0 34446 0 --:--:-- --:--:-- --:--:-- 34446