Retrieving lists of objects using aws-cli fails with error Incorrect Padding
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
New
|
Undecided
|
Unassigned |
Bug Description
I uploaded around 2000 files in one bucket to the swift using aws-cli, but when I try to list them I only get 1000 and after that the request fails.
<Error>
The swift-proxy-server logs show:
Incorrect padding: #012Traceback (most recent call last):#012 File "/usr/lib/
The issue is due to the fact that continuation-token calculated by swift3 is
MjAwMGNvdW50L2Z
but the continuation-token received by the swift3 in the next request is
MjAwMGNvdW50L2Z
This is happening because continuation-token is sent as query param and "==" gets converted into 0x3d hex.
We need to replace %xx in the continuation-token by using unquote.
I think I see how this could happen for swift3: When doing a ?encoding-type=url query, we used to quote() darn near everything [1] -- so the client would get back a response containing something like
<NextContinuat ionToken> MjAwMGNvdW50L2Z pbGUuMTg5OQ% 3D%3D</ NextContinuatio nToken>
which the client dutifully encodes to
?continuation- token=MjAwMGNvd W50L2ZpbGUuMTg5 OQ%253D% 253D
so even when eventlet unquote()s the QUERY_STRING per spec, we get the escaped version.
Two bits of good news: nToken, everything will Just Work; the equals-signs aren't themselves a problem.
- If we stop quoting the NextContinuatio
- We've already done that in s3api [2] -- which explains why I haven't been able to repro the problem on tip-of-master swift.
What version of swift are you using? Can you upgrade to 2.20.0 or later and switch from swift3 to s3api?
[1] https:/ /github. com/openstack/ swift3/ blob/master/ swift3/ etree.py# L85-L103 /github. com/openstack/ swift/commit/ c1c65a7
[2] https:/