PUT Filename with colon(':') from AWS C++ SDK results in SignatureDoesNotMatch

[Shailesh M]

I'm testing Swift access using S3 client (AWS SDK for C++).
I have tested file upload/download successfully with filename consisting alphanum chars.
This particular test fails when filename has ':'.
The same code works with AWS S3 and minio.

The code is trying to create an object with name
backup_catalog_2022-02-18_21:19:57.859.json

If I replace ':' with chars like '-'/'_' it works.

The error is:
SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your key and signing method.

I narrowed down this problem to this method in AWS SDK
StringUtils.cpp
Aws::String StringUtils::URLEncode(const char* unsafe)
-- snip --
        if (IsAlnum(c) || c == '-' || c == '_' || c == '.' || c == '~')
        {
            escaped << (char)c;
        }
        else
        {
            //this unsigned char cast allows us to handle unicode characters.
            escaped << '%' << std::setw(2) << int((unsigned char)c) << std::setw(0);
        }

I suspect the Swift server end does not expect ':' escaped.
As a hack I changed the condition in SDK to
       if (IsAlnum(c) || c == '-' || c == '_' || c == '.' || c == '~' || c == ':')
With this, I don't see any error.

Is this a known issue? Is there a workaround?
I found a very old discussion but did not understand how it was addressed.
https://bugs.launchpad.net/swift/+bug/936998
Do I need to set any env variable on server end?

[Shailesh M (shaileshkm)]

Hi,
I'd appreciate if someone can help me if there is a workaround.

[Tim Burke (1-tim-z)]

What version of Swift is this? Of Python (mainly, 2.7 or 3.x)? Of Eventlet? We've seen similar issues before (see https://bugs.launchpad.net/swift/+bug/1884991), but to my knowledge they should be fixed.

Which signing method is the client using? Specifically, v2 or v4 signatures? I did some tests using ':', '%3A', and '%3a' in URLs locally (Swift 2.29.0+, Python 3.8.10, Eventlet 0.33.0) and it all seemed to work for v2 signatures at least.

[Shailesh M (shaileshkm)]

This is with V4.

How can I find the Swift server version?
Is there a swift client command that I can invoke? I don't have access to the server.

[Shailesh M (shaileshkm)]

My colleague gave me a REST command.
Swift version is 2.28.1.dev19

[Shailesh M (shaileshkm)]

Hi,
Is anyone able to reproduce this issue with V4 signature?

[Thibault Person (thibault-p)]

Hello,
I encounter this error too while setting up a FoundationDB cluster (https://www.foundationdb.org/). I investigate on this issue and it leads me to the Swift AWS signature v4 implementation.

The FoundationDB implementation to backup database to a S3 bucket does not encode path (which contains comma) before sending request but comply with AWS signature calculation: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html.

On the other side, Swift relies on WSGI url encoding for signature calculation despite Amazon recommend to use a custom function for url encoding.

I wrote a patch for this issue and I should submit it as a PR soon.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/swift/+/833913

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/swift/+/833913
Committed: https://opendev.org/openstack/swift/commit/cb8b3cdab262af1d223c0536220400b13c1d0a9a
Submitter: "Zuul (22348)"
Branch: master

commit cb8b3cdab262af1d223c0536220400b13c1d0a9a
Author: Thibault Person <email address hidden>
Date: Tue Mar 15 15:12:57 2022 +0100

    Comply with AWS signature calculation (s3v4)

    The current implementation of s3 signature calculation
    rely on WSGI Url encoding which is discouraged by AWS:
    https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html.
    This leads to reject requests with valid signature.

    This update encode only characters specified by AWS except
    'A'-'Z', 'a'-'z', '0'-'9', '-', '.', '_', and '~' to comply
    AWS signature calculation.

    Fixes LP Bug #1961841

    Change-Id: Ifa8f94544224c3379e7f2805f6f86d0b0a47279a

[Shailesh M (shaileshkm)]

Thanks for fixing this issue.
How can I find in which release this commit is released?

[Tim Burke (1-tim-z)]

The fix is present in swift 2.29.1.