Auth failure with AWS signature-v4

Bug #1411078 reported by Charles Hsu
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Swift3
Fix Released
Undecided
Andrey Pavlov

Bug Description

I known swift3 only support signature v2. And the V4 isn't implement. Maybe we should put it on our blueprint.
Test with s3cmd v1.5.0 and it return a 412 Error.

 $ s3cmd --debug ls s3://test3
DEBUG: ConfigParser: Reading file '/Users/charles/.s3cfg'
DEBUG: ConfigParser: access_key->te...8_chars...r
DEBUG: ConfigParser: access_token->
DEBUG: ConfigParser: add_encoding_exts->
DEBUG: ConfigParser: add_headers->
DEBUG: ConfigParser: bucket_location->US
DEBUG: ConfigParser: cache_file->
DEBUG: ConfigParser: cloudfront_host->192.168.200.201
DEBUG: ConfigParser: default_mime_type->binary/octet-stream
DEBUG: ConfigParser: delay_updates->False
DEBUG: ConfigParser: delete_after->False
DEBUG: ConfigParser: delete_after_fetch->False
DEBUG: ConfigParser: delete_removed->False
DEBUG: ConfigParser: dry_run->False
DEBUG: ConfigParser: enable_multipart->True
DEBUG: ConfigParser: encoding->UTF-8
DEBUG: ConfigParser: encrypt->False
DEBUG: ConfigParser: follow_symlinks->False
DEBUG: ConfigParser: force->False
DEBUG: ConfigParser: get_continue->False
DEBUG: ConfigParser: gpg_command->None
DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_passphrase->...-3_chars...
DEBUG: ConfigParser: guess_mime_type->True
DEBUG: ConfigParser: host_base->192.168.200.201
DEBUG: ConfigParser: host_bucket->192.168.200.201
DEBUG: ConfigParser: human_readable_sizes->False
DEBUG: ConfigParser: invalidate_default_index_on_cf->False
DEBUG: ConfigParser: invalidate_default_index_root_on_cf->True
DEBUG: ConfigParser: invalidate_on_cf->False
DEBUG: ConfigParser: list_md5->False
DEBUG: ConfigParser: log_target_prefix->
DEBUG: ConfigParser: mime_type->
DEBUG: ConfigParser: multipart_chunk_size_mb->15
DEBUG: ConfigParser: preserve_attrs->True
DEBUG: ConfigParser: progress_meter->True
DEBUG: ConfigParser: proxy_host->
DEBUG: ConfigParser: proxy_port->0
DEBUG: ConfigParser: recursive->False
DEBUG: ConfigParser: recv_chunk->4096
DEBUG: ConfigParser: reduced_redundancy->False
DEBUG: ConfigParser: secret_key->te...4_chars...g
DEBUG: ConfigParser: send_chunk->4096
DEBUG: ConfigParser: simpledb_host->
DEBUG: ConfigParser: skip_existing->False
DEBUG: ConfigParser: socket_timeout->300
DEBUG: ConfigParser: urlencoding_mode->normal
DEBUG: ConfigParser: use_https->False
DEBUG: ConfigParser: verbosity->WARNING
DEBUG: ConfigParser: website_error->
DEBUG: ConfigParser: website_index->index.html
DEBUG: Updating Config.Config cache_file ->
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'ls' using UTF-8
DEBUG: Unicodising 's3://test3' using UTF-8
DEBUG: Command: ls
DEBUG: Bucket 's3://test3':
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v4
DEBUG: get_hostname(test3): 192.168.200.201
DEBUG: Canonical Request:
GET
/
delimiter=%2F
host:192.168.200.201
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20150114T103925Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=test:tester/20150114/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=08293376d0c91348cfc11536a3b969d06f19477338e7258a8d81ad4f1dc882d3', 'x-amz-date': '20150114T103925Z'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(test3): 192.168.200.201
DEBUG: ConnMan.get(): creating new connection: http://192.168.200.201
DEBUG: non-proxied HTTPConnection(192.168.200.201)
DEBUG: format_uri(): /test3/?delimiter=/
DEBUG: Sending request method_string='GET', uri='/test3/?delimiter=/', headers={'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=test:tester/20150114/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=08293376d0c91348cfc11536a3b969d06f19477338e7258a8d81ad4f1dc882d3', 'x-amz-date': '20150114T103925Z'}, body=(0 bytes)
DEBUG: Response: {'status': 412, 'headers': {'date': 'Wed, 14 Jan 2015 10:39:25 GMT', 'content-length': '7', 'content-type': 'text/html; charset=UTF-8', 'x-trans-id': 'tx2ca6e281156a4fdb9bfc2-0054b6475d'}, 'reason': 'Precondition Failed', 'data': 'Bad URL'}
DEBUG: ConnMan.put(): connection put back to pool (http://192.168.200.201#1)
DEBUG: S3Error: 412 (Precondition Failed)
DEBUG: HttpHeader: date: Wed, 14 Jan 2015 10:39:25 GMT
DEBUG: HttpHeader: content-length: 7
DEBUG: HttpHeader: content-type: text/html; charset=UTF-8
DEBUG: HttpHeader: x-trans-id: tx2ca6e281156a4fdb9bfc2-0054b6475d
ERROR: Error parsing xml: syntax error: line 1, column 0
ERROR: Bad URL
DEBUG: Not an XML response
ERROR: S3 error: 412 (Precondition Failed):

Revision history for this message
Charles Hsu (charles0126) wrote :
Revision history for this message
Dag Stenstad (dag-stenstad) wrote :

Workaround just to get s3cmd to work is to add "signature_v2 = True" in ~/.s3cfg ...

Revision history for this message
Andrey Pavlov (apavlov-e) wrote :

Same problem is present when using 'aws' tool from Amazon.
But this tool doesn't have workaround with version.

Revision history for this message
Charles Hsu (charles0126) wrote :

@Andrey, Did you try use `S3SignerType` or `AWS3SignerType` for aws java tool?

Here is a similar issue in elasticsearch-cloud-aws, it uses aws java sdk to do backup.
https://github.com/elastic/elasticsearch-cloud-aws/issues/223

Revision history for this message
Andrey Pavlov (apavlov-e) wrote :

@Charles, 'awscli' is a tool based on botocore library that was written on python.
botocore library has 'signature_version' as option but it has a bug - https://github.com/boto/botocore/issues/601

Revision history for this message
Charles Hsu (charles0126) wrote :

@Andery, I haven't try boto3 (newer version of boto) yet, that probably is a bug in boto3. Thanks your info.

Revision history for this message
Andrey Pavlov (apavlov-e) wrote :

because swift3 uses s3token from keystonemiddleware I have filed a bug https://bugs.launchpad.net/keystonemiddleware/+bug/1473042
And now I'm fixing it.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to swift3 (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/211536

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/211933

Changed in swift3:
assignee: nobody → Andrey Pavlov (apavlov-e)
Changed in swift3:
status: New → In Progress
Revision history for this message
Andrey Pavlov (apavlov-e) wrote :

@Charles, I've found a workaround for awscli -
if it is run without region - it uses signature v1.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to swift3 (master)

Reviewed: https://review.openstack.org/211536
Committed: https://git.openstack.org/cgit/stackforge/swift3/commit/?id=52cbf0e48cef4b508a94006be846208e89f9bdb8
Submitter: Jenkins
Branch: master

commit 52cbf0e48cef4b508a94006be846208e89f9bdb8
Author: Andrey Pavlov <email address hidden>
Date: Tue Aug 11 15:18:14 2015 +0300

    remove signature member from request object.

    Remove this field because it doesn't use anywhere.
    Also this is redundant for adding of parsing of signature V4.

    Change-Id: Ia547d01324144ab442ace837870f5d431186f69c
    Related-Bug: #1411078

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on swift3 (master)

Change abandoned by Andrey Pavlov (<email address hidden>) on branch: master
Review: https://review.openstack.org/211933
Reason: windmill...

Revision history for this message
Attila Fazekas (afazekas) wrote :

Another related pending change: https://review.openstack.org/#/c/301165/
I have no idea why hudson-openstack forgotten to mention it.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift3 (master)

Reviewed: https://review.openstack.org/301165
Committed: https://git.openstack.org/cgit/openstack/swift3/commit/?id=a1cc181bd8ef891d3e3b8ed71db9b41c179ced0e
Submitter: Jenkins
Branch: master

commit a1cc181bd8ef891d3e3b8ed71db9b41c179ced0e
Author: Kota Tsuyuzaki <email address hidden>
Date: Wed Aug 12 12:43:37 2015 +0300

    Re:implement AWS signature v4

    New algorithm that supports s3v4 was added.

    What I did in this patch in detail:

    - Implements v4 related code into mix-in class to provide some methods
      for authentication algorithms (e.g. string_to_sign)

    - S3Timestamp everywhere. Old code take a lot of complicated timestamp
      translation from/to datetime, time, date header format (str). This
      patch gathers the translation into "timestamp" property method which
      should be actually handled in the validatation.

    - Run functional tests for both v2/v4 authentication in the same
      environment at the same time which shows evidence that we have complete
      backword compatibilities and we can adopt v4 w/o anything broken.

    *Bonus*
    - Fix some minger bugs for singed urls (almostly expired timestamp),
      for header/query mixture and for unit test case mistake.

    The reason I implemented this from Andrey's original patch is the
    signature v4 stuff is too complicated if we mixes the process/routine
    into same class because of a bunch of if/elif/else statements for header
    handling. (e.g. if 'X-Amz-Date' in req.headers) Note that it is not his
    issue, just AWS is getting complicated algorithms. However, for
    maintainansibility, we need more clear code to find easily which statement
    is supported on v2/v4 to prevent merge buggy code into master. That is why
    I tried to do this. Hopefully this code fits the original author's intention.

    NOTE for operators:
    - Signature V4 is supported only for keystone auth.
    - Set the same value of "region" configuration in keystone to "location" in
      swift3 conf file to enable SigV4.
    - Sigv2 and SigV4 can be used at the same cluster configuration.
    - This stuff has been supported since Keystone 9.0.0.0b1. (We probably
      need to bump the minimum version for keystone in requirements)

    Change-Id: I386abd4ead40f55855657e354fd8ef3fd0d13aa7
    Co-Authored-By: Andrey Pavlov <email address hidden>
    Closes-Bug: #1411078

Changed in swift3:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.