Auth failure with AWS signature-v4

Bug #1411078 reported by Charles Hsu on 2015-01-15
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Swift3
Undecided
Andrey Pavlov

Bug Description

I known swift3 only support signature v2. And the V4 isn't implement. Maybe we should put it on our blueprint.
Test with s3cmd v1.5.0 and it return a 412 Error.

 $ s3cmd --debug ls s3://test3
DEBUG: ConfigParser: Reading file '/Users/charles/.s3cfg'
DEBUG: ConfigParser: access_key->te...8_chars...r
DEBUG: ConfigParser: access_token->
DEBUG: ConfigParser: add_encoding_exts->
DEBUG: ConfigParser: add_headers->
DEBUG: ConfigParser: bucket_location->US
DEBUG: ConfigParser: cache_file->
DEBUG: ConfigParser: cloudfront_host->192.168.200.201
DEBUG: ConfigParser: default_mime_type->binary/octet-stream
DEBUG: ConfigParser: delay_updates->False
DEBUG: ConfigParser: delete_after->False
DEBUG: ConfigParser: delete_after_fetch->False
DEBUG: ConfigParser: delete_removed->False
DEBUG: ConfigParser: dry_run->False
DEBUG: ConfigParser: enable_multipart->True
DEBUG: ConfigParser: encoding->UTF-8
DEBUG: ConfigParser: encrypt->False
DEBUG: ConfigParser: follow_symlinks->False
DEBUG: ConfigParser: force->False
DEBUG: ConfigParser: get_continue->False
DEBUG: ConfigParser: gpg_command->None
DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_passphrase->...-3_chars...
DEBUG: ConfigParser: guess_mime_type->True
DEBUG: ConfigParser: host_base->192.168.200.201
DEBUG: ConfigParser: host_bucket->192.168.200.201
DEBUG: ConfigParser: human_readable_sizes->False
DEBUG: ConfigParser: invalidate_default_index_on_cf->False
DEBUG: ConfigParser: invalidate_default_index_root_on_cf->True
DEBUG: ConfigParser: invalidate_on_cf->False
DEBUG: ConfigParser: list_md5->False
DEBUG: ConfigParser: log_target_prefix->
DEBUG: ConfigParser: mime_type->
DEBUG: ConfigParser: multipart_chunk_size_mb->15
DEBUG: ConfigParser: preserve_attrs->True
DEBUG: ConfigParser: progress_meter->True
DEBUG: ConfigParser: proxy_host->
DEBUG: ConfigParser: proxy_port->0
DEBUG: ConfigParser: recursive->False
DEBUG: ConfigParser: recv_chunk->4096
DEBUG: ConfigParser: reduced_redundancy->False
DEBUG: ConfigParser: secret_key->te...4_chars...g
DEBUG: ConfigParser: send_chunk->4096
DEBUG: ConfigParser: simpledb_host->
DEBUG: ConfigParser: skip_existing->False
DEBUG: ConfigParser: socket_timeout->300
DEBUG: ConfigParser: urlencoding_mode->normal
DEBUG: ConfigParser: use_https->False
DEBUG: ConfigParser: verbosity->WARNING
DEBUG: ConfigParser: website_error->
DEBUG: ConfigParser: website_index->index.html
DEBUG: Updating Config.Config cache_file ->
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'ls' using UTF-8
DEBUG: Unicodising 's3://test3' using UTF-8
DEBUG: Command: ls
DEBUG: Bucket 's3://test3':
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v4
DEBUG: get_hostname(test3): 192.168.200.201
DEBUG: Canonical Request:
GET
/
delimiter=%2F
host:192.168.200.201
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20150114T103925Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=test:tester/20150114/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=08293376d0c91348cfc11536a3b969d06f19477338e7258a8d81ad4f1dc882d3', 'x-amz-date': '20150114T103925Z'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(test3): 192.168.200.201
DEBUG: ConnMan.get(): creating new connection: http://192.168.200.201
DEBUG: non-proxied HTTPConnection(192.168.200.201)
DEBUG: format_uri(): /test3/?delimiter=/
DEBUG: Sending request method_string='GET', uri='/test3/?delimiter=/', headers={'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=test:tester/20150114/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=08293376d0c91348cfc11536a3b969d06f19477338e7258a8d81ad4f1dc882d3', 'x-amz-date': '20150114T103925Z'}, body=(0 bytes)
DEBUG: Response: {'status': 412, 'headers': {'date': 'Wed, 14 Jan 2015 10:39:25 GMT', 'content-length': '7', 'content-type': 'text/html; charset=UTF-8', 'x-trans-id': 'tx2ca6e281156a4fdb9bfc2-0054b6475d'}, 'reason': 'Precondition Failed', 'data': 'Bad URL'}
DEBUG: ConnMan.put(): connection put back to pool (http://192.168.200.201#1)
DEBUG: S3Error: 412 (Precondition Failed)
DEBUG: HttpHeader: date: Wed, 14 Jan 2015 10:39:25 GMT
DEBUG: HttpHeader: content-length: 7
DEBUG: HttpHeader: content-type: text/html; charset=UTF-8
DEBUG: HttpHeader: x-trans-id: tx2ca6e281156a4fdb9bfc2-0054b6475d
ERROR: Error parsing xml: syntax error: line 1, column 0
ERROR: Bad URL
DEBUG: Not an XML response
ERROR: S3 error: 412 (Precondition Failed):

Dag Stenstad (dag-stenstad) wrote :

Workaround just to get s3cmd to work is to add "signature_v2 = True" in ~/.s3cfg ...

Andrey Pavlov (apavlov-e) wrote :

Same problem is present when using 'aws' tool from Amazon.
But this tool doesn't have workaround with version.

Charles Hsu (charles0126) wrote :

@Andrey, Did you try use `S3SignerType` or `AWS3SignerType` for aws java tool?

Here is a similar issue in elasticsearch-cloud-aws, it uses aws java sdk to do backup.
https://github.com/elastic/elasticsearch-cloud-aws/issues/223

Andrey Pavlov (apavlov-e) wrote :

@Charles, 'awscli' is a tool based on botocore library that was written on python.
botocore library has 'signature_version' as option but it has a bug - https://github.com/boto/botocore/issues/601

Charles Hsu (charles0126) wrote :

@Andery, I haven't try boto3 (newer version of boto) yet, that probably is a bug in boto3. Thanks your info.

Andrey Pavlov (apavlov-e) wrote :

because swift3 uses s3token from keystonemiddleware I have filed a bug https://bugs.launchpad.net/keystonemiddleware/+bug/1473042
And now I'm fixing it.

Related fix proposed to branch: master
Review: https://review.openstack.org/211933

Changed in swift3:
assignee: nobody → Andrey Pavlov (apavlov-e)
Changed in swift3:
status: New → In Progress
Andrey Pavlov (apavlov-e) wrote :

@Charles, I've found a workaround for awscli -
if it is run without region - it uses signature v1.

Reviewed: https://review.openstack.org/211536
Committed: https://git.openstack.org/cgit/stackforge/swift3/commit/?id=52cbf0e48cef4b508a94006be846208e89f9bdb8
Submitter: Jenkins
Branch: master

commit 52cbf0e48cef4b508a94006be846208e89f9bdb8
Author: Andrey Pavlov <email address hidden>
Date: Tue Aug 11 15:18:14 2015 +0300

    remove signature member from request object.

    Remove this field because it doesn't use anywhere.
    Also this is redundant for adding of parsing of signature V4.

    Change-Id: Ia547d01324144ab442ace837870f5d431186f69c
    Related-Bug: #1411078

Change abandoned by Andrey Pavlov (<email address hidden>) on branch: master
Review: https://review.openstack.org/211933
Reason: windmill...

Attila Fazekas (afazekas) wrote :

Another related pending change: https://review.openstack.org/#/c/301165/
I have no idea why hudson-openstack forgotten to mention it.

Reviewed: https://review.openstack.org/301165
Committed: https://git.openstack.org/cgit/openstack/swift3/commit/?id=a1cc181bd8ef891d3e3b8ed71db9b41c179ced0e
Submitter: Jenkins
Branch: master

commit a1cc181bd8ef891d3e3b8ed71db9b41c179ced0e
Author: Kota Tsuyuzaki <email address hidden>
Date: Wed Aug 12 12:43:37 2015 +0300

    Re:implement AWS signature v4

    New algorithm that supports s3v4 was added.

    What I did in this patch in detail:

    - Implements v4 related code into mix-in class to provide some methods
      for authentication algorithms (e.g. string_to_sign)

    - S3Timestamp everywhere. Old code take a lot of complicated timestamp
      translation from/to datetime, time, date header format (str). This
      patch gathers the translation into "timestamp" property method which
      should be actually handled in the validatation.

    - Run functional tests for both v2/v4 authentication in the same
      environment at the same time which shows evidence that we have complete
      backword compatibilities and we can adopt v4 w/o anything broken.

    *Bonus*
    - Fix some minger bugs for singed urls (almostly expired timestamp),
      for header/query mixture and for unit test case mistake.

    The reason I implemented this from Andrey's original patch is the
    signature v4 stuff is too complicated if we mixes the process/routine
    into same class because of a bunch of if/elif/else statements for header
    handling. (e.g. if 'X-Amz-Date' in req.headers) Note that it is not his
    issue, just AWS is getting complicated algorithms. However, for
    maintainansibility, we need more clear code to find easily which statement
    is supported on v2/v4 to prevent merge buggy code into master. That is why
    I tried to do this. Hopefully this code fits the original author's intention.

    NOTE for operators:
    - Signature V4 is supported only for keystone auth.
    - Set the same value of "region" configuration in keystone to "location" in
      swift3 conf file to enable SigV4.
    - Sigv2 and SigV4 can be used at the same cluster configuration.
    - This stuff has been supported since Keystone 9.0.0.0b1. (We probably
      need to bump the minimum version for keystone in requirements)

    Change-Id: I386abd4ead40f55855657e354fd8ef3fd0d13aa7
    Co-Authored-By: Andrey Pavlov <email address hidden>
    Closes-Bug: #1411078

Changed in swift3:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers