heat breaks with latest boto

Bug #1122472 reported by Steven Hardy
24
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
High
Steven Hardy
Grizzly
Fix Released
High
Steven Hardy

Bug Description

Latest/recent versions of boto don't work with heat - looks like AWS changed their signature format to a new version which we don't yet support:

Investigation required to decide if we support the new signature format (will require the keystone ec2 api to support it), or figure out how to force boto to use the old format as an interim solution.

# heat-boto resource wordpress_ap3 WikiDatabase
send: 'GET /v1/?Action=DescribeStackResource&ContentType=JSON&LogicalResourceId=WikiDatabase&StackName=wordpress_ap3&Version=2010-05-15 HTTP/1.1\r\nHost: 127.0.0.1:8000\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS4-HMAC-SHA256 Credential=27da451dba8148cbbde31641b32d17eb/20130211/0/127/aws4_request,SignedHeaders=host;x-amz-date,Signature=bb9578c10b212e774d4ff064188a912c5de87f1c33651fc0a16dd5468f0238ba\r\nX-Amz-Date: 20130211T221800Z\r\nUser-Agent: Boto/2.8.0-dev (linux2)\r\n\r\n'
reply: 'HTTP/1.1 400 IncompleteSignature\r\n'
header: Content-Type: application/xml; charset=UTF-8
header: Content-Length: 171
header: Date: Mon, 11 Feb 2013 22:18:00 GMT
ERROR:400 IncompleteSignature
ERROR:<ErrorResponse><Error><Message>The request signature does not conform to AWS standards</Message><Code>IncompleteSignature</Code><Type>Sender</Type></Error></ErrorResponse>
ERROR:Failed to resource. Got error:
ERROR:BotoServerError: 400 IncompleteSignature
ERROR:<ErrorResponse><Error><Message>The request signature does not conform to AWS standards</Message><Code>IncompleteSignature</Code><Type>Sender</Type></Error></ErrorResponse>

Steven Hardy (shardy)
Changed in heat:
status: New → Triaged
importance: Undecided → High
milestone: none → grizzly-3
Revision history for this message
Steven Hardy (shardy) wrote :

So the problem is boto switched to the AWS version 4 signature scheme:

http://docs.aws.amazon.com/general/latest/gr/signature-version-4.html

But we currently expect version 2 signature format (not sure what happened to version 3, no mention on the AWS site):

http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html

Changed in heat:
assignee: nobody → Steven Hardy (shardy)
Revision history for this message
Steven Hardy (shardy) wrote :

Keystone (python-keystonclient) does not yet support v4 signatures, so I raised a blueprint:

https://blueprints.launchpad.net/python-keystoneclient/+spec/ec2signer-v4signatures

Revision history for this message
Steven Hardy (shardy) wrote :

Since the required features are not yet present in keystone, suggest we defer this to H and advise users to stick to a boto version before the switch to v4 signatures as an interim workaround.

So workaround is pip install 'boto==2.5.2' (not sure exactly which version switched CFN to v4 but 2.5.2 definitely works)

Revision history for this message
Steven Hardy (shardy) wrote :

clearing g-3 milestone, we'll have to sort this out in H - I'll speak to the keystone folks and offer to implement the blueprint above which we'll need to fix this bug

Changed in heat:
milestone: grizzly-3 → none
Steven Hardy (shardy)
Changed in heat:
milestone: none → havana-1
status: Triaged → In Progress
Steven Hardy (shardy)
description: updated
Revision history for this message
Steven Hardy (shardy) wrote :

Patch adding the required support to python-keystoneclient up:

https://review.openstack.org/#/c/26013/

When this gets merged I have a patch for ec2token which allows heat to work with the latest boto

Revision history for this message
Steve Baker (steve-stevebaker) wrote :

Steve, could you create a .rst document in heat/doc which contains a testing/compatibility matrix of:

Rows: 1 distro and version per row
Column 1: bundled version of cloud-init
Column 2: bundled version of python-boto
Column 3: testing status of heat/heat-cfntools with this combo
Column 4: links to footnotes with more details (like custom versions of boto required, minimum version of heat, etc)

It doesn't have to be fully fleshed-out by you, it could just start with Fedora 17/18.

This table will be aimed at people choosing what image to use with Heat, or people building their own images. I think it will help a lot, there is stuff in our heads not written anywhere.

Revision history for this message
Steven Hardy (shardy) wrote :

Compatibility matrix started in wiki:

https://wiki.openstack.org/wiki/Heat/BotoCompatibility

We can move it to heat/doc when it's more complete - if we can encourage all existing users to contribute data that would be good!

I'll chase through the v4 fixes next week so we should be working with latest boto soon.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/27828

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/27828
Committed: http://github.com/openstack/heat/commit/8518ca055851e435396712914faa98071e37b863
Submitter: Jenkins
Branch: master

commit 8518ca055851e435396712914faa98071e37b863
Author: Steven Hardy <email address hidden>
Date: Mon Apr 8 14:53:04 2013 +0100

    heat api : Update ec2token middleware for v4 signatures

    Update ec2token so it can verify v4 signature formats.

    Note for v4 signatures to work you currently need to install
    the latest python-keystoneclient, as no release yet contains
    patch ref https://review.openstack.org/#/c/26013/

    This change should be backwards compatible, as older keystoneclient
    versions will simply ignore the additional request keys

    fixes bug #1122472

    Change-Id: Iccc6be7913ab5ca5813a2e0c8f66cda0ccd85a0b

Changed in heat:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in heat:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/grizzly)

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/31568

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/grizzly)

Reviewed: https://review.openstack.org/31568
Committed: http://github.com/openstack/heat/commit/16894252caaace4c3daede236efa42466e34cd09
Submitter: Jenkins
Branch: stable/grizzly

commit 16894252caaace4c3daede236efa42466e34cd09
Author: Steven Hardy <email address hidden>
Date: Mon Apr 8 14:53:04 2013 +0100

    heat api : Update ec2token middleware for v4 signatures

    Update ec2token so it can verify v4 signature formats.

    Note for v4 signatures to work you need keystone to be using
    python-keystoneclient >= 0.2.4, or contain the patch from
    https://review.openstack.org/#/c/26013/

    This change is backwards compatible, as older keystoneclient
    versions will simply ignore the additional request keys

    fixes bug #1122472

    Change-Id: Iccc6be7913ab5ca5813a2e0c8f66cda0ccd85a0b

Thierry Carrez (ttx)
Changed in heat:
milestone: havana-1 → 2013.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers