heat breaks with latest boto

[Steven Hardy]

Latest/recent versions of boto don't work with heat - looks like AWS changed their signature format to a new version which we don't yet support:

Investigation required to decide if we support the new signature format (will require the keystone ec2 api to support it), or figure out how to force boto to use the old format as an interim solution.

# heat-boto resource wordpress_ap3 WikiDatabase
send: 'GET /v1/?Action=DescribeStackResource&ContentType=JSON&LogicalResourceId=WikiDatabase&StackName=wordpress_ap3&Version=2010-05-15 HTTP/1.1\r\nHost: 127.0.0.1:8000\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS4-HMAC-SHA256 Credential=27da451dba8148cbbde31641b32d17eb/20130211/0/127/aws4_request,SignedHeaders=host;x-amz-date,Signature=bb9578c10b212e774d4ff064188a912c5de87f1c33651fc0a16dd5468f0238ba\r\nX-Amz-Date: 20130211T221800Z\r\nUser-Agent: Boto/2.8.0-dev (linux2)\r\n\r\n'
reply: 'HTTP/1.1 400 IncompleteSignature\r\n'
header: Content-Type: application/xml; charset=UTF-8
header: Content-Length: 171
header: Date: Mon, 11 Feb 2013 22:18:00 GMT
ERROR:400 IncompleteSignature
ERROR:<ErrorResponse><Error><Message>The request signature does not conform to AWS standards</Message><Code>IncompleteSignature</Code><Type>Sender</Type></Error></ErrorResponse>
ERROR:Failed to resource. Got error:
ERROR:BotoServerError: 400 IncompleteSignature
ERROR:<ErrorResponse><Error><Message>The request signature does not conform to AWS standards</Message><Code>IncompleteSignature</Code><Type>Sender</Type></Error></ErrorResponse>

[Steven Hardy (shardy)]

So the problem is boto switched to the AWS version 4 signature scheme:

http://docs.aws.amazon.com/general/latest/gr/signature-version-4.html

But we currently expect version 2 signature format (not sure what happened to version 3, no mention on the AWS site):

http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html

[Steven Hardy (shardy)]

Keystone (python-keystonclient) does not yet support v4 signatures, so I raised a blueprint:

https://blueprints.launchpad.net/python-keystoneclient/+spec/ec2signer-v4signatures

[Steven Hardy (shardy)]

Since the required features are not yet present in keystone, suggest we defer this to H and advise users to stick to a boto version before the switch to v4 signatures as an interim workaround.

So workaround is pip install 'boto==2.5.2' (not sure exactly which version switched CFN to v4 but 2.5.2 definitely works)

[Steven Hardy (shardy)]

clearing g-3 milestone, we'll have to sort this out in H - I'll speak to the keystone folks and offer to implement the blueprint above which we'll need to fix this bug

[Steven Hardy (shardy)]

Patch adding the required support to python-keystoneclient up:

https://review.openstack.org/#/c/26013/

When this gets merged I have a patch for ec2token which allows heat to work with the latest boto

[Steve Baker (steve-stevebaker)]

Steve, could you create a .rst document in heat/doc which contains a testing/compatibility matrix of:

Rows: 1 distro and version per row
Column 1: bundled version of cloud-init
Column 2: bundled version of python-boto
Column 3: testing status of heat/heat-cfntools with this combo
Column 4: links to footnotes with more details (like custom versions of boto required, minimum version of heat, etc)

It doesn't have to be fully fleshed-out by you, it could just start with Fedora 17/18.

This table will be aimed at people choosing what image to use with Heat, or people building their own images. I think it will help a lot, there is stuff in our heads not written anywhere.

[Steven Hardy (shardy)]

Compatibility matrix started in wiki:

https://wiki.openstack.org/wiki/Heat/BotoCompatibility

We can move it to heat/doc when it's more complete - if we can encourage all existing users to contribute data that would be good!

I'll chase through the v4 fixes next week so we should be working with latest boto soon.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/27828

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/27828
Committed: http://github.com/openstack/heat/commit/8518ca055851e435396712914faa98071e37b863
Submitter: Jenkins
Branch: master

commit 8518ca055851e435396712914faa98071e37b863
Author: Steven Hardy <email address hidden>
Date: Mon Apr 8 14:53:04 2013 +0100

    heat api : Update ec2token middleware for v4 signatures

    Update ec2token so it can verify v4 signature formats.

    Note for v4 signatures to work you currently need to install
    the latest python-keystoneclient, as no release yet contains
    patch ref https://review.openstack.org/#/c/26013/

    This change should be backwards compatible, as older keystoneclient
    versions will simply ignore the additional request keys

    fixes bug #1122472

    Change-Id: Iccc6be7913ab5ca5813a2e0c8f66cda0ccd85a0b

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/31568

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/31568
Committed: http://github.com/openstack/heat/commit/16894252caaace4c3daede236efa42466e34cd09
Submitter: Jenkins
Branch: stable/grizzly

commit 16894252caaace4c3daede236efa42466e34cd09
Author: Steven Hardy <email address hidden>
Date: Mon Apr 8 14:53:04 2013 +0100

    heat api : Update ec2token middleware for v4 signatures

    Update ec2token so it can verify v4 signature formats.

    Note for v4 signatures to work you need keystone to be using
    python-keystoneclient >= 0.2.4, or contain the patch from
    https://review.openstack.org/#/c/26013/

    This change is backwards compatible, as older keystoneclient
    versions will simply ignore the additional request keys

    fixes bug #1122472

    Change-Id: Iccc6be7913ab5ca5813a2e0c8f66cda0ccd85a0b