cfn-push-stats fails to authenticate

Bug #1372710 reported by Lars Kellogg-Stedman on 2014-09-23
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Heat

Bug Description

I am running Heat (stable/icehouse) on Fedora 20 (using the RDO packages). I am trying to use cfn-push-stats in a Fedora 20 instance, but when running:

    cfn-push-stats --heartbeat

I receive:

 boto.exception.BotoServerError: BotoServerError: 403 AccessDenied
<ErrorResponse><Error><Message>User is not authorized to perform action</Message><Code>AccessDenied</Code><Type>Sender</Type></Error></ErrorResponse>

I have /etc/cfn/cfn-credentials with valid credentials:


These match the credentials that were generated in keystone from my heat template:

keystone ec2-credentials-list --user_id 0d7f702539b04702bf7c84bc3299b0cd
| tenant | access | secret |
| 95d9bbd9b446438a89a353d8adb60704-3f4c5cb0-2355-4f44-8914-50fa435 | aaa8ddeb9c5b40b6ad5a03812ad4dd1e | 46fc263e41ae4d13b9a63ba9835395a3 |

Other tools (e.g., euca-describe-images) are able to authenticate using these credentials.

Both the host and the instance are have python-boto 2.27.0 (on stevebaker's advice I tried upgrading to 2.32.1, but that demonstrates the same behavior).

Lars Kellogg-Stedman (larsks) wrote :

The request getting sent to keystone looks like this:

  "ec2Credentials": {
    "body_hash": "7998222ff85406cf40991aa7e77fc63d5e91743ef474f8d876560f612c0e9028",
    "path": "/v1/",
    "signature": "347dcd4f3c9556d014234dc57e125167eb598a57883c53db207b26ba3c64d091",
    "params": {
      "Action": "PutMetricData",
      "Version": "2010-08-01",
      "MetricData.member.1.MetricName": "Heartbeat",
      "MetricData.member.1.Value": "1",
      "MetricData.member.1.Unit": "Counter",
      "Namespace": "system/linux"
    "verb": "POST",
    "host": "",
    "headers": {
      "Authorization": "AWS4-HMAC-SHA256 Credential=aaa8ddeb9c5b40b6ad5a03812ad4dd1e/20140922/168/192/aws4_request,SignedHeaders=host;x-amz-date,Signature=347dcd4f3c9556d014234dc57e125167eb598a57883c53db207b26ba3c64d091",
      "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
      "X-Amz-Date": "20140922T205704Z",
      "Host": "",
      "User-Agent": "Boto/2.32.1 Python/2.7.5 Linux/3.15.10-200.fc20.x86_64",
      "Accept-Encoding": "identity",
      "Content-Length": "166"
    "access": "aaa8ddeb9c5b40b6ad5a03812ad4dd1e"

Lars Kellogg-Stedman (larsks) wrote :

It looks as if boto and keystone are generating different canonical requests for signing. Boto is signing a request that looks like this:




While keystone is signing a request that looks like this:



That is, keystone includes the Action header in the data to be signed,
while boto does not.

Steve Baker (steve-stevebaker) wrote :

Maybe it is possible for heat-api-cfn to remove the Action header from the request before calling keystone

Angus Salkeld (asalkeld) wrote :

Another option is to try the most common method first, and if we get a failure try the other method (no Action header).

Angus Salkeld (asalkeld) on 2014-09-26
Changed in heat-cfntools:
importance: Undecided → Medium
status: New → Triaged

I also have this problem...issuing this command: /opt/aws/bin/cfn-push-stats --watch restarter-HttpFailureAlarm --service-failure

Dolph Mathews (dolph) on 2014-10-01
Changed in python-keystoneclient:
status: New → Incomplete
Steven Hardy (shardy) wrote :

For some reason this has been marked incomplete for keystoneclient, but FWIW, last time boto broke us like this, I worked around it by changing the logic in keystoneclient dependent on boto version:

It's a bit ugly, but given that boto arbitrarily changes stuff like this periodically it seems like the only option - multiple validation calls to keystone from the API seems suboptimal to say the least.

It makes me wonder what AWS actually does, given that stuff like this appears to change randomly in boto without any change in the signature version - I can only assume the AWS signature check is somehow tolerant of stuff like this, or that they periodically break all their users and boto has to change their stuff to match?

Dolph Mathews (dolph) wrote :

My apologies, I should have explained the Incomplete. It's not clear to me what work needs to happen within keystoneclient, or even if the issue here resides in keystoneclient.

Steven Hardy (shardy) wrote :

@Dolph: I'll try to dig into it a little more and see if a fix in keystoneclient is appropriate in this case, as mentioned above, we did fix a similar issue there previously hence adding it to this bug.

Is there a workaround for this?

Bart Swedrowski (bart613) wrote :

I'd love to see a workaround as well.

Ok I found a workaround. Just downgrade your python-boto package to version 2.12.0. I tried the required version for heat-cfntools, maybe this was broken in a newer version...

Steven Hardy (shardy) wrote :

> That is, keystone includes the Action header in the data to be signed, while boto does not.

How can that be right? Surely this is a boto bug?

The requests in comment #3 indicates the entire query string is missing from the request that is signed, meaning anyone who captures the request can replay whatever data they want into cloudwatch. So IMO we don't want that scheme.

Are we sure the arguments aren't just passed as POST data instead of part of the query string?

Download full text (7.3 KiB)

Yes steven you are right. In the new python boto versions they are sending the arguments via final headers but we are expecting on the query string. See the example.

python boto 2.32.1:

DEBUG [2014-11-12 11:13:48,915] cfn-push-stats called Namespace(cpu_util=False, credential_file='/etc/cfn/cfn-credentials', disk_path='/', disk_space_avail=False, disk_space_used=False, disk_space_util=False, disk_units='megabytes', haproxy=False, haproxy_latency=False, heartbeat=False, mem_avail=False, mem_used=False, mem_util=False, memory_units='megabytes', metric=None, service_failure=True, swap_used=False, swap_util=False, units=None, value=None, verbose=False, watch='restarter-HttpFailureAlarm')
DEBUG [2014-11-12 11:13:48,916] Using access key provided by client.
DEBUG [2014-11-12 11:13:48,916] Using secret key provided by client.
INFO [2014-11-12 11:13:48,916] Sending metric ServiceFailure, Units Counter, Value 1
DEBUG [2014-11-12 11:13:48,917] Method: POST
DEBUG [2014-11-12 11:13:48,917] Path: /v1/
DEBUG [2014-11-12 11:13:48,917] Data:
DEBUG [2014-11-12 11:13:48,917] Headers: {}
DEBUG [2014-11-12 11:13:48,917] Host:
DEBUG [2014-11-12 11:13:48,917] Port: 8003
DEBUG [2014-11-12 11:13:48,917] Params: {'MetricData.member.1.Dimensions.member.2.Value': 'restarter-HttpFailureAlarm', 'MetricData.member.1.Dimensions.member.2.Name': 'AlarmName', 'Namespace': 'system/linux', 'MetricData.member.1.Unit': 'Counter', 'MetricData.member.1.Value': 1, 'Action': 'PutMetricData', 'MetricData.member.1.Dimensions.member.1.Name': 'InstanceId', 'Version': '2010-08-01', 'MetricData.member.1.Dimensions.member.1.Value': u'90b27502-49cf-43f8-804c-fccd304e796d', 'MetricData.member.1.MetricName': 'ServiceFailure'}
DEBUG [2014-11-12 11:13:48,917] establishing HTTP connection: kwargs={'port': 8003, 'timeout': 70}
DEBUG [2014-11-12 11:13:48,917] Token: None
DEBUG [2014-11-12 11:13:48,919] CanonicalRequest:


DEBUG [2014-11-12 11:13:48,919] StringToSign:
DEBUG [2014-11-12 11:13:48,919] Signature:
DEBUG [2014-11-12 11:13:48,919] Final headers: {'Content-Length': '438', 'User-Agent': 'Boto/2.32.1 Python/2.6.6 Linux/2.6.32-431.23.3.el6.x86_64', 'Host': '', 'X-Amz-Date': '20141112T101348Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Authorization': 'AWS4-HMAC-SHA256 Credential=27178e0b9c71478e837c02d17fe1aa4d/20141112/32/10/aws4_request,SignedHeaders=host;x-amz-date,Signature=4086ffd920db62e40af076da8e2a0b02408b2d430eb9c059eca1034a506c7fce'}
DEBUG [2014-11-12 11:13:48,935] Response headers: [('date', 'Wed, 12 Nov 2014 10:13:49 GMT'), ('content-length', '195'), ('content-type', 'application/xml; charset=UTF-8')]
DEBUG [2014-11-12 11:13:48,935] <ErrorResponse><Error><Message>The request signature we calculated does not match the signature you provided</Message><Code>SignatureDoesNotMatch</Code><Type>Sender<...


BTW the functionality was broken in boto v2.20.0.

Actually the bug was introduced in boto 2.18.0.

This was the change that broke it:

I'm having a slightly different error "The request signature we calculated does not match the signature you provided", but i think its related...

Steven Hardy (shardy) wrote :

@bruno-bompastor: Thanks for the info, the keystoneclient code should, in theory, support the v4 signing scheme, but the switch to v4 means a completely new code-path in the signing calculation, so at least now we know where to start looking.

Changed in python-keystoneclient:
assignee: nobody → Bruno Bompastor (bruno-bompastor)
status: Incomplete → Confirmed
Dolph Mathews (dolph) on 2014-11-19
Changed in python-keystoneclient:
importance: Undecided → Wishlist

I was going to propose a patch for this but I noticed that was fixed with this commit:

Its already in juno...

I just have to test it to confirm its fixed.

Changed in python-keystoneclient:
status: Confirmed → Fix Committed
Changed in python-keystoneclient:
assignee: Bruno Bompastor (bruno-bompastor) → nobody
Changed in python-keystoneclient:
status: Fix Committed → Fix Released
Steve Baker (steve-stevebaker) wrote :

I'm going to mark this as Invalid for heat, since it was fixed elsewhere

Changed in heat:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers