cfn-push-stats fails to authenticate

[Lars Kellogg-Stedman]

I am running Heat (stable/icehouse) on Fedora 20 (using the RDO packages). I am trying to use cfn-push-stats in a Fedora 20 instance, but when running:

    cfn-push-stats --heartbeat

I receive:

 boto.exception.BotoServerError: BotoServerError: 403 AccessDenied
<ErrorResponse><Error><Message>User is not authorized to perform action</Message><Code>AccessDenied</Code><Type>Sender</Type></Error></ErrorResponse>

I have /etc/cfn/cfn-credentials with valid credentials:

AWSAccessKeyId=aaa8ddeb9c5b40b6ad5a03812ad4dd1e
AWSSecretKey=46fc263e41ae4d13b9a63ba9835395a3

These match the credentials that were generated in keystone from my heat template:

keystone ec2-credentials-list --user_id 0d7f702539b04702bf7c84bc3299b0cd
+------------------------------------------------------------------+----------------------------------+----------------------------------+
| tenant | access | secret |
+------------------------------------------------------------------+----------------------------------+----------------------------------+
| 95d9bbd9b446438a89a353d8adb60704-3f4c5cb0-2355-4f44-8914-50fa435 | aaa8ddeb9c5b40b6ad5a03812ad4dd1e | 46fc263e41ae4d13b9a63ba9835395a3 |
+------------------------------------------------------------------+----------------------------------+----------------------------------+

Other tools (e.g., euca-describe-images) are able to authenticate using these credentials.

Both the host and the instance are have python-boto 2.27.0 (on stevebaker's advice I tried upgrading to 2.32.1, but that demonstrates the same behavior).

[Lars Kellogg-Stedman (larsks)]

The template behind this is:

http://chunk.io/f/34dbe4a9f168469ea37bd9d23e21b354

[Lars Kellogg-Stedman (larsks)]

The request getting sent to keystone looks like this:

{
  "ec2Credentials": {
    "body_hash": "7998222ff85406cf40991aa7e77fc63d5e91743ef474f8d876560f612c0e9028",
    "path": "/v1/",
    "signature": "347dcd4f3c9556d014234dc57e125167eb598a57883c53db207b26ba3c64d091",
    "params": {
      "Action": "PutMetricData",
      "Version": "2010-08-01",
      "MetricData.member.1.MetricName": "Heartbeat",
      "MetricData.member.1.Value": "1",
      "MetricData.member.1.Unit": "Counter",
      "Namespace": "system/linux"
    },
    "verb": "POST",
    "host": "192.168.200.1:8003",
    "headers": {
      "Authorization": "AWS4-HMAC-SHA256 Credential=aaa8ddeb9c5b40b6ad5a03812ad4dd1e/20140922/168/192/aws4_request,SignedHeaders=host;x-amz-date,Signature=347dcd4f3c9556d014234dc57e125167eb598a57883c53db207b26ba3c64d091",
      "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
      "X-Amz-Date": "20140922T205704Z",
      "Host": "192.168.200.1:8003",
      "User-Agent": "Boto/2.32.1 Python/2.7.5 Linux/3.15.10-200.fc20.x86_64",
      "Accept-Encoding": "identity",
      "Content-Length": "166"
    },
    "access": "aaa8ddeb9c5b40b6ad5a03812ad4dd1e"
  }
}

[Lars Kellogg-Stedman (larsks)]

It looks as if boto and keystone are generating different canonical requests for signing. Boto is signing a request that looks like this:

  POST
  /v1/

  host:192.168.200.1:8003
  x-amz-date:20140923T030939Z

  host;x-amz-date
  7998222ff85406cf40991aa7e77fc63d5e91743ef474f8d876560f612c0e9028

While keystone is signing a request that looks like this:

  POST
  /v1/
  Action=PutMetricData&MetricData.member.1.MetricName=Heartbeat&MetricData.member.1.Unit=Counter&MetricData.member.1.Value=1&Namespace=system%2Flinux&Version=2010-08-01
  host:192.168.200.1:8003
  x-amz-date:20140923T030939Z

  host;x-amz-date
  7998222ff85406cf40991aa7e77fc63d5e91743ef474f8d876560f612c0e9028

That is, keystone includes the Action header in the data to be signed,
while boto does not.

[Steve Baker (steve-stevebaker)]

Maybe it is possible for heat-api-cfn to remove the Action header from the request before calling keystone

[Angus Salkeld (asalkeld)]

Another option is to try the most common method first, and if we get a failure try the other method (no Action header).

[Bruno Bompastor (bruno-bompastor)]

I also have this problem...issuing this command: /opt/aws/bin/cfn-push-stats --watch restarter-HttpFailureAlarm --service-failure

[Steven Hardy (shardy)]

For some reason this has been marked incomplete for keystoneclient, but FWIW, last time boto broke us like this, I worked around it by changing the logic in keystoneclient dependent on boto version:

https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/contrib/ec2/utils.py#L219

It's a bit ugly, but given that boto arbitrarily changes stuff like this periodically it seems like the only option - multiple validation calls to keystone from the API seems suboptimal to say the least.

It makes me wonder what AWS actually does, given that stuff like this appears to change randomly in boto without any change in the signature version - I can only assume the AWS signature check is somehow tolerant of stuff like this, or that they periodically break all their users and boto has to change their stuff to match?

[Dolph Mathews (dolph)]

My apologies, I should have explained the Incomplete. It's not clear to me what work needs to happen within keystoneclient, or even if the issue here resides in keystoneclient.

[Steven Hardy (shardy)]

@Dolph: I'll try to dig into it a little more and see if a fix in keystoneclient is appropriate in this case, as mentioned above, we did fix a similar issue there previously hence adding it to this bug.

[Bruno Bompastor (bruno-bompastor)]

Is there a workaround for this?

[Bart Swedrowski (bart613)]

I'd love to see a workaround as well.

[Bruno Bompastor (bruno-bompastor)]

Ok I found a workaround. Just downgrade your python-boto package to version 2.12.0. I tried the required version for heat-cfntools, maybe this was broken in a newer version...

[Steven Hardy (shardy)]

> That is, keystone includes the Action header in the data to be signed, while boto does not.

How can that be right? Surely this is a boto bug?

The requests in comment #3 indicates the entire query string is missing from the request that is signed, meaning anyone who captures the request can replay whatever data they want into cloudwatch. So IMO we don't want that scheme.

Are we sure the arguments aren't just passed as POST data instead of part of the query string?

[Bruno Bompastor (bruno-bompastor)]

Yes steven you are right. In the new python boto versions they are sending the arguments via final headers but we are expecting on the query string. See the example.

python boto 2.32.1:

DEBUG [2014-11-12 11:13:48,915] cfn-push-stats called Namespace(cpu_util=False, credential_file='/etc/cfn/cfn-credentials', disk_path='/', disk_space_avail=False, disk_space_used=False, disk_space_util=False, disk_units='megabytes', haproxy=False, haproxy_latency=False, heartbeat=False, mem_avail=False, mem_used=False, mem_util=False, memory_units='megabytes', metric=None, service_failure=True, swap_used=False, swap_util=False, units=None, value=None, verbose=False, watch='restarter-HttpFailureAlarm')
DEBUG [2014-11-12 11:13:48,916] Using access key provided by client.
DEBUG [2014-11-12 11:13:48,916] Using secret key provided by client.
INFO [2014-11-12 11:13:48,916] Sending metric ServiceFailure, Units Counter, Value 1
DEBUG [2014-11-12 11:13:48,917] Method: POST
DEBUG [2014-11-12 11:13:48,917] Path: /v1/
DEBUG [2014-11-12 11:13:48,917] Data:
DEBUG [2014-11-12 11:13:48,917] Headers: {}
DEBUG [2014-11-12 11:13:48,917] Host: 10.32.10.151
DEBUG [2014-11-12 11:13:48,917] Port: 8003
DEBUG [2014-11-12 11:13:48,917] Params: {'MetricData.member.1.Dimensions.member.2.Value': 'restarter-HttpFailureAlarm', 'MetricData.member.1.Dimensions.member.2.Name': 'AlarmName', 'Namespace': 'system/linux', 'MetricData.member.1.Unit': 'Counter', 'MetricData.member.1.Value': 1, 'Action': 'PutMetricData', 'MetricData.member.1.Dimensions.member.1.Name': 'InstanceId', 'Version': '2010-08-01', 'MetricData.member.1.Dimensions.member.1.Value': u'90b27502-49cf-43f8-804c-fccd304e796d', 'MetricData.member.1.MetricName': 'ServiceFailure'}
DEBUG [2014-11-12 11:13:48,917] establishing HTTP connection: kwargs={'port': 8003, 'timeout': 70}
DEBUG [2014-11-12 11:13:48,917] Token: None
DEBUG [2014-11-12 11:13:48,919] CanonicalRequest:
POST
/v1/

host:10.32.10.151:8003
x-amz-date:20141112T101348Z

host;x-amz-date
67f215d4c6638325995976310e635a69f4c10c56de61e8d9855e9f80e1f34186
DEBUG [2014-11-12 11:13:48,919] StringToSign:
AWS4-HMAC-SHA256
20141112T101348Z
20141112/32/10/aws4_request
55d0054ff53524a243e0bef360f267ffbac6b31d061a622553b20042053780da
DEBUG [2014-11-12 11:13:48,919] Signature:
4086ffd920db62e40af076da8e2a0b02408b2d430eb9c059eca1034a506c7fce
DEBUG [2014-11-12 11:13:48,919] Final headers: {'Content-Length': '438', 'User-Agent': 'Boto/2.32.1 Python/2.6.6 Linux/2.6.32-431.23.3.el6.x86_64', 'Host': '10.32.10.151:8003', 'X-Amz-Date': '20141112T101348Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Authorization': 'AWS4-HMAC-SHA256 Credential=27178e0b9c71478e837c02d17fe1aa4d/20141112/32/10/aws4_request,SignedHeaders=host;x-amz-date,Signature=4086ffd920db62e40af076da8e2a0b02408b2d430eb9c059eca1034a506c7fce'}
DEBUG [2014-11-12 11:13:48,935] Response headers: [('date', 'Wed, 12 Nov 2014 10:13:49 GMT'), ('content-length', '195'), ('content-type', 'application/xml; charset=UTF-8')]
DEBUG [2014-11-12 11:13:48,935] <ErrorResponse><Error><Message>The request signature we calculated does not match the signature you provided</Message><Code>SignatureDoesNotMatch</Code><Type>Sender<...

[Bruno Bompastor (bruno-bompastor)]

BTW the functionality was broken in boto v2.20.0.

[Bruno Bompastor (bruno-bompastor)]

Actually the bug was introduced in boto 2.18.0.

This was the change that broke it: https://github.com/boto/boto/commit/74bc084bb8c7f9d0f2fd75def01dc6828063b081

I'm having a slightly different error "The request signature we calculated does not match the signature you provided", but i think its related...

[Steven Hardy (shardy)]

@bruno-bompastor: Thanks for the info, the keystoneclient code should, in theory, support the v4 signing scheme, but the switch to v4 means a completely new code-path in the signing calculation, so at least now we know where to start looking.

[Bruno Bompastor (bruno-bompastor)]

I was going to propose a patch for this but I noticed that was fixed with this commit: https://github.com/openstack/python-keystoneclient/commit/cf5e45dd5b1ae9b98698a05e7d39989b6bfd4747

Its already in juno...

I just have to test it to confirm its fixed.

[Steve Baker (steve-stevebaker)]

I'm going to mark this as Invalid for heat, since it was fixed elsewhere