In Heat, instance run cfn-push-stats, which uses boto library, and boto creates EC2 v4 signature.
heat-api-cfn service receives that request, and delegates signature check to Keystone.
Keystone recreates EC2 v4 signature with keystoneclient for checking.
But keystoneclient has its own EC2 v4 signature implementation, and so there is a logic mismatch between boto and keystoneclient.
The following is what has been previously transmitted to the openstack-dev mailing list.
> I am trying Heat instance HA, using RDO Icehouse.
>
> After instance boot, instance push own stats to heat alarm with cfn-push-stats command.
> But cfn-push-stats always failed with error '403 SignatureDoesNotMatch', this message is
> output to /var/log/cfn-push-stats.log.
>
> I debugged client and server side code. (i.e. cfn-push-stats, boto, heat, keystone,
> keystoneclient) And I found curious code mismatch between boto and keystoneclient about
> signature calculation.
>
> Here is a result of debugging, and code examination.
>
> * Client side
>
> cfn-push-stats uses heat-cfntools library, and heat-cfntools do 'POST' request with boto.
> boto perfomes signature calculation. [1]
> for signature calculation, firstly it construct 'CanonicalRequest', some strings are joined.
> And create a digest hash of the CanonicalRequest for signature calculation.
> CanonicalRequest contains CanonicalQueryString, which is transfomed URL query strings.
>
> CanonicalRequest =
> HTTPRequestMethod + '\n' +
> CanonicalURI + '\n' +
> CanonicalQueryString + '\n' +
> CanonicalHeaders + '\n' +
> SignedHeaders + '\n' +
> HexEncode(Hash(RequestPayload))
>
> **AWS original tool (aws-cfn-bootstrap-1.4) and boto uses empty string as
> CanonicalQueryString, when request is POST.**
>
> ----
>
> AWS original tool's code is following.
>
> cfnbootstrap/aws_client.py
>
> 110 class V4Signer(Signer):
>
> 144 (canonical_headers, signed_headers) = self._canonicalize_headers(new_headers)
> 145 canonical_request += canonical_headers + '\n' + signed_headers + '\n'
> 146 canonical_request += hashlib.sha256(self._construct_query(params).encode('utf-8') if verb == 'POST' else '').hexdigest()
>
> ----
>
> boto's code is below.
>
> boto/auth.py
>
> 283 class HmacAuthV4Handler(AuthHandler, HmacKeys):
>
> 393 def canonical_request(self, http_request):
> 394 cr = [http_request.method.upper()]
> 395 cr.append(self.canonical_uri(http_request))
> 396 cr.append(self.canonical_query_string(http_request))
> 397 headers_to_sign = self.headers_to_sign(http_request)
> 398 cr.append(self.canonical_headers(headers_to_sign) + '\n')
> 399 cr.append(self.signed_headers(headers_to_sign))
> 400 cr.append(self.payload(http_request))
> 401 return '\n'.join(cr)
>
> 337 def canonical_query_string(self, http_request):
> 338 # POST requests pass parameters in through the
> 339 # http_request.body field.
> 340 if http_request.method == 'POST':
> 341 return ""
> 342 l = []
> 343 for param in sorted(http_request.params):
> 344 value = boto.utils.get_utf8_value(http_request.params[param])
> 345 l.append('%s=%s' % (urllib.quote(param, safe='-_.~'),
> 346 urllib.quote(value, safe='-_.~')))
> 347 return '&'.join(l)
>
> ----
>
> * Server side
>
> heat-api-cfn queries to keystone in order to check request authorization,
> and keystone uses keystoneclient to check EC2 format request signature.
>
> **keystoneclient uses (non-empty) query string as CanonicalQueryString, even
> though request is POST.**
> And create a digest hash of the CanonicalRequest for signature calculation.
>
> ----
>
> keystoneclient's code is below.
>
> keystoneclient/contrib/ec2/utils.py
>
> 28 class Ec2Signer(object):
>
> 154 def _calc_signature_4(self, params, verb, server_string, path, headers,
> 155 body_hash):
> 156 """Generate AWS signature version 4 string."""
>
> 235 # Create canonical request:
> 236 # http://docs.aws.amazon.com/general/latest/gr/
> 237 # sigv4-create-canonical-request.html
> 238 # Get parameters and headers in expected string format
> 239 cr = "\n".join((verb.upper(), path,
> 240 self._canonical_qs(params),
> 241 canonical_header_str(),
> 242 auth_param('SignedHeaders'),
> 243 body_hash))
>
> 125 @staticmethod
> 126 def _canonical_qs(params):
> 127 """Construct a sorted, correctly encoded query string as required for
> 128 _calc_signature_2 and _calc_signature_4.
> 129 """
> 130 keys = list(params)
> 131 keys.sort()
> 132 pairs = []
> 133 for key in keys:
> 134 val = Ec2Signer._get_utf8_value(params[key])
> 135 val = urllib.parse.quote(val, safe='-_~')
> 136 pairs.append(urllib.parse.quote(key, safe='') + '=' + val)
> 137 qs = '&'.join(pairs)
> 138 return qs
>
> ----
>
> So it should be different from boto(client side) to keystoneclient(server side).
>
> I wrote a patch to resolve this how to treat CanonicalQueryString mismatch,
> My patch honored AWS original tool and boto, so if request is POST,
> if request is POST, 'CanonicalQueryString' is regarded as a empty string.
>
> With my patch, Heat instance HA works fine.
>
> This bug affects Heat and Keystone, but patch is only needed in python-keystoneclient.
> So I will report to python-keystoneclient launchpad and submit a patch to Gerrit.
> Please confirm it.
>
> ----
>
> My environment is RDO Icehouse/CentOS6.5, and package versions is following.
>
> * Client side
>
> cloud-init-0.7.4-2.el6.noarch
> heat-cfntools-1.2.6-2.el6.noarch
> python-boto-2.27.0-1.el6.noarch
>
> * Server side
>
> python-keystoneclient-0.9.0-1.el6.noarch
> python-keystone-2014.1.1-1.el6.noarch
> openstack-keystone-2014.1.1-1.el6.noarch
>
> ----
>
> References
>
> [1] http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
Fix proposed to branch: master
Review: https:/