In Heat, instance run cfn-push-stats, which uses boto library, and boto creates EC2 v4 signature.
heat-api-cfn service receives that request, and delegates signature check to Keystone.
Keystone recreates EC2 v4 signature with keystoneclient for checking.
But keystoneclient has its own EC2 v4 signature implementation, and so there is a logic mismatch
between boto and keystoneclient.
The following is what has been previously transmitted to the openstack-dev mailing list.
> I am trying Heat instance HA, using RDO Icehouse.
>
> After instance boot, instance push own stats to heat alarm with cfn-push-stats command.
> But cfn-push-stats always failed with error '403 SignatureDoesNotMatch', this message is
> output to /var/log/cfn-push-stats.log.
>
> I debugged client and server side code. (i.e. cfn-push-stats, boto, heat, keystone,
> keystoneclient) And I found curious code mismatch between boto and keystoneclient about
> signature calculation.
>
> Here is a result of debugging, and code examination.
>
> * Client side
>
> cfn-push-stats uses heat-cfntools library, and heat-cfntools do 'POST' request with boto.
> boto perfomes signature calculation. [1]
> for signature calculation, firstly it construct 'CanonicalRequest', some strings are joined.
> And create a digest hash of the CanonicalRequest for signature calculation.
> CanonicalRequest contains CanonicalQueryString, which is transfomed URL query strings.
>
> CanonicalRequest =
> HTTPRequestMethod + '\n' +
> CanonicalURI + '\n' +
> CanonicalQueryString + '\n' +
> CanonicalHeaders + '\n' +
> SignedHeaders + '\n' +
> HexEncode(Hash(RequestPayload))
>
> **AWS original tool (aws-cfn-bootstrap-1.4) and boto uses empty string as
> CanonicalQueryString, when request is POST.**
>
> ----
>
> AWS original tool's code is following.
>
> cfnbootstrap/aws_client.py
>
> 110 class V4Signer(Signer):
>
> 144 (canonical_headers, signed_headers) = self._canonicalize_headers(new_headers)
> 145 canonical_request += canonical_headers + '\n' + signed_headers + '\n'
> 146 canonical_request += hashlib.sha256(self._construct_query(params).encode('utf-8') if verb == 'POST' else '').hexdigest()
>
> ----
>
> boto's code is below.
>
> boto/auth.py
>
> 283 class HmacAuthV4Handler(AuthHandler, HmacKeys):
>
> 393 def canonical_request(self, http_request):
> 394 cr = [http_request.method.upper()]
> 395 cr.append(self.canonical_uri(http_request))
> 396 cr.append(self.canonical_query_string(http_request))
> 397 headers_to_sign = self.headers_to_sign(http_request)
> 398 cr.append(self.canonical_headers(headers_to_sign) + '\n')
> 399 cr.append(self.signed_headers(headers_to_sign))
> 400 cr.append(self.payload(http_request))
> 401 return '\n'.join(cr)
>
> 337 def canonical_query_string(self, http_request):
> 338 # POST requests pass parameters in through the
> 339 # http_request.body field.
> 340 if http_request.method == 'POST':
> 341 return ""
> 342 l = []
> 343 for param in sorted(http_request.params):
> 344 value = boto.utils.get_utf8_value(http_request.params[param])
> 345 l.append('%s=%s' % (urllib.quote(param, safe='-_.~'),
> 346 urllib.quote(value, safe='-_.~')))
> 347 return '&'.join(l)
>
> ----
>
> * Server side
>
> heat-api-cfn queries to keystone in order to check request authorization,
> and keystone uses keystoneclient to check EC2 format request signature.
>
> **keystoneclient uses (non-empty) query string as CanonicalQueryString, even
> though request is POST.**
> And create a digest hash of the CanonicalRequest for signature calculation.
>
> ----
>
> keystoneclient's code is below.
>
> keystoneclient/contrib/ec2/utils.py
>
> 28 class Ec2Signer(object):
>
> 154 def _calc_signature_4(self, params, verb, server_string, path, headers,
> 155 body_hash):
> 156 """Generate AWS signature version 4 string."""
>
> 235 # Create canonical request:
> 236 # http://docs.aws.amazon.com/general/latest/gr/
> 237 # sigv4-create-canonical-request.html
> 238 # Get parameters and headers in expected string format
> 239 cr = "\n".join((verb.upper(), path,
> 240 self._canonical_qs(params),
> 241 canonical_header_str(),
> 242 auth_param('SignedHeaders'),
> 243 body_hash))
>
> 125 @staticmethod
> 126 def _canonical_qs(params):
> 127 """Construct a sorted, correctly encoded query string as required for
> 128 _calc_signature_2 and _calc_signature_4.
> 129 """
> 130 keys = list(params)
> 131 keys.sort()
> 132 pairs = []
> 133 for key in keys:
> 134 val = Ec2Signer._get_utf8_value(params[key])
> 135 val = urllib.parse.quote(val, safe='-_~')
> 136 pairs.append(urllib.parse.quote(key, safe='') + '=' + val)
> 137 qs = '&'.join(pairs)
> 138 return qs
>
> ----
>
> So it should be different from boto(client side) to keystoneclient(server side).
>
> I wrote a patch to resolve this how to treat CanonicalQueryString mismatch,
> My patch honored AWS original tool and boto, so if request is POST,
> if request is POST, 'CanonicalQueryString' is regarded as a empty string.
>
> With my patch, Heat instance HA works fine.
>
> This bug affects Heat and Keystone, but patch is only needed in python-keystoneclient.
> So I will report to python-keystoneclient launchpad and submit a patch to Gerrit.
> Please confirm it.
>
> ----
>
> My environment is RDO Icehouse/CentOS6.5, and package versions is following.
>
> * Client side
>
> cloud-init-0.7.4-2.el6.noarch
> heat-cfntools-1.2.6-2.el6.noarch
> python-boto-2.27.0-1.el6.noarch
>
> * Server side
>
> python-keystoneclient-0.9.0-1.el6.noarch
> python-keystone-2014.1.1-1.el6.noarch
> openstack-keystone-2014.1.1-1.el6.noarch
>
> ----
>
> References
>
> [1] http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
In Heat, instance run cfn-push-stats, which uses boto library, and boto creates EC2 v4 signature.
heat-api-cfn service receives that request, and delegates signature check to Keystone.
Keystone recreates EC2 v4 signature with keystoneclient for checking.
But keystoneclient has its own EC2 v4 signature implementation, and so there is a logic mismatch
between boto and keystoneclient.
The following is what has been previously transmitted to the openstack-dev mailing list.
> I am trying Heat instance HA, using RDO Icehouse. tMatch' , this message is cfn-push- stats.log. tring, which is transfomed URL query strings. tring + '\n' + Hash(RequestPay load)) bootstrap- 1.4) and boto uses empty string as tring, when request is POST.** aws_client. py ize_headers( new_headers) sha256( self._construct _query( params) .encode( 'utf-8' ) if verb == 'POST' else '').hexdigest() er(AuthHandler, HmacKeys): request( self, http_request): method. upper() ] self.canonical_ uri(http_ request) ) self.canonical_ query_string( http_request) ) to_sign( http_request) self.canonical_ headers( headers_ to_sign) + '\n') self.signed_ headers( headers_ to_sign) ) self.payload( http_request) ) query_string( self, http_request): http_request. params) : get_utf8_ value(http_ request. params[ param]) quote(param, safe='-_.~'), tring, even contrib/ ec2/utils. py _4(self, params, verb, server_string, path, headers, docs.aws. amazon. com/general/ latest/ gr/ canonical- request. html (verb.upper( ), path, _qs(params) , header_ str(), 'SignedHeaders' ), qs(params) : _get_utf8_ value(params[ key]) parse.quote( val, safe='-_~') urllib. parse.quote( key, safe='') + '=' + val) server side). tring mismatch, String' is regarded as a empty string. keystoneclient. keystoneclient launchpad and submit a patch to Gerrit. 0.7.4-2. el6.noarch 1.2.6-2. el6.noarch boto-2. 27.0-1. el6.noarch keystoneclient- 0.9.0-1. el6.noarch keystone- 2014.1. 1-1.el6. noarch keystone- 2014.1. 1-1.el6. noarch docs.aws. amazon. com/general/ latest/ gr/sigv4- create- canonical- request. html
>
> After instance boot, instance push own stats to heat alarm with cfn-push-stats command.
> But cfn-push-stats always failed with error '403 SignatureDoesNo
> output to /var/log/
>
> I debugged client and server side code. (i.e. cfn-push-stats, boto, heat, keystone,
> keystoneclient) And I found curious code mismatch between boto and keystoneclient about
> signature calculation.
>
> Here is a result of debugging, and code examination.
>
> * Client side
>
> cfn-push-stats uses heat-cfntools library, and heat-cfntools do 'POST' request with boto.
> boto perfomes signature calculation. [1]
> for signature calculation, firstly it construct 'CanonicalRequest', some strings are joined.
> And create a digest hash of the CanonicalRequest for signature calculation.
> CanonicalRequest contains CanonicalQueryS
>
> CanonicalRequest =
> HTTPRequestMethod + '\n' +
> CanonicalURI + '\n' +
> CanonicalQueryS
> CanonicalHeaders + '\n' +
> SignedHeaders + '\n' +
> HexEncode(
>
> **AWS original tool (aws-cfn-
> CanonicalQueryS
>
> ----
>
> AWS original tool's code is following.
>
> cfnbootstrap/
>
> 110 class V4Signer(Signer):
>
> 144 (canonical_headers, signed_headers) = self._canonical
> 145 canonical_request += canonical_headers + '\n' + signed_headers + '\n'
> 146 canonical_request += hashlib.
>
> ----
>
> boto's code is below.
>
> boto/auth.py
>
> 283 class HmacAuthV4Handl
>
> 393 def canonical_
> 394 cr = [http_request.
> 395 cr.append(
> 396 cr.append(
> 397 headers_to_sign = self.headers_
> 398 cr.append(
> 399 cr.append(
> 400 cr.append(
> 401 return '\n'.join(cr)
>
> 337 def canonical_
> 338 # POST requests pass parameters in through the
> 339 # http_request.body field.
> 340 if http_request.method == 'POST':
> 341 return ""
> 342 l = []
> 343 for param in sorted(
> 344 value = boto.utils.
> 345 l.append('%s=%s' % (urllib.
> 346 urllib.quote(value, safe='-_.~')))
> 347 return '&'.join(l)
>
> ----
>
> * Server side
>
> heat-api-cfn queries to keystone in order to check request authorization,
> and keystone uses keystoneclient to check EC2 format request signature.
>
> **keystoneclient uses (non-empty) query string as CanonicalQueryS
> though request is POST.**
> And create a digest hash of the CanonicalRequest for signature calculation.
>
> ----
>
> keystoneclient's code is below.
>
> keystoneclient/
>
> 28 class Ec2Signer(object):
>
> 154 def _calc_signature
> 155 body_hash):
> 156 """Generate AWS signature version 4 string."""
>
> 235 # Create canonical request:
> 236 # http://
> 237 # sigv4-create-
> 238 # Get parameters and headers in expected string format
> 239 cr = "\n".join(
> 240 self._canonical
> 241 canonical_
> 242 auth_param(
> 243 body_hash))
>
> 125 @staticmethod
> 126 def _canonical_
> 127 """Construct a sorted, correctly encoded query string as required for
> 128 _calc_signature_2 and _calc_signature_4.
> 129 """
> 130 keys = list(params)
> 131 keys.sort()
> 132 pairs = []
> 133 for key in keys:
> 134 val = Ec2Signer.
> 135 val = urllib.
> 136 pairs.append(
> 137 qs = '&'.join(pairs)
> 138 return qs
>
> ----
>
> So it should be different from boto(client side) to keystoneclient(
>
> I wrote a patch to resolve this how to treat CanonicalQueryS
> My patch honored AWS original tool and boto, so if request is POST,
> if request is POST, 'CanonicalQuery
>
> With my patch, Heat instance HA works fine.
>
> This bug affects Heat and Keystone, but patch is only needed in python-
> So I will report to python-
> Please confirm it.
>
> ----
>
> My environment is RDO Icehouse/CentOS6.5, and package versions is following.
>
> * Client side
>
> cloud-init-
> heat-cfntools-
> python-
>
> * Server side
>
> python-
> python-
> openstack-
>
> ----
>
> References
>
> [1] http://