| 2016-01-15 18:06:20 |
Jakub Pavlik |
description |
We upgraded environment to OpenStack Kilo 2015.1.2 and then we cannot use heat autoscaling feature anymore.
Keystone version 1:2015.1.2-0ubuntu2~cloud0
Heat version 2015.1.2-0ubuntu1~cloud0
- First we hit authentication failure on ec2tokens heat-api-cfn.log
2016-01-15 14:21:07.505 6590 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
2016-01-15 14:21:07.507 6590 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
2016-01-15 14:21:07.509 6590 INFO heat.api.aws.ec2token [-] Authenticating with http://10.200.26.10:5000/ec2tokens
2016-01-15 14:21:07.533 6590 INFO heat.api.aws.ec2token [-] AWS authentication failure.
2016-01-15 14:21:07.535 6590 INFO eventlet.wsgi.server [-] 10.200.26.11 - - [15/Jan/2016 14:21:07] "POST /v1/signal/arn%3Aopenstack%3Aheat%3A%3A4a28da719dc64deb9d1d5cfa770bbca2%3Astacks%2Fheat_autoscale_cpu_vpc20%2F3a268b22-d436-4cf8-9088-8418702be09a%2Fresources%2Fserver_scaleup_policy?Timestamp=2016-01-15T13%3A14%3A22Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=f042045aabc74728a49059a4c52226bf&SignatureVersion=2&Signature=GomCM13riX1UYu3OpJZ3ERXpfekHkmeACA4DznTL6g8%3D HTTP/1.1" 403 301 0.031258
We applied patch to keystone https://bugs.launchpad.net/openstack-ansible/+bug/1515485 https://git.openstack.org/cgit/openstack/keystone/commit/?id=f66badae63b1544c3bcc83e0c1afc8e9cdc71bab
Then we hit error on python-novaclient auth boot during autoscaling heat-engine.log
2016-01-15 06:43:05.836 11301 ERROR heat.engine.stack [req-fd1fd187-1be5-489a-803b-8c08e491ebd4 - 4a28da719dc64deb9d1d5cfa770bbca2] Exception: (NotFound(),)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack Traceback (most recent call last):
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 593, in validate
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack result = res.validate()
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/nova/server.py", line 1254, in validate
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack limits = self.client_plugin().absolute_limits()
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/nova.py", line 439, in absolute_limits
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack limits = self.client().limits.get()
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/v2/limits.py", line 100, in get
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack return self._get("/limits%s" % query_string, "limits")
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 147, in _get
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack _resp, body = self.api.client.get(url)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 440, in get
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack return self._cs_request(url, 'GET', **kwargs)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 427, in _cs_request
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack self.authenticate()
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 569, in authenticate
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack self._v2_auth(auth_url)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 634, in _v2_auth
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack return self._authenticate(url, body)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 647, in _authenticate
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack **kwargs)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 392, in _time_request
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack resp, body = self.request(url, method, **kwargs)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 386, in request
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack raise exceptions.from_response(resp, body, url, method)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack NotFound: The resource could not be found. (HTTP 404)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack
2016-01-15 06:43:05.838 11301 INFO heat.engine.stack [req-fd1fd187-1be5-489a-803b-8c08e491ebd4 - 4a28da719dc64deb9d1d5cfa770bbca2] Failed to validate: The resource could not be found. (HTTP 404)
It was caused by bad auth_url argument passed into novaclient - http://10.0.170.10:5000/v3/v2.0
We modified /usr/lib/python2.7/dist-packages/heat/engine/clients/os/nova.py to replace v3 to v2.0
args = {
'project_id': self.context.tenant,
'auth_url': self.context.auth_url.replace('/v3','/v2.0'),
'service_type': 'compute',
'username': None,
'api_key': None,
'extensions': extensions,
'endpoint_type': endpoint_type,
'http_log_debug': self._get_client_option('nova',
'http_log_debug'),
'cacert': self._get_client_option('nova', 'ca_file'),
'insecure': self._get_client_option('nova', 'insecure')
}
And then we got following error heat-engine.log
2016-01-15 14:36:05.464 8195 ERROR heat.engine.stack [req-b19963cf-81d3-482d-b31a-27d7b0862b89 - 4a28da719dc64deb9d1d5cfa770bbca2] Exception: (ClientException(),)
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack Traceback (most recent call last):
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 593, in validate
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack result = res.validate()
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/nova/server.py", line 1254, in validate
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack limits = self.client_plugin().absolute_limits()
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/nova.py", line 439, in absolute_limits
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack limits = self.client().limits.get()
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/v2/limits.py", line 100, in get
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack return self._get("/limits%s" % query_string, "limits")
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 147, in _get
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack _resp, body = self.api.client.get(url)
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 439, in get
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack return self._cs_request(url, 'GET', **kwargs)
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 426, in _cs_request
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack self.authenticate()
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 561, in authenticate
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack auth_url = self._v1_auth(auth_url)
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 609, in _v1_auth
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack raise exceptions.from_response(resp, body, url)
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack ClientException: Unknown Error (HTTP 300)
And then we have found that there is only auth_token and no username and password. It seems that python-novaclient cannot use v3 authentication token over v2.0.
Has anybody tested heat autoscaling on latest Kilo update? |
We upgraded environment to OpenStack Kilo 2015.1.2 and then we cannot use heat autoscaling feature anymore.
Keystone version 1:2015.1.2-0ubuntu2~cloud0
Heat version 2015.1.2-0ubuntu1~cloud0
- First we hit authentication failure on ec2tokens heat-api-cfn.log
2016-01-15 14:21:07.505 6590 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
2016-01-15 14:21:07.507 6590 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
2016-01-15 14:21:07.509 6590 INFO heat.api.aws.ec2token [-] Authenticating with http://10.200.26.10:5000/ec2tokens
2016-01-15 14:21:07.533 6590 INFO heat.api.aws.ec2token [-] AWS authentication failure.
2016-01-15 14:21:07.535 6590 INFO eventlet.wsgi.server [-] 10.200.26.11 - - [15/Jan/2016 14:21:07] "POST /v1/signal/arn%3Aopenstack%3Aheat%3A%3A4a28da719dc64deb9d1d5cfa770bbca2%3Astacks%2Fheat_autoscale_cpu_vpc20%2F3a268b22-d436-4cf8-9088-8418702be09a%2Fresources%2Fserver_scaleup_policy?Timestamp=2016-01-15T13%3A14%3A22Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=f042045aabc74728a49059a4c52226bf&SignatureVersion=2&Signature=GomCM13riX1UYu3OpJZ3ERXpfekHkmeACA4DznTL6g8%3D HTTP/1.1" 403 301 0.031258
We applied patch to keystone https://bugs.launchpad.net/openstack-ansible/+bug/1515485 https://git.openstack.org/cgit/openstack/keystone/commit/?id=f66badae63b1544c3bcc83e0c1afc8e9cdc71bab
Then we hit error on python-novaclient auth boot during autoscaling heat-engine.log
2016-01-15 06:43:05.836 11301 ERROR heat.engine.stack [req-fd1fd187-1be5-489a-803b-8c08e491ebd4 - 4a28da719dc64deb9d1d5cfa770bbca2] Exception: (NotFound(),)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack Traceback (most recent call last):
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 593, in validate
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack result = res.validate()
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/nova/server.py", line 1254, in validate
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack limits = self.client_plugin().absolute_limits()
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/nova.py", line 439, in absolute_limits
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack limits = self.client().limits.get()
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/v2/limits.py", line 100, in get
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack return self._get("/limits%s" % query_string, "limits")
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 147, in _get
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack _resp, body = self.api.client.get(url)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 440, in get
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack return self._cs_request(url, 'GET', **kwargs)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 427, in _cs_request
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack self.authenticate()
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 569, in authenticate
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack self._v2_auth(auth_url)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 634, in _v2_auth
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack return self._authenticate(url, body)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 647, in _authenticate
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack **kwargs)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 392, in _time_request
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack resp, body = self.request(url, method, **kwargs)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 386, in request
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack raise exceptions.from_response(resp, body, url, method)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack NotFound: The resource could not be found. (HTTP 404)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack
2016-01-15 06:43:05.838 11301 INFO heat.engine.stack [req-fd1fd187-1be5-489a-803b-8c08e491ebd4 - 4a28da719dc64deb9d1d5cfa770bbca2] Failed to validate: The resource could not be found. (HTTP 404)
It was caused by bad auth_url argument passed into novaclient - http://10.0.170.10:5000/v3/v2.0
We modified /usr/lib/python2.7/dist-packages/heat/engine/clients/os/nova.py to replace v3 to v2.0
args = {
'project_id': self.context.tenant,
'auth_url': self.context.auth_url.replace('/v3','/v2.0'),
'service_type': 'compute',
'username': None,
'api_key': None,
'extensions': extensions,
'endpoint_type': endpoint_type,
'http_log_debug': self._get_client_option('nova',
'http_log_debug'),
'cacert': self._get_client_option('nova', 'ca_file'),
'insecure': self._get_client_option('nova', 'insecure')
}
And then we got following error heat-engine.log
2016-01-15 14:36:05.464 8195 ERROR heat.engine.stack [req-b19963cf-81d3-482d-b31a-27d7b0862b89 - 4a28da719dc64deb9d1d5cfa770bbca2] Exception: (ClientException(),)
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack Traceback (most recent call last):
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 593, in validate
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack result = res.validate()
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/nova/server.py", line 1254, in validate
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack limits = self.client_plugin().absolute_limits()
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/nova.py", line 439, in absolute_limits
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack limits = self.client().limits.get()
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/v2/limits.py", line 100, in get
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack return self._get("/limits%s" % query_string, "limits")
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 147, in _get
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack _resp, body = self.api.client.get(url)
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 439, in get
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack return self._cs_request(url, 'GET', **kwargs)
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 426, in _cs_request
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack self.authenticate()
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 561, in authenticate
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack auth_url = self._v1_auth(auth_url)
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 609, in _v1_auth
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack raise exceptions.from_response(resp, body, url)
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack ClientException: Unknown Error (HTTP 300)
And then we have found that there is only auth_token and no username and password. It seems that python-novaclient cannot use v3 authentication token over v2.0.
https://bugs.launchpad.net/python-novaclient/+bug/1522402
Has anybody tested heat autoscaling on latest Kilo update? |
|
