OpenStack Identity (keystone)

Autoscaling auth failure in OpenStack Kilo 2015.1.2

Bug #1534655 reported by Jakub Pavlik
26
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
New
Undecided
Unassigned
OpenStack Identity (keystone)
Incomplete
Undecided
Unassigned
python-novaclient
New
Undecided
Unassigned
heat (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

We upgraded environment to OpenStack Kilo 2015.1.2 and then we cannot use heat autoscaling feature anymore.

Keystone version 1:2015.1.2-0ubuntu2~cloud0
Heat version 2015.1.2-0ubuntu1~cloud0

- First we hit authentication failure on ec2tokens heat-api-cfn.log

2016-01-15 14:21:07.505 6590 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
2016-01-15 14:21:07.507 6590 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
2016-01-15 14:21:07.509 6590 INFO heat.api.aws.ec2token [-] Authenticating with http://10.200.26.10:5000/ec2tokens
2016-01-15 14:21:07.533 6590 INFO heat.api.aws.ec2token [-] AWS authentication failure.
2016-01-15 14:21:07.535 6590 INFO eventlet.wsgi.server [-] 10.200.26.11 - - [15/Jan/2016 14:21:07] "POST /v1/signal/arn%3Aopenstack%3Aheat%3A%3A4a28da719dc64deb9d1d5cfa770bbca2%3Astacks%2Fheat_autoscale_cpu_vpc20%2F3a268b22-d436-4cf8-9088-8418702be09a%2Fresources%2Fserver_scaleup_policy?Timestamp=2016-01-15T13%3A14%3A22Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=f042045aabc74728a49059a4c52226bf&SignatureVersion=2&Signature=GomCM13riX1UYu3OpJZ3ERXpfekHkmeACA4DznTL6g8%3D HTTP/1.1" 403 301 0.031258

We applied patch to keystone https://bugs.launchpad.net/openstack-ansible/+bug/1515485 https://git.openstack.org/cgit/openstack/keystone/commit/?id=f66badae63b1544c3bcc83e0c1afc8e9cdc71bab

Then we hit error on python-novaclient auth boot during autoscaling heat-engine.log

2016-01-15 06:43:05.836 11301 ERROR heat.engine.stack [req-fd1fd187-1be5-489a-803b-8c08e491ebd4 - 4a28da719dc64deb9d1d5cfa770bbca2] Exception: (NotFound(),)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack Traceback (most recent call last):
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 593, in validate
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack result = res.validate()
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/nova/server.py", line 1254, in validate
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack limits = self.client_plugin().absolute_limits()
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/nova.py", line 439, in absolute_limits
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack limits = self.client().limits.get()
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/v2/limits.py", line 100, in get
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack return self._get("/limits%s" % query_string, "limits")
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 147, in _get
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack _resp, body = self.api.client.get(url)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 440, in get
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack return self._cs_request(url, 'GET', **kwargs)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 427, in _cs_request
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack self.authenticate()
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 569, in authenticate
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack self._v2_auth(auth_url)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 634, in _v2_auth
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack return self._authenticate(url, body)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 647, in _authenticate
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack **kwargs)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 392, in _time_request
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack resp, body = self.request(url, method, **kwargs)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 386, in request
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack raise exceptions.from_response(resp, body, url, method)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack NotFound: The resource could not be found. (HTTP 404)
2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack
2016-01-15 06:43:05.838 11301 INFO heat.engine.stack [req-fd1fd187-1be5-489a-803b-8c08e491ebd4 - 4a28da719dc64deb9d1d5cfa770bbca2] Failed to validate: The resource could not be found. (HTTP 404)

It was caused by bad auth_url argument passed into novaclient - http://10.0.170.10:5000/v3/v2.0

We modified /usr/lib/python2.7/dist-packages/heat/engine/clients/os/nova.py to replace v3 to v2.0

        args = {
            'project_id': self.context.tenant,
            'auth_url': self.context.auth_url.replace('/v3','/v2.0'),
            'service_type': 'compute',
            'username': None,
            'api_key': None,
            'extensions': extensions,
            'endpoint_type': endpoint_type,
            'http_log_debug': self._get_client_option('nova',
                                                      'http_log_debug'),
            'cacert': self._get_client_option('nova', 'ca_file'),
            'insecure': self._get_client_option('nova', 'insecure')
        }

And then we got following error heat-engine.log

2016-01-15 14:36:05.464 8195 ERROR heat.engine.stack [req-b19963cf-81d3-482d-b31a-27d7b0862b89 - 4a28da719dc64deb9d1d5cfa770bbca2] Exception: (ClientException(),)
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack Traceback (most recent call last):
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 593, in validate
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack result = res.validate()
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/nova/server.py", line 1254, in validate
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack limits = self.client_plugin().absolute_limits()
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/nova.py", line 439, in absolute_limits
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack limits = self.client().limits.get()
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/v2/limits.py", line 100, in get
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack return self._get("/limits%s" % query_string, "limits")
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 147, in _get
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack _resp, body = self.api.client.get(url)
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 439, in get
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack return self._cs_request(url, 'GET', **kwargs)
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 426, in _cs_request
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack self.authenticate()
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 561, in authenticate
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack auth_url = self._v1_auth(auth_url)
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 609, in _v1_auth
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack raise exceptions.from_response(resp, body, url)
2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack ClientException: Unknown Error (HTTP 300)

And then we have found that there is only auth_token and no username and password. It seems that python-novaclient cannot use v3 authentication token over v2.0.
 https://bugs.launchpad.net/python-novaclient/+bug/1522402

Has anybody tested heat autoscaling on latest Kilo update?

See original description
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in heat (Ubuntu):
status: New → Confirmed
Jakub Pavlik (pavlk-jakub)
description: updated
Morgan Fainberg (mdrnstm)
tags: removed: keystone
Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

this seems like a novaclient issue not a keystone issue?

Marking as incomplete as i don't think there is anything keystone can do here.

Changed in keystone:
status: New → Incomplete
Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

adding Novaclient as this appears to be related to it's lack of V3 support.

Revision history for this message
melanie witt (melwitt) wrote :

At present, the only way to make the novaclient Python API work with keystone v3 is to construct a keystoneclient Session object and pass it to novaclient Client when creating the client object.

From comment #11 on bug 1522402:

"http://docs.openstack.org/developer/python-novaclient/api.html#usage

Note that you must create a Session object and pass it to novaclient.Client(session=<session object>) when creating the novaclient object. If you don't, novaclient will fall back on its own code for keystone auth and will NOT work with keystone v3."

Revision history for this message
Marek Celoud (marek-celoud) wrote :

We hit the same issue with Liberty. However there is a fix for heat to use keystone session from context in client plugin instead of username,password.

https://github.com/openstack/heat/commit/cd125f328e3e4e9f517278b04347515eba380d5a

This fix is only for Newton. When try in Liberty:

    def _create(self):
        endpoint_type = self._get_client_option('nova', 'endpoint_type')
        extensions = nc.discover_extensions('2')

        args = {
            'session': self.context.keystone_session,
            'extensions': extensions,
            'interface': endpoint_type,
            'service_type': self.COMPUTE,
            'http_log_debug': self._get_client_option('nova',
                                                      'http_log_debug')
        }

We got error:

2016-09-22 16:27:18.355 26510 ERROR heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/nova.py", line 95, in _create
2016-09-22 16:27:18.355 26510 ERROR heat.engine.stack 'session': self.context.keystone_session,
2016-09-22 16:27:18.355 26510 ERROR heat.engine.stack AttributeError: 'RequestContext' object has no attribute 'keystone_session'

When we tried put there "'session': self._keystone_session," instead:

2016-09-22 16:28:18.528 26861 ERROR heat.engine.stack File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 596, in _auth_required
2016-09-22 16:28:18.528 26861 ERROR heat.engine.stack raise exceptions.MissingAuthPlugin(msg)
2016-09-22 16:28:18.528 26861 ERROR heat.engine.stack MissingAuthPlugin: An auth plugin is required to determine endpoint URL

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.