Heat can't create user in keystone

[Daniil Lapshin]

Unable to create heat stack with autoscaling policy, because heat cannot create stack user in keystone. Looks like 'description' field, required by keystone, isn't passed by heat.

Here are snippet from heat-all.log:

<134>Sep 27 05:21:01 node-20 heat-engine 2015-09-27 05:21:01.631 8875 INFO heat.engine.resource [-] CREATE: AutoScalingPolicy "scale_up_policy" Stack "mtestcase" [fe5f99e5-eb94-41ba-a6ce-c79fbbf21dd0]
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource Traceback (most recent call last):
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 435, in _action_recorder
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource yield
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 505, in _do_action
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource yield self.action_handler_task(action, args=handler_args)
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py", line 286, in wrapper
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource step = next(subtask)
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 476, in action_handler_task
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource handler_data = handler(*args)
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/scaling_policy.py", line 90, in handle_create
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource super(AutoScalingPolicy, self).handle_create()
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/signal_responder.py", line 41, in handle_create
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource super(SignalResponder, self).handle_create()
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/stack_user.py", line 34, in handle_create
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource self._create_user()
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/stack_user.py", line 47, in _create_user
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource project_id=self.stack.stack_user_project_id)
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/common/heat_keystoneclient.py", line 406, in create_stack_domain_user
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource return self.create_stack_user(username=username, password=password)
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/common/heat_keystoneclient.py", line 331, in create_stack_user
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource default_project=self.context.tenant_id)
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource return func(*args, **kwargs)
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/users.py", line 74, in create
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource log=not bool(password))
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 150, in _create
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource return self._post(url, body, response_key, return_raw, **kwargs)
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 164, in _post
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource resp, body = self.client.post(url, body=body, **kwargs)
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 635, in post
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource return self._cs_request(url, 'POST', **kwargs)
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 621, in _cs_request
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource return self.request(url, method, **kwargs)
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 596, in request
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource resp = super(HTTPClient, self).request(url, method, **kwargs)
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/keystoneclient/baseclient.py", line 21, in request
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource return self.session.request(url, method, **kwargs)
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource return func(*args, **kwargs)
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 346, in request
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource raise exceptions.from_response(resp, method, url)
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource InternalServerError: An unexpected error prevented the server from fulfilling your request. (HTTP 500)
2015-09-27 05:21:01.631 8875 TRACE heat.engine.resource
<134>Sep 27 05:21:02 node-20 heat-engine 2015-09-27 05:21:02.675 8875 INFO heat.engine.stack [-] Stack CREATE FAILED (mtestcase): Resource CREATE failed: InternalServerError: An unexpected error prevented the server from fulfilling your request. (HTTP 500)
<134>Sep 27 05:21:02 node-20 heat-engine 2015-09-27 05:21:02.688 8875 INFO heat.engine.service [-] Stack create failed, status FAILED

Here are snippet from feystone-all.log:

<14>Sep 27 05:21:01 node-18 keystone-all 10.135.163.19 - - [27/Sep/2015 05:21:01] "OPTIONS / HTTP/1.0" 300 925 0.001276
<11>Sep 27 05:21:01 node-18 keystone-all 'description'
2015-09-27 05:21:01.608 45192 TRACE keystone.common.wsgi Traceback (most recent call last):
2015-09-27 05:21:01.608 45192 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 223, in __call__
2015-09-27 05:21:01.608 45192 TRACE keystone.common.wsgi result = method(context, **params)
2015-09-27 05:21:01.608 45192 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 156, in inner
2015-09-27 05:21:01.608 45192 TRACE keystone.common.wsgi return f(self, context, *args, **kwargs)
2015-09-27 05:21:01.608 45192 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/controllers.py", line 220, in create_user
2015-09-27 05:21:01.608 45192 TRACE keystone.common.wsgi ref = self.identity_api.create_user(ref)
2015-09-27 05:21:01.608 45192 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/notifications.py", line 112, in wrapper
2015-09-27 05:21:01.608 45192 TRACE keystone.common.wsgi result = f(*args, **kwargs)
2015-09-27 05:21:01.608 45192 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 202, in wrapper
2015-09-27 05:21:01.608 45192 TRACE keystone.common.wsgi return f(self, *args, **kwargs)
2015-09-27 05:21:01.608 45192 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 213, in wrapper
2015-09-27 05:21:01.608 45192 TRACE keystone.common.wsgi return f(self, *args, **kwargs)
2015-09-27 05:21:01.608 45192 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 559, in create_user
2015-09-27 05:21:01.608 45192 TRACE keystone.common.wsgi user['description'] = user_ref['description']
2015-09-27 05:21:01.608 45192 TRACE keystone.common.wsgi KeyError: 'description'
2015-09-27 05:21:01.608 45192 TRACE keystone.common.wsgi
<14>Sep 27 05:21:01 node-18 keystone-all 10.135.163.2 - - [27/Sep/2015 05:21:01] "POST /v3/users HTTP/1.1" 500 310 0.037797

MOS version 6.0
Keystone identity backend - LDAP
# dpkg --list | grep -E -e 'heat|keystone'
ii heat-api 2014.2-fuel6.0~mira2 OpenStack orchestration service - ReST API
ii heat-api-cfn 2014.2-fuel6.0~mira2 OpenStack orchestration service - CFN API
ii heat-api-cloudwatch 2014.2-fuel6.0~mira2 OpenStack orchestration service - CloudWatch API
ii heat-common 2014.2-fuel6.0~mira2 OpenStack orchestration service - common files
ii heat-engine 2014.2-fuel6.0~mira2 OpenStack orchestration service - engine
ii keystone 1:2014.2-fuel6.0~mira12 OpenStack identity service - Daemons
ii keystone-doc 1:2014.2-fuel6.0~mira12 OpenStack identity service - Documentation
ii python-heat 2014.2-fuel6.0~mira2 OpenStack orchestration service - Python files
ii python-heatclient 0.2.12-fuel6.0~mira9 client library and CLI for OpenStack Heat
ii python-keystone 1:2014.2-fuel6.0~mira12 OpenStack identity service - Python library
ii python-keystoneclient 1:0.11.1-fuel6.0~mira12 Client library for OpenStack Identity API
ii python-keystonemiddleware 1.2.0-ubuntu5 Middleware for OpenStack Identity (Keystone) - Python 2.x

[Pavlo Shchelokovskyy (pshchelo)]

when/how this was found?

we recently saw such error on a customer env where a certain patch was introduced to Keystone. the patch is not present in any of our branches

for example, if you clone the Keystone repo, checkout the openstack-ci/fuel-6.0/2014.2 branch and search for the failing line

rgrep "user_ref\['description" .

nothing is found! So how do we see this error?

Please ensure that the problem is existing on vanilla versions first. If those lines in Keystone code were introduced by LDAP plugin, target the bug on plugin, not Heat.

[Pavlo Shchelokovskyy (pshchelo)]

checked other Keystone branches in our repo as well

openstack-ci/fuel-6.1/2014.2
openstack-ci/fuel-7.0/2015.1.0
openstack-ci/fuel-8.0/liberty
stable/liberty
master

none of them contain the failing line of code.

Please ensure where did it get from.

[Pavlo Shchelokovskyy (pshchelo)]

besides, in MOS 6.0 Heat + Keystone V3 is not configured at all. Autoscaling would not work there out-of-the-box in any way

[Roman Podoliaka (rpodolyaka)]

A month passed without response. Closing as Invalid.