Internal Server Error while updating project quota when using Keystone LDAP backend

[Bjoern]

[Thu Mar 19 18:57:12.158459 2015] [:error] [pid 4777:tid 139987244758784]
[Thu Mar 19 18:57:12.184901 2015] [:error] [pid 4777:tid 139987244758784] REQ: curl -i --insecure 'http://1.2.3.4:8774/v2/b41a90047bef4e7d9031cc72d692a5c5/limits' -X GET -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: b41a90047bef4e7d9031cc72d692a5c5" -H "X-Auth-Token: {SHA1}b03c7bf01bdb967570a6497da2c7cea1ec452793"
[Thu Mar 19 18:57:12.226996 2015] [:error] [pid 4777:tid 139987244758784] RESP: [200] {'date': 'Thu, 19 Mar 2015 18:57:12 GMT', 'connection': 'keep-alive', 'content-type': 'application/json', 'content-length': '512', 'x-compute-request-id': 'req-ce76d5d7-49df-4986-9738-d23c9953fbd7'}
[Thu Mar 19 18:57:12.227025 2015] [:error] [pid 4777:tid 139987244758784] RESP BODY: {"limits": {"rate": [], "absolute": {"maxServerMeta": 128, "maxPersonality": 5, "totalServerGroupsUsed": 0, "maxImageMeta": 128, "maxPersonalitySize": 10240, "maxServerGroups": 10, "maxSecurityGroupRules": -1, "maxTotalKeypairs": 100, "totalCoresUsed": 8, "totalRAMUsed": 12432, "totalInstancesUsed": 8, "maxSecurityGroups": -1, "totalFloatingIpsUsed": 0, "maxTotalCores": -1, "totalSecurityGroupsUsed": 1, "maxTotalFloatingIps": 10, "maxTotalInstances": -1, "maxTotalRAMSize": -1, "maxServerGroupMembers": 10}}}
[Thu Mar 19 18:57:12.227038 2015] [:error] [pid 4777:tid 139987244758784]
[Thu Mar 19 19:00:16.059014 2015] [:error] [pid 4776:tid 139987244758784] REQ: curl -i --insecure 'http://1.2.3.4:8774/v2/52a54b9b064348108a396f14ce9f23e3/os-quota-sets/b41a90047bef4e7d9031cc72d692a5c5' -X GET -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: 52a54b9b064348108a396f14ce9f23e3" -H "X-Auth-Token: {SHA1}08bec92f7b2951f9532f59c0f9e884338f73449a"
[Thu Mar 19 19:00:16.095317 2015] [:error] [pid 4776:tid 139987244758784] RESP: [200] {'date': 'Thu, 19 Mar 2015 19:00:16 GMT', 'connection': 'keep-alive', 'content-type': 'application/json', 'content-length': '368', 'x-compute-request-id': 'req-e40567ce-3384-4a5b-b1f0-62e12f15f7b8'}
[Thu Mar 19 19:00:16.095365 2015] [:error] [pid 4776:tid 139987244758784] RESP BODY: {"quota_set": {"injected_file_content_bytes": 10240, "metadata_items": 128, "server_group_members": 10, "server_groups": 10, "ram": -1, "floating_ips": 10, "key_pairs": 100, "id": "b41a90047bef4e7d9031cc72d692a5c5", "instances": -1, "security_group_rules": -1, "injected_files": 5, "cores": -1, "fixed_ips": -1, "injected_file_path_bytes": 255, "security_groups": -1}}
[Thu Mar 19 19:00:16.095378 2015] [:error] [pid 4776:tid 139987244758784]
[Thu Mar 19 19:00:18.824318 2015] [:error] [pid 4776:tid 139987244758784] Problem instantiating action class.
[Thu Mar 19 19:00:18.824357 2015] [:error] [pid 4776:tid 139987244758784] Traceback (most recent call last):
[Thu Mar 19 19:00:18.824363 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/openstack_dashboard/wsgi/../../horizon/workflows/base.py", line 368, in action
[Thu Mar 19 19:00:18.824367 2015] [:error] [pid 4776:tid 139987244758784] context)
[Thu Mar 19 19:00:18.824370 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/identity/projects/workflows.py", line 196, in __init__
[Thu Mar 19 19:00:18.824374 2015] [:error] [pid 4776:tid 139987244758784] users_list = [(user.id, user.name) for user in all_users]
[Thu Mar 19 19:00:18.824378 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/openstack_dashboard/wsgi/../../keystoneclient/openstack/common/apiclient/base.py", line 480, in __getattr__
[Thu Mar 19 19:00:18.824382 2015] [:error] [pid 4776:tid 139987244758784] raise AttributeError(k)
[Thu Mar 19 19:00:18.824385 2015] [:error] [pid 4776:tid 139987244758784] AttributeError: name
[Thu Mar 19 19:00:18.824858 2015] [:error] [pid 4776:tid 139987244758784] Internal Server Error: /identity/b41a90047bef4e7d9031cc72d692a5c5/update/
[Thu Mar 19 19:00:18.824873 2015] [:error] [pid 4776:tid 139987244758784] Traceback (most recent call last):
[Thu Mar 19 19:00:18.824877 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 137, in get_response
[Thu Mar 19 19:00:18.824881 2015] [:error] [pid 4776:tid 139987244758784] response = response.render()
[Thu Mar 19 19:00:18.824897 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/template/response.py", line 105, in render
[Thu Mar 19 19:00:18.824901 2015] [:error] [pid 4776:tid 139987244758784] self.content = self.rendered_content
[Thu Mar 19 19:00:18.824904 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/template/response.py", line 82, in rendered_content
[Thu Mar 19 19:00:18.824906 2015] [:error] [pid 4776:tid 139987244758784] content = template.render(context)
[Thu Mar 19 19:00:18.824911 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 140, in render
[Thu Mar 19 19:00:18.824915 2015] [:error] [pid 4776:tid 139987244758784] return self._render(context)
[Thu Mar 19 19:00:18.824918 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 134, in _render
[Thu Mar 19 19:00:18.824922 2015] [:error] [pid 4776:tid 139987244758784] return self.nodelist.render(context)
[Thu Mar 19 19:00:18.824926 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
[Thu Mar 19 19:00:18.824929 2015] [:error] [pid 4776:tid 139987244758784] bit = self.render_node(node, context)
[Thu Mar 19 19:00:18.824932 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/template/debug.py", line 78, in render_node
[Thu Mar 19 19:00:18.824935 2015] [:error] [pid 4776:tid 139987244758784] return node.render(context)
[Thu Mar 19 19:00:18.824938 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/template/defaulttags.py", line 506, in render
[Thu Mar 19 19:00:18.824941 2015] [:error] [pid 4776:tid 139987244758784] output = self.nodelist.render(context)
[Thu Mar 19 19:00:18.824943 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
[Thu Mar 19 19:00:18.824946 2015] [:error] [pid 4776:tid 139987244758784] bit = self.render_node(node, context)
[Thu Mar 19 19:00:18.824949 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/template/debug.py", line 78, in render_node
[Thu Mar 19 19:00:18.824952 2015] [:error] [pid 4776:tid 139987244758784] return node.render(context)
[Thu Mar 19 19:00:18.824954 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/template/loader_tags.py", line 53, in render
[Thu Mar 19 19:00:18.824957 2015] [:error] [pid 4776:tid 139987244758784] result = self.nodelist.render(context)
[Thu Mar 19 19:00:18.824960 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
[Thu Mar 19 19:00:18.824962 2015] [:error] [pid 4776:tid 139987244758784] bit = self.render_node(node, context)
[Thu Mar 19 19:00:18.824965 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/template/debug.py", line 78, in render_node
[Thu Mar 19 19:00:18.824968 2015] [:error] [pid 4776:tid 139987244758784] return node.render(context)
[Thu Mar 19 19:00:18.824970 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/template/defaulttags.py", line 196, in render
[Thu Mar 19 19:00:18.824973 2015] [:error] [pid 4776:tid 139987244758784] nodelist.append(node.render(context))
[Thu Mar 19 19:00:18.824976 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/template/defaulttags.py", line 298, in render
[Thu Mar 19 19:00:18.824979 2015] [:error] [pid 4776:tid 139987244758784] match = condition.eval(context)
[Thu Mar 19 19:00:18.824981 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/template/defaulttags.py", line 867, in eval
[Thu Mar 19 19:00:18.824990 2015] [:error] [pid 4776:tid 139987244758784] return self.value.resolve(context, ignore_failures=True)
[Thu Mar 19 19:00:18.824992 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 585, in resolve
[Thu Mar 19 19:00:18.824995 2015] [:error] [pid 4776:tid 139987244758784] obj = self.var.resolve(context)
[Thu Mar 19 19:00:18.824998 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 735, in resolve
[Thu Mar 19 19:00:18.825001 2015] [:error] [pid 4776:tid 139987244758784] value = self._resolve_lookup(context)
[Thu Mar 19 19:00:18.825003 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 789, in _resolve_lookup
[Thu Mar 19 19:00:18.825006 2015] [:error] [pid 4776:tid 139987244758784] current = current()
[Thu Mar 19 19:00:18.825009 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/openstack_dashboard/wsgi/../../horizon/workflows/base.py", line 439, in has_required_fields
[Thu Mar 19 19:00:18.825012 2015] [:error] [pid 4776:tid 139987244758784] return any(field.required for field in self.action.fields.values())
[Thu Mar 19 19:00:18.825015 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/openstack_dashboard/wsgi/../../horizon/workflows/base.py", line 368, in action
[Thu Mar 19 19:00:18.825018 2015] [:error] [pid 4776:tid 139987244758784] context)
[Thu Mar 19 19:00:18.825021 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/identity/projects/workflows.py", line 196, in __init__
[Thu Mar 19 19:00:18.825024 2015] [:error] [pid 4776:tid 139987244758784] users_list = [(user.id, user.name) for user in all_users]
[Thu Mar 19 19:00:18.825027 2015] [:error] [pid 4776:tid 139987244758784] File "/usr/local/lib/python2.7/dist-packages/openstack_dashboard/wsgi/../../keystoneclient/openstack/common/apiclient/base.py", line 480, in __getattr__
[Thu Mar 19 19:00:18.825030 2015] [:error] [pid 4776:tid 139987244758784] raise AttributeError(k)
[Thu Mar 19 19:00:18.825033 2015] [:error] [pid 4776:tid 139987244758784] AttributeError: name

[Bjoern (bjoern-t)]

I'm running Horizon 2014.2.1 on Juno. Currently it is not clear why the Project > Modify Quota bails out.
I was able to modify quotas using nova but not though Horizon.

[Bjoern (bjoern-t)]

Installed clients inside the Horizon container :

python-ceilometerclient==1.0.9
python-cinderclient==1.1.1
python-glanceclient==0.15.0
python-heatclient==0.2.10
python-keystoneclient==1.0.0
python-neutronclient==2.3.9.post26
python-novaclient==2.18.1
python-saharaclient==0.7.6
python-swiftclient==2.2.0
python-troveclient==1.0.5

Installed Keystone components:

keystone==2014.2.1
keystonemiddleware==1.3.1
python-keystoneclient==1.0.0
python-ldap==2.3.13

[Bjoern (bjoern-t)]

All. This error is related to the number of users inside the LDAP sub-tree. Currently we do have 4824 users (1110 disabled users) inside the LDAP sub tree which keystone is reporting. All user information is collected while pulling up the project quota and that seems to throw off django while building generating the template response.
page_size seems to have an additional effect. Once I set the page_size to 0, I get the Project Quotas listed but the User list is not generated inside this template and horizon aborts the request with :

[Thu Mar 19 20:03:13.968444 2015] [:error] [pid 6122:tid 140350672860928]
[Thu Mar 19 20:03:14.231283 2015] [:error] [pid 6122:tid 140350672860928] Recoverable error: An unexpected error prevented the server from fulfilling your request: {'desc': 'Size limit exceeded'} (Disable debug mode to suppress these details.) (HTTP 500)

[Richard Jones (r1chardj0n3s)]

'Size limit exceeded' is an error returned by Active Directory in response to a search resulting in over 1000 results (eg ref http://blog.tirasa.net/getting-around-active-directory-search.html).

Pagination or filtering could be an option here.

Keystone API v2 has a "limit" parameter (with no facility for fetching past the limit, as far as I can tell) and v3 has pagination via "per_page" and "page". Unfortunately, the user list interface here needs a complete re-work as it assumes the entire user list is loaded; filtering is done on the client side.

I'll note that the new Identity listing using Angular will run into this issue (hi, Thai!) since it applies no limit to its user listing fetch.

[Richard Jones (r1chardj0n3s)]

I believe the correct fix for this is to enable a user_filter in the LDAP keystone configuration.

http://docs.openstack.org/admin-guide-cloud/content/configuring-keystone-for-ldap-backend-identity.html

I will investigate un-breaking the interface even in the face of poor filtering.

[Timur Sufiev (tsufiev-x)]

Hello, Bjoern!

We had the same 'Size limit exceeded' LDAP while updating quotas for a Project, but after pressing 'Save' button we also had the problems with revoking the admin role for current user in a tenant. I wonder whether you did encounter the same problem?

[Bjoern (bjoern-t)]

I got my error inside the Project > Modify Quota menu, just trying to load the quotas even. I do agree that a user filter should be in place to reduce result set but Horizon should be smart enough to support paging in this case.

[Rob Cresswell (robcresswell-deactivatedaccount)]

Marking undecided until its verified. It's pretty difficult for us to recreate this style of scale bug; if it still exists, it may require a video call / youtube recording or something.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Dashboard (Horizon) because there has been no activity for 60 days.]