OpenStack Identity (keystone)

Keystone v3 user/tenant lookup by name via OpenStack CLI client fails

Bug #1454309 reported by Edmund Rhudy on 2015-05-12

This bug affects 4 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Identity (keystone)	Fix Released	High	Edmund Rhudy	OpenStack Identity (keystone) 8.0.0 "liberty"
	Kilo	Fix Released	High	Dolph Mathews	OpenStack Identity (keystone) 2015.1.3

Bug Description

When using the openstack CLI client to look up users/tenants by name (e.g., openstack user show admin or openstack openstack project show AdminTenant), it fails with a 500 and the following traceback:

2015-05-12 09:27:22.483530 2015-05-12 09:27:22.483 31012 DEBUG keystone.common.ldap.core [-] LDAP search: base=ou=People,dc=local,dc=lan scope=2 filterstr=(&(&None(sn=admin))(objectClass=inetOrgPerson)) attrs=['cn', 'userPassword', 'enabled', 'sn', 'mail'] attrsonly=0 search_s /usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py:931
2015-05-12 09:27:22.483677 2015-05-12 09:27:22.483 31012 DEBUG keystone.common.ldap.core [-] LDAP unbind unbind_s /usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py:904
2015-05-12 09:27:22.485831 2015-05-12 09:27:22.483 31012 ERROR keystone.common.wsgi [-] {'desc': 'Bad search filter'}
2015-05-12 09:27:22.485874 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi Traceback (most recent call last):
2015-05-12 09:27:22.485881 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 239, in __call__
2015-05-12 09:27:22.485885 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi result = method(context, **params)
2015-05-12 09:27:22.485897 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 202, in wrapper
2015-05-12 09:27:22.485901 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi return f(self, context, filters, **kwargs)
2015-05-12 09:27:22.485904 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/controllers.py", line 223, in list_users
2015-05-12 09:27:22.485908 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi hints=hints)
2015-05-12 09:27:22.485911 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 52, in wrapper
2015-05-12 09:27:22.485915 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi return f(self, *args, **kwargs)
2015-05-12 09:27:22.485919 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 342, in wrapper
2015-05-12 09:27:22.485922 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi return f(self, *args, **kwargs)
2015-05-12 09:27:22.485926 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 353, in wrapper
2015-05-12 09:27:22.485930 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi return f(self, *args, **kwargs)
2015-05-12 09:27:22.485933 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 791, in list_users
2015-05-12 09:27:22.485937 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi ref_list = driver.list_users(hints)
2015-05-12 09:27:22.485941 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap.py", line 82, in list_users
2015-05-12 09:27:22.485944 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi return self.user.get_all_filtered(hints)
2015-05-12 09:27:22.485948 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap.py", line 269, in get_all_filtered
2015-05-12 09:27:22.485951 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi return [self.filter_attributes(user) for user in self.get_all(query)]
2015-05-12 09:27:22.485964 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py", line 1863, in get_all
2015-05-12 09:27:22.485968 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi for x in self._ldap_get_all(ldap_filter)
2015-05-12 09:27:22.485972 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py", line 1467, in _ldap_get_all
2015-05-12 09:27:22.485975 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi attrs)
2015-05-12 09:27:22.485979 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py", line 944, in search_s
2015-05-12 09:27:22.485983 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi attrlist_utf8, attrsonly)
2015-05-12 09:27:22.485986 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py", line 541, in search_s
2015-05-12 09:27:22.485995 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi attrlist, attrsonly)
2015-05-12 09:27:22.485999 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 552, in search_s
2015-05-12 09:27:22.486002 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout)
2015-05-12 09:27:22.486009 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 545, in search_ext_s
2015-05-12 09:27:22.486013 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi msgid = self.search_ext(base,scope,filterstr,attrlist,attrsonly,serverctrls,clientctrls,timeout,sizelimit)
2015-05-12 09:27:22.486017 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 541, in search_ext
2015-05-12 09:27:22.486036 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi timeout,sizelimit,
2015-05-12 09:27:22.486040 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 99, in _ldap_call
2015-05-12 09:27:22.486044 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi result = func(*args,**kwargs)
2015-05-12 09:27:22.486047 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi FILTER_ERROR: {'desc': 'Bad search filter'}
2015-05-12 09:27:22.486050 2015-05-12 09:27:22.483 31012 TRACE keystone.common.wsgi

The LDAP filter string is being composed in a way that causes None to be substituted in at one point: (&(&None(sn=admin))(objectClass=inetOrgPerson))

I traced it through the code and found that the problem method is keystone.common.ldap.core.BaseLdap.filter_query (line 1674 of keystone/common/ldap/core.py on the stable/kilo branch). The method argument query is None by default, which ends up being substituted into the query string later on. Changing the default value of query to an empty string causes things to function as expected.

(I am waiting on internal permission to contribute code, so I haven't created a PR for this at this time.)

Tags:

Revision history for this message

Dolph Mathews (dolph) wrote on 2015-05-13:

Can you confirm which release you're seeing this in? If this is in a stable release, it definitely needs to be fixed.

tags:	added: ldap
Changed in keystone:
importance:	Undecided → High
status:	New → Triaged
tags:	added: kilo-backport-potential

Revision history for this message

Edmund Rhudy (erhudy) wrote on 2015-05-13:

I experienced this issue under Kilo using package python-keystone-2015.1.0-0ubuntu1~cloud0, obtained from the Canonical cloud repository at http://ubuntu-cloud.archive.canonical.com/.

I am still working on isolating the circumstances under which it occurs; as detailed above, I was using the Keystone v3 API via the new openstack CLI. I haven't tested yet against v2.0 using either openstack or the legacy Keystone client.

Revision history for this message

Edmund Rhudy (erhudy) wrote on 2015-05-13:

Bug appears to only be present when using API v3; user list works in both, user show by ID works in both, user show by name only works against v2.0. (Below commands were run with the credentials already supplied in environment variables):

Revision history for this message

Edmund Rhudy (erhudy) wrote on 2015-05-18:

keystone-1454309.patch Edit (565 bytes, text/plain)

Revision history for this message

Thomas Hsiao (thomas-hsiao) wrote on 2015-05-19:

I encountered the same blocking issue for ldap domain user lookup by name for Kilo 2015.1.0.
I also tested your proposal and the issue seemed getting resolved. I am wondering if you are going to contribute that code very soon or…

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-05-20: Fix proposed to keystone (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/184622

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-05-21: Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/184824

Changed in keystone:
assignee:	nobody → Edmund Rhudy (erhudy)
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-05-21: Change abandoned on keystone (stable/kilo)

Change abandoned by Edmund Rhudy (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/184622
Reason: abandoning this review and creating new one off master

OpenStack Infra (hudson-openstack) on 2015-05-22

Changed in keystone:
assignee:	Edmund Rhudy (erhudy) → Guang Yee (guang-yee)

Edmund Rhudy (erhudy) on 2015-05-22

Changed in keystone:
assignee:	Guang Yee (guang-yee) → Edmund Rhudy (erhudy)

OpenStack Infra (hudson-openstack) on 2015-05-22

Changed in keystone:
assignee:	Edmund Rhudy (erhudy) → Brant Knudson (blk-u)

Brant Knudson (blk-u) on 2015-05-22

Changed in keystone:
assignee:	Brant Knudson (blk-u) → Edmund Rhudy (erhudy)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-05-22: Fix merged to keystone (master)

Reviewed: https://review.openstack.org/184824
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=2c6db4a3bb9e1718744b0e5b03af050fd2866182
Submitter: Jenkins
Branch: master

commit 2c6db4a3bb9e1718744b0e5b03af050fd2866182
Author: Edmund Rhudy <email address hidden>
Date: Thu May 21 12:42:40 2015 -0400

Make sure LDAP filter is constructed correctly

    This fixes an issue where, when querying Keystone via the v3 API, you
    would get back an invalid LDAP filter, because None was coerced to the
    string "None" and inserted into the middle of the query.

Change-Id: I9d45a4dca265b69e261f134118bb30c8cd128166
Closes-Bug: 1454309

Changed in keystone:
status:	In Progress → Fix Committed

Revision history for this message

Edmund Rhudy (erhudy) wrote on 2015-05-22:

#10

Fix was committed to Liberty.

Dolph, do you want this to be backported for Keystone? We're patching Keystone locally, but it would be nice to not have to do so.

Revision history for this message

Guang Yee (guang-yee) wrote on 2015-05-22:

#11

I think we need to backport.

Doug Hellmann (doug-hellmann) on 2015-06-23

Changed in keystone:
milestone:	none → liberty-1
status:	Fix Committed → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-07-15: Fix proposed to keystone (stable/kilo)

#12

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/201924

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-07-15: Change abandoned on keystone (stable/kilo)

#13

Change abandoned by Eric Brown (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/201924
Reason: Dup of https://review.openstack.org/#/c/184622/

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-08-14: Fix merged to keystone (stable/kilo)

#14

Reviewed: https://review.openstack.org/184622
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=bc96d9f63db29137b4a7f8727dd6ec0d6c848736
Submitter: Jenkins
Branch: stable/kilo

commit bc96d9f63db29137b4a7f8727dd6ec0d6c848736
Author: Edmund Rhudy <email address hidden>
Date: Thu May 21 12:42:40 2015 -0400

Make sure LDAP filter is constructed correctly

Conflicts:
keystone/tests/unit/common/test_ldap.py

The conflict is due to imports being reorganized in the same area as
this change.

    Change-Id: I9d45a4dca265b69e261f134118bb30c8cd128166
    Closes-Bug: 1454309
    (cherry picked from commit 2c6db4a3bb9e1718744b0e5b03af050fd2866182)

Thierry Carrez (ttx) on 2015-10-15