OpenStack Identity (keystone)

ldap unicode issue with mapping id generator

Bug #1419187 reported by Kevin Fox on 2015-02-06

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Identity (keystone)	Fix Released	Critical	Henry Nash	OpenStack Identity (keystone) 2015.1.0 "kilo"
	Juno	Fix Released	Critical	Henry Nash	OpenStack Identity (keystone) 2014.2.3

Bug Description

exception when listing users in ldap:

2015-02-06 14:52:21.934 27199 ERROR keystone.common.wsgi [-] 'ascii' codec can't encode character u'\xe9' in position 19: ordinal not in range(128)
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi Traceback (most recent call last):
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 223, in __call__
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi result = method(context, **params)
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/identity/controllers.py", line 48, in get_users
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi CONF.identity.default_domain_id)
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/manager.py", line 47, in wrapper
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi return f(self, *args, **kwargs)
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 202, in wrapper
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi return f(self, *args, **kwargs)
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 213, in wrapper
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi return f(self, *args, **kwargs)
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 620, in list_users
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi ref_list, domain_scope, driver, mapping.EntityType.USER)
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 328, in _set_domain_id_and_mapping
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi x, domain_id, driver, entity_type) for x in ref]
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 325, in _set_domain_id_and_mapping
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi ref, domain_id, driver, entity_type, conf)
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 359, in _set_domain_id_and_mapping_for_single_ref
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi local_entity, public_id)
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/identity/mapping_backends/sql.py", line 70, in create_id_mapping
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi public_id = self.id_generator_api.generate_public_ID(entity)
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/identity/id_generators/sha256.py", line 27, in generate_public_ID
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi m.update(mapping[key])
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 19: ordinal not in range(128)
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi
2015-02-06 14:52:21.959 27199 INFO eventlet.wsgi.server [-] 172.20.96.65 - - [06/Feb/2015 14:52:21] "GET /v2.0/users HTTP/1.1" 500 291 23.150229

'local_entity' is here:
https://github.com/openstack/keystone/blob/c4c8d0b99a0404f4dcdb2f87c48fe15ee1526197/keystone/identity/mapping_backends/sql.py#L66

its: {'local_id': u'^PNNL T&Q Communiqu\xe9', 'domain_id':
'default', 'entity_type': 'user'}

Tags:

Revision history for this message

Kevin Fox (kevpn) wrote on 2015-02-06:

Oh. This is rdo juno on centos 7.

Revision history for this message

Morgan Fainberg (mdrnstm) wrote on 2015-02-07:

This could be related to: https://bugs.launchpad.net/keystone/+bug/1172106

Revision history for this message

Morgan Fainberg (mdrnstm) wrote on 2015-02-07:

This bug is with the mapping id generator not properly handling encoding for text strings.

Changed in keystone:
importance:	Undecided → Critical
status:	New → Triaged

Revision history for this message

Morgan Fainberg (mdrnstm) wrote on 2015-02-07:

Confirmed to affect Juno

Revision history for this message

Morgan Fainberg (mdrnstm) wrote on 2015-02-07:

Key Snippets from the traceback:

2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/identity/mapping_backends/sql.py", line 70, in create_id_mapping
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi public_id = self.id_generator_api.generate_public_ID(entity)
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/identity/id_generators/sha256.py", line 27, in generate_public_ID
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi m.update(mapping[key])
2015-02-06 14:52:21.934 27199 TRACE keystone.common.wsgi UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 19: ordinal not in range(128)

Henry Nash (henry-nash) on 2015-02-07

Changed in keystone:
assignee:	nobody → Henry Nash (henry-nash)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-08: Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/153844

Changed in keystone:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-08:

Fix proposed to branch: master
Review: https://review.openstack.org/153846

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-09: Fix merged to keystone (master)

Reviewed: https://review.openstack.org/153844
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=635ff8f307a3891d5898aed0212b42ad0d89a43e
Submitter: Jenkins
Branch: master

commit 635ff8f307a3891d5898aed0212b42ad0d89a43e
Author: Henry Nash <email address hidden>
Date: Sun Feb 8 09:35:50 2015 +0000

Improve testing of unicode id mapping

    We currently don't test that our id mapping correctly supports
    unicode - and in fact, it doesn't. This patch adds a test that
    demonstrates this. A follow-on patch will fix the issue.

Change-Id: Ief0597bbd0f1486f23eb799b68a85b590b7e35ef
Partial-Bug: 1419187

Changed in keystone:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-09:

Reviewed: https://review.openstack.org/153846
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=4f0107e43491ec5ed829b83eae7be32fea5ba659
Submitter: Jenkins
Branch: master

commit 4f0107e43491ec5ed829b83eae7be32fea5ba659
Author: Henry Nash <email address hidden>
Date: Sun Feb 8 09:44:12 2015 +0000

Make identity id mapping handle unicode

    Idenitity id mapping is used to create public ids for local entities,
    typically stored in LDAP backends. Part of the mapping involves
    creating a hash of the local identifiers - but this hashing did
    not correctly handle unicode. This patch fixes this.

Change-Id: Icc2a6bc4a7e88004bbe6f86d3a96cff07be4c6f9
Closes-Bug: 1419187

Morgan Fainberg (mdrnstm) on 2015-02-09

Changed in keystone:
milestone:	none → kilo-3
summary:	- ldap unicode issue + ldap unicode issue with mapping id generator

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-12: Fix proposed to keystone (stable/juno)

#10

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/155279

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-12:

#11

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/155282

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-13:

#12

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/155767

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-13: Change abandoned on keystone (stable/juno)

#13

Change abandoned by henry-nash (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/155279
Reason: Now that commit ID has been corrected, this patch is now here: https://review.openstack.org/#/c/155767/

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-13: Fix proposed to keystone (stable/juno)

#14

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/155769

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-13: Change abandoned on keystone (stable/juno)

#15

Change abandoned by henry-nash (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/155282
Reason: Correcting commit ID means that this change is now here: https://review.openstack.org/#/c/155769/

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-16: Fix merged to keystone (stable/juno)

#16

Reviewed: https://review.openstack.org/155767
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=a322ceea9df5c4733e09bfe24510bf58e4f37d9e
Submitter: Jenkins
Branch: stable/juno

commit a322ceea9df5c4733e09bfe24510bf58e4f37d9e
Author: Henry Nash <email address hidden>
Date: Sun Feb 8 09:35:50 2015 +0000

Improve testing of unicode id mapping

    We currently don't test that our id mapping correctly supports
    unicode - and in fact, it doesn't. This patch adds a test that
    demonstrates this. A follow-on patch will fix the issue.

(cherry picked from commit 635ff8f307a3891d5898aed0212b42ad0d89a43e)
Partial-Bug: 1419187

Change-Id: Ief0597bbd0f1486f23eb799b68a85b590b7e35ef

tags:

added: in-stable-juno

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-16:

#17

Reviewed: https://review.openstack.org/155769
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=ddfa37eefee567806027860af0412c00b4a2c384
Submitter: Jenkins
Branch: stable/juno

commit ddfa37eefee567806027860af0412c00b4a2c384
Author: Henry Nash <email address hidden>
Date: Sun Feb 8 09:44:12 2015 +0000

Make identity id mapping handle unicode

    Identity id mapping is used to create public ids for local entities,
    typically stored in LDAP backends. Part of the mapping involves
    creating a hash of the local identifiers - but this hashing did
    not correctly handle unicode. This patch fixes this.

(cherry picked from commit 4f0107e43491ec5ed829b83eae7be32fea5ba659)
Closes-Bug: 1419187

Change-Id: Icc2a6bc4a7e88004bbe6f86d3a96cff07be4c6f9

Thierry Carrez (ttx) on 2015-03-19

Changed in keystone:
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2015-04-30

Changed in keystone:
milestone:	kilo-3 → 2015.1.0

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.