non-ascii usernames broken in nova

[Andrew Bogott]

Taking mtaylor's suggestion, I've been trying to create a user with the name 'måņŧÿ'. It fails in various ways. Horizon just says 'Error: Error creating user: Unhandled error.' Nova-manage gives more info:

$ nova-manage user create måņŧÿ
2012-02-09 20:16:46,518 ERROR nova.exception [req-67983579-0a3a-4004-b702-3ba4ebec5eea None None] DB exception wrapped.
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/opt/stack/nova/nova/exception.py", line 104, in _wrap
(nova.exception): TRACE: return f(*args, **kwargs)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1400, in flush
(nova.exception): TRACE: self._flush(objects)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1481, in _flush
(nova.exception): TRACE: flush_context.execute()
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 302, in execute
(nova.exception): TRACE: rec.execute(self)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 446, in execute
(nova.exception): TRACE: uow
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/mapper.py", line 1887, in _save_obj
(nova.exception): TRACE: execute(statement, params)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1191, in execute
(nova.exception): TRACE: params)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1271, in _execute_clauseelement
(nova.exception): TRACE: return self.__execute_context(context)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1302, in __execute_context
(nova.exception): TRACE: context.parameters[0], context=context)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in _cursor_execute
(nova.exception): TRACE: context)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1394, in _cursor_execute
(nova.exception): TRACE: context)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 299, in do_execute
(nova.exception): TRACE: cursor.execute(statement, parameters)
(nova.exception): TRACE: ProgrammingError: (ProgrammingError) You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. u'INSERT INTO users (created_at, updated_at, deleted_at, deleted, id, name, access_key, secret_key, is_admin) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)' ('2012-02-10 02:16:46.515383', None, None, 0, 'm\xc3\xa5\xc5\x86\xc5\xa7\xc3\xbf', None, '30d6e169-64be-40ef-8f00-872939918505', 'd943602e-1875-4938-ab62-70eba6ac9ebb', 0)
(nova.exception): TRACE:
Command failed, please check log for more info
2012-02-09 20:16:46,570 CRITICAL nova [-] User %(user)s already exists.

[Andrew Bogott (andrewbogott)]

Probably we just need to decode some args. I'm inveestigating.

[Andrew Bogott (andrewbogott)]

It looks like we are generally fast and loose with encoding in nova. I have a patch that fixes one particular case, but we need a general solution:

1) Adopt a standard for what encoding is used for all implicitly-encoded IO. (I would propose that that standard be UTF8 rather than ASCII.)

2) At all points where strings enter Python (e.g. commandline args) immediately decode them into unicode (which can unambiguously contain all possible 8-bit encodings.)

3) At all points where 'unicodes' exit Python (being written to stdout or a log file or a database) explicitly encode them as appropriate (generally UTF8, again, especially if we're ever going to read them back in.)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/4209
Committed: http://github.com/openstack/nova/commit/93cee6047f74e256c3695451b5dc51256208f76f
Submitter: Jenkins
Branch: master

commit 93cee6047f74e256c3695451b5dc51256208f76f
Author: Andrew Bogott <email address hidden>
Date: Wed Feb 15 13:08:27 2012 -0600

    Decode nova-manage args into unicode.

    This improves handling of non-ASCII args... it's not
    exactly a comprehensive solution, though.

    Partial (demonstration) fix for bug 931667.

    Change-Id: Icc2aabe9c0c521732003f7013438630ebc8d2238

[Andy Chong (andycjw)]

I have tested and it seems user with name 'måņŧÿ' is successfully created with the change in nova
should this bug be considered fix in horizon as well?

[Andrew Bogott (andrewbogott)]

Yes, I believe it is fixed.