unicode characters in keypair name cause 500 server errors

[Gabriel Hurley]

We need to sanitize the keypair name input to only accept ASCII characters. Otherwise you get this:

REQ BODY: {"keypair": {"name": "yay\u090"}}

RESP:{'date': 'Thu, 01 Dec 2011 20:39:06 GMT', 'status': '500', 'content-length': '133', 'content-type': 'application/json; charset=UTF-8'} {"cloudServersFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

[Jesse Andrews (anotherjesse)]

Not sure if we want to only allow ASCII... I think it is (or should be) valid to have servers named with unicode - so we should allow unicode in keynames?

[Gabriel Hurley (gabriel-hurley)]

I'm 100% in agreement that it *should* handle unicode characters. But right now Nova raises an unhandled exception if you try to, so we should disable it o our end until such time as Nova does support it.

[Gabe Westmaas (westmaas)]

This is definitely an issue. What is the right expectation here? If a customer specifies unicode in the server name, the unix hostname will not contain those characters, correct? What if all the characters are unicode? In addition, I think the initial migrations might have to be changed, since this affects how they are stored in the db. I haven't traced this at all, but I don't think its straightforward to support unicode. In the meantime perhaps we should disallow users from using unicode in the hostname.

Here's a Traceback just for fun:

2012-01-15 12:38:35,641 ERROR nova.api.openstack.v2 [f00369f6-ec7d-4239-b71d-b6d6affd300a username projectid] Caught error: 'utf8' codec can't decode byte 0xa8 in position 5: invalid start byte
(nova.api.openstack.v2): TRACE: Traceback (most recent call last):
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/v2/__init__.py", line 59, in __call__
(nova.api.openstack.v2): TRACE: return req.get_response(self.application)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 919, in get_response
(nova.api.openstack.v2): TRACE: application, catch_exc_info=False)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 887, in call_application
(nova.api.openstack.v2): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/keystone/middleware/auth_token.py", line 209, in __call__
(nova.api.openstack.v2): TRACE: return self._forward_request(env, start_response, proxy_headers)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/keystone/middleware/auth_token.py", line 325, in _forward_request
(nova.api.openstack.v2): TRACE: return self.app(env, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE: return resp(environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
(nova.api.openstack.v2): TRACE: resp = self.call_func(req, *args, **self.kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
(nova.api.openstack.v2): TRACE: return self.func(req, *args, **kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/wsgi.py", line 501, in __call__
(nova.api.openstack.v2): TRACE: response = req.get_response(self.application)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 919, in get_response
(nova.api.openstack.v2): TRACE: application, catch_exc_info=False)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 887, in call_application
(nova.api.openstack.v2): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE: return resp(environ, start_response)
(nova.api.openstack...

[justinsb (justin-fathomdb)]

Related to Bug #925748

[Devin Carlen (devcamcar)]

@bcwaldon: Why did you mark as incomplete? This seems to have a pretty simple repro.

[Brian Waldon (bcwaldon)]

No idea, I went on a bug triaging spree and may have mis-marked this one.

[Gabriel Hurley (gabriel-hurley)]

I did some work on this the other day. The issue is actually in Nova/deployment:

Nova handles the unicode characters correctly at the Python level, but if not backed by a DB that supports a unicode character set it raises a useless exception. Even more unfortunately, MySQL has a 1000-byte limit for key names, which is exceeded by something in Nova's migrations when run with a 3-byte unicode charset instead of a 1-byte latin charset. PostgreSQL does not have this problem.

[Gabriel Hurley (gabriel-hurley)]

Bumping to 2012.1 since there isn't a patch yet.

[adapaka bhavaniprasad (adapaka-prasad)]

what is the keypair in openstack and how it will be usefull to openstack?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/4787

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/4787
Committed: http://github.com/openstack/nova/commit/27013e87f53cccbfd2e203f065e416437ba08987
Submitter: Jenkins
Branch: master

commit 27013e87f53cccbfd2e203f065e416437ba08987
Author: Mike Lundy <email address hidden>
Date: Fri Mar 2 00:15:04 2012 +0000

    Fail gracefully when the db doesn't speak unicode

    When sqlalchemy/mysql doesn't contain the charset=utf8 or use_unicode=1
    parameters (and the default mysql connection charset is not utf8),
    sqlalchemy will connect to mysql in ascii mode; In ascii mode, it will
    attempt to reencode all input data to latin-1 regardless of the
    database/table collation setting. This catches that error and turns it
    into an invalid parameter.

    (The error message associated with this is something like
    UnicodeEncodeError: 'latin-1' codec can't encode character u'\u2026' in
    position 30: ordinal not in range(256))

    This fixes bug 944034 and is related to bug 898808

    Change-Id: I082b7568ef9e9d2104e13aa58d207535ef201bd3

[Mike Lundy (novas0x2a)]

I haven't tested the above fix in the context of this bug (just in terms of bug 944034) but I suspect you will now get a 400 instead of a 500 when you use a non-ascii keypair with a mysql connection that doesn't support non-ascii.

There might also be a bit of a documentation fix necessary here: if you back nova with mysql and you want to use unicode in various place that are stored in the db, you must use a sql_connection string with charset=utf8 or use_unicode=1 (see http://docs.sqlalchemy.org/en/latest/dialects/mysql.html#unicode) or set the default connection charset to utf8 in mysql.

Quoting from the above link:

"Without this setting, many MySQL server installations default to a latin1 encoding for client connections, which has the effect of all data being converted into latin1, even if you have utf8 or another character set configured on your tables and columns."

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/4783
Committed: http://github.com/openstack/horizon/commit/86bb083c9d24f770e0a14c50ee8f53453f938deb
Submitter: Jenkins
Branch: master

commit 86bb083c9d24f770e0a14c50ee8f53453f938deb
Author: Joshua McKenty <email address hidden>
Date: Thu Mar 1 23:40:25 2012 +0000

    Form errors should be very apparent.

      * Adding the required control-group css class to the form fields.
      * bug #898808

    Change-Id: I1c09200e1e58204f5e651d57967f4e3aa4c41919