OpenStack Dashboard (Horizon)

unicode characters in keypair name cause 500 server errors

Reported by Gabriel Hurley on 2011-12-01
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Low
Mike Lundy
OpenStack Dashboard (Horizon)
Low
Joshua McKenty

Bug Description

We need to sanitize the keypair name input to only accept ASCII characters. Otherwise you get this:

REQ BODY: {"keypair": {"name": "yay\u090"}}

RESP:{'date': 'Thu, 01 Dec 2011 20:39:06 GMT', 'status': '500', 'content-length': '133', 'content-type': 'application/json; charset=UTF-8'} {"cloudServersFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

Jesse Andrews (anotherjesse) wrote :

Not sure if we want to only allow ASCII... I think it is (or should be) valid to have servers named with unicode - so we should allow unicode in keynames?

Gabriel Hurley (gabriel-hurley) wrote :

I'm 100% in agreement that it *should* handle unicode characters. But right now Nova raises an unhandled exception if you try to, so we should disable it o our end until such time as Nova does support it.

Jake Dahn (jakedahn) on 2011-12-02
Changed in horizon:
status: New → Confirmed
assignee: nobody → Jake Dahn (jakedahn)
Devin Carlen (devcamcar) on 2011-12-17
Changed in horizon:
milestone: none → essex-4
importance: Undecided → Low
Gabe Westmaas (westmaas) wrote :
Download full text (6.8 KiB)

This is definitely an issue. What is the right expectation here? If a customer specifies unicode in the server name, the unix hostname will not contain those characters, correct? What if all the characters are unicode? In addition, I think the initial migrations might have to be changed, since this affects how they are stored in the db. I haven't traced this at all, but I don't think its straightforward to support unicode. In the meantime perhaps we should disallow users from using unicode in the hostname.

Here's a Traceback just for fun:

2012-01-15 12:38:35,641 ERROR nova.api.openstack.v2 [f00369f6-ec7d-4239-b71d-b6d6affd300a username projectid] Caught error: 'utf8' codec can't decode byte 0xa8 in position 5: invalid start byte
(nova.api.openstack.v2): TRACE: Traceback (most recent call last):
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/v2/__init__.py", line 59, in __call__
(nova.api.openstack.v2): TRACE: return req.get_response(self.application)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 919, in get_response
(nova.api.openstack.v2): TRACE: application, catch_exc_info=False)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 887, in call_application
(nova.api.openstack.v2): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/keystone/middleware/auth_token.py", line 209, in __call__
(nova.api.openstack.v2): TRACE: return self._forward_request(env, start_response, proxy_headers)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/keystone/middleware/auth_token.py", line 325, in _forward_request
(nova.api.openstack.v2): TRACE: return self.app(env, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE: return resp(environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
(nova.api.openstack.v2): TRACE: resp = self.call_func(req, *args, **self.kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
(nova.api.openstack.v2): TRACE: return self.func(req, *args, **kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/wsgi.py", line 501, in __call__
(nova.api.openstack.v2): TRACE: response = req.get_response(self.application)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 919, in get_response
(nova.api.openstack.v2): TRACE: application, catch_exc_info=False)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 887, in call_application
(nova.api.openstack.v2): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE: return resp(environ, start_response)
(nova.api.openstack...

Read more...

Brian Waldon (bcwaldon) on 2012-01-20
Changed in nova:
status: New → Incomplete
justinsb (justin-fathomdb) wrote :

Related to Bug #925748

Devin Carlen (devcamcar) wrote :

@bcwaldon: Why did you mark as incomplete? This seems to have a pretty simple repro.

Brian Waldon (bcwaldon) wrote :

No idea, I went on a bug triaging spree and may have mis-marked this one.

Changed in nova:
status: Incomplete → Confirmed
Gabriel Hurley (gabriel-hurley) wrote :

I did some work on this the other day. The issue is actually in Nova/deployment:

Nova handles the unicode characters correctly at the Python level, but if not backed by a DB that supports a unicode character set it raises a useless exception. Even more unfortunately, MySQL has a 1000-byte limit for key names, which is exceeded by something in Nova's migrations when run with a 3-byte unicode charset instead of a 1-byte latin charset. PostgreSQL does not have this problem.

Gabriel Hurley (gabriel-hurley) wrote :

Bumping to 2012.1 since there isn't a patch yet.

Changed in horizon:
assignee: Jake Dahn (jakedahn) → nobody
milestone: essex-4 → 2012.1
Thierry Carrez (ttx) on 2012-02-29
Changed in horizon:
milestone: 2012.1 → essex-rc1

what is the keypair in openstack and how it will be usefull to openstack?

Changed in horizon:
assignee: nobody → Joshua McKenty (joshua-mckenty)
Changed in nova:
assignee: nobody → Joshua McKenty (joshua-mckenty)

Fix proposed to branch: master
Review: https://review.openstack.org/4787

Changed in nova:
assignee: Joshua McKenty (joshua-mckenty) → Mike Lundy (novas0x2a)
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/4787
Committed: http://github.com/openstack/nova/commit/27013e87f53cccbfd2e203f065e416437ba08987
Submitter: Jenkins
Branch: master

commit 27013e87f53cccbfd2e203f065e416437ba08987
Author: Mike Lundy <email address hidden>
Date: Fri Mar 2 00:15:04 2012 +0000

    Fail gracefully when the db doesn't speak unicode

    When sqlalchemy/mysql doesn't contain the charset=utf8 or use_unicode=1
    parameters (and the default mysql connection charset is not utf8),
    sqlalchemy will connect to mysql in ascii mode; In ascii mode, it will
    attempt to reencode all input data to latin-1 regardless of the
    database/table collation setting. This catches that error and turns it
    into an invalid parameter.

    (The error message associated with this is something like
    UnicodeEncodeError: 'latin-1' codec can't encode character u'\u2026' in
    position 30: ordinal not in range(256))

    This fixes bug 944034 and is related to bug 898808

    Change-Id: I082b7568ef9e9d2104e13aa58d207535ef201bd3

Changed in nova:
status: In Progress → Fix Committed
Mike Lundy (novas0x2a) wrote :

I haven't tested the above fix in the context of this bug (just in terms of bug 944034) but I suspect you will now get a 400 instead of a 500 when you use a non-ascii keypair with a mysql connection that doesn't support non-ascii.

There might also be a bit of a documentation fix necessary here: if you back nova with mysql and you want to use unicode in various place that are stored in the db, you must use a sql_connection string with charset=utf8 or use_unicode=1 (see http://docs.sqlalchemy.org/en/latest/dialects/mysql.html#unicode) or set the default connection charset to utf8 in mysql.

Quoting from the above link:

"Without this setting, many MySQL server installations default to a latin1 encoding for client connections, which has the effect of all data being converted into latin1, even if you have utf8 or another character set configured on your tables and columns."

Devin Carlen (devcamcar) on 2012-03-02
Changed in nova:
milestone: none → essex-rc1
importance: Undecided → Medium
importance: Medium → Low

Reviewed: https://review.openstack.org/4783
Committed: http://github.com/openstack/horizon/commit/86bb083c9d24f770e0a14c50ee8f53453f938deb
Submitter: Jenkins
Branch: master

commit 86bb083c9d24f770e0a14c50ee8f53453f938deb
Author: Joshua McKenty <email address hidden>
Date: Thu Mar 1 23:40:25 2012 +0000

    Form errors should be very apparent.

      * Adding the required control-group css class to the form fields.
      * bug #898808

    Change-Id: I1c09200e1e58204f5e651d57967f4e3aa4c41919

Changed in horizon:
status: Confirmed → Fix Committed
Thierry Carrez (ttx) on 2012-03-20
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-03-20
Changed in horizon:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in horizon:
milestone: essex-rc1 → 2012.1
Thierry Carrez (ttx) on 2012-04-05
Changed in nova:
milestone: essex-rc1 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers