filter doesn't handle unicode charaters

[Matthias Runge]

1 go to project/instances
2. insert 'ölk' into filter field
3. enter filter
4.
UnicodeEncodeError at /project/instances/

'ascii' codec can't encode character u'\xf6' in position 0: ordinal not in range(128)

Request Method: GET
Request URL: http://localhost:8000/project/instances/
Django Version: 1.8.2
Exception Type: UnicodeEncodeError
Exception Value:

'ascii' codec can't encode character u'\xf6' in position 0: ordinal not in range(128)

Exception Location: /usr/lib64/python2.7/urllib.py in urlencode, line 1347
Python Executable: /usr/bin/python
Python Version: 2.7.10
Python Path:

['/home/mrunge/work/horizon',
 '/usr/lib64/python27.zip',
 '/usr/lib64/python2.7',
 '/usr/lib64/python2.7/plat-linux2',
 '/usr/lib64/python2.7/lib-tk',
 '/usr/lib64/python2.7/lib-old',
 '/usr/lib64/python2.7/lib-dynload',
 '/usr/lib64/python2.7/site-packages',
 '/usr/lib64/python2.7/site-packages/gtk-2.0',
 '/usr/lib/python2.7/site-packages',
 '/home/mrunge/work/horizon/openstack_dashboard']

[hogetaro (hogetaro)]

there seems to be two problems

1) horizon does not encode unicode string before setting it into url param
 novaclient/v2/servers.py: 593
            query_string = "?%s" % parse.urlencode(new_qparams)

it should encode unicode string to utf-8 before urlencode

2) but even encoding it to utf-8 , nova api cant handle it (500 error happen at api call)

2015-07-10 11:20:33.035 ERROR nova.api.openstack [req-2e7f9436-fa14-4b84-95cb-4f21e423d9aa admin admin] Caught error: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)
2015-07-10 11:20:33.035 TRACE nova.api.openstack Traceback (most recent call last):
2015-07-10 11:20:33.035 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/__init__.py", line 126, in __call__
2015-07-10 11:20:33.035 TRACE nova.api.openstack return req.get_response(self.application)
2015-07-10 11:20:33.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
2015-07-10 11:20:33.035 TRACE nova.api.openstack application, catch_exc_info=False)
2015-07-10 11:20:33.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
2015-07-10 11:20:33.035 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2015-07-10 11:20:33.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-10 11:20:33.035 TRACE nova.api.openstack return resp(environ, start_response)
2015-07-10 11:20:33.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2015-07-10 11:20:33.035 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2015-07-10 11:20:33.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2015-07-10 11:20:33.035 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2015-07-10 11:20:33.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 434, in __call__
2015-07-10 11:20:33.035 TRACE nova.api.openstack response = req.get_response(self._app)
2015-07-10 11:20:33.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
2015-07-10 11:20:33.035 TRACE nova.api.openstack application, catch_exc_info=False)
2015-07-10 11:20:33.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
2015-07-10 11:20:33.035 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2015-07-10 11:20:33.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-10 11:20:33.035 TRACE nova.api.openstack return resp(environ, start_response)
2015-07-10 11:20:33.035 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-10 11:20:33.035 TRACE nova.api.openstack return resp(environ, start_response)
2015-07-...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/211850

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/224422

[Richard Jones (r1chardj0n3s)]

Fix for nova https://review.openstack.org/#/c/224431/

[Richard Jones (r1chardj0n3s)]

Seems like the auto-linking doesn't want to play along.

glanceclient patch is https://review.openstack.org/#/c/224991/

[Rob Cresswell (robcresswell-deactivatedaccount)]

I propose removing this from Horizon RC1, as it is a client based issue. However, I'm unsure if we want to temporarily patch for release, and leave a TODO for when the client bugs merge.

[Matthias Runge (mrunge)]

I removed this from rc1, but would argue, horizon shouldn't throw an exception, no matter if it's a *-client bug or not.

[Richard Jones (r1chardj0n3s)]

Matthias, it's not Horizon throwing the exception, it's the client code that has a bug. I don't think it's reasonable for us to try/except around every client code call just so we don't traceback when there's a bug in that code.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/224422
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=485201453a5d7a9b6b6b81d737edfce7fefb97a2
Submitter: Jenkins
Branch: master

commit 485201453a5d7a9b6b6b81d737edfce7fefb97a2
Author: Richard Jones <email address hidden>
Date: Thu Sep 17 15:05:35 2015 +1000

    Encode unicode filter arguments for server list

    Previously unicode arguments were not encoded so unicode
    exceptions were thrown when the arguments were URL encoded.

    Change-Id: I69a5ac8dcf584ad5b5604caf3c4cb16bb59d3fe3
    Partial-Bug: 1472999

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/224431
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1c003473353a511142b30ade1aa92f6fb1633e45
Submitter: Jenkins
Branch: master

commit 1c003473353a511142b30ade1aa92f6fb1633e45
Author: Richard Jones <email address hidden>
Date: Thu Sep 17 15:09:33 2015 +1000

    Allow filtering using unicode characters

    Current code forces the filter arguments to be converted
    to a byte string using str() which prevents filtering
    using unicode characters. This patch fixes that.

    Change-Id: I3abef5da4c2614806011440a4443e869920b4afd
    Partial-Bug: 1472999

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rob Cresswell (<email address hidden>) on branch: master
Review: https://review.openstack.org/211850
Reason: Abandoned as per discussion in comments.

[OpenStack Infra (hudson-openstack)]

Change abandoned by melanie witt (<email address hidden>) on branch: master
Review: https://review.openstack.org/200649
Reason: I think this was fixed by https://review.openstack.org/#/c/224422

Please restore if that's not the case.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/249101

[OpenStack Infra (hudson-openstack)]

Change abandoned by Glance Bot (<email address hidden>) on branch: master
Review: https://review.openstack.org/224991

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dave Walker (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/249101
Reason:
stable/kilo closed for 2015.1.4

This release is now pending its final release and no freeze exception has
been seen for this changeset. Therefore, I am now abandoning this change.

If this is not correct, please urgently raise a thread on openstack-dev.

More details at: https://wiki.openstack.org/wiki/StableBranch

[Ian Cordasco (icordasc)]

The change on glanceclient was abandoned due to inactivity. I've updated the status to reflect that.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean McGinnis (<email address hidden>) on branch: master
Review: https://review.opendev.org/224991
Reason: I don't think this is still an issue. Feel free to restore and update if it is and if you would like to continue with this.

[Cyril Roelandt (cyril-roelandt)]

$ glance image-list --tag ölk

^ this command does not crash with a recent version of glanceclient running on Python3, so I'll have to agree with Sean and consider this bug invalid now. Feel free to reopen if needed.