python-keystoneclient

keystoneclient cannot log non-ascii data

Bug #1453953 reported by Chad Roberts on 2015-05-11

This bug affects 6 people

Affects		Status	Importance	Assigned to	Milestone
	python-keystoneclient	Fix Released	High	Trevor McKay

Bug Description

This bug was originally written against Sahara, but it appears to be a Horizon issue instead, so I'm reporting it here.

When trying to upload the spark-example.jar from the Sahara edp-examples, it fails with the message "Danger: There was an error submitting the form. Please try again."

In the logs, the stack trace looks like this:

Internal Server Error: /project/data_processing/job_binaries/create-job-binary
Traceback (most recent call last):
  File "/home/croberts/src/horizon/.venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/croberts/src/horizon/horizon/decorators.py", line 36, in dec
    return view_func(request, *args, **kwargs)
  File "/home/croberts/src/horizon/horizon/decorators.py", line 52, in dec
    return view_func(request, *args, **kwargs)
  File "/home/croberts/src/horizon/horizon/decorators.py", line 36, in dec
    return view_func(request, *args, **kwargs)
  File "/home/croberts/src/horizon/horizon/decorators.py", line 84, in dec
    return view_func(request, *args, **kwargs)
  File "/home/croberts/src/horizon/.venv/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/croberts/src/horizon/.venv/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/croberts/src/horizon/.venv/lib/python2.7/site-packages/django/views/generic/edit.py", line 173, in post
    return self.form_valid(form)
  File "/home/croberts/src/horizon/horizon/forms/views.py", line 173, in form_valid
    exceptions.handle(self.request)
  File "/home/croberts/src/horizon/horizon/exceptions.py", line 364, in handle
    six.reraise(exc_type, exc_value, exc_traceback)
  File "/home/croberts/src/horizon/horizon/forms/views.py", line 170, in form_valid
    handled = form.handle(self.request, form.cleaned_data)
  File "/home/croberts/src/horizon/openstack_dashboard/dashboards/project/data_processing/job_binaries/forms.py", line 183, in handle
    _("Unable to create job binary"))
  File "/home/croberts/src/horizon/horizon/exceptions.py", line 364, in handle
    six.reraise(exc_type, exc_value, exc_traceback)
  File "/home/croberts/src/horizon/openstack_dashboard/dashboards/project/data_processing/job_binaries/forms.py", line 169, in handle
    bin_url = self.handle_internal(request, context)
  File "/home/croberts/src/horizon/openstack_dashboard/dashboards/project/data_processing/job_binaries/forms.py", line 216, in handle_internal
    _("Unable to upload job binary"))
  File "/home/croberts/src/horizon/horizon/exceptions.py", line 364, in handle
    six.reraise(exc_type, exc_value, exc_traceback)
  File "/home/croberts/src/horizon/openstack_dashboard/dashboards/project/data_processing/job_binaries/forms.py", line 212, in handle_internal
    request.FILES["job_binary_file"].read())
  File "/home/croberts/src/horizon/openstack_dashboard/api/sahara.py", line 332, in job_binary_internal_create
    data=data)
  File "/home/croberts/src/horizon/.venv/lib/python2.7/site-packages/saharaclient/api/job_binary_internals.py", line 31, in create
    'job_binary_internal', dump_json=False)
  File "/home/croberts/src/horizon/.venv/lib/python2.7/site-packages/saharaclient/api/base.py", line 110, in _update
    resp = self.api.put(url, **kwargs)
  File "/home/croberts/src/horizon/.venv/lib/python2.7/site-packages/keystoneclient/adapter.py", line 179, in put
    return self.request(url, 'PUT', **kwargs)
  File "/home/croberts/src/horizon/.venv/lib/python2.7/site-packages/saharaclient/api/client.py", line 46, in request
    return super(HTTPClient, self).request(*args, **kwargs)
  File "/home/croberts/src/horizon/.venv/lib/python2.7/site-packages/keystoneclient/adapter.py", line 95, in request
    return self.session.request(url, method, **kwargs)
  File "/home/croberts/src/horizon/.venv/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/home/croberts/src/horizon/.venv/lib/python2.7/site-packages/keystoneclient/session.py", line 371, in request
    logger=logger)
  File "/home/croberts/src/horizon/.venv/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/home/croberts/src/horizon/.venv/lib/python2.7/site-packages/keystoneclient/session.py", line 195, in _http_log_request
    logger.debug(' '.join(string_parts))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfd in position 14: ordinal not in range(128)

Revision history for this message

Ken Chen (ken-chen-i) wrote on 2015-05-21:

I found this is a keystoneclient bug. I already reported it in https://bugs.launchpad.net/keystone/+bug/1457279

Revision history for this message

Dolph Mathews (dolph) wrote on 2015-05-27:

From Ken Chen in https://bugs.launchpad.net/python-keystoneclient/+bug/1457279

------
In keystoneclient/session.py file, _http_log_request method, we have below codes:
if data:
string_parts.append("-d '%s'" % data)

logger.debug(' '.join(string_parts))

However, if data is not in ascii chars, this might cause error like:
UnicodeEncodeError: 'ascii' codec can't encode character u'\\xbb' in position 10: ordinal not in range(128)
This is also the reason of bug https://bugs.launchpad.net/horizon/+bug/1453953
------

summary:	- [data processing] Unable to upload job binaries + keystoneclient cannot log non-ascii data
affects:	horizon → python-keystoneclient
tags:	removed: sahara
Changed in python-keystoneclient:
status:	New → Triaged
importance:	Undecided → High

OpenStack Infra (hudson-openstack) on 2015-05-27

Changed in python-keystoneclient:
assignee:	nobody → Nikita Konovalov (nkonovalov)
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-11-19: Change abandoned on python-keystoneclient (master)

Change abandoned by Nikita Konovalov (<email address hidden>) on branch: master
Review: https://review.openstack.org/183514

Revision history for this message

Steve Martinelli (stevemar) wrote on 2015-11-27:

unassigning since the patch is abandoned

Changed in python-keystoneclient:
assignee:	Nikita Konovalov (nkonovalov) → nobody

Boris Bobrov (bbobrov) on 2015-12-01

Changed in python-keystoneclient:
status:	In Progress → New
status:	New → Confirmed
assignee:	nobody → Boris Bobrov (bbobrov)

Revision history for this message

Davanum Srinivas (DIMS) (dims-v) wrote on 2015-12-07:

We should see if encodeutils.safe_encode from oslo.utils will work to fix this problem

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-02-09: Fix proposed to python-keystoneclient (master)

Fix proposed to branch: master
Review: https://review.openstack.org/278027

Changed in python-keystoneclient:
assignee:	Boris Bobrov (bbobrov) → Trevor McKay (tmckay)
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-02-12: Fix merged to python-keystoneclient (master)

Reviewed: https://review.openstack.org/278027
Committed: https://git.openstack.org/cgit/openstack/python-keystoneclient/commit/?id=96fbfeab973cf61cd5ba129056cf4e6634d805b8
Submitter: Jenkins
Branch: master

commit 96fbfeab973cf61cd5ba129056cf4e6634d805b8
Author: Trevor McKay <email address hidden>
Date: Tue Feb 9 13:40:39 2016 -0500

Handle exception on UnicodeDecodError in logging of request

    If the logging of an HTTP request causes a UnicodeDecodeError,
    modify the log entry using oslo_utils.encodeutils.safe_decode
    with errors='replace' and try again

Co-Authored-By: Nikita Konovalov <email address hidden>

Change-Id: Ic365c654ebca4045208c6c30e232665145db7b4c
Closes-Bug: #1453953