Unable to upload files with non-ASCII name to swift container

[Vincent Hou]

If a file name is non-ASCII, it is unable to upload to the swift container.

[Wed Jun 06 13:06:09 2012] [error] DEBUG:django.db.backends:(0.001) SELECT "django_session"."session_key", "django_session"."session_d
ata", "django_session"."expire_date" FROM "django_session" WHERE ("django_session"."session_key" = e667f753bd7458b1cc20ef174e12ddca A
ND "django_session"."expire_date" > 2012-06-06 13:06:09.114140 ); args=('e667f753bd7458b1cc20ef174e12ddca', u'2012-06-06 13:06:09.1141
40')
[Wed Jun 06 13:06:09 2012] [error] DEBUG:horizon.api.keystone:Creating a new keystoneclient connection to http://9.119.148.161:5000/v2
.0.
[Wed Jun 06 13:06:09 2012] [error] DEBUG:keystoneclient.client:REQ: curl -i http://9.119.148.161:5000/v2.0/tenants -X GET -H "User-Age
nt: python-keystoneclient" -H "X-Auth-Token: 9aa4d071931149c8a03b89ae13872001"
[Wed Jun 06 13:06:09 2012] [error]
[Wed Jun 06 13:06:09 2012] [error] DEBUG:keystoneclient.client:RESP: {'status': '200', 'content-length': '231', 'content-location': u'
http://9.119.148.161:5000/v2.0/tenants', 'vary': 'X-Auth-Token', 'date': 'Wed, 06 Jun 2012 05:06:09 GMT', 'content-type': 'application
/json'}
[Wed Jun 06 13:06:09 2012] [error] RESP BODY: {"tenants_links": [], "tenants": [{"enabled": true, "description": null, "name": "admin"
, "id": "3d2a25b5cf6140f8a6a4ac98ab317dfc"}, {"enabled": true, "description": null, "name": "demo", "id": "f890a4e479c94a6bb8d586b8ba2
c2266"}]}
[Wed Jun 06 13:06:09 2012] [error]
[Wed Jun 06 13:06:17 2012] [error] DEBUG:django.db.backends:(0.001) SELECT "django_session"."session_key", "django_session"."session_d
ata", "django_session"."expire_date" FROM "django_session" WHERE ("django_session"."session_key" = e667f753bd7458b1cc20ef174e12ddca A
ND "django_session"."expire_date" > 2012-06-06 13:06:17.200633 ); args=('e667f753bd7458b1cc20ef174e12ddca', u'2012-06-06 13:06:17.2006
33')
[Wed Jun 06 13:06:17 2012] [error] DEBUG:horizon.api.swift:Swift connection created using token "585857ca648f419ea07b17f885b2845e" and
 url "http://9.119.148.161:8080/v1/AUTH_f890a4e479c94a6bb8d586b8ba2c2266"
[Wed Jun 06 13:06:17 2012] [error] CRITICAL:openstack_dashboard:Unhandled Exception in of type "<type 'exceptions.UnicodeEncodeError'>
" in dashboard.
[Wed Jun 06 13:06:17 2012] [error] Traceback (most recent call last):
[Wed Jun 06 13:06:17 2012] [error] File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_resp
onse
[Wed Jun 06 13:06:17 2012] [error] response = callback(request, *callback_args, **callback_kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/decorators.py", line 40, in dec
[Wed Jun 06 13:06:17 2012] [error] return view_func(request, *args, **kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/decorators.py", line 55, in dec
[Wed Jun 06 13:06:17 2012] [error] return view_func(request, *args, **kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/decorators.py", line 40, in dec
[Wed Jun 06 13:06:17 2012] [error] return view_func(request, *args, **kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/decorators.py", line 129, in dec
[Wed Jun 06 13:06:17 2012] [error] return view_func(request, *args, **kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/usr/local/lib/python2.7/dist-packages/django/views/generic/base.py", line 48, in view
[Wed Jun 06 13:06:17 2012] [error] return self.dispatch(request, *args, **kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/usr/local/lib/python2.7/dist-packages/django/views/generic/base.py", line 69, in dispatch
[Wed Jun 06 13:06:17 2012] [error] return handler(request, *args, **kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/forms/views.py", line 84, in post
[Wed Jun 06 13:06:17 2012] [error] return self.get(self, request, *args, **kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/forms/views.py", line 64, in get
[Wed Jun 06 13:06:17 2012] [error] form, handled = self.maybe_handle()
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/forms/views.py", line 59, in mayb
e_handle
[Wed Jun 06 13:06:17 2012] [error] self.form, self.handled = form.maybe_handle(self.request, **kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/forms/base.py", line 101, in maybe_handle
[Wed Jun 06 13:06:17 2012] [error] exceptions.handle(request)
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/forms/base.py", line 99, in maybe_handle
[Wed Jun 06 13:06:17 2012] [error] return form, form.handle(request, form.cleaned_data)
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/dashboards/nova/containers/forms.py", line 75, in handle
[Wed Jun 06 13:06:17 2012] [error] exceptions.handle(request, _("Unable to upload object."))
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/dashboards/nova/containers/forms.py", line 72, in handle
[Wed Jun 06 13:06:17 2012] [error] obj.sync_metadata()
[Wed Jun 06 13:06:17 2012] [error] File "/usr/local/lib/python2.7/dist-packages/cloudfiles/utils.py", line 45, in decorator
[Wed Jun 06 13:06:17 2012] [error] return f(*args, **kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/usr/local/lib/python2.7/dist-packages/cloudfiles/storage_object.py", line 231, in sync_metadata
[Wed Jun 06 13:06:17 2012] [error] data='')
[Wed Jun 06 13:06:17 2012] [error] File "/usr/local/lib/python2.7/dist-packages/cloudfiles/connection.py", line 186, in make_request
[Wed Jun 06 13:06:17 2012] [error] self.connection.request(method, path, data, headers)
[Wed Jun 06 13:06:17 2012] [error] File "/usr/lib/python2.7/httplib.py", line 958, in request
[Wed Jun 06 13:06:17 2012] [error] self._send_request(method, url, body, headers)
[Wed Jun 06 13:06:17 2012] [error] File "/usr/lib/python2.7/httplib.py", line 991, in _send_request
[Wed Jun 06 13:06:17 2012] [error] self.putheader(hdr, value)
[Wed Jun 06 13:06:17 2012] [error] File "/usr/lib/python2.7/httplib.py", line 938, in putheader
[Wed Jun 06 13:06:17 2012] [error] hdr = '%s: %s' % (header, '\\r\\n\\t'.join([str(v) for v in values]))
[Wed Jun 06 13:06:17 2012] [error] UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)
[Wed Jun 06 13:06:17 2012] [error] ERROR:django.request:Internal Server Error: /nova/containers/\xe4\xbd\xa0\xe5\xa5\xbd/upload
[Wed Jun 06 13:06:17 2012] [error] Traceback (most recent call last):
[Wed Jun 06 13:06:17 2012] [error] File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
[Wed Jun 06 13:06:17 2012] [error] response = callback(request, *callback_args, **callback_kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/decorators.py", line 40, in dec
[Wed Jun 06 13:06:17 2012] [error] return view_func(request, *args, **kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/decorators.py", line 55, in dec
[Wed Jun 06 13:06:17 2012] [error] return view_func(request, *args, **kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/decorators.py", line 40, in dec
[Wed Jun 06 13:06:17 2012] [error] return view_func(request, *args, **kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/decorators.py", line 129, in dec
[Wed Jun 06 13:06:17 2012] [error] return view_func(request, *args, **kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/usr/local/lib/python2.7/dist-packages/django/views/generic/base.py", line 48, in view
[Wed Jun 06 13:06:17 2012] [error] return self.dispatch(request, *args, **kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/usr/local/lib/python2.7/dist-packages/django/views/generic/base.py", line 69, in dispatch
[Wed Jun 06 13:06:17 2012] [error] return handler(request, *args, **kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/forms/views.py", line 84, in post
[Wed Jun 06 13:06:17 2012] [error] return self.get(self, request, *args, **kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/forms/views.py", line 64, in get
[Wed Jun 06 13:06:17 2012] [error] form, handled = self.maybe_handle()
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/forms/views.py", line 59, in maybe_handle
[Wed Jun 06 13:06:17 2012] [error] self.form, self.handled = form.maybe_handle(self.request, **kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/forms/base.py", line 101, in maybe_handle
[Wed Jun 06 13:06:17 2012] [error] exceptions.handle(request)
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/forms/base.py", line 99, in maybe_handle
[Wed Jun 06 13:06:17 2012] [error] return form, form.handle(request, form.cleaned_data)
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/dashboards/nova/containers/forms.py", line 75, in handle
[Wed Jun 06 13:06:17 2012] [error] exceptions.handle(request, _("Unable to upload object."))
[Wed Jun 06 13:06:17 2012] [error] File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/dashboards/nova/containers/forms.py", line 72, in handle
[Wed Jun 06 13:06:17 2012] [error] obj.sync_metadata()
[Wed Jun 06 13:06:17 2012] [error] File "/usr/local/lib/python2.7/dist-packages/cloudfiles/utils.py", line 45, in decorator
[Wed Jun 06 13:06:17 2012] [error] return f(*args, **kwargs)
[Wed Jun 06 13:06:17 2012] [error] File "/usr/local/lib/python2.7/dist-packages/cloudfiles/storage_object.py", line 231, in sync_metadata
[Wed Jun 06 13:06:17 2012] [error] data='')
[Wed Jun 06 13:06:17 2012] [error] File "/usr/local/lib/python2.7/dist-packages/cloudfiles/connection.py", line 186, in make_request
[Wed Jun 06 13:06:17 2012] [error] self.connection.request(method, path, data, headers)
[Wed Jun 06 13:06:17 2012] [error] File "/usr/lib/python2.7/httplib.py", line 958, in request
[Wed Jun 06 13:06:17 2012] [error] self._send_request(method, url, body, headers)
[Wed Jun 06 13:06:17 2012] [error] File "/usr/lib/python2.7/httplib.py", line 991, in _send_request
[Wed Jun 06 13:06:17 2012] [error] self.putheader(hdr, value)
[Wed Jun 06 13:06:17 2012] [error] File "/usr/lib/python2.7/httplib.py", line 938, in putheader
[Wed Jun 06 13:06:17 2012] [error] hdr = '%s: %s' % (header, '\\r\\n\\t'.join([str(v) for v in values]))
[Wed Jun 06 13:06:17 2012] [error] UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)

[Tihomir Trifonov (ttrifonov)]

Actually this is not quite a Horizon bug. It could be fixed in Horizon, but I think it is better to fix the main problem, and not side effects.

The problem here is that object upload connects to Swift service, and passes original filename of the uploaded file as a HTTP header. By definition, Unicode symbols are not allowed for HTTP headers. They should be quoted in ISO-8859-1 accorging to RFC 2047 (http://www.ietf.org/rfc/rfc2047.txt).
The actual place where this should be done is cloudfiles/connection.py, row 177, right before "isinstance(hdrs, dict) and headers.update(hdrs)" it's needed a encoding to be performed to header values:

    from urllib import urlencode, quote
.......

        if hdrs:
            for key, value in hdrs.items():
                hdrs[key] = quote(value.encode('utf-8'))

        isinstance(hdrs, dict) and headers.update(hdrs)

quote() function will perform the needed quoting in ISO-8859-1, and everything works fine then.

[Vincent Hou (houshengbo)]

Thank you for your explanation, Tihomir.
Well, one more question. Is it a good approach to put the file name in the http header? Why not in the body? Any advantages to do it that way?

[Tihomir Trifonov (ttrifonov)]

Ups,

Vincent, I haven't received a notification for your reply for some reason. Sorry for the late reply.
Horizon adds the original file name in the metadata:

            obj.metadata['orig-filename'] = object_file.name

And then the metadata is passed through as HTTP headers.

Is it better or not - this could be a good argument. However at the moment all metadata is sent in headers.

But back on the bug - is it a filename or anything else, the http headers should be escaped anyway.

[Vincent Hou (houshengbo)]

Thank you, Tihomir.
It is the filename, the filename of a local file to be uploaded into a swift container. For example, if it is name 测试.txt, which is a Chinese name, the bug will be raised when upload it to a container via the horizon interface.

[Gabriel Hurley (gabriel-hurley)]

I just tried this with python-swiftclient and found even worse results than with cloudfiles. I'm bumping this out of Horizon's milestones until such time as swift wants to address their unicode support in their clients/API.

[Gabriel Hurley (gabriel-hurley)]

Superseded by https://bugs.launchpad.net/horizon/+bug/1008940