Upload failed if filename is unicode string in formpost (python3)

Bug #1858259 reported by cho seongsoo on 2020-01-04
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
High
Unassigned

Bug Description

If the filename in form-data is unicode string, the proxy-server occure exception.

I try to reproduce this in test code to change the filename with unicode. But the all test case passed.

This is my workaround solution to fix this problem.
I changed this line.
https://github.com/openstack/swift/blob/stable/train/swift/common/middleware/formpost.py#L368

```
filename = attributes['filename'].encode('utf-8', 'replace').decode('latin-1')
subenv['PATH_INFO'] += filename or 'filename'
```

I have referenced how the eventlet's wsgi process the path_info.
https://github.com/eventlet/eventlet/pull/497/files

======================

Here is a error log

```
2019-12-23T14:00:03.889135+09:00 server proxy-server: Error: An error occurred:
Traceback (most recent call last):
 File "/usr/lib/python3/dist-packages/swift/common/middleware/proxy_logging.py", line 398, in iter_response
   yield chunk
GeneratorExit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/usr/lib/python3/dist-packages/swift/common/middleware/catch_errors.py", line 75, in handle_request
   resp = self._app_call(env)
 File "/usr/lib/python3/dist-packages/swift/common/wsgi.py", line 1282, in _app_call
   resp = self.app(env, self._start_response)
 File "/usr/lib/python3/dist-packages/swift/common/middleware/gatekeeper.py", line 122, in __call__
   return self.app(env, gatekeeper_response)
 File "/usr/lib/python3/dist-packages/swift/common/middleware/proxy_logging.py", line 415, in __call__
   iterable = self.app(env, my_start_response)
 File "/usr/lib/python3/dist-packages/swift/common/middleware/healthcheck.py", line 52, in __call__
   return self.app(env, start_response)
 File "/usr/lib/python3/dist-packages/swift/common/middleware/memcache.py", line 109, in __call__
   return self.app(env, start_response)
 File "/usr/lib/python3/dist-packages/swift/common/middleware/listing_formats.py", line 138, in __call__
   return self.app(env, start_response)
 File "/usr/lib/python3/dist-packages/swift/common/swob.py", line 1538, in _wsgify_self
   return func(self, Request(env))(env, start_response)
 File "/usr/lib/python3/dist-packages/swift/common/swob.py", line 1538, in _wsgify_self
   return func(self, Request(env))(env, start_response)
 File "/usr/lib/python3/dist-packages/swift/common/middleware/tempurl.py", line 504, in __call__
   return self.app(env, start_response)
 File "/usr/lib/python3/dist-packages/swift/common/middleware/ratelimit.py", line 314, in __call__
   return self.app(env, start_response)
 File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
   resp = self.call_func(req, *args, **kw)
 File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
   return self.func(req, *args, **kwargs)
 File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __call__
   response = req.get_response(self._app)
 File "/usr/lib/python3/dist-packages/webob/request.py", line 1314, in send
   application, catch_exc_info=False)
 File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
   app_iter = application(self.environ, start_response)
 File "/usr/lib/python3/dist-packages/swift/common/middleware/s3api/s3api.py", line 309, in __call__
   return resp(env, start_response)
 File "/usr/lib/python3/dist-packages/swift/common/middleware/s3api/s3api.py", line 173, in __call__
   return self.app(env, start_response)
 File "/usr/lib/python3/dist-packages/swift/common/middleware/s3api/s3token.py", line 262, in __call__
   return self._app(environ, start_response)
 File "/usr/lib/python3/dist-packages/swift/common/middleware/keystoneauth.py", line 234, in __call__
   return self.app(environ, keystone_start_response)
 File "/usr/lib/python3/dist-packages/swift/common/middleware/formpost.py", line 230, in __call__
   env, attrs['boundary'])
 File "/usr/lib/python3/dist-packages/swift/common/middleware/formpost.py", line 293, in _translate_form
   self._perform_subrequest(env, attributes, fp, keys)
 File "/usr/lib/python3/dist-packages/swift/common/middleware/formpost.py", line 428, in _perform_subrequest
   close_if_possible(reiterate(self.app(subenv, _start_response)))
 File "/usr/lib/python3/dist-packages/swift/common/utils.py", line 4185, in close_if_possible
   return close_method()
 File "/usr/lib/python3/dist-packages/swift/common/utils.py", line 3890, in close
   close_method()
 File "/usr/lib/python3/dist-packages/swift/common/middleware/proxy_logging.py", line 409, in iter_response
   start_time, time.time(), resp_headers=resp_headers)
 File "/usr/lib/python3/dist-packages/swift/common/middleware/proxy_logging.py", line 224, in log_request
   if req.path.startswith('/v1/'):
 File "/usr/lib/python3/dist-packages/swift/common/swob.py", line 1042, in path
   self.environ['PATH_INFO'])
 File "/usr/lib/python3/dist-packages/swift/common/swob.py", line 309, in wsgi_quote
   raise TypeError('Expected a WSGI string; got %r' % wsgi_str)
TypeError: Expected a WSGI string; got '/v1/AUTH_318617090a71423b938ee42a932b1742/tcon/03 안녕, 나의 눈부신 비행기.mp3' (txn: tx93fa2d6b14e447309042f-005e0049d3)
```

Tim Burke (1-tim-z) wrote :
Download full text (6.5 KiB)

Funny -- I tried uploading a file named ☃ (snowman) and my browser seems to have turned that into ... and HTML entity?? "☃"

But putting it in the upload prefix got me a traceback! A little different from yours, though:

Jan 7 23:45:34 saio proxy-server: get_keys(): from callback: 'latin-1' codec can't encode character '\u2603' in position 24: ordinal not in range(256):
Traceback (most recent call last):
  File "/vagrant/swift/swift/common/middleware/crypto/crypto_utils.py", line 170, in get_keys
    keys = fetch_crypto_keys(key_id=key_id)
  File "/vagrant/swift/swift/common/middleware/crypto/keymaster.py", line 130, in fetch_crypto_keys
    path, secret_id=secret_id)
  File "/vagrant/swift/swift/common/middleware/crypto/keymaster.py", line 299, in create_key
    return hmac.new(key, wsgi_to_bytes(path),
  File "/vagrant/swift/swift/common/swob.py", line 279, in wsgi_to_bytes
    return wsgi_str.encode('latin1')
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2603' in position 24: ordinal not in range(256)

I can get a similar error using curl to get a snowman in the filename:

$ curl -F redirect=/v1/AUTH_test/c/thanks.html -F maxmax_file_count=3 -F expires=1578445456 -F signature=b60d8d455deb53f8a2a50a794ab155c8a9925b5c -F file0="@/home/vagrant/.s3cfg;filename=☃" https://saio/v1/AUTH_test/c/upload_ -v

So I tried disabling encryption -- which caused a pretty fun blow-up, this time coming out on STDERR:

Exception ignored in:
<generator object ProxyLoggingMiddleware.__call__.<locals>.iter_response at 0x7f8910b5dc50>
Traceback (most recent call last):
File "/vagrant/swift/swift/common/middleware/proxy_logging.py", line 409, in iter_response
start_time, time.time(), resp_headers=resp_headers)
File "/vagrant/swift/swift/common/middleware/proxy_logging.py", line 224, in log_request
if req.path.startswith('/v1/'):
File "/vagrant/swift/swift/common/swob.py", line 1070, in path
self.environ['PATH_INFO'])
File "/vagrant/swift/swift/common/swob.py", line 309, in wsgi_quote
raise TypeError('Expected a WSGI string; got %r' % wsgi_str)
TypeError
:
Expected a WSGI string; got '/v1/AUTH_test/c/upload_☃'

Which is looking more familiar. Differences might be explained by changes on master since train to quote more paths when logging? At any rate, that's certainly close enough for me to call this confirmed!

Probably off-topic: My first try with curl led to... something strange:

$ curl -F max_file_size=102400 -F max_file_count=3 -F expires=1578444242 -F signature=df2b9b797125c7ff2dbe320206538f6190f17a78 -F file0="@~/.s3cfg;filename=☃" https://saio/v1/AUTH_test/c/let_it_☃_ -v
Warning: setting file ~/.s3cfg failed!
* Trying 127.0.1.1...
... <snip> ...
> POST /v1/AUTH_test/c/let_it_☃_ HTTP/1.1
> Host: saio
> User-Agent: curl/7.58.0
> Accept: */*
> Transfer-Encoding: chunked
> Content-Type: multipart/form-data; boundary=------------------------fe289f46b0bc0e60
> Expect: 100-continue
>
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23...

Read more...

Changed in swift:
status: New → Confirmed
importance: Undecided → High

Reviewed: https://review.opendev.org/701497
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=9483630ae130c5b2616f8617da25bcb51b647208
Submitter: Zuul
Branch: master

commit 9483630ae130c5b2616f8617da25bcb51b647208
Author: Tim Burke <email address hidden>
Date: Tue Jan 7 21:16:37 2020 -0800

    py3: Fix formpost unicode filename issues

    Previously, we took the native string filename attribute and put it
    directly in the (WSGI string) PATH_INFO field. Now, we convert it to
    a WSGI string first.

    Change-Id: I30e3beb8707b88c36bd3cdc7a0887d069e943ba6
    Closes-Bug: #1858259

Changed in swift:
status: Confirmed → Fix Released

Reviewed: https://review.opendev.org/703654
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=48099c9823934bbda9fca82a49fc990fac3a9928
Submitter: Zuul
Branch: stable/train

commit 48099c9823934bbda9fca82a49fc990fac3a9928
Author: Tim Burke <email address hidden>
Date: Tue Jan 7 21:16:37 2020 -0800

    py3: Fix formpost unicode filename issues

    Previously, we took the native string filename attribute and put it
    directly in the (WSGI string) PATH_INFO field. Now, we convert it to
    a WSGI string first.

    Change-Id: I30e3beb8707b88c36bd3cdc7a0887d069e943ba6
    Closes-Bug: #1858259
    (cherry picked from commit 9483630ae130c5b2616f8617da25bcb51b647208)

tags: added: in-stable-train

This issue was fixed in the openstack/swift 2.24.0 release.

Download full text (40.5 KiB)

Reviewed: https://review.opendev.org/713632
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=79bd2e59e5c15ee84814ec1c4f0893176ba79412
Submitter: Zuul
Branch: feature/losf

commit f2ffd900593db2829a39a073f0c033d82985c40f
Author: Clay Gerrard <email address hidden>
Date: Fri Feb 28 11:09:51 2020 -0600

    Apply limit to list versioned containers

    Change-Id: I28e062273d673c4f07cd3c5da088aa790b77a599
    Closes-Bug: #1863841

commit dc40779307095b8d0b2761b77b9cb2904ec721ae
Author: Clay Gerrard <email address hidden>
Date: Fri Feb 28 10:00:25 2020 -0600

    Use float consistently for proxy timeout settings

    Change-Id: I433c97df99193ec31c863038b9b6fd20bb3705b8

commit 55049beda5b9d7038a3604a87f28312d7702ccb2
Author: Tim Burke <email address hidden>
Date: Fri Feb 28 18:59:32 2020 -0800

    tests: Use timedelta to adjust dates, not string manipulations

    Change-Id: I8f65ccd7f2a79d5b877bfbef0274fb7857e21391

commit 3b65a5998cc921d2763cf1a9ec1e40b88491262d
Author: Tim Burke <email address hidden>
Date: Wed Jan 10 06:16:41 2018 +0000

    Fix up some Content-Type handling in account/container listings

    Update content type on 204 (not just 200) to properly handle HEAD
    requests from xml/txt listings.

    Add "Vary: Accept" header to listings, since otherwise, browsers may
    serve the wrong content type from cache (even though we *would have*
    sent the *right* type if it actually sent the request).

    Change-Id: Iaa333aaca36a8dc2df65d38ef2173e3a6e2000ee

commit ecca23eb806e11cf6517f0456483da7a065350a8
Author: Clay Gerrard <email address hidden>
Date: Fri Feb 21 15:33:21 2020 -0600

    Extend eventlet_debug logging to GreenAsyncPile

    Change-Id: Ibd9fe5c9a1e75b86eb7d540594d5cf516758e17e

commit 0fb3371484f1d0f629d0b0e33f6aafbff0e43ee9
Author: Sam Morrison <email address hidden>
Date: Tue Feb 18 10:17:50 2020 +1100

    Delay importing swiftclient until after monkey-patching

    Commit message below partly copied from nova:

    Eventlet monkey patching should be as early as possible

    We were seeing infinite recursion opening an ssl socket when running
    various combinations of python3, eventlet, and urllib3. It is not
    clear exactly what combination of versions are affected, but for
    background there is an example of this issue documented here:

    https://github.com/eventlet/eventlet/issues/371

    The immediate cause in swift's case was that we were calling
    eventlet.monkey_patch() after importing swiftclient (which imported
    requests, which finally imported urllib3).

    We only use the imported function in one place, however; hold off on
    importing until we actually need it to ensure that monkey patching
    happens first. Note that we *don't* want to monkey-patch at import time,
    as we've previously had bugs related to import-time side-effects.

    Change-Id: I24f4bcc3d62dc37fd9559032bfd25f5b15f98745
    Closes-bug: #1863680
    Related-bug: #1380815

commit a5afe767581d2cb97cf3690067e6d626c7682c2c
Author: Tim Burke <email address hidden>
Date: Wed Feb 19 10:09:49 2020 -0800

    Revert "Make roll...

tags: added: in-feature-losf
Tim Burke (1-tim-z) on 2020-09-18
tags: added: python3
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.