String "..%c0%af" causes 500 errors in multiple locations

[Charles Neill]

While doing some testing on Keystone using Syntribos (https://github.com/openstack/syntribos), our team (myself, Michael Dong, Rahul U Nair, Vinay Potluri, Aastha Dixit, and Khanak Nangia) noticed that we got 500 status codes when the string "..%c0%af" was inserted in various places in the URL for different types of requests.

Here are some examples:

=========

DELETE /v3/policies/..%c0%af HTTP/1.1
Host: [REDACTED]:5000
Connection: close
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.11.0
X-Auth-Token: [REDACTED]
Content-Length: 0

HTTP/1.1 500 Internal Server Error
Date: Tue, 16 Aug 2016 22:04:27 GMT
Server: Apache/2.4.7 (Ubuntu)
Vary: X-Auth-Token
X-Distribution: Ubuntu
x-openstack-request-id: req-238fd5a9-be45-41f2-893a-97b513b27af3
Content-Length: 143
Connection: close
Content-Type: application/json

{"error": {"message": "An unexpected error prevented the server from fulfilling your request.", "code": 500, "title": "Internal Server Error"}}

=========

PATCH /v3/policies/..%c0%af HTTP/1.1
Host: [REDACTED]:5000
Connection: close
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.11.0
Content-type: application/json
X-Auth-Token: [REDACTED]
Content-Length: 70

{"type": "--serialization-mime-type--", "blob": "--serialized-blob--"}

HTTP/1.1 500 Internal Server Error
Date: Tue, 16 Aug 2016 22:05:36 GMT
Server: Apache/2.4.7 (Ubuntu)
Vary: X-Auth-Token
X-Distribution: Ubuntu
x-openstack-request-id: req-57a41600-02b4-4d2a-b3e9-40f7724d65f2
Content-Length: 143
Connection: close
Content-Type: application/json

{"error": {"message": "An unexpected error prevented the server from fulfilling your request.", "code": 500, "title": "Internal Server Error"}}

=========

GET /v3/domains/0426ac1e48f642ef9544c2251e07e261/groups/..%c0%af/roles HTTP/1.1
Host: [REDACTED]:5000
Connection: close
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.11.0
X-Auth-Token: [REDACTED]

HTTP/1.1 500 Internal Server Error
Date: Tue, 16 Aug 2016 22:07:09 GMT
Server: Apache/2.4.7 (Ubuntu)
Vary: X-Auth-Token
X-Distribution: Ubuntu
x-openstack-request-id: req-02313f77-63c6-4aa8-a87e-e3d2a13ad6b7
Content-Length: 143
Connection: close
Content-Type: application/json

{"error": {"message": "An unexpected error prevented the server from fulfilling your request.", "code": 500, "title": "Internal Server Error"}}

=========

I've marked this as a security issue as a precaution in case it turns out that there is a more serious vulnerability underlying these errors. We have no reason to suspect that there is a greater vulnerability at this time, but given the many endpoints this seems to affect, I figured caution was worthwhile since this may be a framework-wide issue. Feel free to make this public if it is determined not to be security-impacting.

Here is a (possibly incomplete) list of affected endpoints. Inserting the string "..%c0%af" in any or all of the spots labeled "HERE" should yield a 500 error. As you can see, virtually all v3 endpoints exhibit this behavior.

=========

[GET|PATCH|DELETE] /v3/endpoints/[HERE]

[GET|PATCH] /v3/domains/[HERE]
GET /v3/domains/[HERE]/groups/[HERE]/roles
[HEAD|PUT|DELETE] /v3/domains/[HERE]/groups/[HERE]/roles/[HERE]
GET /v3/domains/[HERE]/users/[HERE]/roles
[HEAD|DELETE] /v3/domains/[HERE]/users/[HERE]/roles/[HERE]

[GET|PATCH|DELETE] /v3/groups/[HERE]
[HEAD|PUT|DELETE] /v3/groups[HERE]/users/[HERE]

[POST|DELETE] /v3/keys/[HERE]

[GET|PATCH|DELETE] /v3/policies/[HERE]
[GET|PUT|DELETE] /v3/policies/[HERE]/OS-ENDPOINT-POLICY/endpoints/[HERE]
[GET|HEAD] /v3/policies/[HERE]/OS-ENDPOINT-POLICY/policy
[GET|PUT|DELETE] /v3/policies/[HERE]/OS-ENDPOINT-POLICY/services/[HERE]
[PUT|DELETE] /v3/policies/[HERE]/OS-ENDPOINT-POLICY/services/[HERE]
[GET|PUT|DELETE] /v3/policies/[HERE]/OS-ENDPOINT-POLICY/services/regions/[HERE]

[GET|PATCH|DELETE] /v3/projects/[HERE]
[DELETE|PATCH] /v3/projects/[HERE]/cascade
GET /v3/projects/[HERE]/groups/[HERE]/roles
GET /v3/projects/[HERE]/users/[HERE]/roles
[HEAD|PUT|DELETE] /v3/projects/[HERE]/groups/[HERE]/roles/[HERE]

[GET|PATCH|DELETE] /v3/regions/[HERE]

[PATCH|DELETE] /v3/roles/[HERE]

[GET|PATCH|DELETE] /v3/services/[HERE]

[GET|PATCH|DELETE] /v3/users/[HERE]
GET /v3/users/[HERE]/groups
POST /v3/users/[HERE]/password
GET /v3/users/[HERE]/projects

GET /v3/OS-OAUTH1/users/[HERE]/access_tokens/[HERE]/roles/[HERE]
[GET|PATCH|DELETE] /v3/OS-OAUTH1/consumers/[HERE]
[GET|DELETE] /v3/OS-OAUTH1/users/[HERE]/access_tokens/[HERE]

[David Stanek (dstanek)]

This happens because webob tries to decode the path as UTF-8, but that is actually malformed UTF-8. This is probably something that should be caught in keystone.common.wsgi and returned as a 400, but I don't think these is any security issues. Is it possible to use this to exploit keystone is some way.

Log from a bad request:

2016-08-16 23:45:25.192 14345 ERROR keystone.common.wsgi [req-dd271afd-aa0c-4fd2-90f2-6366327e9bba - - - - -] 'utf8' codec can't decode byte 0xc0 in position 12: invalid start byte
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi Traceback (most recent call last):
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/opt/stack/keystone/keystone/common/wsgi.py", line 372, in _inner
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi return method(self, request)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/opt/stack/keystone/keystone/common/wsgi.py", line 432, in __call__
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi response = request.get_response(self.application)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi application, catch_exc_info=False)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi app_iter = application(self.environ, start_response)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi resp = self.call_func(req, *args, **self.kwargs)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi return self.func(req, *args, **kwargs)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/opt/stack/keystone/keystone/common/wsgi.py", line 687, in __call__
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi if request.path_info != '/':
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi return req.encget(key, encattr=encattr)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi return val.decode(encoding)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi return codecs.utf_8_decode(input, errors, True)
2016-08-16 23:45:25.192 14345 TRACE keystone.common.wsgi UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 12: invalid start byte
2016-08-16 23:45:25.192 14345 TRACE keystone.common....

[Morgan Fainberg (mdrnstm)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Morgan Fainberg (mdrnstm)]

For what it is worth, this doesn't look like a security flaw that can be exploited directly as far as I can see from the trace david stanek provided. However, I will defer to the keystone-coresec team to determine if this actually does result in an exploit.

[Morgan Fainberg (mdrnstm)]

@David Stanek/Keystone Core-Sec,

If you would like to classify this as not a security bug, please let us know so that it can be moved to a public bug.

[Dolph Mathews (dolph)]

+1 for fixing the issue in public.

[Steve Martinelli (stevemar)]

I don't see a reason to keep this private.

[Morgan Fainberg (mdrnstm)]

I have marked this bug as public and since this is not a security related bug, the OSSA task has been marked as "Wont Fix".

Thanks Steve and Dolph for the quick response.

[Robert Clark (robert-clark)]

Seems reasonable to me. Well done on finding this Syntribos team - this tool is showing real promise. I'm expecting you to find some interesting higher-severity issues in the future!

[Charles Neill (charles-neill)]

This affects a number of other OpenStack projects in a similar way, including:

- Neutron
- Cinder
- Glance

More projects may be affected that we are unaware of.

=====================================================
Example traceback from Neutron
=====================================================

2016-09-28 00:16:43.342 1218 DEBUG neutron.wsgi [-] (1218) accepted ('10.0.2.2', 50029) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:868
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/local/lib/python2.7/dist-packages/oslo_log/handlers.py", line 76, in format
    return logging.StreamHandler.format(self, record)
  File "/usr/lib/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/local/lib/python2.7/dist-packages/oslo_log/formatters.py", line 297, in format
    return logging.Formatter.format(self, record)
  File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
    record.message = record.getMessage()
  File "/usr/lib/python2.7/logging/__init__.py", line 328, in getMessage
    msg = msg % self.args
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1164, in as_text
    bytes = self.as_bytes()
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1135, in as_bytes
    url = self.url
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 504, in url
    url = self.path_url
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 476, in path_url
    bpath_info = bytes_(self.path_info, self.url_encoding)
  File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget
    return req.encget(key, encattr=encattr)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget
    return val.decode(encoding)
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 11: invalid start byte
Logged from file catch_errors.py, line 41
2016-09-28 00:16:43.489 1218 INFO neutron.wsgi [req-37cfe540-c134-4ec0-91d2-56687f38ffd5 admin -] 10.0.2.2 - - [28/Sep/2016 00:16:43] "PUT /v2.0/flavors/..%c0%af HTTP/1.1" 500 414 0.140984

=====================================================
Example traceback from Cinder
=====================================================

2016-09-27 23:50:00.142 5986 DEBUG eventlet.wsgi.server [-] (5986) accepted ('10.0.2.2', 49862) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:868
2016-09-27 23:50:00.198 ERROR cinder.api.middleware.fault [req-7ec3610c-9cdc-4f7d-b69c-36c385d0fa10 admin] Caught error: <type 'exceptions.UnicodeDecodeError'> 'utf8' codec can't decode byte 0xc0 in position 3: invalid start byte
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault Traceback (most recent call last):
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/opt/stack/cinder/cinder/api/middleware/fault.py", line 79, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return ...

[Armando Migliaccio (armando-migliaccio)]

I am unclear to where the issue lies, but should this be fixed centrally rather than being delegated to the individual projects?

[Sean McGinnis (sean-mcginnis)]

This does seem like something that should be fixed centrally. Otherwise this will pop up repeatedly as projects come and go.

[John Perkins (john-d-perkins-deactivatedaccount)]

Webob maintainer 'bertjwregeer' has been pinged regarding this issue and there the fix is in progress: https://github.com/Pylons/webob/issues/161

[Bert JW Regeer (bertjwregeer)]

The "fix" in WebOb is going to be checking for UnicodeDecodeError and turning that into an URLDecodeError. There would still be a requirement for something upstream of the Request object to catch the exception and properly send it out.

URLDecodeError will be a sub-class of UnicodeDecodeError and HTTPBadRequest, so that it is a valid WebOb exc.WSGIHTTPException and thus can be used as a response object...

There is no direct fix that will suddenly handle this case directly, code that touches PATH and other such variables will simply raise a better exception than a generic UnicodeDecodeError.