Content-Type 'multipart/form-data' causes 500 on POST/PUT/PATCH requests

[Charles Neill]

It appears that, despite the restrictions on Content-Type headers for valid requests, using "multipart/form-data" as the content type on otherwise-valid POST, PUT, and PATCH requests will cause Barbican to throw a 500 error. This could be a bug in an upstream dependency like pecan or paste.

-----------------------------------
Example request / response:

POST /v1/secrets HTTP/1.1
Host: localhost:9311
Content-Length: 246
X-Project-Id: [valid project-id]
Accept: application/json
Connection: keep-alive
X-Auth-Token: [valid token]
Content-Type: multipart/form-data

{"name": "AES key", "algorithm": "aes", "payload_content_type": "application/octet-stream", "payload_content_encoding": "base64", "mode": "cbc", "bit_length": 256, "payload": "gF6+lLoF3ohA9aPRpt+6bQ==", "expiration": "2018-02-28T19:14:44.180394"}

HTTP/1.0 500 Internal Server Error
Server: PasteWSGIServer/0.5 Python/2.7.5
Date: Mon, 03 Aug 2015 17:09:29 GMT
Content-type: text/plain
Content-length: 22

Internal Server Error

-----------------------------------
Stacktrace:

Exception happened during processing of request from ('127.0.0.1', 61306)
Traceback (most recent call last):
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/paste/httpserver.py", line 1085, in process_request_in_thread
    self.finish_request(request, client_address)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 649, in __init__
    self.handle()
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/paste/httpserver.py", line 459, in handle
    BaseHTTPRequestHandler.handle(self)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/paste/httpserver.py", line 454, in handle_one_request
    self.wsgi_execute()
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/paste/httpserver.py", line 304, in wsgi_execute
    self.wsgi_start_response)
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/paste/urlmap.py", line 216, in __call__
    return app(environ, start_response)
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 639, in __call__
    return self._call_app(env, start_response)
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 559, in _call_app
    return self._app(env, _fake_start_response)
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "~/tests/barbican/os-clean/barbican/barbican/api/middleware/__init__.py", line 66, in __call__
    response = req.get_response(self.application)
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/webob/request.py", line 1317, in send
    application, catch_exc_info=False)
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/webob/request.py", line 1281, in call_application
    app_iter = application(self.environ, start_response)
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/pecan/core.py", line 812, in __call__
    return super(Pecan, self).__call__(environ, start_response)
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/pecan/core.py", line 660, in __call__
    controller, args, kwargs = self.find_controller(state)
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/pecan/core.py", line 830, in find_controller
    controller, args, kw = super(Pecan, self).find_controller(_state)
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/pecan/core.py", line 524, in find_controller
    params = req.params
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/pecan/core.py", line 54, in __getattribute__
    return WebObRequest.__getattribute__(self, name)
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/webob/request.py", line 853, in params
    params = NestedMultiDict(self.GET, self.POST)
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/pecan/core.py", line 54, in __getattribute__
    return WebObRequest.__getattribute__(self, name)
  File "~/.virtualenvs/barb/lib/python2.7/site-packages/webob/request.py", line 808, in POST
    keep_blank_values=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/cgi.py", line 498, in __init__
    self.read_multi(environ, keep_blank_values, strict_parsing)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/cgi.py", line 612, in read_multi
    raise ValueError, 'Invalid boundary in multipart form: %r' % (ib,)
ValueError: Invalid boundary in multipart form: ''

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/208662

[OpenStack Infra (hudson-openstack)]

Change abandoned by Fernando Diaz (<email address hidden>) on branch: master
Review: https://review.openstack.org/208545
Reason: Hasn't been worked on in a while

[OpenStack Infra (hudson-openstack)]

Change abandoned by Charles Neill (<email address hidden>) on branch: master
Review: https://review.openstack.org/208622
Reason: Abandoning in favor of Syntribos

[OpenStack Infra (hudson-openstack)]

Change abandoned by Charles Neill (<email address hidden>) on branch: master
Review: https://review.openstack.org/208662
Reason: Abandoning in favor of Syntribos

[Dave McCowan (dave-mccowan)]

I recreated this on the current master branch. cgi.py is raising the exception. It does not seem Barbican code ever gets involved.

Partial traceback from Barbican log:

2017-03-20 18:07:28.970 TRACE barbican File "/usr/lib/python2.7/cgi.py", line 621, in read_multi
2017-03-20 18:07:28.970 TRACE barbican raise ValueError, 'Invalid boundary in multipart form: %r' % (ib,)
2017-03-20 18:07:28.970 TRACE barbican ValueError: Invalid boundary in multipart form: ''
2

Command to recreate:

curl -vvv -X POST -H "content-type:multipart/form-data" -H "X-Auth-Token: $TOKEN" -d '{"payload": "my-secret-here", "payload_content_type": "text/plain"}' http://localhost:9311/v1/secrets

[Grzegorz Grasza (xek)]

Closing out bugs created before migration to StoryBoard. Please re-open if you are of the opinion it is still current.