keystone under apache can't handle request chunking

[David Patterson]

Method to reproduce stack trace for apache config with chunking enabled
========================================================

1. If using devstack, configure it to enable chunking in apache, by adding "WSGIChunkedRequest On" in the file devstack/files/apache-keystone.template (in both sections) and start devstack

2. Run from the command line:

openstack -v --debug server list (or any simple command)

3. Look in the output for the very first "curl" command that looks something like this:

curl -i --insecure -X POST http://192.168.51.21:5000/v2.0/tokens -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-keystoneclient" -d '{"auth": {"tenantName": "demo", "passwordCredentials": {"username": "demo", "password": "admin"}}}'

4. The above "curl" command succeeds. Modify it to use chunking by adding -H "Transfer-Encoding: chunked" and run it:

curl -i --insecure -X POST http://192.168.51.21:5000/v2.0/tokens -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-keystoneclient" -H "Transfer-Encoding: chunked" -d '{"auth": {"tenantName": "demo", "passwordCredentials": {"username": "demo", "password": "admin"}}}'

5. You will get a cli error:

HTTP/1.1 500 Internal Server Error
Date: Wed, 16 Jul 2014 19:01:58 GMT
Server: Apache/2.2.22 (Ubuntu)
Vary: X-Auth-Token
Content-Length: 215
Connection: close
Content-Type: application/json

{"error": {"message": "An unexpected error prevented the server from fulfilling your request: request data read error (Disable debug mode to suppress these details.)", "code": 500, "title": "Internal Server Error"}}

6. The keystone log will show a stack trace:

[Wed Jul 16 19:02:43 2014] [error] 13693 ERROR keystone.common.wsgi [-] request data read error
[Wed Jul 16 19:02:43 2014] [error] 13693 TRACE keystone.common.wsgi Traceback (most recent call last):
[Wed Jul 16 19:02:43 2014] [error] 13693 TRACE keystone.common.wsgi File "/opt/stack/keystone/keystone/common/wsgi.py", line 414, in __call__
[Wed Jul 16 19:02:43 2014] [error] 13693 TRACE keystone.common.wsgi response = self.process_request(request)
[Wed Jul 16 19:02:43 2014] [error] 13693 TRACE keystone.common.wsgi File "/opt/stack/keystone/keystone/middleware/core.py", line 112, in process_request
[Wed Jul 16 19:02:43 2014] [error] 13693 TRACE keystone.common.wsgi params_json = request.body
[Wed Jul 16 19:02:43 2014] [error] 13693 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 677, in _body__get
[Wed Jul 16 19:02:43 2014] [error] 13693 TRACE keystone.common.wsgi self.make_body_seekable() # we need this to have content_length
[Wed Jul 16 19:02:43 2014] [error] 13693 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 922, in make_body_seekable
[Wed Jul 16 19:02:43 2014] [error] 13693 TRACE keystone.common.wsgi self.copy_body()
[Wed Jul 16 19:02:43 2014] [error] 13693 TRACE keystone.common.wsgi File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 938, in copy_body
[Wed Jul 16 19:02:43 2014] [error] 13693 TRACE keystone.common.wsgi self.body = self.body_file_raw.read()
[Wed Jul 16 19:02:43 2014] [error] 13693 TRACE keystone.common.wsgi File "/opt/stack/keystone/keystone/common/utils.py", line 306, in read
[Wed Jul 16 19:02:43 2014] [error] 13693 TRACE keystone.common.wsgi result = self.data.read()
[Wed Jul 16 19:02:43 2014] [error] 13693 TRACE keystone.common.wsgi IOError: request data read error
[Wed Jul 16 19:02:43 2014] [error] 13693 TRACE keystone.common.wsgi

[Morgan Fainberg (mdrnstm)]

If I'm reading the information on the google groups archive for mod_wsgi, this is an issue with mod_wsgi and not something that can be fixed in Keystone directly. This is related to the 3.X tree for mod_wsgi, which *should* be fixed in the 4.X series. It appears that Ubuntu 14.04 (Trusty) has the 3.4 release of mod_wsgi

See: https://groups.google.com/forum/#!topic/modwsgi/Rk-cXTGSCHQ

This is going to take some extra work to determine if anything can be done prior to using a 4.X release of mod_wsgi.

[Graham Dumpleton (graham-dumpleton)]

Two issues here.

1. Has Keystone even been modified to violate the WSGI specification so as to allow chunked request content? Simply enabling that mod_wsgi option will not help if your application hasn't been modified so as to ignore the WSGI specification and try and work anyway. For portability to different WSGI servers, you shouldn't be violating the WSGI specification.

2. That option will not work with mod_wsgi daemon mode. That is still the case even with latest mod_wsgi versions.

[Samuel Bercovici (samuelb)]

Please note that prior moving KeyStone to work by default in apache, working in chunked mode was supported.
So if you configure devstack to not use apache (KEYSTONE_USE_MOD_WSGI=False in local.conf) chunking works.

This may create an issue to code who works with the KeyStone API in chunking model and used to work prior the change to work by default with apache.

[Morgan Fainberg (mdrnstm)]

@graham,

We have not modified keystone to violate wsgi spec and we recommend daemon mode. We should not be violating the wsgi spec. I want to support other wsgi servers and deployment options (not just eventlet and mod_wsgi). So I don't think we should break portability.

@samuel
If eventlet allows chunked and it violates the spec, I would argue it was a bug in eventlet and should either :

a) be documented that chunked only works under eventlet and that it is not guaranteed to continue working as it is in violation of the wsgi spec

B) disable chunked support even in eventlet with documentation that it violates the spec.

I am inclined to say the right answer is option (a).

[Dolph Mathews (dolph)]

David: What's the use case for attempting chunking in the first place?

[David Patterson (david-patterson)]

It is a long story. The version of jersey/apachehttpclient4 we used in our OpenStack Java libraries always used chunking. It always worked with OpenStack until Keystone moved to using apache as the default in devstack. (It is the only project that uses apache as default, other than horizon.)

Of course, our luck is that there is a bug in the version of jersey/apachehttpclient4 we were using that didn't allow turning off chunking. We had to upgrade jersey and turn off chunking for requests not needing it. (We chose to only use chunking on application/octet-stream requests such as uploding an image.) To upgrade jersey we had to (re)shade asm.

All that from a seemingly small, innocuous config change. :)

[Lance Bragstad (lbragstad)]

Adding the meeting-topic tag to this discuss in #openstack-meeting if we can improve the documentation before Juno RC1.

[Lance Bragstad (lbragstad)]

Discussed this in the Keystone weekly meeting and we've decided that we should target a documentation fix for Juno RC1. The proper fix belongs upstream in Apache (cc. morganfainberg).

[Morgan Fainberg (mdrnstm)]

There is no proper fix, the wsgi spec pretty much precludes the use of request chunking short of violating the spec. In short, under apache request chunking wont be supported as outlined in my previous comment.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/120274

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/120274
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=05c01e26d1e62298489ba3894576faf3c352e2c9
Submitter: Jenkins
Branch: master

commit 05c01e26d1e62298489ba3894576faf3c352e2c9
Author: Morgan Fainberg <email address hidden>
Date: Tue Sep 9 15:44:17 2014 -0700

    Document mod_wsgi doesn't support chunked encoding

    Document in logical places that keystone under mod_wsgi will not
    support chunked encoding.

    Change-Id: I957059be560fa65e2f0d5166d9490b2c7bab9f17
    Closes-Bug: #1346211