User Operations Fail in case if Keystone's endpoint list is larger than 8 KB

[Sanchit Malhotra]

If Swift Server is associated with Keystone Server which has endpoints of all openstack key services distributed across various regions, then while authenticating a user requests, the whole list is appended in request header.

For example:
'X-Service-Catalog': '[{"endpoints": [{"adminURL": "http://192.168.136.199:35357/", "region": "RegionOne", "internalURL": "http://192.168.136.199:35357/", "publicURL": "http://192.168.136.199:5000"}], "type": "identity", "name": "keystone"}, {"endpoints": [{"adminURL": "http://192.168.136.198:8777", "region": "regionA"}, {"adminURL": "http://192.168.136.198:8777", "region": "regionB"}, {"adminURL": "http://192.168.136.198:8777", "region": "RegionC"}, {"adminURL": "http://192.168.136.198:8777", "region": "RegionD"}, {"adminURL": "http://192.168.136.198:8777", "region": "RegionE"}, {"adminURL": "http://192.168.136.198:8777", "region": "RegionF"}, {"adminURL": "http://192.168.136.198:8777", ..............

If suppose the size of list of endpoints is larger than 8 KB, then the request would fail at WSGI as it accepts header Line of a max size of 8 KB.

File: eventlet/wsgi.py
{code}
MAX_HEADER_LINE = 8192

class FileObjectForHeaders(object):

    def __init__(self, fp):
        self.fp = fp
        self.total_header_size = 0

    def readline(self, size=-1):
        sz = size
        if size < 0:
            sz = MAX_HEADER_LINE
        rv = self.fp.readline(sz)
        if size < 0 and len(rv) >= MAX_HEADER_LINE:
            raise HeaderLineTooLong()
        self.total_header_size += len(rv)
        if self.total_header_size > MAX_TOTAL_HEADER_SIZE:
            raise HeadersTooLarge()
        return rv
{code}

Following is the error:

root@swift:~# swift upload abad .bashrc --debug
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://192.168.136.199:5000/v2.0/tokens
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 192.168.136.199
DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 21389
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 192.168.136.192
DEBUG:requests.packages.urllib3.connectionpool:"PUT /v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad HTTP/1.1" 400 40
INFO:swiftclient:REQ: curl -i http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad -X PUT -H "Content-Length: 0" -H "X-Auth-Token: 3260f6aa815b4767a1022e168766e035"
INFO:swiftclient:RESP STATUS: 400 Bad Request
INFO:swiftclient:RESP HEADERS: {u'Date': u'Wed, 18 Jan 2017 11:51:13 GMT', u'Content-Length': u'40', u'Content-Type': u'text/plain', u'X-Trans-Id': u'tx75aae9d07c4848e9bf168-00587f56b1'}
INFO:swiftclient:RESP BODY: Header value too long: X-Service-Catalog
ERROR:swiftclient:Container PUT failed: http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad 400 Bad Request Header value too long: X-Service-Catalog
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1565, in _retry
    service_token=self.service_token, **kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 976, in put_container
    http_response_content=body)
ClientException: Container PUT failed: http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad 400 Bad Request Header value too long: X-Service-Catalog
ERROR:swiftclient.service:Container PUT failed: http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad 400 Bad Request Header value too long: X-Service-Catalog
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/swiftclient/service.py", line 1554, in _create_container_job
    container, headers, response_dict=create_response
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1643, in put_container
    response_dict=response_dict)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1565, in _retry
    service_token=self.service_token, **kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 976, in put_container
    http_response_content=body)
ClientException: Container PUT failed: http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad 400 Bad Request Header value too long: X-Service-Catalog
Warning: failed to create container 'abad': 400 Bad Request: Header value too long: X-Service-Catalog
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://192.168.136.199:5000/v2.0/tokens
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 192.168.136.199
DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 21389
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 192.168.136.192
DEBUG:requests.packages.urllib3.connectionpool:"HEAD /v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad/.bashrc HTTP/1.1" 400 0
INFO:swiftclient:REQ: curl -i http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad/.bashrc -I -H "X-Auth-Token: 3ea0c82a1b26469c82dac9452e7f13de"
INFO:swiftclient:RESP STATUS: 400 Header Line Too Long
INFO:swiftclient:RESP HEADERS: {u'Date': u'Wed, 18 Jan 2017 11:51:13 GMT', u'Content-Length': u'0', u'Content-Type': u'text/html; charset=UTF-8', u'X-Trans-Id': u'tx8bb737da355b48a481e17-00587f56b1'}
ERROR:swiftclient:Object HEAD failed: http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad/.bashrc 400 Header Line Too Long
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1565, in _retry
    service_token=self.service_token, **kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1166, in head_object
    http_response_content=body)
ClientException: Object HEAD failed: http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad/.bashrc 400 Header Line Too Long
ERROR:swiftclient.service:Object HEAD failed: http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad/.bashrc 400 Header Line Too Long
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/swiftclient/service.py", line 1783, in _upload_object_job
    headers = conn.head_object(container, obj)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1657, in head_object
    return self._retry(None, head_object, container, obj, headers=headers)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1565, in _retry
    service_token=self.service_token, **kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1166, in head_object
    http_response_content=body)
ClientException: Object HEAD failed: http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad/.bashrc 400 Header Line Too Long
Object HEAD failed: http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad/.bashrc 400 Header Line Too Long

[Donagh McCabe (donagh-mccabe)]

What is your setting for include_service_catalog?

By default, the value is "false". This was originally done for two reasons:

- saves space in the environment [since Swift does not use/need the catalog]
- prevented the above issue

[Having said this, I had thought the header-too-long problem had been fixed elsewhere, but maybe I miss-remember and the only solution used was to set include_service_catalog=false]

Note: Swift itself does need X-Service-Catalog. The Keystone audit middleware *does* use it, but unless you have good reason to use audit middleware, you can set include_service_catalog=false.

[Donagh McCabe (donagh-mccabe)]

FYI: https://review.openstack.org/#/c/59371

[Sanchit Malhotra (isanchitm)]

>What is your setting for include_service_catalog?

Unfortunately, my proxy configuration file does not include the configuration parameter "include_service_catalog" under auth_token filter.

But I suppose this setting is just for applicable for "X-Service-Catalog" request header.

[Tim Burke (1-tim-z)]

Our sample config makes it seem like that defaults to False [1], but it definitely defaults to True [2] and has for a very long time [3].

Seems like we should clarify what the default is and add a note about why you may want to change the option to False, similar to what we do for delay_auth_decision.

[1] https://github.com/openstack/swift/blob/2.15.1/etc/proxy-server.conf-sample#L385
[2] https://github.com/openstack/keystonemiddleware/blob/4.17.0/keystonemiddleware/auth_token/_opts.py#L149
[3] https://github.com/openstack/python-keystoneclient/commit/a97b293

[Tim Burke (1-tim-z)]

*Maybe* we could also get away with having swift's keystoneauth middleware drop any x-service-catalog headers it finds? I'm not sure who uses it or where they would be in the pipeline, though.