Token auth fails when token is larger than 8k

[Steve Baker]

The following tests fail when there are 8 or more endpoints registered with keystone
tempest.api.compute.test_auth_token.AuthTokenTestJSON.test_v3_token
tempest.api.compute.test_auth_token.AuthTokenTestXML.test_v3_token

Steps to reproduce:
- run devstack with the following services (the heat h-* apis push the endpoint count over the threshold
  ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-obj,n-cpu,n-sch,horizon,mysql,rabbit,sysstat,tempest,s-proxy,s-account,s-container,s-object,cinder,c-api,c-vol,c-sch,n-cond,heat,h-api,h-api-cfn,h-api-cw,h-eng,n-net
- run the failing tempest tests, eg
  testr run test_v3_token
- results in the following errors:
ERROR: tempest.api.compute.test_auth_token.AuthTokenTestJSON.test_v3_token
tags: worker-0
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tempest/api/compute/test_auth_token.py", line 48, in test_v3_token
    self.servers_v3.list_servers()
  File "tempest/services/compute/json/servers_client.py", line 138, in list_servers
    resp, body = self.get(url)
  File "tempest/common/rest_client.py", line 269, in get
    return self.request('GET', url, headers)
  File "tempest/common/rest_client.py", line 394, in request
    resp, resp_body)
  File "tempest/common/rest_client.py", line 443, in _error_checker
    resp_body = self._parse_resp(resp_body)
  File "tempest/common/rest_client.py", line 327, in _parse_resp
    return json.loads(body)
  File "/usr/lib64/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
======================================================================
ERROR: tempest.api.compute.test_auth_token.AuthTokenTestXML.test_v3_token
tags: worker-0
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tempest/api/compute/test_auth_token.py", line 48, in test_v3_token
    self.servers_v3.list_servers()
  File "tempest/services/compute/xml/servers_client.py", line 181, in list_servers
    resp, body = self.get(url, self.headers)
  File "tempest/common/rest_client.py", line 269, in get
    return self.request('GET', url, headers)
  File "tempest/common/rest_client.py", line 394, in request
    resp, resp_body)
  File "tempest/common/rest_client.py", line 443, in _error_checker
    resp_body = self._parse_resp(resp_body)
  File "tempest/common/rest_client.py", line 519, in _parse_resp
    return xml_to_json(etree.fromstring(body))
  File "lxml.etree.pyx", line 2993, in lxml.etree.fromstring (src/lxml/lxml.etree.c:63285)
  File "parser.pxi", line 1617, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:93571)
  File "parser.pxi", line 1495, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:92370)
  File "parser.pxi", line 1011, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:89010)
  File "parser.pxi", line 577, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:84711)
  File "parser.pxi", line 676, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:85816)
  File "parser.pxi", line 627, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:85308)
XMLSyntaxError: None
Ran 2 tests in 2.497s (+0.278s)
FAILED (id=214, failures=2)

- run keystone endpoint-delete on endpoints until there is 7 endpoints
- failing tests should now pass

[Steve Baker (steve-stevebaker)]

This bug is currently blocking heat from being enabled in tempest gating https://review.openstack.org/#/c/32002

[Dolph Mathews (dolph)]

What is the JSON/XML response returned from keystone in either case? I'm guessing there's something wrong with a specific endpoint, rather than an issue with the specific number of endpoints.

[Steve Baker (steve-stevebaker)]

I've done some testing and I really do thing the working endpoint limit is 8.

Steps to reproduce:
- keystone endpoint-delete until there are 6 endpoints
- confirm test_v3_token works with 6 endpoints
- create a new endpoint
  keystone service-create --name ep1 --type test1
  keystone endpoint-create --region RegionOne --service-id <serviceid> --publicurl http://example.com --adminurl http://example.com --internalurl http://example.com
- confirm test_v3_token works with 7 endpoints
- create a new endpoint
  keystone service-create --name ep2 --type test2
  keystone endpoint-create --region RegionOne --service-id <serviceid> --publicurl http://example.com --adminurl http://example.com --internalurl http://example.com
- confirm test_v3_token fails with 8 endpoints
- delete endpoint ep1
- confirm test_v3_token works with 7 endpoints

Is there some simple curl foo I can use to make keystone v3 requests?

[Steve Baker (steve-stevebaker)]

In the testing mentioned above, the token for 8 endpoints is 8448 bytes. For 7 endpoints its 7640.

I'd bet a beer in Hong Kong that something is truncating the token header to 8k somewhere.

[Steve Baker (steve-stevebaker)]

The v2 catalog format is more compact, hence smaller tokens. I created enough endpoints to make a v2 token over 8k (20 endpoints total)

This results in many calls to keystone, then test_auth_token.py not running any tests at all

[Dolph Mathews (dolph)]

This would be resolved by https://blueprints.launchpad.net/keystone/+spec/catalog-optional

[Steve Baker (steve-stevebaker)]

What could we do in the short term so that heat tests can run on gating?

[Dolph Mathews (dolph)]

Fixing this would provide a small improvement here https://bugs.launchpad.net/keystone/+bug/1164458

[Steve Baker (steve-stevebaker)]

I've found a fix, but it would have to be applied to every project that uses eventlet.wsgi:
http://lists.openstack.org/pipermail/openstack-dev/2013-June/010366.html

[Steve Baker (steve-stevebaker)]

Fixed in nova with https://review.openstack.org/#/c/33362/

[Chmouel Boudjnah (chmouel)]

https://review.openstack.org/#/c/39882/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/64641
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=aa1ba1f0176af286506e4188628e5acbb8f7f0e6
Submitter: Jenkins
Branch: master

commit aa1ba1f0176af286506e4188628e5acbb8f7f0e6
Author: Shao Kai Li <email address hidden>
Date: Thu Jan 2 00:51:41 2014 -0500

    Raise max header size to accommodate large tokens.

    The max header is exceeded in the following scenario
    - Auth tokens built with a keystone v3 API catalog
    - A catalog with approximately 8 or more endpoints defined

    An equivalent fix will be required for all projects which use
    eventlet.wsgi.

    Copied from https://review.openstack.org/#/c/33362/

    Closes-Bug: 1190149

    Change-Id: I3fcb196b25a7a477b799c8dfa2ff5115977ca06d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/66332

[Florent Flament (florentflament)]

Neutron fix has been previsouly proposed by Zhang Hua here:
https://review.openstack.org/#/c/56511/

Trying and reactivating this abandonned patch.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/66615

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/66332
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=df8be62e58d8eb36cef43a7f3b594ff0a0e1ccb6
Submitter: Jenkins
Branch: master

commit df8be62e58d8eb36cef43a7f3b594ff0a0e1ccb6
Author: Florent Flament <email address hidden>
Date: Mon Jan 13 13:43:59 2014 +0000

    Allow users to customize max header size

    Also increases the default value from 8192 to 16384

    The former max header (8192) is exceeded in the following scenario
    - Auth tokens built with a keystone v3 API catalog
    - A catalog with approximately 8 or more endpoints defined

    Equivalent fixes have been proposed in other projects

    Change-Id: Ic714166440e2aa08b619ab1e5e5e0cf3cea3e309
    Closes-Bug: 1190149

[Florent Flament (florentflament)]

Zhang Hua's Neutron fix has been reactivated:

https://review.openstack.org/#/c/56511/

[Florent Flament (florentflament)]

An interesting comment from Mark McClain about redefining constants:
https://review.openstack.org/#/c/56511/2/neutron/wsgi.py

I agree with him that redefining constants should be avoided when possible, since it can possibly create unexpected behaviors if the constant is redefined to different values in different parts of the code.

However, in the case of the eventlet.wsgi.MAX_HEADER_LINE constant, I don't see any better solution until eventlet's code is updated:
https://bitbucket.org/eventlet/eventlet/src/66cc547edabfb7b45afaead69f029cae7f51f161/eventlet/wsgi.py#cl-189

We should be careful about not redefining this constant several times (grep -r MAX_HEADER_LINE .)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/67601

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/66615
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=865243c16751e18e28b60ef03ecaada44c22cd13
Submitter: Jenkins
Branch: master

commit 865243c16751e18e28b60ef03ecaada44c22cd13
Author: Florent Flament <email address hidden>
Date: Tue Jan 14 15:58:38 2014 +0000

    Fix Error 400 Header Line Too Long

    Fix Error 400 Header Line Too Long when using Identity v3 PKI Tokens

    Uses swift.conf max_header_size option to set wsgi.MAX_HEADER_LINE,
    allowing the operator to customize this parameter.

    The default value has been let to 8192 to avoid unexpected
    configuration change on deployed platforms. The max_header_size option
    has to be increased (for example to 16384), to accomodate for large
    Identity v3 PKI tokens, including more than 7 catalog entries.

    The default max header line size of 8192 is exceeded in the following
    scenario:
    - Auth tokens generated by Keystone v3 API include the catalog.
    - Keystone's catalog contains more than 7 services.

    Similar fixes have been merged in other projects.

    Change-Id: Ia838b18331f57dfd02b9f71d4523d4059f38e600
    Closes-Bug: 1190149

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/67800

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/70289

[Steve Baker (steve-stevebaker)]

Actually I'm going to mark this as critical for heat, since I think our switch to keystone v3 (and its bigger tokens) will trigger this in the common case.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: feature/ec
Review: https://review.openstack.org/70442

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/67800
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=933f1dca3662b0667c90a3fc143c5245fe6b3f91
Submitter: Jenkins
Branch: master

commit 933f1dca3662b0667c90a3fc143c5245fe6b3f91
Author: Florent Flament <email address hidden>
Date: Mon Jan 20 10:07:53 2014 +0000

    Allow operators to customize max header size

    HTTP messages max header line size has been increased from 8K to 16K
    by default to allow using tokens including up to 14-15 catalog
    entries. The same issue (https://bugs.launchpad.net/swift/+bug/119014)
    may occur again in the future, if Keystone's catalog grows further.

    Allowing operators to customize the max header size, will allow them
    to have Nova working whatever the size of the catalog (if the option
    is properly set).

    Related-Bug: #1190149
    DocImpact

    Change-Id: I96694ac9bc242267cfc9f4d0c18b2b7fe0000460

[Kai Qiang Wu(Kennan) (wkqwu)]

If my understand correctly, the new change make max header line as configurable field in conf file, keystone need follow that.

Seems keystone hard code that.

[Steve Baker (steve-stevebaker)]

It looks like we're converging on doing this as a conf options rather than hardcoded.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/67601
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=be163cf1028aff50413a89060bb8badfa7d51ad5
Submitter: Jenkins
Branch: master

commit be163cf1028aff50413a89060bb8badfa7d51ad5
Author: Florent Flament <email address hidden>
Date: Fri Jan 17 10:54:27 2014 +0000

    Allow operators to customize max header size

    HTTP messages max header line size has been increased from 8K to 16K
    by default to allow using tokens including up to 14-15 catalog
    entries. The same issue
    (https://bugs.launchpad.net/swift/+bug/1190149) may occur again in
    the future, if keystone's catalog grows further.

    Allowing operators to customize the max header size, will allow them
    to have Cinder working whatever the size of the catalog (if the option
    is properly set).

    Change-Id: Iac269d96fadc96c1e9383dc03422f2d87196dd9d
    Related-Bug: #1190149
    DocImpact

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/70289
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=0b02feb20d4485d0c6d486c5a72b814ce3bdf9e5
Submitter: Jenkins
Branch: master

commit 0b02feb20d4485d0c6d486c5a72b814ce3bdf9e5
Author: Steve Baker <email address hidden>
Date: Mon Feb 3 08:45:14 2014 +1300

    Raise the default max header to accommodate large tokens

    PKI tokens hit the default limit if there is enough
    services defined in the keystone catalog.

    Also the v3 catalog is larger than the v2 catalog which would explain
    why this bug is being hit just now.

    This change adds the configuration option max_header_line to each of the
    API confurations which has a default of 16384.

    Closes-Bug: #1190149
    Change-Id: I5da09aa08a1242c5e356bd8bf532baa9347ce075

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/havana
Review: https://review.openstack.org/75732

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/havana
Review: https://review.openstack.org/75747

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/75784

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/75786

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/77108

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/75784
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=defcf235c158e93938c93f556383939515852c88
Submitter: Jenkins
Branch: stable/havana

commit defcf235c158e93938c93f556383939515852c88
Author: Steve Baker <email address hidden>
Date: Mon Feb 3 08:45:14 2014 +1300

    Raise the default max header to accommodate large tokens

    PKI tokens hit the default limit if there is enough
    services defined in the keystone catalog.

    Also the v3 catalog is larger than the v2 catalog which would explain
    why this bug is being hit just now.

    This change adds the configuration option max_header_line to each of the
    API confurations which has a default of 16384.

    Closes-Bug: #1190149
    Change-Id: I5da09aa08a1242c5e356bd8bf532baa9347ce075
    (cherry picked from commit 0b02feb20d4485d0c6d486c5a72b814ce3bdf9e5)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/75786
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=4df858db82a8684f19bdc25bf1747bdafd4bd627
Submitter: Jenkins
Branch: master

commit 4df858db82a8684f19bdc25bf1747bdafd4bd627
Author: lvdongbing <email address hidden>
Date: Mon Feb 24 15:57:20 2014 +0800

    Raise max header size to accommodate large tokens

    The max header is exceeded in the following scenario
    - Auth tokens built with a keystone v3 API catalog
    - A catalog with approximately 8 or more endpoints defined

    An equivalent fix will be required for all projects which use
    eventlet.wsgi.

    Closes-Bug: #1190149

    Change-Id: Ia0882e43395f1025137173b51aab147433b54255

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/77108
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=1690a739cd94efe6cb2af8e794a4acdf59de54cd
Submitter: Jenkins
Branch: stable/havana

commit 1690a739cd94efe6cb2af8e794a4acdf59de54cd
Author: Florent Flament <email address hidden>
Date: Mon Jan 13 13:43:59 2014 +0000

    Raise max header size to accommodate large tokens

    The former max header (8192) is exceeded in the following scenario
    - Auth tokens built with a keystone v3 API catalog
    - A catalog with approximately 8 or more endpoints defined

    Equivalent fixes have been proposed in other projects

    Change-Id: Ic714166440e2aa08b619ab1e5e5e0cf3cea3e309
    Closes-Bug: 1190149

[Alan Pevec (apevec)]

Proposed to Cinder stable/havana https://review.openstack.org/78513

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/78513
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=5b9fc5c132b0ace37c3c99f2f9a32bc88450403c
Submitter: Jenkins
Branch: stable/havana

commit 5b9fc5c132b0ace37c3c99f2f9a32bc88450403c
Author: Shao Kai Li <email address hidden>
Date: Thu Jan 2 00:51:41 2014 -0500

    Raise max header size to accommodate large tokens

    The max header is exceeded in the following scenario
    - Auth tokens built with a keystone v3 API catalog
    - A catalog with approximately 8 or more endpoints defined

    An equivalent fix will be required for all projects which use
    eventlet.wsgi.

    Copied from https://review.openstack.org/#/c/33362/

    Closes-Bug: 1190149

    Change-Id: I3fcb196b25a7a477b799c8dfa2ff5115977ca06d
    (cherry picked from commit aa1ba1f0176af286506e4188628e5acbb8f7f0e6)

[Adam Gandelman (gandelman-a)]

Note this has been Fix Released in the 2013.2.3 releases of Cinder, Glance and Heat. I can't seem to update the status on this bug without launchpad error.

[Adam Young (ayoung)]

Note that a recent commit to Master mitigates the effect of this: it removes two values from each endpoint, significantly shrinking the size of the service catalog.

http://git.openstack.org/cgit/openstack/keystone/commit/?id=6f4cade9ed980b7c87abf5f7aa17bf56874252fa

A future enhancement will compress the tokens, but it is a little too invasive to get working for Icehouse.

[Openstack Gerrit (openstack-gerrit)]

Related fix proposed to branch: master
Review: https://review.openstack.org/90130

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/90130
Committed: https://git.openstack.org/cgit/stackforge/manila/commit/?id=979c2c25409a33247955c199cbe8f897477e3a07
Submitter: Jenkins
Branch: master

commit 979c2c25409a33247955c199cbe8f897477e3a07
Author: vponomaryov <email address hidden>
Date: Thu Apr 24 19:03:11 2014 +0300

    Raise max header size to accommodate large tokens

    Related-Bug: #1190149

    Change-Id: I14a64c538b8d35549747ae74c2aea192fccbed9b

[Valeriy Ponomaryov (vponomaryov)]

Who has permissions to set status "fix released" please set it to manila's project.
Thanks in advance.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/133967

[Amrith Kumar (amrith)]

I don't believe this is a problem in Trove which already defaults to 16k for tokens. Now if 16k isn't sufficient, there's a configuration parameter.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Thomas Bechtold (<email address hidden>) on branch: master
Review: https://review.openstack.org/133967
Reason: consensus is to leave he default.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/166083

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/166083
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=07dc4fd895d883d3ca9339d6725d0ac46a7d7e94
Submitter: Jenkins
Branch: master

commit 07dc4fd895d883d3ca9339d6725d0ac46a7d7e94
Author: Deliang Fan <email address hidden>
Date: Thu Mar 19 19:48:39 2015 -0700

    Raise the default max header to accommodate large tokens

    PKI tokens hit the default limit if there is enough
    services defined in the keystone catalog.

    So sahara should allow users to customize max header size and
    also increase the default value from 8192 to 16384.

    Change-Id: If3daff1ba18f7fcd4cf3b7d9b4152b551d8ad277
    Closes-Bug: 1190149

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/203286

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/203288

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/203289

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/203289
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=7a6faa63b898861ce2af2295c283a036bffcce87
Submitter: Jenkins
Branch: stable/juno

commit 7a6faa63b898861ce2af2295c283a036bffcce87
Author: Kirill Zaitsev <email address hidden>
Date: Sat Jul 18 02:24:51 2015 +0300

    Allow setting eventlet.wsgi.MAX_HEADER_LINE

    This commit allows setting wsgi MAX_HEADER_LINE. This is done to allow
    large keystone tokens. Also removes seeimngly unused run_server
    function, to not cause confustion.

    Change-Id: I2848c08d0758f39cb988079a8fabf36d6d2a2c98
    Closes-Bug: #1190149

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/203286
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=20756379cd07e1194543e7b09444efb3412f3db8
Submitter: Jenkins
Branch: master

commit 20756379cd07e1194543e7b09444efb3412f3db8
Author: Kirill Zaitsev <email address hidden>
Date: Sat Jul 18 02:24:51 2015 +0300

    Allow setting eventlet.wsgi.MAX_HEADER_LINE

    This commit allows setting wsgi MAX_HEADER_LINE. This is done to allow
    large keystone tokens. Also removes seeimngly unused run_server
    function, to not cause confustion.

    Change-Id: I2848c08d0758f39cb988079a8fabf36d6d2a2c98
    Closes-Bug: #1190149

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/203288
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=99a727e346172f73da6cc6a23edee87348b8a100
Submitter: Jenkins
Branch: stable/kilo

commit 99a727e346172f73da6cc6a23edee87348b8a100
Author: Kirill Zaitsev <email address hidden>
Date: Sat Jul 18 02:24:51 2015 +0300

    Allow setting eventlet.wsgi.MAX_HEADER_LINE

    This commit allows setting wsgi MAX_HEADER_LINE. This is done to allow
    large keystone tokens. Also removes seeimngly unused run_server
    function, to not cause confustion.

    Change-Id: I2848c08d0758f39cb988079a8fabf36d6d2a2c98
    Closes-Bug: #1190149

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/murano 2015.1.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/murano 2015.1.1 release.