Found the following error in the keystone logs of our Mitaka deployment.
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi [req-b256411d-ba72-4076-b644-ef08a5400ab2 66725b90caea4963b1b4f91f90ab1dee ab149973d5b84459bd3ece44074ec2aa - default default] too many values to unpack
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi Traceback (most recent call last):
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 249, in __call__
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi result = method(context, **params)
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 156, in inner
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi context['subject_token_id']))
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 124, in wrapped
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi __ret_val = __f(*args, **kwargs)
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/token/provider.py", line 208, in validate_token
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi token = self._validate_token(unique_id)
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1053, in decorate
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi should_cache_fn)
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 657, in get_or_create
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi async_creator) as value:
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 158, in __enter__
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi return self._enter()
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 98, in _enter
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi generated = self._enter_create(createdtime)
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 149, in _enter_create
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi created = self.creator()
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 625, in gen_value
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi created_value = creator()
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1049, in creator
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi return fn(*arg, **kw)
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/token/provider.py", line 298, in _validate_token
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi return self.driver.validate_non_persistent_token(token_id)
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/token/providers/common.py", line 772, in validate_non_persistent_token
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi audit_info=audit_ids)
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/token/providers/common.py", line 526, in get_token_data
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi project_id, trust)
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/token/providers/common.py", line 469, in _populate_service_catalog
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi user_id, project_id)
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 124, in wrapped
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi __ret_val = __f(*args, **kwargs)
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1053, in decorate
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi should_cache_fn)
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 657, in get_or_create
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi async_creator) as value:
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 158, in __enter__
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi return self._enter()
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 91, in _enter
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi value = value_fn()
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 610, in get_value
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi value = self.backend.get(key)
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/cache/_context_cache.py", line 96, in get
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi value = self.proxied.get(key)
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/backends/memcached.py", line 161, in get
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi value = self.client.get(key)
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_cache/backends/memcache_pool.py", line 36, in _run_method
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi return getattr(client, __name)(*args, **kwargs)
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/memcache.py", line 1093, in get
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi return self._get('get', key)
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/memcache.py", line 1077, in _get
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi return _unsafe_get()
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/memcache.py", line 1059, in _unsafe_get
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi server, raise_exception=True
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/memcache.py", line 1211, in _expectvalue
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi resp, rkey, flags, len = line.split()
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi ValueError: too many values to unpack
2016-07-06 05:14:30.311 18314 ERROR keystone.common.wsgi
2016-07-06 05:14:30.321 18314 INFO eventlet.wsgi.server [req-b256411d-ba72-4076-b644-ef08a5400ab2 66725b90caea4963b1b4f91f90ab1dee ab149973d5b84459bd3ece44074ec2aa - default default] 10.111.109.87,10.111.109.89 - - [06/Jul/2016 05:14:30] "GET /v3/auth/tokens HTTP/1.1" 500 400 0.115269
And another...
2016-07-06 05:15:06.273 18314 DEBUG keystone.middleware.auth [req-de5f209c-27a5-4fdb-81af-34e01c2ca43a - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. _build_auth_context /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:71
2016-07-06 05:15:06.276 18314 INFO keystone.common.wsgi [req-de5f209c-27a5-4fdb-81af-34e01c2ca43a - - - - -] POST http://10.111.109.81:35357/v2.0/tokens
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi [req-de5f209c-27a5-4fdb-81af-34e01c2ca43a - - - - -] too many values to unpack
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi Traceback (most recent call last):
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 249, in __call__
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi result = method(context, **params)
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_log/versionutils.py", line 165, in wrapped
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi return func_or_cls(*args, **kwargs)
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/token/controllers.py", line 130, in authenticate
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi user_ref['id'], tenant_ref['id'])
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 124, in wrapped
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi __ret_val = __f(*args, **kwargs)
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1053, in decorate
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi should_cache_fn)
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 657, in get_or_create
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi async_creator) as value:
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 158, in __enter__
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi return self._enter()
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 91, in _enter
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi value = value_fn()
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 610, in get_value
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi value = self.backend.get(key)
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/cache/_context_cache.py", line 96, in get
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi value = self.proxied.get(key)
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/backends/memcached.py", line 161, in get
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi value = self.client.get(key)
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_cache/backends/memcache_pool.py", line 36, in _run_method
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi return getattr(client, __name)(*args, **kwargs)
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/memcache.py", line 1093, in get
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi return self._get('get', key)
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/memcache.py", line 1077, in _get
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi return _unsafe_get()
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/memcache.py", line 1059, in _unsafe_get
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi server, raise_exception=True
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/memcache.py", line 1211, in _expectvalue
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi resp, rkey, flags, len = line.split()
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi ValueError: too many values to unpack
2016-07-06 05:15:06.385 18314 ERROR keystone.common.wsgi
2016-07-06 05:15:06.390 18314 INFO eventlet.wsgi.server [req-de5f209c-27a5-4fdb-81af-34e01c2ca43a - - - - -] 10.111.109.191,10.111.109.89 - - [06/Jul/2016 05:15:06] "POST /v2.0/tokens HTTP/1.1" 500 400 0.118120
This is a very strange error. On the surface it looks like memcached is returning an unexpected VALUE line. Are you able to dig into memcache and look for the same key? I'm curious as to what response you get. Below is an example telnet session where I set a key named 'key' to the value 'test' and then read it out of the cache. I also look for a key 'not_there' which is a key that is not in memcached.
telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
set key 0 60 4
test
STORED
get key
VALUE key 0 4
test
END
get not_there
END
quit
Connection closed by foreign host.